Single target tracking method, electronic device, storage medium and chip

Abstract

The present application provides a single target tracking method, electronic device, storage medium and chip. In the method, a correlation calculation module obtained by custom programming is used to equivalently replace the relevant functional modules in the target tracking model, so that the correlation calculation module is no longer restricted by the target platform framework, and the dynamic convolution calculation can be accelerated to achieve the effect of improving the target tracking performance.

Description

Technical Field

[0001] The present application relates to the field of intelligent terminal technology, and in particular to a single target tracking method, electronic device, storage medium and chip. Background Art

[0002] Single target tracking is an important technology in the field of computer vision and is widely used in many fields. Its main purpose is to achieve real-time tracking of a single target in a video sequence. Tracking means giving a target frame in a certain video image frame and continuously locating the target in subsequent video image frames.

[0003] For example, single target tracking technology can be applied to camera applications of smart terminals such as mobile phones. When users use camera applications to take pictures, the camera applications can track the target and focus based on the target tracking results to improve the shooting effect. However, the target tracking performance of the camera application will directly affect the user experience. Therefore, how to optimize the target tracking performance of the camera application is a technical problem that needs to be solved. Summary of the invention

[0004] In order to solve the above technical problems, the present application provides a single target tracking method, electronic device, storage medium and chip. In this method, the dynamic convolution calculation involved in the target tracking process is accelerated through custom programming to achieve the effect of improving the target tracking performance.

[0005] In a first aspect, an embodiment of the present application provides a single target tracking method. The method is applied in an electronic device, and the electronic device includes: a template feature extraction module, a video frame feature extraction module, a correlation calculation module, a matching score calculation module, and a target coordinate determination module. Among them, the template feature extraction module, the video frame feature extraction module, the matching score calculation module, and the target coordinate determination module are generated based on the target platform framework, and the correlation calculation module is generated based on custom programming;

[0006] The method includes:

[0007] In response to the user selecting the tracking object in the first preview image frame, the template feature extraction module extracts the template features corresponding to the tracking object;

[0008] A video frame feature extraction module extracts a feature map of a second preview image frame; wherein the second preview image frame is an image frame acquired after the first preview image frame;

[0009] The correlation calculation module reads the template features and the feature map of the second preview image frame, performs dynamic convolution acceleration calculation on the feature map of the second preview image frame and the template features, and obtains a correlation calculation result corresponding to the second preview image frame;

[0010] The matching score calculation module calculates a matching score of the tracking object corresponding to the second preview image frame according to the correlation calculation result corresponding to the second preview image frame;

[0011] The target coordinate determination module calculates the position information of the tracking object in the second preview image frame according to the correlation calculation result corresponding to the second preview image frame, and generates a target tracking result according to the position information of the tracking object in the second preview image frame and the second preview image frame when the tracking object matching score corresponding to the second preview image frame meets a preset condition.

[0012] In this way, the correlation calculation module obtained by custom programming is used to equivalently replace the relevant functional modules in the target tracking model, so that the correlation calculation module is no longer restricted by the target platform framework, and the dynamic convolution calculation can be accelerated to achieve the effect of improving the target tracking performance.

[0013] According to the first aspect, each element in the feature map of the second preview image frame is integer data, and each element in the template feature is integer data; each integer data is predefined with a conversion ratio and an offset, wherein the conversion ratio and the offset are used to convert the integer data into floating-point data.

[0014] For example, for integer data M, the corresponding conversion scale and offset are scale_M and offset_M respectively, then “(M+offset_M)×scale_M” can represent the floating-point data corresponding to the integer data M.

[0015] According to the first aspect, or any implementation of the first aspect above, the correlation calculation module reads the template feature and the feature map of the second preview image frame, and performs dynamic convolution acceleration calculation on the feature map of the second preview image frame and the template feature, including:

[0016] The correlation calculation module performs multiplication calculation on the first element a in the feature map of the second preview image frame and the second element b in the template feature;

[0017] Wherein, when calculating the product c of the first element a and the second element b, the product c is solved based on the equivalent floating-point data of the first element a and the equivalent floating-point data of the second element b, and the product c is integer data;

[0018] When solving the product c, the correlation calculation module uses the predetermined first target integer data for multiplication calculation, and performs a data binary right shift n1 bits on the product result corresponding to the first target integer data; wherein the first target integer data is the expansion of the relevant floating-point data involved in solving the product c by 2 n1The relevant floating-point data is obtained by multiplying the first element a and the second element b, and the conversion ratio of the product c, and n1 is an integer.

[0019] In this way, when solving the product c based on the equivalent floating-point data of the first element a and the equivalent floating-point data of the second element b, the operation of floating-point data is no longer involved. The equivalent division operation is also performed through the data binary shift method, which improves the computing performance and achieves the effect of accelerating the dynamic convolution operation. Moreover, since the integer operator used is pre-calculated, it will not affect the accuracy of the calculation result.

[0020] According to the first aspect, or any implementation of the first aspect above, the data arrangement format of the feature map of the second preview image frame is a HWC format.

[0021] According to the first aspect, or any implementation of the first aspect above, the correlation calculation module reads the template feature and the feature map of the second preview image frame, and performs dynamic convolution acceleration calculation on the feature map of the second preview image frame and the template feature, including:

[0022] A data vector in the channel number dimension of the feature map of the second preview image frame is used as a processing unit, and dynamic convolution calculation of the feature map of the second preview image frame and the template feature is implemented based on vectorized instructions.

[0023] In this way, when performing dynamic convolution operations, using a data vector in the channel number dimension in the feature map as the processing unit can effectively improve the efficiency of dynamic convolution operations and alleviate the pressure on device memory, performance, and power consumption.

[0024] According to the first aspect, or any implementation of the first aspect, the vectorized instruction includes at least one of the following:

[0025] Vector addition instructions, vector multiplication instructions, vector fetch instructions, vector store instructions, vector data arrangement and transformation instructions, and vector data bit width expansion and reduction instructions.

[0026] According to the first aspect, or any implementation of the first aspect above, a data vector on the channel number dimension in the feature map of the second preview image frame is used as a processing unit, and dynamic convolution acceleration calculation of the feature map of the second preview image frame and the template feature is implemented based on a vectorized instruction, including:

[0027] Perform multiplication calculation on the first data vector A in the channel number dimension in the feature map of the second preview image frame and the third element d in the template feature;

[0028] Wherein, when calculating the product data vector E of the first data vector A and the third element d, the product vector E is solved based on the equivalent floating-point data vector of the first data vector A and the equivalent floating-point data of the third element d, and the product data vector E is an integer data vector;

[0029] When solving the product data vector E, a predetermined second target integer data is used for multiplication calculation, and the product result corresponding to the second target integer data is subjected to a data binary right shift operation of n2 bits; wherein the second target integer data is the expansion of the relevant floating-point data involved in solving the product vector E by 2 n2 The relevant floating-point data is obtained by multiplying the first data vector A, the third element d and the product vector E, and n2 is an integer.

[0030] In this way, when the data vector is used as the processing unit to accelerate the dynamic convolution operation, the operation of floating-point data will not be involved. The equivalent division operation is also performed through the data binary shift method, which further improves the computing performance and makes the target tracking algorithm delay meet the usage requirements.

[0031] According to the first aspect, or any implementation of the first aspect above, the method further includes:

[0032] A video frame feature extraction module extracts a feature map of a third preview image frame; wherein the third preview image frame is an image frame acquired after the second preview image frame;

[0033] The correlation calculation module reads the template features and the feature map of the third preview image frame, performs dynamic convolution acceleration calculation on the feature map of the third preview image frame and the template features, and obtains a correlation calculation result corresponding to the third preview image frame;

[0034] The matching score calculation module calculates a tracking object matching score corresponding to the third preview image frame according to a correlation calculation result corresponding to the third preview image frame;

[0035] The target coordinate determination module calculates the position information of the tracked object in the third preview image frame according to the correlation calculation result corresponding to the third preview image frame, and generates a target tracking result according to the position information of the tracked object in the third preview image frame and the third preview image frame when the tracking object matching score corresponding to the third preview image frame meets a preset condition.

[0036] In this way, when the tracking object is not updated, the template image of the tracking object only needs to be input once, and then the template feature extraction module only needs to perform the template feature extraction operation once. Regardless of which video image frame to be tracked is used for target tracking processing, the correlation calculation module only needs to read the template features in the memory for dynamic convolution calculation, which reduces the data processing amount of the template feature extraction module.

[0037] In a second aspect, an embodiment of the present application provides an electronic device. The electronic device includes: one or more processors; a memory; and one or more computer programs, wherein the one or more computer programs are stored in the memory, and when the computer programs are executed by the one or more processors, the electronic device executes the single target tracking method of the first aspect and any one of the first aspects.

[0038] The second aspect and any implementation of the second aspect correspond to the first aspect and any implementation of the first aspect respectively. The technical effects corresponding to the second aspect and any implementation of the second aspect can refer to the technical effects corresponding to the above-mentioned first aspect and any implementation of the first aspect, which will not be repeated here.

[0039] In a third aspect, an embodiment of the present application provides a computer-readable storage medium. The computer-readable storage medium includes a computer program, and when the computer program is executed on an electronic device, the electronic device executes the single target tracking method of the first aspect and any one of the first aspects.

[0040] The third aspect and any implementation of the third aspect correspond to the first aspect and any implementation of the first aspect, respectively. The technical effects corresponding to the third aspect and any implementation of the third aspect can refer to the technical effects corresponding to the first aspect and any implementation of the first aspect, which will not be repeated here.

[0041] In a fourth aspect, an embodiment of the present application provides a computer program product, including a computer program, which, when executed, enables a computer to execute a single target tracking method as described in the first aspect or any one of the first aspects.

[0042] The fourth aspect and any implementation of the fourth aspect correspond to the first aspect and any implementation of the first aspect, respectively. The technical effects corresponding to the fourth aspect and any implementation of the fourth aspect can refer to the technical effects corresponding to the above-mentioned first aspect and any implementation of the first aspect, which will not be repeated here.

[0043] In a fifth aspect, the present application provides a chip, which includes one or more interface circuits and one or more processors; the interface circuit is used to receive a signal from a memory of an electronic device and send a signal to the processor, the signal including a computer instruction stored in the memory; when the processor executes the computer instruction, the electronic device executes the single target tracking method of the first aspect and any one of the first aspect.

[0044] The fifth aspect and any implementation of the fifth aspect correspond to the first aspect and any implementation of the first aspect, respectively. The technical effects corresponding to the fifth aspect and any implementation of the fifth aspect can refer to the technical effects corresponding to the first aspect and any implementation of the first aspect, which will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] Figure 1 A schematic diagram of an application scenario of single target tracking is shown as an example;

[0046] Figure 2 is a schematic diagram of an exemplary single target tracking method;

[0047] Figure 3 A schematic diagram of modules involved in the single target tracking method provided in an embodiment of the present application;

[0048] Figure 4a is a schematic diagram for exemplarily showing the method of taking vector data in a feature map;

[0049] Figure 4b The integrated schematic diagram of the data computing unit is shown as an example;

[0050] Figure 5 The figure is a schematic diagram showing the application effect of a single target tracking method;

[0051] Figure 6 A schematic diagram of the input of the single target tracking method provided in the embodiment of the present application;

[0052] Figure 7 A schematic diagram of an application scenario of single target tracking is shown as an example;

[0053] Figure 8 is a schematic diagram of the hardware structure of an electronic device shown as an example;

[0054] Fig. 9 The software structure diagram of an electronic device is shown as an example. DETAILED DESCRIPTION

[0055] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0056] The term "and / or" in this article is merely a description of the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone.

[0057] The terms "first" and "second" in the description and claims of the embodiments of the present application are used to distinguish different objects rather than to describe a specific order of objects. For example, a first target object and a second target object are used to distinguish different target objects rather than to describe a specific order of target objects.

[0058] In the embodiments of the present application, words such as "exemplary" or "for example" are used to indicate examples, illustrations or descriptions. Any embodiment or design described as "exemplary" or "for example" in the embodiments of the present application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "exemplary" or "for example" is intended to present related concepts in a specific way.

[0059] In the description of the embodiments of the present application, unless otherwise specified, the meaning of "multiple" refers to two or more than two. For example, multiple processing units refer to two or more processing units; multiple systems refer to two or more systems.

[0060] Single object tracking (SOT) is an important technology in the field of computer vision and is widely used in many fields. Its main purpose is to achieve real-time tracking of a single target in a video sequence. Tracking means giving a target frame in a certain video image frame and continuously locating the target in subsequent video image frames. Generally speaking, single object tracking can be understood as giving a target to be tracked in a certain video image frame, tracking the target in subsequent image frames and marking the target frame. Among them, the target frame can be a rectangular frame used to indicate the position and size of the tracked target.

[0061] For example, single target tracking technology can be applied to camera applications of smart terminals such as mobile phones. When a user uses the camera application to take a photo, the camera application can track the target and focus based on the target tracking results to improve the shooting effect.

[0062] Figure 1An example application scenario is shown. When the target tracking function of the camera application is turned on, if the user clicks on the subject, the camera application will automatically track and focus on the subject. Figure 1 As shown in (1), the camera application displays a photo preview interface 101. When the user needs to use the photo object 1011 displayed in the photo preview interface 101 as a target to be tracked, the user clicks the photo object 1011 in the photo preview interface 101. In response to the user operation, the camera application uses the photo object 1011 as a target to be tracked, generates and displays a target frame 1012 corresponding to the photo object 1011, as shown in FIG. Figure 1 As shown in (2), the target frame 1012 is used to indicate the position and size of the photographed object 1011. Then, the camera application tracks the photographed object 1011 in the preview image frames collected subsequently and marks the target frame.

[0063] As the phone's camera lens moves, or the subject moves, the video image frames displayed in the camera app's shooting preview interface will also be refreshed. However, the camera app's target tracking performance may not match the refresh rate of the preview image frames, which affects the smoothness of the camera app's target tracking. In this way, when the app's target tracking performance cannot meet the refresh rate of the preview image frames, some preview image frames may display the target frame corresponding to the object to be tracked, while some preview image frames may not display the target frame corresponding to the object to be tracked, resulting in the problem that the target frame appears and disappears from time to time.

[0064] For example, after the user selects the photographed object 1011, the target frame corresponding to the photographed object 1011 cannot be displayed in some preview image frames subsequently collected by the camera application. Figure 1 The photo preview interface 102 shown in (3) of FIG. 100; and in the partial preview image frames subsequently collected by the camera application, a target frame corresponding to the photographed object 1011 can be displayed. At this time, you can refer to Figure 1 The target frame 1013 corresponding to the photographed object 1011 is displayed in the photo preview interface 103 shown in (4).

[0065] It is understandable that when a target frame corresponding to the object to be tracked is displayed in the preview image frame, it indicates that the camera application can successfully track the corresponding object; and when the object to be tracked is displayed in the preview image frame but no target frame corresponding to it is displayed, it indicates that the camera application has not achieved target tracking of the object based on the preview image frame. When the camera application fails to achieve target tracking based on a preview image frame, the camera application cannot perform a focus operation based on the target tracking result in this preview image frame, thereby affecting the shooting effect.

[0066] Considering that the target tracking performance of camera applications will directly affect the user experience, how to optimize the target tracking performance of camera applications is a technical problem that needs to be solved.

[0067] Regarding single target tracking, we can understand it as a template matching problem, that is, the target object in the first video image frame is used as a template, which matches the video image frames collected later. Figure 2 As shown in Figure 1, the input of the target tracking model is the template image of the tracking object and the video image frame to be tracked. After the inference of the target tracking model, the corresponding target tracking result can be output. Figure 2 ,The target tracking result can be understood as marking a target frame corresponding to the tracking object on the video image frame to be tracked.

[0068] like Figure 2 As shown, in one implementation method, the target tracking model is generated based on the XX platform. Specifically, a precompiled model file corresponding to the target tracking model can be generated based on the XX platform, and then the precompiled model file can be loaded into the mobile phone memory based on the SDK (Software Development Kit) of the XX platform to obtain the target tracking model. Then, the camera application of the mobile phone can perform single target tracking processing based on the target tracking model. However, the target tracking model is limited by the limitations of the XX platform framework, and there may be performance bottlenecks, resulting in the camera application being unable to match the refresh rate of the video image frame when performing target tracking based on the target tracking model, affecting the smoothness of the camera application when performing single target tracking.

[0069] In order to solve this problem, the embodiment of the present application provides a single target tracking method. In this method, the target tracking model is functionally split, the correlation calculation module that seriously affects the overall performance of the model is identified, and this module is equivalently replaced using custom programming, so that the correlation calculation part is no longer restricted by the framework of the XX platform, thereby improving the performance of single target tracking.

[0070] like Figure 3As shown, the target tracking model can be equivalently divided into a template feature extraction module, a video frame feature extraction module, a correlation calculation module, a matching score calculation module and a target coordinate determination module. Among them, the template feature extraction module is used to extract features of the tracking object template image through CNN (Convolutional Neural Networks) to obtain the template features of the tracking object. The template feature can be understood as a convolution kernel. The video frame feature extraction module is used to extract features of the video image frame to be tracked through CNN to obtain a feature map corresponding to the video image frame to be tracked. The correlation calculation module is used to perform dynamic convolution calculation based on the template features and the feature map to obtain the correlation calculation result. The matching score calculation module is used to calculate the tracking object matching score based on the correlation calculation result. Among them, the tracking object matching score can be used to indicate the matching degree between the object detected in the video image frame and the tracking object (or the tracking object template image). For example, the higher the tracking object matching score, the higher the matching degree. The target coordinate determination module is used to calculate the position information of the tracked object in the video image frame, such as the coordinate information of the target frame, according to the correlation calculation result, and generate the target tracking result based on the position information of the tracked object in the video image frame and the video image frame to be tracked when the tracking object matching score meets the preset conditions (for example, the tracking object matching score is higher than the preset threshold).

[0071] Through analysis, in the above-mentioned multiple template feature extraction modules, the reason for the bottleneck of the target tracking model performance is the correlation calculation module. Therefore, in the embodiment of the present application, the function of the correlation calculation module is realized by custom programming, and the functional module pre-compiled based on the XX platform and loaded based on the platform SDK is no longer used. That is, in the target tracking model provided in the embodiment of the present application, the correlation calculation module is generated based on custom programming, and other functional modules (i.e., template feature extraction module, video frame feature extraction module, matching score calculation module, target coordinate determination module) can still be generated based on the XX platform, that is, based on the XX platform, a pre-compiled model file corresponding to each functional module is generated, and then the corresponding pre-compiled model file is loaded into the mobile phone memory based on the SDK of the XX platform to obtain the corresponding functional module (or functional model).

[0072] In the embodiment of the present application, the template feature extraction module extracts features from the template image of the tracking object, obtains the template features H[m,n] corresponding to the tracking object, and stores the template features H[m,n] in the device memory, where m and n are the dimensions of the template features.

[0073] The video frame feature extraction module extracts features from the video image frame to be tracked, obtains the feature map X[h,w,c] corresponding to the video frame image, and stores the feature map X[h,w,c] in the device memory. Among them, h represents the number of pixels in the height direction of the feature map, w represents the number of pixels in the width direction of the feature map, and c represents the number of channels of the feature map.

[0074] The correlation calculation module reads the template features H[m,n] and the feature map X[h,w,c] corresponding to the video image frame to be tracked in the device memory, and performs dynamic convolution calculation based on the template features H[m,n] and the feature map X[h,w,c]. The correlation calculation result can be expressed as:

[0075]

[0076] In the embodiment of the present application, each element in the template feature H[m,n] output by the template feature extraction module and the feature map X[h,w,c] output by the video frame feature extraction module is an integer data. Therefore, each time X[i+m,j+n,c]·H[m,n] is calculated, a multiplication operation of integer data is involved.

[0077] Considering that there may be an overflow problem when multiplying integer data, the embodiment of the present application adopts a floating-point data equivalent method to solve the problem of integer data multiplication operation.

[0078] In this embodiment, for each integer data in the correlation calculation module, a corresponding conversion ratio (scale) and offset (offset) are customized. The conversion ratio (scale) and offset (offset) can be used to convert a data from integer data to floating-point data. For example, for integer data M, the corresponding conversion ratio and offset are scale_M and offset_M, respectively, then "(M+offset_M)×scale_M" can represent the floating-point data corresponding to the integer data M. When the number of bits of the integer data is p bit, the corresponding conversion ratio (scale) is 1 / (2 p -1). For example, if the number of bits of an integer data is 8 bits, the corresponding conversion ratio (scale) is 1 / 255.

[0079] In this embodiment, a consistent data quantization form is used for a feature map X[h,w,c], and the conversion ratio (scale) and offset (offset) of each integer element in the same feature map X[h,w,c] are consistent; a consistent data quantization form is used for a template feature H[m,n], and the conversion ratio (scale) and offset (offset) of each integer element in the same template feature H[m,n] are also consistent.

[0080] For example, when calculating X[i+m,j+n,c]·H[m,n], it is necessary to calculate the product of integer data a (elements in X[i+m,j+n,c]) and integer data b (elements in H[m,n]). Assume that a×b=c. Where a, b, and c are all integer data; the conversion ratio and offset corresponding to a are scale_a and offset_a respectively, and "(a+offset_a)×scale_a" can represent the floating-point data corresponding to the integer data a; the conversion ratio and offset corresponding to b are scale_b and offset_b respectively, and "(b+offset_b)×scale_b" can represent the floating-point data corresponding to the integer data b; the conversion ratio and offset corresponding to c are scale_c and offset_c respectively, and "(c+offset_c)×scale_c" can represent the floating-point data corresponding to the integer data c.

[0081] Thus, a×b=c, which is equivalent to (a+offset_a)×scale_a×(b+offset_b)×scale_b=(c+offset_c)×scale_c (Formula 1). Since a, offset_a, scale_a, b, offset_b, scale_b, offset_c, and scale_c are all known quantities, c can be solved by Formula 1.

[0082] Among them, c+offset_c=(scale_a×scale_b / scale_c)×(a+offset_a)×(b+offset_b) (Formula 2).

[0083] In Formula 2, scale_a×scale_b / scale_c is floating-point data, and (a+offset_a)×(b+offset_b) is integer data. Therefore, when solving c based on Formula 2, it is necessary to multiply the floating-point data and the integer data. Considering that the calculation related to floating-point data is time-consuming, in this embodiment, the floating-point data in Formula 2 is fixed-point processed to improve the calculation performance.

[0084] Since scale_a, scale_b, and scale_c are all known, in this embodiment, the floating point data "scale_a×scale_b / scale_c" can be magnified by 2 in advance. n1 times into integer data (temporarily call this integer data x'), that is, pre-compressing the floating point data "scale_a×scale_b / scale_c" into integer data, reducing the data memory usage. The correlation calculation module can directly use this integer data. Among them, n1 is an integer. That is, x'=(scale_a×scale_b / scale_c)×2 n1 Therefore, Formula 2 can be optimized and adjusted to: c+offset_c=x'×(a+offset_a)×(b+offset_b) / 2 n1 (Formula 3).

[0085] Where x' = (scale_a × scale_b / scale_c) × 2 n1 is a known quantity obtained in advance and is an integer data. Therefore, c can be directly solved based on Formula 3. When solving c based on Formula 3, x'×(a+offset_a)×(b+offset_b) is a multiplication operation of integer data. Considering that the division operation is also time-consuming, binary shift can be used to achieve "division by 2" when solving c based on Formula 3. n1 " operation. Among them, by shifting an integer data to the right by n1 bits, the absolute value of the data can be changed to 1 / 2 of the original value. n1 .

[0086] Thus, when the correlation calculation module calculates X[i+m,j+n,c]·H[m,n], for any element (integer data) multiplication calculation, the pre-calculated integer operator (such as (scale_a×scale_b / scale_c)×2 n1 ) replaces the floating-point operator (scale_a×scale_b / scale_c) involved in the calculation process, that is, directly uses the pre-calculated integer operator for calculation, and processes the corresponding calculation result based on the binary shift operation (such as shifting n1 bits to the right), so as to achieve equivalent replacement. As a result, the calculation performance of the correlation calculation module is improved, and the accuracy of the data calculation result can be guaranteed to be as close as possible to the data calculation result obtained based on the floating-point operator.

[0087] In an embodiment of the present application, the data arrangement format of the feature map X[h, w, c] is in HWC format. Among them, the HWC format refers to a format in which the image size is arranged in the order of height, width and number of channels. In the HWC format, the pixel data of each pixel is arranged in the memory in sequence, with the height being the first dimension, the width being the second dimension, and the number of channels being the third dimension. Therefore, in the memory, the data of the feature map X[h, w, c] is continuous in the dimension of the number of channels. Furthermore, the correlation calculation module can read multiple data with the same height dimension, the same width dimension, different and continuous number of channels in the feature map X[h, w, c] in the memory each time as a vector for processing.

[0088] like Figure 4a As shown, element X(h,w,1), element X(h,w,2), element X(h,w,3), …, element X(h,w,x), …, element X(h,w,c) are c continuous data with the same height dimension, the same width dimension, and different channel number dimensions. These data can be read out from the memory as a vector [X(h,w,1), X(h,w,2), X(h,w,3), …, X(h,w,x), …, X(h,w,c)] by the correlation calculation module.

[0089] Thus, when the correlation calculation module calculates X[i+m,j+n,c]·H[m,n], it can use the data vectors [X(h,w,1), X(h,w,2), X(h,w,3), ..., X(h,w,x), ..., X(h,w,c)] in the channel number dimension as processing units for calculation. The correlation calculation module can implement vector-related processing based on vector addition instructions, vector multiplication instructions, vector fetch instructions, vector storage instructions, vector data arrangement transformation instructions, and vector data bit width expansion and reduction instructions during the calculation process.

[0090] In this way, when calculating X[i+m,j+n,c]·H[m,n], the product of the integer data vector A=[X(i+m,j+n,1), X(i+m,j+n,2), X(i+m,j+n,3),…, X(i+m,j+n,x),…, X(i+m,j+n,c)] and the integer data d (elements in H[m,n]) can be calculated, thereby accelerating the efficiency of dynamic convolution calculation.

[0091] Assume that A×d=E. Where A and E are both integer data vectors. Where A×d=E, it can be equivalent to (A+offset_A)×scale_A×(d+offset_d)×scale_d=(E+offset_E)×scale_E (Formula 4).

[0092] Since A, offset_A, scale_A, d, offset_d, scale_d, offset_E, and scale_E are all known quantities, E can be solved using Formula 4.

[0093] Among them, E+offset_E=(scale_A×scale_d / scale_E)×(A+offset_A)×(d+offset_d) (Equation 5).

[0094] In Formula 5, scale_A×scale_d / scale_E is floating-point data, and the data involved in the data operation (A+offset_A)×(d+offset_d) are all integer data. Therefore, when solving E based on Formula 5, it is necessary to multiply the floating-point data and the integer data vector. Considering that the calculation related to floating-point data is time-consuming, in this embodiment, the floating-point data in Formula 5 is fixed-point processed to improve the calculation performance.

[0095] Since scale_A, scale_d, and scale_E are all known, in this embodiment, the floating point data "scale_A×scale_d / scale_E" can be magnified by 2 in advance. n2 times into integer data (temporarily call this integer data X'), that is, compressing the floating-point data "scale_A×scale_d / scale_E" into integer data in advance, reducing the data memory usage. The correlation calculation module can directly use the integer data. Among them, n2 is an integer, n2 can be the same as n1, and n2 can also be different from n1. The values ​​of n1 and n2 are related to the data quantization form of the feature map and template features, and specifically to the scale of the integer elements in the feature map and template features. That is, X'=(scale_A×scale_d / scale_E)×2 n2 Therefore, equation 5 can be optimized and adjusted to: E+offset_E=X'×(A+offset_A)×(d+offset_d) / 2 n2 (Formula 6).

[0096] Thus, when the correlation calculation module calculates X[i+m,j+n,c]·H[m,n], the vector in the channel number dimension of X[i+m,j+n,c] is used as the processing unit to perform the dot product calculation of the vector X[i+m,j+n,c] and the vector H[m,n]. The pre-calculated integer operator (such as (scale_A×scale_d / scale_E)×2 n2) replaces the floating-point operators (scale_A×scale_d / scale_E) involved in the calculation process, that is, directly uses the pre-calculated integer operators for calculation, and processes the corresponding calculation results based on the binary shift operation (such as shifting n2 bits to the right), so as to achieve equivalent replacement. As a result, the computing performance of the correlation calculation module is improved, and the accuracy of the data calculation results can be guaranteed to be as close as possible to the data calculation results obtained based on floating-point operators. Moreover, when calculating X[i+m,j+n,c]·H[m,n], multiple multiplication calculation units are integrated together through vectorized instructions, which greatly alleviates the pressure on device memory, performance, and power consumption.

[0097] For example, refer to Figure 4b As shown, since X(h,w,1), X(h,w,2), X(h,w,3), ..., X(h,w,c) are stored continuously in the device memory, the multiplication calculation of X(h,w,1) and H(m,n), the multiplication calculation of X(h,w,2) and H(m,n), the multiplication calculation of X(h,w,3) and H(m,n), ..., the multiplication calculation of X(h,w,c) and H(m,n) can be integrated into the multiplication calculation of the vector [X(h,w,1), X(h,w,2), X(h,w,3), ..., X(h,w,c)] and H(m,n), and the corresponding calculation processing is completed based on the vectorized instruction.

[0098] After the correlation calculation module obtains the correlation calculation result, the correlation calculation result is stored in the device memory for use by the matching score calculation module and the target coordinate determination module to obtain the target tracking result. The processing logic of the matching score calculation module and the target coordinate determination module can refer to the existing technology and will not be repeated here.

[0099] In the technical solution provided in the embodiment of the present application, the correlation calculation module is implemented based on custom programming and is no longer limited by the limitations of the XX platform framework, which improves the computing performance, avoids the problem of unstable target tracking, and can better support the update and change of the tracking target.

[0100] When the technical solution provided by the embodiment of the present application is applied to a mobile phone, the target tracking performance of the camera application is optimized, and can match the refresh rate of the preview image frame, thereby improving the smoothness of the camera application in performing target tracking. Figure 5As shown in (1), the camera application displays a photo preview interface 201. When the user needs to use the photo object 2011 displayed in the photo preview interface 201 as a target to be tracked, the user clicks the photo object 2011 in the photo preview interface 201. In response to the user operation, the camera application uses the photo object 2011 as a target to be tracked, generates and displays a target frame 2012 corresponding to the photo object 2011, as shown in FIG. Figure 5 As shown in (2). Then, the camera application tracks the subject 2011 in the preview image frames collected subsequently and marks the target frame. As the mobile phone camera lens moves, or the subject moves, the video image frame displayed on the camera application shooting preview interface will also be refreshed. Since the target tracking performance of the camera application is optimized, the target frame corresponding to the subject 2011 can be displayed in the preview image frames collected subsequently by the camera application. At this time, you can refer to Figure 5 In the photo preview interface 202 shown in (3), a target frame 2013 corresponding to the photographed object 2011 is displayed. In this way, in the preview interface displayed by the camera application, there will be no problem of the target frame appearing sometimes and sometimes not, and then the camera application will not be unable to focus based on the target tracking result and affect the shooting effect.

[0101] In such Figure 2 In the existing scheme shown, since the target tracking model is generated based on the XX platform framework, when the tracking object is not updated, the target tracking model needs to input the template image of the tracking object and the video image frame to be tracked each time it performs target tracking processing. That is, when the target tracking processing is performed on the video image frame 1 to be tracked, the input of the target tracking model is the template image of the tracking object and the video image frame 1 to be tracked; when the target tracking processing is performed on the video image frame 2 to be tracked, the input of the target tracking model is the template image of the tracking object and the video image frame 2 to be tracked; when the target tracking processing is performed on the video image frame 3 to be tracked, the input of the target tracking model is the template image of the tracking object and the video image frame 3 to be tracked, and so on. In this way, each time the target tracking model performs target tracking processing, it is necessary to extract the template feature H[m,n] from the template image, which increases the amount of data processing and reduces the computing performance. Moreover, in view of the defects of the XX platform such as memory scheduling, the target tracking model generated based on the XX platform cannot well support the update scheduling of the tracking object, resulting in poor target tracking performance.

[0102] In the scheme shown in this application, Figure 6As shown, when the tracking object is not updated, the template image of the tracking object only needs to be input once, and then the template feature extraction module only needs to perform the extraction operation of the template feature H[m,n] once and store H[m,n] in the device memory. Whether it is the target tracking processing for the video image frame 1 to be tracked, or the target tracking processing for the video image frame 2 to be tracked, the video image frame 3 to be tracked, etc., the correlation calculation module only needs to read the template feature H[m,n] in the memory, until the tracking object is updated, the template image of the tracking object needs to be input into the template feature extraction module for template feature extraction. In this way, relative to Figure 2 The existing scheme shown reduces the amount of data processing and improves the computing performance. Moreover, even if the tracked object is updated frequently, it will not affect the target tracking performance of the camera application.

[0103] Figure 7 The example shows the application scenario of switching the tracking object. Figure 7 As shown in (1), the camera application displays a photo preview interface 301. When the user needs to use the photo object 3011 displayed in the photo preview interface 301 as a target to be tracked, the user clicks the photo object 3011 in the photo preview interface 301. In response to the user operation, the camera application uses the photo object 3011 as a target to be tracked, generates and displays a target frame 3012 corresponding to the photo object 3011, as shown in FIG. Figure 7 As shown in (2). Then, the camera application tracks the subject 2011 in the preview image frames collected subsequently and marks the target frame. As the mobile phone camera lens moves, or the subject moves, the video image frame displayed on the camera application shooting preview interface will also be refreshed. Since the target tracking performance of the camera application is optimized, the target frame corresponding to the subject 3011 can be displayed in the preview image frames collected subsequently by the camera application. At this time, you can refer to Figure 7 When the user needs to adjust the tracking object, for example, to set the photographing object 3013 displayed in the photographing preview interface 302 as the target to be tracked, the user clicks the photographing object 3013 in the photographing preview interface 302. In response to the user operation, the camera application sets the photographing object 3013 as the target to be tracked again, generates and displays a target frame 3014 corresponding to the photographing object 3013, as shown in FIG. Figure 7 As shown in (4) in the figure, after that, the target application no longer tracks the photographed object 3011, but tracks the photographed object 3013.

[0104] Since the correlation calculation module is not restricted by the XX platform framework, when the tracking object is updated, it only needs to re-read the updated template features in the memory and perform dynamic convolution calculations. This will not affect the target tracking performance, ensuring the stability of target tracking and thus improving the shooting effect.

[0105] like Figure 8 FIG. 1 is a schematic diagram of the structure of the electronic device 100. Optionally, the electronic device 100 may be a terminal, which may also be referred to as a terminal device. The terminal may be a device with a shooting function such as a cellular phone or a tablet computer, which is not limited in this application. It should be noted that the schematic diagram of the structure of the electronic device 100 may be applicable to Figure 1 , Figure 5 , Figure 7 It should be understood that Figure 8 The electronic device 100 shown is only one example of an electronic device, and the electronic device 100 may have more or fewer components than shown in the figure, may combine two or more components, or may have a different configuration of components. Figure 8 The various components shown in the EMBODIMENTS 2000 may be implemented in hardware, software, or a combination of hardware and software, including one or more signal processing and / or application specific integrated circuits.

[0106] The electronic device 100 may include: a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, an earphone interface 170D, a sensor module 180, a button 190, a motor 191, an indicator 192, a camera 193, a display screen 194, and a subscriber identification module (SIM) card interface 195, etc. The sensor module 180 may include a pressure sensor, a gyroscope sensor, an acceleration sensor, a temperature sensor, a motion sensor, an air pressure sensor, a magnetic sensor, a distance sensor, a proximity light sensor, a fingerprint sensor, a touch sensor, an ambient light sensor, a bone conduction sensor, etc.

[0107] The processor 110 may include one or more processing units, for example, the processor 110 may include an application processor (AP), a modem processor, a graphics processor (GPU), an image signal processor (ISP), a controller, a memory, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU), etc. Different processing units may be independent devices or integrated into one or more processors.

[0108] The controller may be the nerve center and command center of the electronic device 100. The controller may generate an operation control signal according to the instruction operation code and the timing signal to complete the control of fetching and executing instructions.

[0109] A memory may also be provided in the processor 110 for storing instructions and data. In some embodiments, the memory in the processor 110 is a cache memory.

[0110] The electronic device 100 implements the display function through a GPU, a display screen 194, and an application processor. The GPU is a microprocessor for image processing, which connects the display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. The processor 110 may include one or more GPUs that execute program instructions to generate or change display information.

[0111] The display screen 194 is used to display images, videos, etc. The display screen 194 includes a display panel. In some embodiments, the electronic device 100 may include 1 or N display screens 194, where N is a positive integer greater than 1.

[0112] The electronic device 100 can realize the shooting function through ISP, camera 193, video codec, GPU, display screen 194 and application processor.

[0113] ISP is used to process the data fed back by camera 193. For example, when taking a photo, the shutter is opened, and the light is transmitted to the camera photosensitive element through the lens. The light signal is converted into an electrical signal, and the camera photosensitive element transmits the electrical signal to ISP for processing and converts it into an image visible to the naked eye. ISP can also perform algorithm optimization on the noise, brightness, and skin color of the image. ISP can also optimize the exposure, color temperature and other parameters of the shooting scene. In some embodiments, ISP can be set in camera 193.

[0114] The camera 193 is used to capture still images or videos. The object generates an optical image through the lens and projects it onto the photosensitive element. The photosensitive element can be a charge coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the optical signal into an electrical signal, and then passes the electrical signal to the ISP to be converted into a digital image signal. The ISP outputs the digital image signal to the DSP for processing. The DSP converts the digital image signal into an image signal in a standard RGB, YUV or other format. In some embodiments, the electronic device 100 may include 1 or N cameras 193, where N is a positive integer greater than 1.

[0115] The external memory interface 120 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the electronic device 100. The external memory card communicates with the processor 110 via the external memory interface 120 to implement a data storage function.

[0116] The internal memory 121 can be used to store computer executable program codes, which include instructions. The processor 110 executes various functional applications and data processing of the electronic device 100 by running the instructions stored in the internal memory 121. The internal memory 121 may include a program storage area and a data storage area. Among them, the program storage area may store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc. The data storage area may store data created during the use of the electronic device 100 (such as audio data, a phone book, etc.), etc. In addition, the internal memory 121 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, a universal flash storage (UFS), etc.

[0117] The software system of the electronic device 100 may adopt a layered architecture, an event-driven architecture, a micro-core architecture, a micro-service architecture, or a cloud architecture. The embodiment of the present application takes the Android system of the layered architecture as an example to exemplify the software structure of the electronic device 100.

[0118] Fig. 9 It is a software structure block diagram of the electronic device 100 according to an embodiment of the present application.

[0119] The layered architecture of the electronic device 100 divides the software into several layers, each layer has a clear role and division of labor. The layers communicate with each other through software interfaces. In some embodiments, the Android system is divided into four layers, from top to bottom, namely, the application layer, the application framework layer, the hardware abstraction layer (HAL), and the kernel layer.

[0120] Understandably, Fig. 9 For example, between the application framework layer and the HAL layer, there may also be an Android runtime and a system library layer. Among them, the Android Runtime includes a core library and a virtual machine, which is responsible for the scheduling and management of the Android system. The system library may include multiple functional modules. For example: a surface manager, a media library, a 3D graphics processing library (e.g., OpenGL ES), a 2D graphics engine (e.g., SGL), etc.

[0121] The application layer can include a series of application packages.

[0122] like Fig. 9 As shown, the application package may include applications such as call, calendar, map, navigation, music, video, short message, camera, gallery, etc.

[0123] The application framework layer provides an application programming interface (API) and a programming framework for the applications in the application layer. The application framework layer includes some predefined functions.

[0124] like Fig. 9 As shown, the application framework layer may include a window manager, a content provider, a view system, a phone manager, a resource manager, a notification manager, and the like.

[0125] The window manager is used to manage window programs. The window manager can obtain the display screen size, determine whether there is a status bar, lock the screen, capture the screen, etc.

[0126] Content providers are used to store and retrieve data and make it accessible to applications. The data may include videos, images, audio, calls made and received, browsing history and bookmarks, phone books, etc.

[0127] The view system includes visual controls, such as controls for displaying text, controls for displaying images, etc. The view system can be used to build applications. A display interface can be composed of one or more views. For example, a display interface including a text notification icon can include a view for displaying text and a view for displaying images.

[0128] The phone manager is used to provide communication functions of the terminal device 100, such as management of call status (including connection, disconnection, etc.).

[0129] The resource manager provides various resources for applications, such as localized strings, icons, images, layout files, video files, and so on.

[0130] The notification manager allows applications to display notification information in the status bar. It can be used to convey notification-type messages and can disappear automatically after a short stay without user interaction. For example, notification information is used to inform the completion of downloads, message reminders, etc. Notification information can also appear in the system top status bar in the form of icons or scroll bar text, such as notifications of applications running in the background, or in the form of dialog windows on the screen. Notification information can also be, for example, text information prompted in the status bar, a prompt sound, vibration of electronic equipment, flashing indicator lights, etc.

[0131] It should be pointed out that the camera application can also call the content provider, resource manager, notification manager, window manager, view system, etc. according to actual business needs, and the embodiments of the present application do not impose any restrictions on this.

[0132] The kernel layer is a layer between hardware and software. The kernel layer includes at least a camera driver. The camera driver can be used to drive a hardware module with a shooting function, such as a camera sensor. In other words, the camera driver is responsible for data interaction with the camera sensor. Of course, the kernel layer may also include a display driver, an audio driver, a sensor driver, etc., and the embodiments of the present application do not impose any restrictions on this.

[0133] In addition, the HAL layer can encapsulate the drivers in the kernel layer and provide a calling interface to the application framework layer, shielding the implementation details of the low-level hardware. Fig. 9 As shown, the above HAL layer may include Camera HAL and the like.

[0134] Camera HAL is the core software framework of Camera, which may include a template feature extraction module, a video frame feature extraction module, a matching score calculation module, a correlation calculation module, a target coordinate determination module, etc. In addition, Camera HAL may also include other functional modules, including but not limited to an image signal processing front end (ImageSingnal Processing Front End, IFE) module, a spatial alignment transform (Spatial AlignmentTransform, SAT) module, an image signal processing back end (ImageSingnal Processing Post End, IPE) module, etc., which are not limited in this embodiment.

[0135] Among them, the template feature extraction module, video frame feature extraction module, matching score calculation module, and target coordinate determination module are functional modules generated based on the XX platform framework, while the correlation calculation module does not rely on the XX platform framework, but is a functional module generated based on custom programming.

[0136] In this embodiment, the template feature extraction module is used to extract features from the template image block of the tracking object through CNN to obtain the template features of the tracking object. The video frame feature extraction module is used to extract features from the video image frame to be tracked through CNN to obtain a feature map corresponding to the video image frame to be tracked. The correlation calculation module is used to perform dynamic convolution calculation based on the template features and the feature map to obtain a correlation calculation result. The matching score calculation module is used to calculate the matching score of the tracking object according to the correlation calculation result. The target coordinate determination module is used to calculate the position information of the tracking object in the video image frame, such as the coordinate information of the target frame, according to the correlation calculation result, and generate a target tracking result based on the position information of the tracking object in the video image frame and the video image frame to be tracked when the matching score of the tracking object meets the preset conditions.

[0137] Understandably, Fig. 9 The layers in the software structure shown and the components included in each layer do not constitute a specific limitation on the electronic device 100. In other embodiments of the present application, the electronic device 100 may include more or fewer layers than shown, and each layer may include more or fewer components, which is not limited in the present application.

[0138] It is understandable that in order to implement the single target tracking method in the embodiment of the present application, the electronic device includes hardware and / or software modules corresponding to the execution of each function. In combination with the algorithm steps of each example described in the embodiment disclosed herein, the present application can be implemented in the form of hardware or a combination of hardware and computer software. Whether a function is executed in the form of hardware or computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application in combination with the embodiments, but such implementation should not be considered to be beyond the scope of the present application.

[0139] This embodiment also provides a computer storage medium, in which computer instructions are stored. When the computer instructions are executed on an electronic device, the electronic device executes the above-mentioned related method steps to implement the single target tracking method in the above-mentioned embodiment.

[0140] This embodiment further provides a computer program product. When the computer program product is run on a computer, the computer is enabled to execute the above-mentioned related steps to implement the single target tracking method in the above-mentioned embodiment.

[0141] In addition, an embodiment of the present application also provides a device, which can specifically be a chip, component or module, and the device may include a connected processor and memory; wherein the memory is used to store computer-executable instructions, and when the device is running, the processor can execute the computer-executable instructions stored in the memory so that the chip executes the single target tracking method in the above-mentioned method embodiments.

[0142] An embodiment of the present application also provides a chip, which includes one or more interface circuits and one or more processors; the interface circuit is used to receive signals from the memory of the electronic device and send signals to the processor, the signals including computer instructions stored in the memory; when the processor executes the computer instructions, the electronic device executes the single target tracking method in the above-mentioned method embodiments.

[0143] Among them, the electronic device (such as a mobile phone, etc.), computer storage medium, computer program product or chip provided in this embodiment is used to execute the corresponding method provided above. Therefore, the beneficial effects that can be achieved can refer to the beneficial effects in the corresponding method provided above, and will not be repeated here.

[0144] Through the description of the above implementation methods, technical personnel in the relevant field can understand that for the convenience and simplicity of description, only the division of the above-mentioned functional modules is used as an example. In actual applications, the above-mentioned functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above.

[0145] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic, for example, the division of modules or units is only a logical function division, and there may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0146] As described above, the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present application.

Claims

1. A single target tracking method, characterized in that: Applied in an electronic device, the electronic device comprises: a template feature extraction module, a video frame feature extraction module, a correlation calculation module, a matching score calculation module and a target coordinate determination module; Wherein, the template feature extraction module, the video frame feature extraction module, the matching score calculation module and the target coordinate determination module are generated based on the target platform framework, and the correlation calculation module is generated based on custom programming; The method comprises: In response to a user selecting a tracking object in the first preview image frame, the template feature extraction module extracts a template feature corresponding to the tracking object; The video frame feature extraction module extracts a feature map of a second preview image frame; wherein the second preview image frame is an image frame acquired after the first preview image frame; The correlation calculation module reads the template feature and the feature map of the second preview image frame, performs dynamic convolution acceleration calculation on the feature map of the second preview image frame and the template feature, and obtains a correlation calculation result corresponding to the second preview image frame; wherein each element in the feature map of the second preview image frame is integer data, each element in the template feature is integer data, each integer data is predefined with a conversion ratio and an offset, and the conversion ratio and the offset are used to convert the integer data into floating-point data; The correlation calculation module reads the template feature and the feature map of the second preview image frame, and performs dynamic convolution acceleration calculation on the feature map of the second preview image frame and the template feature, including: Perform multiplication calculation on a first element a in the feature map of the second preview image frame and a second element b in the template feature; Wherein, when calculating the product c of the first element a and the second element b, the product c is solved based on the equivalent floating-point data of the first element a and the equivalent floating-point data of the second element b, and the product c is integer data; When solving the product c, a predetermined first target integer data is used for multiplication calculation, and a product result corresponding to the first target integer data is subjected to a data binary right shift operation of n1 bits; wherein the first target integer data is a result of expanding the relevant floating-point data involved in solving the product c by 2 n1 times, the relevant floating-point data is determined according to the conversion ratio of the first element a, the conversion ratio of the second element b and the conversion ratio of the product c, and n1 is an integer; The matching score calculation module calculates a tracking object matching score corresponding to the second preview image frame according to a correlation calculation result corresponding to the second preview image frame; The target coordinate determination module calculates the position information of the tracked object in the second preview image frame according to the correlation calculation result corresponding to the second preview image frame, and generates a target tracking result according to the position information of the tracked object in the second preview image frame and the second preview image frame when the tracking object matching score corresponding to the second preview image frame meets a preset condition.

2. The method according to claim 1, characterized in that The data arrangement format of the feature map of the second preview image frame is the HWC format.

3. The method according to claim 2, characterized in that The correlation calculation module reads the template feature and the feature map of the second preview image frame, and performs dynamic convolution acceleration calculation on the feature map of the second preview image frame and the template feature, including: A data vector in the channel number dimension in the feature map of the second preview image frame is used as a processing unit, and dynamic convolution calculation of the feature map of the second preview image frame and the template feature is implemented based on vectorized instructions.

4. The method according to claim 3, characterized in that The vectorized instruction includes at least one of the following: Vector addition instructions, vector multiplication instructions, vector fetch instructions, vector store instructions, vector data arrangement and transformation instructions, and vector data bit width expansion and reduction instructions.

5. The method according to claim 3, characterized in that: Taking a data vector in the channel number dimension in the feature map of the second preview image frame as a processing unit, implementing dynamic convolution acceleration calculation of the feature map of the second preview image frame and the template feature based on a vectorized instruction, including: Perform multiplication calculation on a first data vector A in the channel number dimension in the feature map of the second preview image frame and a third element d in the template feature; Wherein, when calculating the product data vector E of the first data vector A and the third element d, the product vector E is solved based on the equivalent floating-point data vector of the first data vector A and the equivalent floating-point data of the third element d, and the product data vector E is an integer data vector; When solving the product data vector E, a predetermined second target integer data is used for multiplication calculation, and a product result corresponding to the second target integer data is subjected to a data binary right shift operation of n2 bits; wherein the second target integer data is a result of expanding the relevant floating-point data involved in solving the product vector E by 2 n2 The relevant floating-point data is obtained by multiplying the conversion ratio of the first data vector A, the conversion ratio of the third element d and the conversion ratio of the product vector E, and n2 is an integer.

6. The method according to claim 1, characterized in that Also includes: The video frame feature extraction module extracts a feature map of a third preview image frame; wherein the third preview image frame is an image frame acquired after the second preview image frame; The correlation calculation module reads the template feature and the feature map of the third preview image frame, performs dynamic convolution acceleration calculation on the feature map of the third preview image frame and the template feature, and obtains a correlation calculation result corresponding to the third preview image frame; The matching score calculation module calculates a tracking object matching score corresponding to the third preview image frame according to a correlation calculation result corresponding to the third preview image frame; The target coordinate determination module calculates the position information of the tracking object in the third preview image frame according to the correlation calculation result corresponding to the third preview image frame, and generates a target tracking result according to the position information of the tracking object in the third preview image frame and the third preview image frame when the tracking object matching score corresponding to the third preview image frame meets a preset condition.

7. An electronic device, characterized in that: include: one or more processors; Memory; and one or more computer programs, wherein the one or more computer programs are stored in the memory, and when the computer programs are executed by the one or more processors, the electronic device executes the single target tracking method as described in any one of claims 1-6.

8. A computer-readable storage medium comprising a computer program, characterized in that: When the computer program is executed on an electronic device, the electronic device executes the single target tracking method according to any one of claims 1 to 6.

9. A chip, characterized in that: It includes one or more interface circuits and one or more processors; the interface circuit is used to receive a signal from a memory of an electronic device and send the signal to the processor, the signal including a computer instruction stored in the memory; when the processor executes the computer instruction, the electronic device executes the single target tracking method as described in any one of claims 1 to 6.

Images