Single target tracking method, device, terminal and storage medium

Abstract

The present invention discloses a single target tracking method, device, terminal and storage medium, the method comprising: inputting a reference image into a feature extraction network to obtain a first feature image output by the feature extraction network, inputting an image to be tracked into the feature extraction network to obtain a second feature image output by the feature extraction network, wherein the reference image includes target marking information; inputting the first feature image into an encoder to obtain an intermediate feature image output by the encoder; inputting the intermediate feature image and the second feature image into a decoder to obtain output data of the decoder, and determining a target tracking result in the image to be tracked according to the output data of the decoder, wherein the decoder includes a cross-correlation operation. When the present invention tracks a target in an image to be tracked, the decoder combines a cross-correlation operation, realizes more accurate feature fusion, and reduces interference from background or other information.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to a single target tracking method, device, terminal and storage medium. Background Art

[0002] Target tracking technology is widely used in actual production and life scenarios. Typical application scenarios include intelligent video surveillance systems, intelligent human-computer interaction, intelligent transportation and visual navigation systems. As the application scenarios of target tracking technology become more and more extensive, target tracking models need to face more and more complex scenarios. However, the accuracy of existing target tracking models is not high.

[0003] Therefore, the prior art still needs to be improved and enhanced. Summary of the invention

[0004] In view of the above-mentioned defects in the prior art, the present invention provides a single target tracking method, device, terminal and storage medium, aiming to solve the problem of low accuracy of the target tracking model in the prior art.

[0005] In order to solve the above technical problems, the technical solution adopted by the present invention is as follows:

[0006] A first aspect of the present invention provides a single target tracking method, the method comprising:

[0007] Inputting a reference image into a feature extraction network to obtain a first feature image output by the feature extraction network, inputting an image to be tracked into the feature extraction network to obtain a second feature image output by the feature extraction network, wherein the reference image includes target marking information;

[0008] Inputting the first feature image into an encoder to obtain an intermediate feature image output by the encoder;

[0009] The intermediate feature image and the second feature image are input into a decoder to obtain output data of the decoder, and a target tracking result in the image to be tracked is determined according to the output data of the decoder, wherein the decoder includes a cross-correlation operation.

[0010] The single target tracking method, wherein before inputting the image to be tracked into the feature extraction network, further comprises:

[0011] Preprocessing the initial image to be tracked according to the target tracking result of the first image to obtain the image to be tracked;

[0012] The first image is a previous frame of the initial image to be tracked.

[0013] The single target tracking method, wherein the preprocessing of the initial image to be tracked comprises:

[0014] Cutting out a partial area from the initial image to be tracked as the image to be tracked;

[0015] The coordinates of the center point of the image to be tracked are the same as the coordinates of the center point of the target in the first image, and the area of ​​the image to be tracked is N times the area of ​​the area occupied by the target in the first image, where N>1.

[0016] The single target tracking method, wherein the step of inputting the intermediate feature image and the second feature image into a decoder to obtain output data of the decoder comprises:

[0017] Decomposing the intermediate feature image to obtain a plurality of first intermediate features and a plurality of second intermediate features;

[0018] Wherein, the size of each of the first intermediate features is C*1*1, C is the number of channels of the intermediate feature image, and the feature value in the jth first intermediate feature is the feature value of the jth pixel position in each channel of the intermediate feature image; the size of each of the second intermediate features is H0W0*1*1, H0 and W0 are respectively the length and width of the image in each channel of the intermediate feature image, and the feature value in the i-th second intermediate feature is the feature value in the i-th channel of the intermediate feature image;

[0019] In the decoder, a cross-correlation operation is performed based on the first intermediate feature, the second intermediate feature and the second feature image, and output data of the decoder is obtained based on the result of the cross-correlation operation.

[0020] The single target tracking method, wherein the performing a cross-correlation operation based on the first intermediate feature, the second intermediate feature and the second feature image comprises:

[0021] Inputting the second feature image into the multi-head attention module in the decoder to obtain attention features;

[0022] Performing a first cross-correlation operation on the attention feature and the first intermediate feature to obtain a first cross-correlation operation result;

[0023] A second cross-correlation operation is performed on the first cross-correlation operation result and the second intermediate feature to obtain a second cross-correlation operation result.

[0024] The single target tracking method, wherein the formula for the first cross-correlation operation is:

[0025]

[0026] Wherein, * represents the cross-correlation operation; represents the attention feature; H and W are the length and width of the feature map in each channel of the attention feature respectively; T j is the jth first intermediate feature; P is the first cross-correlation operation result, the first cross-correlation operation is composed of multiple first data, P j is the jth first data;

[0027] The formula for the second cross-correlation operation is:

[0028] O={O i |O i =T i *P} i∈{1，...，C} ;

[0029] Among them, T i is the i-th second intermediate feature; O is the second cross-correlation operation result, the second cross-correlation operation is composed of multiple second data, O i is the i-th second data.

[0030] The single target tracking method, wherein the output data of the decoder includes a target candidate result and a matching degree corresponding to the target candidate result; and determining the target tracking result in the image to be tracked according to the output data of the decoder includes:

[0031] According to the matching degrees corresponding to the target candidate results, the target candidate result with the highest matching degree is selected as the target tracking.

[0032] A second aspect of the present invention provides a single target tracking device, comprising:

[0033] A feature extraction module, wherein the feature extraction module is used to input a reference image into a feature extraction network to obtain a first feature image output by the feature extraction network, and input an image to be tracked into the feature extraction network to obtain a second feature image output by the feature extraction network, wherein the reference image includes target marking information;

[0034] An intermediate feature acquisition module, the intermediate feature acquisition module is used to input the first feature image into an encoder and acquire an intermediate feature image output by the encoder;

[0035] A tracking result acquisition module, wherein the tracking result acquisition module is used to input the intermediate feature image and the second feature image into a decoder to obtain output data of the decoder, and determine the target tracking result in the image to be tracked according to the output data of the decoder, wherein the decoder includes a cross-correlation operation.

[0036] According to a third aspect of the present invention, a terminal is provided, comprising a processor and a computer-readable storage medium communicatively connected to the processor, wherein the computer-readable storage medium is suitable for storing a plurality of instructions, and the processor is suitable for calling the instructions in the computer-readable storage medium to execute the steps of implementing any one of the single target tracking methods described above.

[0037] According to a fourth aspect of the present invention, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores one or more programs, and the one or more programs can be executed by one or more processors to implement the steps of any of the single target tracking methods described above.

[0038] Compared with the prior art, the present invention provides a single target tracking method, wherein the single target tracking method inputs a reference image into a feature extraction network to obtain a first feature image output by the feature extraction network, inputs an image to be tracked into the feature extraction network to obtain a second feature image output by the feature extraction network, wherein the reference image includes target tag information; inputs the first feature image into an encoder to obtain an intermediate feature image output by the encoder; inputs the intermediate feature image and the second feature image into a decoder to obtain output data of the decoder, and determines the target tracking result in the image to be tracked according to the output data of the decoder, wherein the decoder includes a cross-correlation operation. When the present invention tracks a target in an image to be tracked, a cross-correlation operation is combined in the decoder to achieve more accurate feature fusion and reduce interference from background or other information. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1 A flow chart of an embodiment of a single target tracking method provided by the present invention;

[0040] Figure 2 A model structure diagram of an embodiment of a single target tracking method provided by the present invention;

[0041] Figure 3 A schematic diagram of two-stage cross-correlation of an embodiment of a single target tracking method provided by the present invention;

[0042] Figure 4 A schematic diagram of two-stage cross-correlation of an embodiment of a single target tracking method provided by the present invention;

[0043] Figure 5 A structural principle diagram of an embodiment of a single target tracking device provided by the present invention;

[0044] Figure 6 A schematic diagram of the principles of an embodiment of a terminal provided by the present invention. DETAILED DESCRIPTION

[0045] In order to make the purpose, technical solution and effect of the present invention clearer and more specific, the present invention is further described in detail with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0046] Those skilled in the art will appreciate that, unless expressly stated, the singular forms '"a", "an", "said" and "the" used herein may also include plural forms. It should be further understood that the term "comprising" used in the specification of the present invention refers to the presence of the features, integers, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or groups thereof. It should be understood that when we refer to an element as being "connected" or "coupled" to another element, it may be directly connected or coupled to the other element, or there may be intermediate elements. In addition, the "connection" or "coupling" used herein may include wireless connection or wireless coupling. The term "and / or" used herein includes all or any unit and all combinations of one or more associated listed items.

[0047] It will be understood by those skilled in the art that, unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as generally understood by those skilled in the art in the art to which the present invention belongs. It should also be understood that terms such as those defined in general dictionaries should be understood to have meanings consistent with the meanings in the context of the prior art, and will not be interpreted with idealized or overly formal meanings unless specifically defined as herein.

[0048] The single target tracking method provided by the present invention can be applied to a terminal with computing capability, and the terminal can execute the single target tracking method provided by the present invention to track the target in a video sequence.

[0049] Embodiment 1

[0050] This embodiment is a single target tracking method. When tracking a target, a single target tracking model based on Transformer combined with cross-correlation is constructed. The model takes the search region (Search Region) and the template (Template) as input. The two branches first extract features through the backbone network (Backbone) of the same structure, and then further extract and fuse features through the Transformer combined with cross-correlation. Finally, the two branches output classification and regression predictions respectively. The final result is screened by the size of the classification value, and the final tracking result is generated after post-processing.

[0051] like Figure 1As shown, in one embodiment of the single target tracking method provided by the present invention, the single target tracking comprises the steps of:

[0052] S100, inputting a reference image into a feature extraction network to obtain a first feature image output by the feature extraction network, inputting an image to be tracked into the feature extraction network to obtain a second feature image output by the feature extraction network, wherein the reference image includes target marking information.

[0053] Reference Figure 2 The single target tracking model adopted in this embodiment is a dual-branch input, the two branches are a template and a search area, the template is the reference image, and the search area is the image to be tracked, wherein, in this embodiment, the reference image is the first frame in the video sequence, the reference image includes target marking information, and the target marking information is the marking information manually marked with a box, that is, the target to be tracked is marked with a box in the reference image, it can be understood that the reference image can also be any frame in the video sequence before the image to be tracked, in a possible implementation, the reference image can be updated every once in a while, so that the target marking information in the reference image can provide more accurate reference information for the target tracking in the image to be tracked. Each frame of the video sequence to be tracked except the reference image is sequentially input into the single target tracking model in this embodiment to obtain the target tracking result of each frame of the video sequence to be tracked except the reference image.

[0054] The reference image is input into the feature extraction network to obtain a first feature image output by the feature extraction network, and the image to be tracked is input into the feature extraction network to obtain a second feature image output by the feature extraction network. That is to say, in this embodiment, the structure and parameters of the network used to extract the initial features of the reference image and the image to be tracked are the same.

[0055] In a possible implementation, the image to be tracked may be any frame except the first frame in the video sequence. In this embodiment, in order to improve efficiency, any frame except the first frame in the video sequence is further processed as the image to be tracked. Before inputting the image to be tracked into the feature extraction network, the method further includes:

[0056] S110, preprocessing the initial image to be tracked according to the target tracking result of the first image to obtain the image to be tracked;

[0057] S120: The first image is a previous frame of the initial image to be tracked.

[0058] The initial image to be tracked is each frame in the video sequence to be tracked except the reference image. Before the image to be tracked is input into the feature extraction network, the image to be tracked is preprocessed according to the tracking result of the previous frame of the image to be tracked in the video sequence to be tracked. In this embodiment, the previous frame of the image to be tracked in the video sequence to be tracked is the first image.

[0059] Specifically, the preprocessing of the initial image to be tracked includes:

[0060] Cutting out a partial area from the initial image to be tracked as the image to be tracked;

[0061] The coordinates of the center point of the image to be tracked are the same as the coordinates of the center point of the target in the first image, and the area of ​​the image to be tracked is N times the area of ​​the area occupied by the target in the first image, where N>1.

[0062] Specifically, the coordinates of the initial tracking image corresponding to the coordinates of the center point of the first image are used as the center point, and an area N times the area occupied by the target in the first image is cut out as the image to be tracked. Wherein N is a number greater than 1, and in this embodiment, N is 4.

[0063] The single target tracking method described in this embodiment further includes the steps of:

[0064] S200: Input the first feature image to an encoder to obtain an intermediate feature image output by the encoder.

[0065] S300, inputting the intermediate feature image and the second feature image into a decoder to obtain output data of the decoder, and determining a target tracking result in the image to be tracked according to the output data of the decoder, wherein the decoder includes a cross-correlation operation.

[0066] In the tracking framework provided in this embodiment, the encoder and decoder adopt the Transformer structure, which is located after the feature extraction network. Its function is to further extract and fuse features. Compared with the ordinary convolutional network structure, the unique structure of the Transformer can more effectively capture global semantic information, thereby reducing semantic loss. Fusion of the Transformer structure with the feature extraction network can effectively solve the problem that the cross-correlation operation in the feature extraction network can only calculate the similarity between local image blocks, thereby causing global semantic loss.

[0067] Specifically, the Transformer structure is as follows Figure 3As shown in the figure, at the input end, the encoder is connected to the template branch and the decoder is connected to the search region branch. In the internal structure, the position encoding is used to increase the position prior, the multi-head attention module is used to capture global semantic information, and the feedforward network and the normalization layer are used to connect different network modules to build a deep network.

[0068] The first feature image extracted by the feature extraction network in the template branch is input to the encoder, and the intermediate feature image output by the encoder is obtained, which is represented by T. Wherein, C is the number of channels of the intermediate feature image, and H0 and W0 are the length and width of the image in each channel of the intermediate feature image, respectively.

[0069] The intermediate feature image and the second feature image extracted by the feature extraction network in the search area branch are input into a decoder to obtain output data of the decoder.

[0070] The step of inputting the intermediate feature image and the second feature image into a decoder to obtain output data of the decoder includes:

[0071] S310, disassembling the intermediate feature image to obtain a plurality of first intermediate features and a plurality of second intermediate features;

[0072] Wherein, the size of each of the first intermediate features is C*1*1, C is the number of channels of the intermediate feature image, and the feature value in the jth first intermediate feature is the feature value of the jth pixel position in each channel of the intermediate feature image; the size of each of the second intermediate features is H0W0*1*1, H0 and W0 are respectively the length and width of the image in each channel of the intermediate feature image, and the feature value in the i-th second intermediate feature is the feature value in the i-th channel of the intermediate feature image;

[0073] Specifically, the first intermediate feature is a feature image obtained by splitting the intermediate feature image T according to channels. In this embodiment, T j represents the jth first intermediate feature; the second intermediate feature is a feature image obtained by splitting the intermediate feature image T according to the spatial position. In this embodiment, T i represents the i-th said first intermediate feature.

[0074] S320. In the decoder, a cross-correlation operation is performed according to the first intermediate feature, the second intermediate feature and the second feature image, and output data of the decoder is obtained according to a result of the cross-correlation operation.

[0075] The performing of a cross-correlation operation according to the first intermediate feature, the second intermediate feature and the second feature image comprises:

[0076] S321, inputting the second feature image into the multi-head attention module in the decoder to obtain attention features;

[0077] S322, performing a first cross-correlation operation on the attention feature and the first intermediate feature to obtain a first cross-correlation operation result;

[0078] S323: Perform a second cross-correlation operation on the first cross-correlation operation result and the second intermediate feature to obtain a second cross-correlation operation result.

[0079] Different from the general Transformer structure, in this embodiment, a two-stage cross-correlation module (Two-stagePixel-wiseCorrelation) is added after the multi-head attention module of the decoder in the Transformer structure to fuse the features extracted by the two branches. Furthermore, the cross-correlation operation combined with the Transformer structure proposed in this embodiment further improves the accuracy of feature fusion after multiple stacking of the decoder. After processing, the decoder based on cross-correlation finally generates the output result of the Transformer structure. The two cross-correlation structures are as follows: Figure 4 shown.

[0080] Specifically, the second feature image is input into the multi-head attention module in the decoder to obtain the attention feature, where the attention feature is represented by S. H and W are the length and width of the feature map in each channel of the attention feature, respectively;

[0081] Performing a first cross-correlation operation on the attention feature and the first intermediate feature to obtain a first cross-correlation operation result;

[0082] Wherein, the formula for the first cross-correlation operation is:

[0083]

[0084] Wherein, * represents the cross-correlation operation; represents the attention feature; T j is the jth first intermediate feature; P is the first cross-correlation operation result, The first cross-correlation operation is composed of a plurality of first data, P j is the jth first data. Specifically, the specific implementation of the first cross-correlation operation can be subdivided into cross-correlation operations at H0*W0 positions. Pj is the result of each operation. These results together constitute the result of the first operation.

[0085] Perform a second cross-correlation operation on the first cross-correlation operation result and the second intermediate feature to obtain a second cross-correlation operation result;

[0086] Wherein, the formula for the second cross-correlation operation is:

[0087] O={O i |O i =T i *P} i∈{1，...，C}

[0088] Among them, T i is the i-th second intermediate feature; O is the second cross-correlation operation result, the second cross-correlation operation is composed of multiple second data, O i is the i-th second data. Specifically, the specific implementation of the second cross-correlation operation can be subdivided into cross-correlation operations at C positions. i These results together constitute the result of the second operation.

[0089] The main purpose of the first stage of cross-correlation calculation is to fuse the spatial position information at the pixel level, which can effectively improve the accuracy of feature fusion.

[0090] In the first stage of cross-correlation calculation, after the template is split at the pixel level, a large amount of interference information is introduced while improving the accuracy of the fused information. Therefore, in this embodiment, the second stage of cross-correlation operation is further designed in the single target tracking model for processing. The specific method is that the second stage of cross-correlation operation decomposes the intermediate features on the channel. The purpose of doing this is to take into account the correlation between the channel and the object category, so splitting the channel can make the feature fusion more focused on a specific type of tracking target.

[0091] Inputting the second cross-correlation operation result into a feedforward network layer of a decoder to obtain output data of the decoder, wherein the output data of the decoder includes a target candidate result and a matching degree corresponding to the target candidate result; determining the target tracking result in the image to be tracked according to the output data of the decoder includes:

[0092] S330: According to the matching degrees corresponding to the target candidate results, select the target candidate result with the highest matching degree as the target tracking.

[0093] Reference Figure 2It can be seen that the output data of the decoder includes a state estimation branch and a target classification branch. The state estimation branch is a target candidate result, and the target candidate result includes the position and shape of each candidate result, which is represented by boxes of different sizes; the target classification branch is the matching degree corresponding to the target candidate result, including the score value of each target candidate result on the image to be tracked, and the score value is used to judge the matching degree between the target candidate result and the tracking target. According to the matching degree corresponding to the target candidate result, the target candidate result with the highest matching degree is selected as the target tracking.

[0094] When initializing the single target tracking model in the single target tracking method provided in this embodiment, the weights of the trained network are selected as the initialization weights of the feature extraction network, and then the feature extraction network is trained together with the remaining model networks to obtain the final weights.

[0095] In summary, this embodiment provides a single target tracking method, inputs a reference image into a feature extraction network to obtain a first feature image output by the feature extraction network, inputs an image to be tracked into the feature extraction network to obtain a second feature image output by the feature extraction network, wherein the reference image includes target marking information; inputs the first feature image into an encoder to obtain an intermediate feature image output by the encoder; inputs the intermediate feature image and the second feature image into a decoder to obtain output data of the decoder, and determines the target tracking result in the image to be tracked according to the output data of the decoder, wherein the decoder includes a cross-correlation operation. When the present invention tracks a target in an image to be tracked, a cross-correlation operation is combined in the decoder to achieve more accurate feature fusion and reduce interference from background or other information.

[0096] It should be understood that, although the various steps in the flowcharts given in the accompanying drawings of the present specification are displayed in sequence according to the indications of the arrows, these steps are not necessarily executed in sequence in the order indicated by the arrows. Unless there is a clear description in this article, there is no strict order restriction on the execution of these steps, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowchart may include multiple sub-steps or multiple stages, and these sub-steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these sub-steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the sub-steps or stages of other steps.

[0097] Those of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided by the present invention may include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0098] Embodiment 2

[0099] Based on the above embodiments, the present invention also provides a single target tracking device. Figure 5 As shown, the single target tracking device comprises:

[0100] A feature extraction module, the feature extraction module is used to input a reference image into a feature extraction network, obtain a first feature image output by the feature extraction network, input an image to be tracked into the feature extraction network, obtain a second feature image output by the feature extraction network, wherein the reference image includes target marking information, as specifically described in the first embodiment;

[0101] An intermediate feature acquisition module, the intermediate feature acquisition module is used to input the first feature image into an encoder to acquire an intermediate feature image output by the encoder, as specifically described in Embodiment 1;

[0102] A tracking result acquisition module, the tracking result acquisition module is used to input the intermediate feature image and the second feature image into a decoder to obtain output data of the decoder, and determine the target tracking result in the image to be tracked according to the output data of the decoder, wherein the decoder includes a cross-correlation operation, as specifically described in Example 1.

[0103] Embodiment 3

[0104] Based on the above embodiments, the present invention also provides a terminal, such as Figure 6 As shown, the terminal includes a processor 10 and a memory 20. Figure 6 Only some components of the terminal are shown, but it should be understood that it is not required to implement all of the components shown, and more or fewer components may be implemented instead.

[0105] In some embodiments, the memory 20 may be an internal storage unit of the terminal, such as a hard disk or memory of the terminal. In other embodiments, the memory 20 may also be an external storage device of the terminal, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. equipped on the terminal. Further, the memory 20 may also include both an internal storage unit of the terminal and an external storage device. The memory 20 is used to store application software and various types of data installed in the terminal. The memory 20 may also be used to temporarily store data that has been output or is to be output. In one embodiment, a single target tracking program 30 is stored on the memory 20, and the single target tracking program 30 can be executed by the processor 10, thereby realizing the single target tracking method in the present application.

[0106] In some embodiments, the processor 10 may be a central processing unit (CPU), a microprocessor or other chip, used to run the program code or process data stored in the memory 20, such as executing the single target tracking method.

[0107] In one embodiment, when the processor 10 executes the single target tracking program 30 in the memory 20, the following steps are implemented:

[0108] Inputting a reference image into a feature extraction network to obtain a first feature image output by the feature extraction network, inputting an image to be tracked into the feature extraction network to obtain a second feature image output by the feature extraction network, wherein the reference image includes target marking information;

[0109] Inputting the first feature image into an encoder to obtain an intermediate feature image output by the encoder;

[0110] The intermediate feature image and the second feature image are input into a decoder to obtain output data of the decoder, and a target tracking result in the image to be tracked is determined according to the output data of the decoder, wherein the decoder includes a cross-correlation operation.

[0111] Before inputting the image to be tracked into the feature extraction network, the method further includes:

[0112] Preprocessing the initial image to be tracked according to the target tracking result of the first image to obtain the image to be tracked;

[0113] The first image is a previous frame of the initial image to be tracked.

[0114] The initial image to be tracked is preprocessed, including:

[0115] Cutting out a partial area from the initial image to be tracked as the image to be tracked;

[0116] The coordinates of the center point of the image to be tracked are the same as the coordinates of the center point of the target in the first image, and the area of ​​the image to be tracked is N times the area of ​​the area occupied by the target in the first image, where N>1.

[0117] The step of inputting the intermediate feature image and the second feature image into a decoder to obtain output data of the decoder includes:

[0118] Decomposing the intermediate feature image to obtain a plurality of first intermediate features and a plurality of second intermediate features;

[0119] Wherein, the size of each of the first intermediate features is C*1*1, C is the number of channels of the intermediate feature image, and the feature value in the jth first intermediate feature is the feature value of the jth pixel position in each channel of the intermediate feature image; the size of each of the second intermediate features is H0W0*1*1, H0 and W0 are respectively the length and width of the image in each channel of the intermediate feature image, and the feature value in the i-th second intermediate feature is the feature value in the i-th channel of the intermediate feature image;

[0120] In the decoder, a cross-correlation operation is performed based on the first intermediate feature, the second intermediate feature and the second feature image, and output data of the decoder is obtained based on the result of the cross-correlation operation.

[0121] The performing of a cross-correlation operation according to the first intermediate feature, the second intermediate feature and the second feature image comprises:

[0122] Inputting the second feature image into the multi-head attention module in the decoder to obtain attention features;

[0123] Performing a first cross-correlation operation on the attention feature and the first intermediate feature to obtain a first cross-correlation operation result;

[0124] A second cross-correlation operation is performed on the first cross-correlation operation result and the second intermediate feature to obtain a second cross-correlation operation result.

[0125] Wherein, the formula for the first cross-correlation operation is:

[0126]

[0127] Wherein, * represents the cross-correlation operation; represents the attention feature; H and W are the length and width of the feature map in each channel of the attention feature respectively; T j is the jth first intermediate feature; P is the first cross-correlation operation result, the first cross-correlation operation is composed of multiple first data, P j is the jth first data;

[0128] The formula for the second cross-correlation operation is:

[0129] O={O i |O i =T i *P} i∈{1，...，C} ;

[0130] Among them, T i is the i-th second intermediate feature; O is the second cross-correlation operation result, the second cross-correlation operation is composed of multiple second data, O i is the i-th second data.

[0131] The output data of the decoder includes a target candidate result and a matching degree corresponding to the target candidate result; and determining the target tracking result in the image to be tracked according to the output data of the decoder includes:

[0132] According to the matching degrees corresponding to the target candidate results, the target candidate result with the highest matching degree is selected as the target tracking.

[0133] Embodiment 4

[0134] The present invention also provides a computer-readable storage medium, in which one or more programs are stored. The one or more programs can be executed by one or more processors to implement the steps of the single target tracking method as described above.

[0135] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A single target tracking method, characterized in that: The single target tracking method comprises: Inputting a reference image into a feature extraction network to obtain a first feature image output by the feature extraction network, inputting an image to be tracked into the feature extraction network to obtain a second feature image output by the feature extraction network, wherein the reference image includes target marking information; Inputting the first feature image into an encoder to obtain an intermediate feature image output by the encoder; Inputting the intermediate feature image and the second feature image into a decoder to obtain output data of the decoder, and determining a target tracking result in the image to be tracked according to the output data of the decoder; The step of inputting the intermediate feature image and the second feature image into a decoder to obtain output data of the decoder includes: Decomposing the intermediate feature image to obtain a plurality of first intermediate features and a plurality of second intermediate features; Wherein, the size of each of the first intermediate features is C*1*1, C is the number of channels of the intermediate feature image, and the feature value in the jth first intermediate feature is the feature value of the jth pixel position in each channel of the intermediate feature image; the size of each of the second intermediate features is H0W0*1*1, H0 and W0 are respectively the length and width of the image in each channel of the intermediate feature image, and the feature value in the i-th second intermediate feature is the feature value in the i-th channel of the intermediate feature image; In the decoder, inputting the second feature image into a multi-head attention module in the decoder to obtain attention features; Performing a first cross-correlation operation on the attention feature and the first intermediate feature to obtain a first cross-correlation operation result; Perform a second cross-correlation operation on the first cross-correlation operation result and the second intermediate feature to obtain a second cross-correlation operation result; The output data of the decoder is obtained according to the second cross-correlation operation result.

2. The single target tracking method according to claim 1, characterized in that: Before inputting the image to be tracked into the feature extraction network, the method further includes: Preprocessing the initial image to be tracked according to the target tracking result of the first image to obtain the image to be tracked; The first image is a previous frame of the initial image to be tracked.

3. The single target tracking method according to claim 2, characterized in that: The preprocessing of the initial image to be tracked includes: Cutting out a partial area from the initial image to be tracked as the image to be tracked; The coordinates of the center point of the image to be tracked are the same as the coordinates of the center point of the target in the first image, and the area of ​​the image to be tracked is N times the area of ​​the area occupied by the target in the first image, where N>1.

4. The single target tracking method according to claim 1, characterized in that: The formula for the first cross-correlation operation is: Wherein, * represents the cross-correlation operation; represents the attention feature; H and W are the length and width of the feature map in each channel of the attention feature respectively; T j is the jth first intermediate feature; P is the first cross-correlation operation result, the first cross-correlation operation is composed of multiple first data, P j is the jth first data; The formula for the second cross-correlation operation is: O={O i |O i =T i *P} i∈{1,...,C} ; Among them, T i is the i-th second intermediate feature; O is the second cross-correlation operation result, the second cross-correlation operation is composed of multiple second data, O i is the i-th second data.

5. The single target tracking method according to claim 1, characterized in that: The output data of the decoder includes a target candidate result and a matching degree corresponding to the target candidate result; The step of determining the target tracking result in the image to be tracked according to the output data of the decoder comprises: According to the matching degrees corresponding to the target candidate results, the target candidate result with the highest matching degree is selected as the target tracking.

6. A single target tracking device, characterized in that: include: A feature extraction module, wherein the feature extraction module is used to input a reference image into a feature extraction network to obtain a first feature image output by the feature extraction network, and input an image to be tracked into the feature extraction network to obtain a second feature image output by the feature extraction network, wherein the reference image includes target marking information; An intermediate feature acquisition module, the intermediate feature acquisition module is used to input the first feature image into an encoder and acquire an intermediate feature image output by the encoder; A tracking result acquisition module, the tracking result acquisition module is used to input the intermediate feature image and the second feature image into a decoder, obtain output data of the decoder, and determine the target tracking result in the image to be tracked according to the output data of the decoder; The step of inputting the intermediate feature image and the second feature image into a decoder to obtain output data of the decoder includes: Decomposing the intermediate feature image to obtain a plurality of first intermediate features and a plurality of second intermediate features; Wherein, the size of each of the first intermediate features is C*1*1, C is the number of channels of the intermediate feature image, and the feature value in the jth first intermediate feature is the feature value of the jth pixel position in each channel of the intermediate feature image; the size of each of the second intermediate features is H0W0*1*1, H0 and W0 are respectively the length and width of the image in each channel of the intermediate feature image, and the feature value in the i-th second intermediate feature is the feature value in the i-th channel of the intermediate feature image; In the decoder, inputting the second feature image into a multi-head attention module in the decoder to obtain attention features; Performing a first cross-correlation operation on the attention feature and the first intermediate feature to obtain a first cross-correlation operation result; Perform a second cross-correlation operation on the first cross-correlation operation result and the second intermediate feature to obtain a second cross-correlation operation result; The output data of the decoder is obtained according to the second cross-correlation operation result.

7. A terminal, characterized in that: The terminal includes: a processor, a computer-readable storage medium communicatively connected to the processor, the computer-readable storage medium being suitable for storing a plurality of instructions, and the processor being suitable for calling the instructions in the computer-readable storage medium to execute the steps of implementing the single target tracking method described in any one of claims 1 to 5 above.

8. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores one or more programs, and the one or more programs can be executed by one or more processors to implement the steps of the single target tracking method according to any one of claims 1-5.

Images