Target tracking method, device and equipment based on adaptive time context aggregation

Abstract

The present invention relates to the field of computer vision technology, and in particular to a target tracking method, device, and equipment based on adaptive temporal context aggregation. The method comprises: S1: normalizing the images of the target template and the two branches of the search area to obtain the normalized images of the two branches; S2: inputting the normalized images of the two branches into a feature extraction network to obtain a target template feature map and a search area feature map correspondingly; S3: inputting the target template feature map and the search area feature map into a similarity matching network to perform a deep cross-correlation operation to obtain a response map; S4: inputting the response map into an adaptive temporal context aggregation network to perform an adaptive temporal context aggregation operation to obtain an enhanced response map; S5: inputting the enhanced response map into an anchor-free classification regression network to perform classification and regression of the target to obtain a target tracking result. The present invention can improve the tracking performance of the target while almost not increasing the amount of additional calculations.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and in particular to a target tracking method, device and equipment based on adaptive temporal context aggregation. Background Art

[0002] Aerial ground tracking is one of the important tasks of remote sensing detection. It is used in many fields such as aerial photography, intelligent monitoring and military detection. Compared with ordinary target tracking, aerial ground tracking has many problems to be solved due to its complex application scenarios and platform limitations. Compared with general target tracking, aerial ground tracking faces many challenges such as low image resolution, susceptibility to occlusion, small targets, and interference from similar targets. First of all, the occlusion, scale change and scale change of the image have high requirements on the feature extraction ability of the model. Most of the existing correlation filter tracking algorithms are implemented using artificially designed features or combined features, so they are suitable for tracking scenes with simple backgrounds and are difficult to cope with the above challenges.

[0003] Due to challenges such as low resolution, small objects, and interference from similar targets, accurate and effective feature extraction becomes very difficult. In order to overcome the above difficulties, the acquisition of contextual information of continuous frames is crucial. However, most existing aerial trackers only use template frames and current frame information, while ignoring the temporal contextual information of continuous frames. Secondly, since aerial ground tracking needs to be deployed on a platform with limited computing resources, how to balance computing and accuracy is also a major problem faced by aerial ground tracking.

[0004] Since the target pixels in the aerial ground tracking scene are small and the discrimination is low, it is difficult to extract rich features for tracking. Therefore, the context information and rich motion information of consecutive frames are of great significance for guiding the aerial ground tracking of low-resolution, small and similar targets that are difficult to extract discriminative features. However, most of the existing trackers ignore the temporal background information. Summary of the invention

[0005] In order to solve the problems of low resolution, small targets, interference from similar targets in images of aerial-to-ground scenes, and the difficulty in achieving accurate and robust tracking due to the fact that the existing twin neural networks ignore the temporal context information of consecutive frames, the present invention provides a target tracking method, device, and equipment based on adaptive temporal context aggregation. On the basis of the existing twin neural networks, an adaptive temporal context aggregation network is proposed, which cleverly utilizes the calculation results of the previous frames of images while taking into account speed, thereby improving the target tracking performance with almost no additional calculation amount.

[0006] The target tracking method based on adaptive time context aggregation proposed in the present invention specifically includes the following steps:

[0007] S1: Divide the image sequence to be recognized into the target template and the search area of ​​the twin neural network, and normalize the images of the two branches of the target template and the search area to obtain the normalized images of the two branches;

[0008] S2: Input the normalized images of the two branches into the feature extraction network to obtain the target template feature map and the search area feature map respectively;

[0009] S3: Input the target template feature map and the search area feature map into the similarity matching network for deep cross-correlation operation to obtain a response map;

[0010] S4: inputting the response map into the adaptive time context aggregation network to perform an adaptive time context aggregation operation to obtain a reinforced response map;

[0011] S5: Input the enhanced response map into the anchor-free classification and regression network to classify and regress the target to obtain the target tracking result.

[0012] Preferably, step S1 specifically includes the steps of:

[0013] S11: taking the first frame image of the image sequence to be identified as a target template, and taking the remaining frame images of the image sequence to be identified as a search area;

[0014] S12: The image frame of the target template is cropped and normalized to 127*127 pixels, and the image frames of the search area are cropped and normalized to 255*255 pixels.

[0015] Preferably, the feature extraction network adopts AlexNet convolutional neural network, and the AlexNet convolutional neural network is used to extract the last three layers of feature maps of the normalized images of the two branches, and correspondingly obtain the target template feature map and the search area feature map.

[0016] Preferably, the similarity matching network includes a cross-correlation module, a Concat module and a 1*1 ordinary convolution layer. The target template feature map and the search area feature map are input into the cross-correlation module. The cross-correlation module is used to perform a cross-correlation operation on the search area feature map using the target template feature map as a convolution kernel. The Concat module is used to splice the feature maps output by the cross-correlation module, and after a convolution operation is performed by a 1*1 ordinary convolution layer, a response map is output.

[0017] The response graph is calculated as:

[0018]

[0019] R = Conv(Concat(R i )) (2);

[0020] Among them, L i is the feature map of the i-th layer, Z is the target template feature map, X is the search area feature map, is the cross-correlation operation, R i It is the feature map output by the cross-correlation operation between the i-th layer feature map of the target template feature map and the i-th layer feature map of the search area feature map, R is the response map, Concat is the concatenation operation, and Conv is the 1×1 convolution operation.

[0021] Preferably, the adaptive time context aggregation network comprises a time context encoder and a time context decoder, wherein:

[0022] The temporal context encoder is used to obtain a feature map R with temporal prior information by combining the context information of the current frame and adjacent frames of the response map. e and adjustment parameters;

[0023] The temporal context decoder is used to decode the feature map R e The splicing and fusion are performed, and the time context information of the current frame of the response map is enhanced by combining the adjustment parameters to obtain the enhanced response map.

[0024] Preferably, step S4 specifically includes the following steps:

[0025] S41: Input the current frame and adjacent frames of the response graph into the temporal context encoder, and perform linear mapping, splicing and fusion operations in sequence to obtain a feature graph R with temporal prior information e :

[0026] R e =Conv(Concat(Linear(R t ,...,R t-n ))) (3);

[0027] Among them, Linear is a linear mapping; R t is the current frame of the response graph, R t-n is the tnth frame image which is n frames away from the current frame of the response image;

[0028] S42: Input the current frame of the response graph into the temporal context decoder for linear mapping to obtain a feature graph R, and combine the feature graph R and the feature graph R e Perform splicing and fusion to obtain a feature map R with aggregated information d :

[0029] R d =Conv(Concat(Linear(R e ,R t ))) (4);

[0030] S43: The feature map R e Perform a global average pooling operation in the temporal context encoder and use the ReLu activation function for activation to obtain the adjustment parameter w:

[0031] w = ReLu(GAP(Conv(R e ))) (5);

[0032] Among them, ReLu is the ReLu activation function, GAP is the global average pooling operation;

[0033] S44: Input the adjustment parameter w to the temporal context decoder and compare it with the feature map R d Multiply to obtain the feature map R`:

[0034] R`=w·R d (6);

[0035] S45: Perform residual connection and summation operations on the feature map R` to obtain the enhanced response map R t *:

[0036] R t *=R` + Linear(R t ) (7).

[0037] Preferably, the anchor-free classification regression network includes a classification branch and a regression branch. The classification branch and the regression branch are processed in parallel. The classification branch is used to perform foreground and background discrimination on the pixels of the enhanced response image. The regression branch is used to draw a tracking frame of the target by taking each pixel point of the enhanced response image as the center and performing regression calculation on the distance between each pixel point and the upper boundary, lower boundary, left boundary and right boundary.

[0038] The target tracking device based on adaptive time context aggregation provided by the present invention includes:

[0039] A normalization unit, used for dividing the image sequence to be recognized into a target template and a search area of ​​the twin neural network, and normalizing the images of the two branches of the target template and the search area to obtain normalized images of the two branches;

[0040] A feature extraction unit is used to input the normalized images of the two branches into the feature extraction network, and obtain a target template feature map and a search area feature map accordingly;

[0041] A similarity matching network unit is used to input the target template feature map and the search area feature map into the similarity matching network for deep cross-correlation operation to obtain a response map;

[0042] An adaptive time context aggregation unit, used for inputting the response graph into an adaptive time context aggregation network to perform an adaptive time context aggregation operation to obtain an enhanced response graph;

[0043] The anchor-free classification and regression unit is used to input the enhanced response map into the anchor-free classification and regression network to classify and regress the target and obtain the target tracking result.

[0044] The computer device provided by the present invention comprises:

[0045] at least one processor; and

[0046] a memory communicatively connected to at least one processor; wherein,

[0047] The memory stores instructions that can be executed by at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform a target tracking method based on adaptive time context aggregation.

[0048] The present invention provides a non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to enable a computer to execute a target tracking method based on adaptive time context aggregation.

[0049] Compared with the prior art, the present invention can achieve the following beneficial effects:

[0050] (1) Based on the existing twin neural network, the present invention proposes an adaptive temporal context aggregation network, including a temporal context encoder and a temporal context decoder. Through the structure of the temporal context encoder and the temporal context decoder, effective inter-frame context information is adaptively introduced into the tracking process of the current frame of the response graph, so that the tracking device proposed by the present invention can effectively cope with challenges such as low resolution, small targets, and interference from similar targets, and realize robust and accurate tracking of aerial ground targets.

[0051] (2) The adaptive temporal context aggregation network proposed in the present invention improves the tracking performance of the device proposed in the present invention by cleverly utilizing the results of calculations of the previous frames of images without increasing the amount of additional calculations. Therefore, the tracking performance of the target is improved while taking into account the tracking speed. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] Figure 1 It is a model framework diagram of a target tracking method based on adaptive time context aggregation provided by an embodiment of the present invention;

[0053] Figure 2 is a flow chart of a target tracking method based on adaptive time context aggregation provided by an embodiment of the present invention;

[0054] Figure 3 is a schematic diagram of a network structure of a similarity matching network provided according to an embodiment of the present invention;

[0055] Figure 4 is a schematic diagram of a network structure of an adaptive time context aggregation network provided according to an embodiment of the present invention;

[0056] Figure 5 is a structural schematic diagram of a target tracking device based on adaptive time context aggregation provided by an embodiment of the present invention;

[0057] Figure 6 It is a structural block diagram of a computer device provided according to an embodiment of the present invention.

[0058] Figure numerals: normalization unit 1, feature extraction unit 2, similarity matching network unit 3, adaptive time context aggregation unit 4, anchor-free classification regression unit 5, computer device 6, external device 7, processing unit 8, bus 9, network adapter 10, input / output (I / O) interface 11, display 12, system memory 13, random access memory (RAM) 14, cache memory 15, storage system 16, program / utility 17, program module 18. DETAILED DESCRIPTION

[0059] Hereinafter, embodiments of the present invention will be described with reference to the accompanying drawings. In the following description, the same modules are represented by the same reference numerals. In the case of the same reference numerals, their names and functions are also the same. Therefore, the detailed description thereof will not be repeated.

[0060] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and do not constitute a limitation of the present invention.

[0061] Figure 1 A model framework of a target tracking method based on adaptive time context aggregation provided according to an embodiment of the present invention.

[0062] like Figure 1As shown, the target tracking method based on adaptive time context aggregation proposed in the present invention mainly includes a feature extraction network, a similarity matching network, an adaptive time context aggregation network and an anchor-free classification regression network, wherein the feature extraction network adopts an AlexNet convolutional neural network, the target template and the search area of ​​the twin neural network are normalized, and the normalized images of the two branches obtained are input into the feature extraction network to extract the last three layers of feature maps, and obtain the target template feature map and the search area feature map. The target template feature map and the search area feature map are input into the similarity matching network for deep cross-correlation operation to obtain a response map. The response map is input into the adaptive time context aggregation network and enhanced by the adaptive time context aggregation operation. Finally, the classification head and regression head in the anchor-free classification regression network are used to classify and regress the input enhanced response map to obtain the tracking result of the target.

[0063] Figure 2 A process of a target tracking method based on adaptive time context aggregation is provided according to an embodiment of the present invention.

[0064] like Figure 2 As shown, the target tracking method based on adaptive time context aggregation provided by the embodiment of the present invention specifically includes the following steps:

[0065] S1: Divide the image sequence to be identified into a target template and a search area of ​​the twin neural network, and normalize the images of the two branches of the target template and the search area to obtain normalized images of the two branches.

[0066] Step S1 specifically includes the following steps:

[0067] S11: taking the first frame image of the image sequence to be recognized as a target template, and taking the remaining frame images of the image sequence to be recognized as a search area.

[0068] S12: The image frame of the target template is cropped and normalized to 127*127 pixels, and the image frames of the search area are cropped and normalized to 255*255 pixels.

[0069] S2: Input the normalized images of the two branches into the feature extraction network to obtain the target template feature map and the search area feature map respectively.

[0070] The feature extraction network adopts AlexNet convolutional neural network, and AlexNet convolutional neural network is used to extract the last three layers of feature maps of the normalized images of the two branches, and correspondingly obtain the target template feature map and the search area feature map.

[0071] S3: Input the target template feature map and the search area feature map into the similarity matching network for deep cross-correlation operation to obtain a response map.

[0072] S4: Input the response map into the adaptive time context aggregation network to perform an adaptive time context aggregation operation to obtain an enhanced response map.

[0073] Step S4 specifically includes the following steps:

[0074] S41: Input the current frame and adjacent frames of the response graph into the temporal context encoder, and perform linear mapping, splicing and fusion operations in sequence to obtain a feature graph R with temporal prior information e :

[0075] R e =Conv(Concat(Linear(R t ,...,R t-n ))) (3);

[0076] Among them, Linear is a linear mapping; R t is the current frame of the response graph, R t-n is the tnth frame image which is n frames away from the current frame of the response image;

[0077] S42: Input the current frame of the response graph into the temporal context decoder for linear mapping to obtain a feature graph R, and combine the feature graph R and the feature graph R e Perform splicing and fusion to obtain a feature map R with aggregated information d :

[0078] R d =Conv(Concat(Linear(R e ,R t ))) (4);

[0079] S43: The feature map R e Perform a global average pooling operation in the temporal context encoder and use the ReLu activation function for activation to obtain the adjustment parameter w:

[0080] w = ReLu(GAP(Conv(R e ))) (5);

[0081] Among them, ReLu is the ReLu activation function, GAP is the global average pooling operation;

[0082] S44: Input the adjustment parameter w to the temporal context decoder and compare it with the feature map R d Multiply to obtain the feature map R`:

[0083] R`=w·Rd (6);

[0084] S45: Perform residual connection and summation operations on the feature map R` to obtain the enhanced response map R t *:

[0085] R t *=R` + Linear(R t ) (7).

[0086] S5: Input the enhanced response map into the anchor-free classification and regression network to classify and regress the target to obtain the target tracking result.

[0087] The anchor-free classification regression network includes a classification branch and a regression branch. The classification branch and the regression branch are processed in parallel. The classification branch is used to distinguish the foreground and background of the pixels of the enhanced response map. The regression branch is used to draw the tracking frame of the target by regressing the distance between each pixel point and the upper boundary, lower boundary, left boundary and right boundary with each pixel point in the enhanced response map as the center.

[0088] Since convolutional neural networks calculate feature maps layer by layer in series, shallow feature layers contain high-resolution low-semantic features, while deep feature layers focus more on low-resolution high-semantic information features. If only the last layer of feature maps of the convolutional neural network is used for fusion, high-resolution low-semantic features will be ignored. Therefore, in order to take into account both deep and shallow features and achieve more comprehensive information fusion, we use the method of cross-correlating and fusing multiple feature layers to generate response maps.

[0089] Figure 3 The network structure of the similarity matching network provided according to the embodiment of the present invention is shown.

[0090] like Figure 3 As shown in the figure, the similarity matching network includes a cross-correlation module, a Concat module and a 1*1 ordinary convolution layer. The target template feature map and the search area feature map are input into the cross-correlation module. The cross-correlation module is used to use the target template feature map as a convolution kernel to perform a cross-correlation operation on the search area feature map. The Concat module is used to splice the feature maps output by the cross-correlation module, and after the convolution operation is performed by the 1*1 ordinary convolution layer, the response map is output.

[0091] The response graph is calculated as:

[0092]

[0093] R = Conv(Concat(R i )) (2);

[0094] Among them, L iis the feature map of the i-th layer, Z is the target template feature map, X is the search area feature map, is the cross-correlation operation, R i The feature map output by cross-correlating the i-th layer feature map of the target template feature map and the i-th layer feature map of the search area feature map contains the similarity information between the search area feature map and the target template feature. R is the response map, Concat is the concatenation operation, and Conv is the 1×1 convolution operation.

[0095] Figure 4 The network structure of the adaptive time context aggregation network provided according to an embodiment of the present invention is shown.

[0096] like Figure 4 As shown, the adaptive time context aggregation network includes a time context encoder and a time context decoder, wherein,

[0097] The temporal context encoder is used to obtain a feature map R with temporal prior information by combining the context information of the current frame and adjacent frames of the response map. e and adjustment parameters;

[0098] The temporal context decoder is used to decode the feature map R e The splicing and fusion are performed, and the time context information of the current frame of the response graph is enhanced by combining the adjustment parameters to obtain the enhanced response graph.

[0099] Figure 5 The structure of a target tracking device based on adaptive time context aggregation provided according to an embodiment of the present invention is shown.

[0100] like Figure 5 As shown, the target tracking device based on adaptive time context aggregation provided by the embodiment of the present invention includes:

[0101] A normalization unit 1 is used to divide the image sequence to be identified into a target template and a search area of ​​the twin neural network, and normalize the images of the two branches of the target template and the search area to obtain normalized images of the two branches;

[0102] Feature extraction unit 2, used for inputting the normalized images of the two branches into the feature extraction network, and correspondingly obtaining the target template feature map and the search area feature map;

[0103] A similarity matching network unit 3 is used to input the target template feature map and the search area feature map into the similarity matching network for deep cross-correlation operation to obtain a response map;

[0104] An adaptive time context aggregation unit 4 is used to input the response graph into an adaptive time context aggregation network to perform an adaptive time context aggregation operation to obtain an enhanced response graph;

[0105] The anchor-free classification and regression unit 5 is used to input the enhanced response map into the anchor-free classification and regression network to classify and regress the target, and obtain the target tracking result.

[0106] Figure 6 It is a structural schematic diagram of a computer device 6 provided in an embodiment of the present invention. Figure 6 A block diagram of an exemplary computer device 6 suitable for use in implementing embodiments of the present invention is shown. Figure 6 The computer device 6 shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present invention.

[0107] like Figure 6 As shown, computer device 6 is in the form of a general-purpose computing device. Computer device 6 is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. Electronic devices can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present invention described and / or required herein.

[0108] Components of the computer device 6 may include, but are not limited to, one or more processors or processing units 8, a system memory 13, and a bus 9 that connects various system components (including the system memory 13 and the processing unit 8).

[0109] Bus 9 represents one or more of several types of bus structures, including a memory bus or memory controller, a peripheral bus, an accelerated graphics port, a processor or a local bus using any of a variety of bus structures. For example, these architectures include but are not limited to Industry Standard Architecture (ISA) bus, Micro Channel Architecture (MAC) bus, Enhanced ISA bus, Video Electronics Standards Association (VESA) local bus and Peripheral Component Interconnect (PCI) bus.

[0110] The computer device 6 typically includes a variety of computer system readable media. These media can be any available media that can be accessed by the computer device 6, including volatile and non-volatile media, removable and non-removable media.

[0111] System memory 13 may include computer system readable media in the form of volatile memory, such as random access memory (RAM) 14 and / or cache memory 15. Computer device 6 may further include other removable / non-removable, volatile / non-volatile computer system storage media. By way of example only, storage system 16 may be used to read and write non-removable, non-volatile magnetic media ( Figure 6 not shown, usually called a "hard drive"). Although Figure 6 Not shown in the figure, a disk drive for reading and writing to a removable non-volatile disk (e.g., a "floppy disk"), and an optical disk drive for reading and writing to a removable non-volatile optical disk (e.g., a CD-ROM, a DVD-ROM, or other optical media) may be provided. In these cases, each drive may be connected to the bus 9 via one or more data medium interfaces. The memory 28 may include at least one program product having a set (e.g., at least one) of program modules that are configured to perform the functions of the various embodiments of the present invention.

[0112] A program / utility 17 having a set (at least one) of program modules 18 may be stored, for example, in a memory 28, such program modules 18 including but not limited to an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment. The program modules 18 generally perform the functions and / or methods of the embodiments described herein.

[0113] The computer device 6 may also communicate with one or more external devices 7 (e.g., keyboard, pointing device, display 12, etc.), may also communicate with one or more devices that enable a user to interact with the computer device 6, and / or communicate with any device that enables the computer device 6 to communicate with one or more other computing devices (e.g., network card, modem, etc.). Such communication may be performed via an input / output (I / O) interface 11. Furthermore, the computer device 6 may also communicate with one or more networks (e.g., a local area network (LA6), a wide area network (WA6), and / or a public network, such as the Internet) via a network adapter 10. As shown, the network adapter 10 communicates with other modules of the computer device 6 via a bus 9. It should be understood that, although not shown in the figure, other hardware and / or software modules may be used in conjunction with the computer device 6, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0114] The processing unit 8 executes various functional applications and data processing by running the programs stored in the system memory 13, such as implementing the target tracking method based on the wide spectrum correlation fusion network provided in the embodiment of the present invention.

[0115] An embodiment of the present invention also provides a non-transitory computer-readable storage medium storing computer instructions, on which a computer program is stored, wherein when the program is executed by a processor, the target tracking method based on a broad spectrum correlation fusion network provided in all the inventive embodiments of the present application is implemented.

[0116] The computer storage medium of the embodiment of the present invention may adopt any combination of one or more computer-readable media. The computer-readable medium may be a computer-readable signal medium or a computer-readable storage medium. More specific examples (non-exhaustive list) of computer-readable storage media include: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this document, a computer-readable storage medium may be any tangible medium containing or storing a program that may be used by or in combination with an instruction execution system, an apparatus, or a device.

[0117] Computer-readable signal media may include data signals propagated in baseband or as part of a carrier wave, which carry computer-readable program code. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Computer-readable signal media may also be any computer-readable medium other than a computer-readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0118] The program code included in the computer readable medium can be transmitted with any appropriate medium, including but not limited to wireless, electric wire, optical cable, RF, etc., or any suitable combination of the above. The computer program code for performing the operation of the present invention can be written in one or more programming languages ​​or their combinations, and the programming language includes object-oriented programming languages ​​such as Java, Smalltalk, C++, and also includes conventional procedural programming languages-such as "C" language or similar programming languages. The program code can be executed completely on the user's computer, partially on the user's computer, as an independent software package, partially on the user's computer and partially on the remote computer, or completely on the remote computer or server. In the case of a remote computer, the remote computer can be connected to the user's computer by any type of network including a local area network (LA6) or a wide area network (WA6), or it can be connected to an external computer (for example, using an Internet service provider to connect to the Internet).

[0119] An embodiment of the present invention further provides a computer program product, including a computer program, which implements the above-mentioned target tracking method based on the broad spectrum correlation fusion network when executed by a processor.

[0120] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps described in the disclosure of the present invention can be performed in parallel, sequentially or in different orders, as long as the desired results of the technical solution disclosed in the present invention can be achieved, and this document does not limit this.

[0121] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A target tracking method based on adaptive temporal context aggregation, characterized in that: The specific steps include: S1: Divide the image sequence to be identified into a target template and a search area of ​​the twin neural network, and normalize the images of the two branches of the target template and the search area to obtain normalized images of the two branches; S2: Input the normalized images of the two branches into the feature extraction network to obtain the target template feature map and the search area feature map respectively; S3: Inputting the target template feature map and the search area feature map into a similarity matching network to perform a deep cross-correlation operation to obtain a response map; S4: inputting the response graph into an adaptive time context aggregation network to perform an adaptive time context aggregation operation to obtain an enhanced response graph; The adaptive time context aggregation network includes a time context encoder and a time context decoder, wherein: The temporal context encoder is used to obtain a feature map with temporal prior information by combining context information of the current frame and adjacent frames of the response map and adjustment parameters; The temporal context decoder is used to decode the feature map Performing splicing and fusion, and combining the adjustment parameters to enhance the temporal context information of the current frame of the response graph, to obtain an enhanced response graph; The step S4 specifically includes the following steps: S41: Input the current frame and adjacent frames of the response graph into the temporal context encoder, and perform linear mapping, splicing and fusion operations in sequence to obtain a feature graph with temporal prior information. : （1）； in, is a linear mapping; is the current frame of the response graph, is the tnth frame image n frames away from the current frame of the response image, Concat is a concatenation operation, Conv is Convolution operation; S42: Input the current frame of the response graph into the temporal context decoder for linear mapping to obtain a feature graph , the feature map and the characteristic graph Perform splicing and fusion to obtain a feature map with aggregated information : （2）； S43: The feature map A global average pooling operation is performed in the temporal context encoder, and a ReLu activation function is used for activation to obtain the adjustment parameter w: （3）； Among them, ReLu is the ReLu activation function, GAP is the global average pooling operation; S44: Input the adjustment parameter w into the temporal context decoder and compare it with the feature map Multiply to get the feature map : （4）； S45: The feature map Perform residual connection and summation operations to obtain the enhanced response graph : （5）； S5: Inputting the enhanced response map into the anchor-free classification and regression network to classify and regress the target, and obtaining the tracking result of the target.

2. The target tracking method based on adaptive temporal context aggregation according to claim 1, characterized in that: The step S1 specifically includes the following steps: S11: taking the first frame image of the image sequence to be identified as a target template, and taking the remaining frame images of the image sequence to be identified as a search area; S12: Crop and normalize the image frame of the target template to 127 127 pixels, and the image frames in the search area are cropped and normalized to 255 255 pixels.

3. The target tracking method based on adaptive temporal context aggregation according to claim 1, characterized in that: The feature extraction network adopts AlexNet convolutional neural network, and the AlexNet convolutional neural network is used to extract the last three layers of feature maps of the normalized images of the two branches, and correspondingly obtain the target template feature map and the search area feature map.

4. The target tracking method based on adaptive temporal context aggregation according to claim 3 is characterized in that: The similarity matching network includes a cross-correlation module, a Concat module and a 1 ordinary convolution layer, the target template feature map and the search area feature map are input into the cross-correlation module, the cross-correlation module is used to use the target template feature map as a convolution kernel to perform a cross-correlation operation on the search area feature map, and the Concat module is used to splice the feature maps output by the cross-correlation module and pass through the 1 1 After the ordinary convolution layer performs the convolution operation, the response map is output; The calculation formula of the response graph is: （6）； （7）； in, is the i-th layer feature map, Z is the target template feature map, X is the search area feature map, For the cross-correlation operation, is the feature map output by performing a cross-correlation operation on the i-th layer feature map of the target template feature map and the i-th layer feature map of the search area feature map, R is the response map, Concat is a concatenation operation, and Conv is Convolution operation.

5. The target tracking method based on adaptive temporal context aggregation according to claim 1, characterized in that: The anchor-free frame classification regression network includes a classification branch and a regression branch. The classification branch is processed in parallel with the regression branch. The classification branch is used to perform foreground and background discrimination on the pixels of the enhanced response image. The regression branch is used to draw a tracking frame of the target by taking each pixel point of the enhanced response image as the center and performing regression calculation on the distance between each pixel point and the upper boundary, lower boundary, left boundary and right boundary.

6. A target tracking device based on adaptive temporal context aggregation, characterized in that: include: A normalization unit, used for dividing the image sequence to be identified into a target template and a search area of ​​the twin neural network, and normalizing the images of the two branches of the target template and the search area to obtain normalized images of the two branches; A feature extraction unit is used to input the normalized images of the two branches into the feature extraction network, and obtain a target template feature map and a search area feature map accordingly; A similarity matching network unit, used for inputting the target template feature map and the search area feature map into a similarity matching network for deep cross-correlation operation to obtain a response map; An adaptive time context aggregation unit, used for inputting the response graph into an adaptive time context aggregation network to perform an adaptive time context aggregation operation to obtain an enhanced response graph; The adaptive time context aggregation network includes a time context encoder and a time context decoder, wherein: The temporal context encoder is used to obtain a feature map with temporal prior information by combining context information of the current frame and adjacent frames of the response map and adjustment parameters; The temporal context decoder is used to decode the feature map Performing splicing and fusion, and combining the adjustment parameters to enhance the temporal context information of the current frame of the response graph, to obtain an enhanced response graph; The specific steps include: S41: Input the current frame and adjacent frames of the response graph into the temporal context encoder, and perform linear mapping, splicing and fusion operations in sequence to obtain a feature graph with temporal prior information. : （8）； in, is a linear mapping; is the current frame of the response graph, is the tnth frame image n frames away from the current frame of the response image, Concat is a concatenation operation, Conv is Convolution operation; S42: Input the current frame of the response graph into the temporal context decoder for linear mapping to obtain a feature graph , the feature map and the characteristic graph Perform splicing and fusion to obtain a feature map with aggregated information : （9）； S43: The feature map A global average pooling operation is performed in the temporal context encoder, and a ReLu activation function is used for activation to obtain the adjustment parameter w: （10）； Among them, ReLu is the ReLu activation function, GAP is the global average pooling operation; S44: Input the adjustment parameter w into the temporal context decoder and compare it with the feature map Multiply to get the feature map : （11）； S45: The feature map Perform residual connection and summation operations to obtain the enhanced response graph : （12）； The anchor-free classification and regression unit is used to input the enhanced response map into the anchor-free classification and regression network to classify and regress the target, so as to obtain the tracking result of the target.

7. A computer device, characterized in that: include: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute the target tracking method based on adaptive time context aggregation as described in any one of claims 1 to 5.

8. A non-transitory computer-readable storage medium storing computer instructions, characterized in that: The computer instructions are used to enable the computer to execute the target tracking method based on adaptive time context aggregation according to any one of claims 1 to 5.

Images