Target tracking method, device and equipment based on broad spectrum correlation fusion network

Abstract

The present invention relates to the field of computer vision technology, and in particular to a target tracking method, device, and equipment based on a broad spectrum correlation fusion network, the method comprising: S1: normalizing the target template of the twin neural network and the input images of the two branches of the search area to obtain a normalized image; S2: inputting the normalized image into the residual network for feature extraction, and correspondingly obtaining the last three layers of feature blocks; S3: inputting the last three layers of feature blocks of the two branches into the Transformer fusion neck network for feature fusion and enhancement, and correspondingly obtaining the first and second feature blocks; S4: performing a deep cross-correlation operation on the first and second feature blocks, and obtaining a response map by merging and reducing the scale operation; S5: inputting the response map into the spatiotemporal dependency fusion module and the prediction head in sequence to obtain the target tracking result. The present invention can adaptively complete the efficient fusion and optimization update of feature maps at different levels.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and in particular to a target tracking method, device and equipment based on a broad spectrum correlation fusion network. Background Art

[0002] The target tracking task is a basic and challenging problem in the field of computer vision. It aims to determine the subsequent position of the target and generate a predicted aiming frame given only the information of the initial position of the target in the first frame. Initially, it was a task that simulated human visual attention and cognitive mechanisms and gradually became an important task in the field of computer vision. Recently, benefiting from the rapid development of interdisciplinary subjects, target tracking has been widely used in intelligent video surveillance, drone applications, assisted autonomous driving and other fields. In order to achieve efficient and accurate tracking, we need to distinguish the two attributes of the target foreground and background and overcome various ambiguous interfering feature information. For example, in the scene where the target is partially occluded, the features of the car become insignificant and may only be detected as local features and become difficult to distinguish. In the scene of scale change, the multi-scale features of the car make successful tracking have high requirements for features at different levels.

[0003] Most of the existing traditional algorithms are based on generative models to achieve target tracking, which requires complex detection feature operators and uses manually designed features to perform template matching to complete the target tracking task. For example, the optical flow method for moving target tracking still divides the moving target tracking task into two steps. The first step is to calculate the optical flow information of two frames of images, and the second step is to select a suitable threshold to segment the optical flow map. It fails to integrate a suitable network framework, making it difficult to select the threshold. It is not applicable to aerial images with high platform freedom of movement and more complex movements, and there are problems such as slow speed, more missed detections, and inaccurate and incomplete moving target tracking.

[0004] In recent years, the optimal target tracking method based on twin neural networks has become a very efficient method to solve the tracking problem. It has achieved a balance between accuracy and real-time performance while greatly improving performance, and has become a hot research field based on deep learning methods. The core idea of ​​the twin neural network tracking method is to use two identical feature extraction network branches, which are used for feature extraction of target templates and search areas respectively, and convert the tracking problem into a similarity matching problem between the features of the two branches through the operation process of the cross-correlation layer. Finally, the best matching search area is obtained through the subsequent classification and regression network to complete the tracking. In recent years, the development trend of twin neural networks has shown that how to effectively utilize features at different levels is the key to improving algorithm performance. One way is through linear multi-scale context fusion. For example, SiamCAR and SiamBAN achieve feature fusion by directly summing or channel cascading the feature blocks extracted by the backbone extraction network. Other works enable the network to obtain richer global context information by designing efficient local encoding modeling layers or expanding the receptive field by decomposing the feature information structure of the operation. Although existing methods of utilizing features have achieved the use of contextual information to track targets of different scales, linear fusion or local modeling cannot fully utilize feature information from a global perspective, and the pixel-level cross-correlation between features at different levels is often ignored, which is also necessary for accurate tracking. Summary of the invention

[0005] In order to solve the problems of existing target tracking methods based on twin neural networks, linear fusion or local modeling cannot fully utilize feature information from a global perspective, and the pixel-level cross-correlation between features at different levels is often ignored, as well as tracking drift and low accuracy caused by target occlusion, scale and perspective changes, complex backgrounds, etc., the present invention provides a target tracking method, device, and equipment based on a broad-spectrum correlation fusion network, which can adaptively complete the efficient fusion and optimization update of feature maps at different levels, so as to enhance the network's ability to distinguish the feature expression of target tracking, thereby improving the performance of the device in remote sensing tracking scenarios.

[0006] The target tracking method based on the broad spectrum correlation fusion network provided by the present invention specifically comprises the following steps:

[0007] S1: Normalize the target template of the twin neural network and the input images of the two branches of the search area to obtain normalized images of the two branches.

[0008] S2: Input the normalized images of the two branches into the residual network for feature extraction, and obtain the last three layers of feature blocks of the two branches.

[0009] S3: The last three layers of feature blocks of the two branches are input into the Transformer fusion neck network for feature fusion and feature enhancement, and the first feature block and the second feature block are obtained accordingly.

[0010] Step S3 specifically includes the following steps:

[0011] S31: The last three layers of feature blocks of the two branches are input into three layers of independent coordinate context encoders to model the long-distance correlation of two spatial dimensions, and correspondingly obtain three feature blocks with long-distance correlation of two spatial dimensions.

[0012] S32: The three feature blocks of the two branches are respectively input into the adjustment layer for channel adjustment, and three target feature blocks with a target number of channels are correspondingly obtained.

[0013] S33: The three target feature blocks of the two branches are hierarchically input into the Transformer cascade fusion module for global correlation modeling, and the first feature block and the second feature block with global correlation are correspondingly obtained.

[0014] S4: Perform a deep cross-correlation operation on the first feature block and the second feature block, and obtain a response map by merging and reducing the scale.

[0015] S5: The response map is sequentially input into the spatiotemporal dependency fusion module and the prediction head for target classification and regression to obtain the target tracking result.

[0016] Preferably, step S31 includes:

[0017] S311: The coordinate context encoder inputs any one of the last three layers of feature blocks into the residual structure for feature decomposition to obtain feature maps X1 and X2.

[0018] S312: After the feature map X1 and the feature map X2 are globally pooled, channel cascade, 2D convolution, normalization layer and nonlinear layer operations are performed in sequence to obtain an aggregate feature map f:

[0019] f=δ(F1([z h ,z w ])) (1);

[0020] Among them, δ is the normalization layer, F1 is the nonlinear layer transformation, z h is the output feature of the pooling kernel with height h, z w is the output feature of the pooling kernel with width w;

[0021] S313: Split the aggregated feature map into feature map y1 and feature map y2 along two directions of the spatial dimension, perform 2D convolution operation and Sigmoid normalization processing on feature map y1 and feature map y2 in sequence, and obtain optimized weights in two directions.

[0022] S314: Fuse the optimization weights in the two directions, and apply the feature map X1 and the feature map X2 to the fused optimization weights to obtain an updated feature map Z.

[0023] S315: Calculate the feature block input in step S31 by using the Gamma function, and add the feature map Z to the calculation result of the Gamma function, and output the feature map Y by the following formula:

[0024] Y = y + Gamma · X (2);

[0025] Wherein, y is the feature map Z, and X is the feature block input in step S311.

[0026] Preferably, step S33 specifically includes:

[0027] After dimensionality adjustment, the feature blocks of each layer are flattened into sequence information using convolution operations and used as the input of the subsequent Transformer cascade fusion module, where the Transformer cascade fusion module contains five pooling strategy encoder ETF modules. The fourth-layer features output by the adjustment layer are used as the input of the query variable Q of the three-way ETF module in parallel, and the third-layer features, fourth-layer features and fifth-layer features output by the adjustment layer are used as the key-value pair inputs of the three-way ETF module in parallel respectively. The calculation result of the ETF module whose key-value pair input is the fourth-layer feature is used as the input of the QKV triplet of the next ETF module, and the output result is input into another ETF module. Multiple self-attention enhancements of the fourth-layer features are completed through the cascaded ETF modules.

[0028] The calculation formula of the ETF module is:

[0029] ETF(Q,K,V)=Norm(I+MLP(I))

[0030] I=Norm(Q+MutiHead(Q,Norm&AvgPool(K,V))) (3);

[0031] Among them, Norm is the normalization layer of the ETF module, AvgPool is the average pooling of the ETF module, MLP is the fully connected layer of the ETF module, I is the intermediate variable of the residual structure of the ETF module, MutiHead is the multi-head attention mechanism, Q is the query variable Q, K is the key index value in the key-value pair, and V is the corresponding value of the key index value.

[0032] Preferably, step S4 comprises:

[0033] By performing layered convolution on the corresponding layers of the target template and the search area, the response map of each level of cross-correlation operation is calculated using a deep cross-correlation operation, and then the response map R that combines the deep features and the shallow features is output by merging and reducing the scale:

[0034]

[0035] in, represents the deep cross-correlation operation, B is the corresponding layer of the target template, and C is the corresponding layer of the search area.

[0036] Preferably, the spatiotemporal dependency fusion module includes a channel dimension fusion encoder and a mixed space dimension fusion encoder, wherein:

[0037] The response map is input into the average pooling layer and the maximum pooling layer of the channel dimension fusion encoder for pooling processing, and the feature map after pooling is activated by the Mish activation function in the shared MLP submodule to obtain two weights with the same number of channels as the response map. The two weights are added and normalized by the Sigmoid function to obtain the channel dimension fusion weight.

[0038] The response map is processed by the Conv1 layer to obtain a feature map N. The feature map N is sequentially subjected to maximum pooling and average pooling to obtain a feature map N1. The vertical tensor group obtained by changing the dimension of the feature map N is multiplied by the horizontal tensor group obtained by changing the dimension and transposing the feature map N2. The feature map N2 is subjected to average pooling to obtain a feature map N3. The feature maps N1 and N3 are added and normalized by the Sigmoid function to obtain the spatial dimension fusion weight.

[0039] The channel dimension fusion weight and the spatial dimension fusion weight are multi-dimensionally multiplied and then added to obtain the feature map M. The feature map M and the response map are channel-concatenated to obtain the feature map M1. The feature map M1 is processed through the Conv3 layer to obtain the optimized feature map M2.

[0040] The calculation formula of feature map M2 is:

[0041] SDF(F)=conv3[F,F×M c +F×M s ](5);

[0042] Among them, F is the response graph, conv3 is the operation of Conv3 layer, M c is the channel dimension fusion weight, M s is the fusion weight of the spatial dimension.

[0043] Preferably, step S5 specifically includes the following steps:

[0044] S51: Input the response map into the spatiotemporal dependency fusion module to obtain the feature map M2.

[0045] S52: Input the feature map M2 into the prediction head for target classification and regression to obtain the target tracking result.

[0046] Preferably, step S52 includes:

[0047] The prediction head adopts a classification and regression network based on the anchor-free mechanism to directly classify and regress the candidate boxes at each position in the response map to obtain the target tracking result. The response map represents the linear mapping of the input image of the search area in space with the target template as the weight, and each position in the response map is mapped to the corresponding position in the original image space of the search area.

[0048] Preferably, the resolutions of the normalized images of the two branches are 127*127 and 255*255 respectively, the target number of channels is 256 channels, the scale of the first feature block is 7*7*256, and the scale of the second feature block is 31*31*256.

[0049] Preferably, the adjustment layer adopts a dimension adjustment strategy of a multi-layer convolutional layer to uniformly adjust the number of channels of the three target feature blocks of the two branches after channel cascading to the target channel, including:

[0050] The number of channels of the three target feature blocks of the two branches after channel cascading is uniformly adjusted from the original [512, 1024, 2048] channels to [256, 256, 256] channels, and the dimensions of the number of channels of the three target feature blocks of the two branches are adjusted to 3*256.

[0051] The target tracking device based on the broad spectrum correlation fusion network provided by the present invention comprises:

[0052] The normalization unit is used to normalize the input images of the two branches of the target template and the search area of ​​the twin neural network to obtain normalized images of the two branches.

[0053] The feature extraction unit is used to input the normalized images of the two branches into the residual network for feature extraction to obtain the last three layers of feature blocks of the two branches.

[0054] The Transformer fusion neck unit is used to perform feature fusion and feature enhancement on the last three layers of feature blocks of the two branches, and obtain the first feature block and the second feature block accordingly.

[0055] The similarity matching network unit is used for performing a deep cross-correlation operation on the first feature block and the second feature block, and obtaining a response map by merging and reducing the scale operation.

[0056] The prediction unit is used to input the response map into the spatiotemporal dependency fusion module and the prediction head in sequence to perform target classification and regression to obtain the target tracking result.

[0057] The computer device provided by the present invention comprises:

[0058] at least one processor; and

[0059] a memory communicatively connected to at least one processor; wherein,

[0060] The memory stores instructions that can be executed by at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform a target tracking method based on a wide-spectrum correlation fusion network.

[0061] The present invention provides a non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to enable a computer to execute a target tracking method based on a broad spectrum correlation fusion network.

[0062] Compared with the prior art, the present invention can achieve the following beneficial effects:

[0063] (1) The target tracking method based on the broad-spectrum correlation fusion network proposed in the present invention can adaptively complete the efficient fusion and optimization update of feature maps at different levels to enhance the network's ability to distinguish the feature expressions of target tracking, thereby improving the performance of the target tracking device based on the broad-spectrum correlation fusion network in remote sensing tracking scenarios.

[0064] (2) The present invention proposes a better correlation fusion scheme for features at different depth levels, which relies on the rich context dependency information obtained in adaptive global modeling to improve the feature expression capabilities of feature maps at different levels. BRIEF DESCRIPTION OF THE DRAWINGS

[0065] Figure 1 is a flow chart of a target tracking method based on a broad spectrum correlation fusion network provided according to an embodiment of the present invention;

[0066] Figure 2 It is a model framework diagram of a target tracking method based on a broad spectrum correlation fusion network provided in accordance with an embodiment of the present invention;

[0067] Figure 3 is a structural block diagram of a coordinate context encoder provided according to an embodiment of the present invention;

[0068] FIG4( a) is a structural block diagram of a channel dimension fusion encoder provided according to an embodiment of the present invention;

[0069] FIG4( b ) is a block diagram of a hybrid spatial dimension fusion encoder according to an embodiment of the present invention;

[0070] FIG4( c ) is a structural block diagram of a spatiotemporal dependency fusion network provided according to an embodiment of the present invention;

[0071] Figure 5 is a structural block diagram of a target tracking device based on a broad spectrum correlation fusion network provided according to an embodiment of the present invention;

[0072] Figure 6 It is a structural block diagram of a computer device according to an embodiment of the present invention.

[0073] Figure numerals: normalization unit 1, feature extraction unit 2, Transformer fusion neck unit 3, similarity matching network unit 4, prediction unit 5, computer device 6, external device 7, processing unit 8, bus 9, network adapter 10, input / output (I / O) interface 11, display 12, system memory 13, random access memory (RAM) 14, cache memory 15, storage system 16, program / utility 17, program module 18. DETAILED DESCRIPTION

[0074] Hereinafter, embodiments of the present invention will be described with reference to the accompanying drawings. In the following description, the same modules are represented by the same reference numerals. In the case of the same reference numerals, their names and functions are also the same. Therefore, the detailed description thereof will not be repeated.

[0075] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and do not constitute a limitation of the present invention.

[0076] Figure 1 The flowchart of the target tracking method based on the broad spectrum correlation fusion network provided according to an embodiment of the present invention is shown.

[0077] like Figure 1 As shown, the target tracking method based on the broad spectrum correlation fusion network provided by the embodiment of the present invention specifically includes the following steps:

[0078] S1: Normalize the target template of the twin neural network and the input images of the two branches of the search area to obtain normalized images of the two branches.

[0079] The resolutions of the normalized images of the two branches are 127*127 and 255*255 respectively.

[0080] S2: Input the normalized images of the two branches into the residual network for feature extraction, and obtain the last three layers of feature blocks of the two branches.

[0081] The residual network can use the ResNet-50 feature extraction network.

[0082] S3: The last three layers of feature blocks of the two branches are input into the Transformer fusion neck network for feature fusion and feature enhancement, and the first feature block and the second feature block are obtained accordingly.

[0083] Step S3 specifically includes the following steps:

[0084] S31: The last three layers of feature blocks of the two branches are input into three layers of independent coordinate context encoders to model the long-distance correlation of two spatial dimensions, and correspondingly obtain three feature blocks with long-distance correlation of two spatial dimensions.

[0085] Step S31 specifically includes the following steps:

[0086] S311: The coordinate context encoder inputs any one of the last three layers of feature blocks into the residual structure for feature decomposition to obtain feature maps X1 and X2.

[0087] S312: After the feature map X1 and the feature map X2 are globally pooled, channel cascade, 2D convolution, normalization layer and nonlinear layer operations are performed in sequence to obtain an aggregate feature map f:

[0088] f=δ(F1([z h ,z w ])) (1);

[0089] Among them, δ is the normalization layer, F1 is the nonlinear layer transformation, z h is the output feature of the pooling kernel with height h, z w is the output feature of the pooling kernel with width w;

[0090] S313: Split the aggregated feature map into feature map y1 and feature map y2 along two directions of the spatial dimension, perform 2D convolution operation and Sigmoid normalization processing on feature map y1 and feature map y2 in sequence, and obtain optimized weights in two directions.

[0091] S314: Fuse the optimization weights in the two directions, and apply the feature map X1 and the feature map X2 to the fused optimization weights to obtain an updated feature map Z.

[0092] S315: Calculate the feature block input in step S31 by using the Gamma function, and add the feature map Z to the calculation result of the Gamma function, and output the feature map Y by the following formula:

[0093] Y = y + Gamma · X (2);

[0094] Wherein, y is the feature map Z, and X is the feature block input in step S311.

[0095] S32: The three feature blocks of the two branches are respectively input into the adjustment layer for channel adjustment, and three target feature blocks with a target number of channels are correspondingly obtained.

[0096] S33: The three target feature blocks of the two branches are hierarchically input into the Transformer cascade fusion module for global correlation modeling, and the first feature block and the second feature block with global correlation are correspondingly obtained.

[0097] Step S33 specifically includes:

[0098] After dimensionality adjustment, the feature blocks of each layer are flattened into sequence information using convolution operations and used as the input of the subsequent Transformer cascade fusion module, where the Transformer cascade fusion module contains five pooling strategy encoder ETF modules. The fourth-layer features output by the adjustment layer are used as the input of the query variable Q of the three-way ETF module in parallel, and the third-layer features, fourth-layer features and fifth-layer features output by the adjustment layer are used as the key-value pair inputs of the three-way ETF module in parallel respectively. The calculation result of the ETF module whose key-value pair input is the fourth-layer feature is used as the input of the QKV triplet of the next ETF module, and the output result is input into another ETF module. Multiple self-attention enhancements of the fourth-layer features are completed through the cascaded ETF modules, thereby enhancing the expression of the middle-layer features containing balanced spatial and semantic information.

[0099] The calculation formula of the ETF module is:

[0100] ETF(Q,K,V)=Norm(I+MLP(I))

[0101] I=Norm(Q+MutiHead(Q,Norm&AvgPool(K,V))) (3);

[0102] Among them, Norm is the normalization layer of the ETF module, AvgPool is the average pooling of the ETF module, MLP is the fully connected layer of the ETF module, I is the intermediate variable of the residual structure of the ETF module, MutiHead is the multi-head attention mechanism, Q is the query variable Q, K is the key index value in the key-value pair, and V is the corresponding value of the key index value.

[0103] S4: Perform a deep cross-correlation operation on the first feature block and the second feature block, and obtain a response map by merging and reducing the scale.

[0104] Step S4 includes:

[0105] By performing layered convolution on the corresponding layers of the target template and the search area, the response map of each level of cross-correlation operation is calculated using a deep cross-correlation operation, and then the response map R that combines the deep features and the shallow features is output by merging and reducing the scale:

[0106]

[0107] in, represents the deep cross-correlation operation, B is the corresponding layer of the target template, and C is the corresponding layer of the search area.

[0108] S5: The response map is sequentially input into the spatiotemporal dependency fusion module and the prediction head for target classification and regression to obtain the target tracking result.

[0109] Step S5 specifically includes the following steps:

[0110] S51: Input the response map into the spatiotemporal dependency fusion module to obtain the feature map M2.

[0111] S52: Input the feature map M2 into the prediction head for target classification and regression to obtain the target tracking result.

[0112] Step S52 includes:

[0113] The prediction head adopts a classification and regression network based on the anchor-free mechanism to directly classify and regress the candidate boxes at each position in the response map to obtain the target tracking result. The response map represents the linear mapping of the input image of the search area in space with the target template as the weight, and each position in the response map is mapped to the corresponding position in the original image space of the search area.

[0114] The embodiment of the present invention adopts a classification regression network based on the anchor-free mechanism, and directly classifies and regresses the candidate frame at each pixel position through the classification head and regression head network, without setting prior information to initialize the anchor frame size, thereby breaking through the limitations of the scale and receptive field of the fixed aiming frame, and minimizing the limitations and interference of the set prior information on the classification regression network, avoiding the introduction of multi-layer hyperparameter information that is difficult to optimize, and ensuring the computational efficiency of the classification regression network. The center head calculates the center score of the predicted frame at each pixel position to eliminate the interference of some abnormal pixels on the accuracy of the generated aiming frame.

[0115] Figure 2The model framework of the target tracking method based on the broad spectrum correlation fusion network provided according to an embodiment of the present invention is shown.

[0116] like Figure 2 As shown in the figure, the model network results of this method mainly include the following four main parts: feature extraction backbone network, Transformer fusion neck, similarity matching network, and prediction head. Among them, the feature extraction backbone network adopts ResNet-50 feature extraction network, and the Transformer fusion neck is composed of coordinate context encoder, adjustment layer and Transformer cascade fusion module. The coordinate context encoder is used to realize the shallow spatial position information update based on long-distance dependency; the adjustment layer is used to adjust the channel; the Transformer cascade fusion module is used to further realize the branch communication of multi-way feature calculation for the output features of the CCE module. The similarity matching network is used to obtain the response map through deep cross-correlation operation. The prediction head includes a classification head and a regression head. The response map is processed by the spatiotemporal dependency fusion module, that is, the dependencies of multiple dimensions are used to better balance the semantics and spatial information of the deep feature map, and the balanced response map is input into the prediction head to realize the classification and regression of the target.

[0117] Figure 3 FIG. 2 shows the structure of a coordinate context encoder provided according to an embodiment of the present invention.

[0118] like Figure 3 As shown, its working principle and process are as described in step S31. The channel relationship and long-term dependency are encoded through precise position information, thereby breaking through the limitation of the receptive field, obtaining the global receptive field and encoding precise position information, and realizing the shallow spatial position information update based on long-distance dependency.

[0119] Figure 4(a) shows the structure of a channel dimension fusion encoder provided according to an embodiment of the present invention; Figure 4(b) shows the structure of a mixed space dimension fusion encoder provided according to an embodiment of the present invention; Figure 4(c) shows a structural block diagram of a spatiotemporal dependency fusion network provided according to an embodiment of the present invention.

[0120] As shown in Figure 4(a)-(c), the spatiotemporal dependency fusion module includes a channel dimension fusion encoder and a mixed space dimension fusion encoder, where:

[0121] The response map is input into the average pooling layer and the maximum pooling layer of the channel dimension fusion encoder for pooling processing, and the feature map after pooling is activated by the Mish activation function in the shared MLP submodule to obtain two weights with the same number of channels as the response map. The two weights are added and normalized by the Sigmoid function to obtain the channel dimension fusion weight:

[0122] M c =Sigmiod(MLP(MaxPool(F))+MLP(AvgPool(F))) (5);

[0123] Among them, F is the response graph, Sigmiod is the Sigmoid function, and MaxPool is the maximum pooling process.

[0124] The response map is processed through the Conv1 layer to obtain feature map N. Feature map N is processed by maximum pooling and average pooling in turn to obtain feature map N1. At the same time, the vertical tensor group obtained by changing the dimension of feature map N is multiplied by the horizontal tensor group obtained by changing the dimension and transposing the feature map N2. Feature map N2 is processed by average pooling to obtain feature map N3. Feature map N1 and feature map N3 are added and normalized by Sigmoid function to obtain spatial dimension fusion weight:

[0125]

[0126] Among them, conv2 is the dimension adjustment convolution layer, reshape is the operation of changing the shape of the tensor, reshape&transpose is the operation of changing the shape of the tensor and transposing it, and M s1 and M s2 To calculate M s A process variable with no actual physical meaning.

[0127] The channel dimension fusion weight and the spatial dimension fusion weight are multi-dimensionally multiplied and then added to obtain the feature map M, the feature map M and the response map are channel-concatenated to obtain the feature map M1, and the feature map M1 is processed through the Conv3 layer to obtain the optimized feature map M2;

[0128] The calculation formula of feature map M2 is:

[0129] SDF(F)=conv3[F,F×M c +F×M s ] (7);

[0130] Among them, F is the response graph, conv3 is the operation of Conv3 layer, M c is the channel dimension fusion weight, M s is the fusion weight of the spatial dimension.

[0131] The target number of channels is 256 channels, the scale of the first feature block is 7*7*256, and the scale of the second feature block is 31*31*256.

[0132] The adjustment layer adopts the dimension adjustment strategy of multi-layer convolutional layers to uniformly adjust the number of channels of the three target feature blocks of the two branches after channel cascading to the target channel, including:

[0133] The number of channels of the three target feature blocks of the two branches after channel cascading is uniformly adjusted from the original [512, 1024, 2048] channels to [256, 256, 256] channels, and the dimensions of the number of channels of the three target feature blocks of the two branches are adjusted to 3*256.

[0134] Figure 5 The structure of a target tracking device based on a broad spectrum correlation fusion network provided according to an embodiment of the present invention is shown.

[0135] like Figure 5 As shown, the target tracking device based on the broad spectrum correlation fusion network provided by the present invention includes:

[0136] A normalization unit 1 is used to normalize the input images of the target template and the two branches of the search area of ​​the twin neural network to obtain normalized images of the two branches;

[0137] Feature extraction unit 2, used for inputting the normalized images of the two branches into the residual network for feature extraction, and obtaining the last three layers of feature blocks of the two branches;

[0138] Transformer fusion neck unit 3 is used to perform feature fusion and feature enhancement on the last three layers of feature blocks of the two branches, and obtain the first feature block and the second feature block accordingly;

[0139] A similarity matching network unit 4 is used to perform a deep cross-correlation operation on the first feature block and the second feature block, and obtain a response map by merging and reducing the scale operation;

[0140] The prediction unit 5 is used to input the response map into the spatiotemporal dependency fusion module and the prediction head in sequence to perform target classification and regression to obtain the target tracking result.

[0141] Accordingly, according to an embodiment of the present invention, the present invention also provides a computer device, a readable storage medium and a computer program product.

[0142] Figure 6 It is a schematic diagram of the structure of a computer device 6 provided in an embodiment of the present invention. Figure 6 A block diagram of an exemplary computer device 6 suitable for use in implementing embodiments of the present invention is shown. Figure 6 The computer device 6 shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present invention.

[0143] like Figure 6As shown, computer device 6 is in the form of a general-purpose computing device. Computer device 6 is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. Electronic devices can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present invention described and / or required herein.

[0144] Components of the computer device 6 may include, but are not limited to, one or more processors or processing units 8, a system memory 13, and a bus 9 that connects various system components (including the system memory 13 and the processing unit 8).

[0145] Bus 9 represents one or more of several types of bus structures, including a memory bus or memory controller, a peripheral bus, an accelerated graphics port, a processor or a local bus using any of a variety of bus structures. For example, these architectures include but are not limited to Industry Standard Architecture (ISA) bus, Micro Channel Architecture (MAC) bus, Enhanced ISA bus, Video Electronics Standards Association (VESA) local bus and Peripheral Component Interconnect (PCI) bus.

[0146] The computer device 6 typically includes a variety of computer system readable media. These media can be any available media that can be accessed by the computer device 6, including volatile and non-volatile media, removable and non-removable media.

[0147] System memory 13 may include computer system readable media in the form of volatile memory, such as random access memory (RAM) 14 and / or cache memory 15. Computer device 6 may further include other removable / non-removable, volatile / non-volatile computer system storage media. By way of example only, storage system 16 may be used to read and write non-removable, non-volatile magnetic media ( Figure 6 not shown, usually called a "hard drive"). Although Figure 6 Not shown in the figure, a disk drive for reading and writing to a removable non-volatile disk (e.g., a "floppy disk"), and an optical disk drive for reading and writing to a removable non-volatile optical disk (e.g., a CD-ROM, a DVD-ROM, or other optical media) may be provided. In these cases, each drive may be connected to the bus 9 via one or more data medium interfaces. The memory 28 may include at least one program product having a set (e.g., at least one) of program modules that are configured to perform the functions of the various embodiments of the present invention.

[0148] A program / utility 17 having a set (at least one) of program modules 18 may be stored, for example, in a memory 28, such program modules 18 including but not limited to an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment. The program modules 18 generally perform the functions and / or methods of the embodiments described herein.

[0149] The computer device 6 may also communicate with one or more external devices 7 (e.g., keyboard, pointing device, display 12, etc.), may also communicate with one or more devices that enable a user to interact with the computer device 6, and / or communicate with any device that enables the computer device 6 to communicate with one or more other computing devices (e.g., network card, modem, etc.). Such communication may be performed via an input / output (I / O) interface 11. Furthermore, the computer device 6 may also communicate with one or more networks (e.g., a local area network (LA6), a wide area network (WA6), and / or a public network, such as the Internet) via a network adapter 10. As shown, the network adapter 10 communicates with other modules of the computer device 6 via a bus 9. It should be understood that, although not shown in the figure, other hardware and / or software modules may be used in conjunction with the computer device 6, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0150] The processing unit 8 executes various functional applications and data processing by running the programs stored in the system memory 13, such as implementing the target tracking method based on the wide spectrum correlation fusion network provided in the embodiment of the present invention.

[0151] An embodiment of the present invention also provides a non-transitory computer-readable storage medium storing computer instructions, on which a computer program is stored, wherein when the program is executed by a processor, the target tracking method based on a broad spectrum correlation fusion network provided in all the inventive embodiments of the present application is implemented.

[0152] The computer storage medium of the embodiment of the present invention may adopt any combination of one or more computer-readable media. The computer-readable medium may be a computer-readable signal medium or a computer-readable storage medium. More specific examples (non-exhaustive list) of computer-readable storage media include: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this document, a computer-readable storage medium may be any tangible medium containing or storing a program that may be used by or in combination with an instruction execution system, an apparatus, or a device.

[0153] Computer-readable signal media may include data signals propagated in baseband or as part of a carrier wave, which carry computer-readable program code. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Computer-readable signal media may also be any computer-readable medium other than a computer-readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0154] The program code included in the computer readable medium can be transmitted with any appropriate medium, including but not limited to wireless, electric wire, optical cable, RF, etc., or any suitable combination of the above. The computer program code for performing the operation of the present invention can be written in one or more programming languages ​​or their combinations, and the programming language includes object-oriented programming languages ​​such as Java, Smalltalk, C++, and also includes conventional procedural programming languages-such as "C" language or similar programming languages. The program code can be executed completely on the user's computer, partially on the user's computer, as an independent software package, partially on the user's computer and partially on the remote computer, or completely on the remote computer or server. In the case of a remote computer, the remote computer can be connected to the user's computer by any type of network including a local area network (LA6) or a wide area network (WA6), or it can be connected to an external computer (for example, using an Internet service provider to connect to the Internet).

[0155] An embodiment of the present invention further provides a computer program product, including a computer program, which implements the above-mentioned target tracking method based on the broad spectrum correlation fusion network when executed by a processor.

[0156] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps described in the disclosure of the present invention can be performed in parallel, sequentially or in different orders, as long as the desired results of the technical solution disclosed in the present invention can be achieved, and this document does not limit this.

[0157] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A target tracking method based on a broad spectrum correlation fusion network, characterized in that: The specific steps include: S1: normalize the target template of the twin neural network and the input images of the two branches of the search area to obtain normalized images of the two branches; S2: Input the normalized images of the two branches into the residual network for feature extraction, and obtain the last three layers of feature blocks of the two branches; S3: Input the last three layers of feature blocks of the two branches into the Transformer fusion neck network for feature fusion and feature enhancement, and obtain the first feature block and the second feature block accordingly; The step S3 specifically includes the following steps: S31: Input the last three layers of feature blocks of the two branches into three layers of independent coordinate context encoders to model the long-distance correlation of two spatial dimensions, and correspondingly obtain three feature blocks with long-distance correlation of two spatial dimensions; S32: inputting the three feature blocks of the two branches into the adjustment layer for channel adjustment respectively, and correspondingly obtaining three target feature blocks with a target number of channels; S33: The three target feature blocks of the two branches are hierarchically input into the Transformer cascade fusion module for global correlation modeling, and a first feature block and a second feature block with global correlation are correspondingly obtained; S4: performing a deep cross-correlation operation on the first feature block and the second feature block, and obtaining a response map by merging and reducing the scale; S5: The response map is sequentially input into the spatiotemporal dependency fusion module and the prediction head to perform target classification and regression to obtain the target tracking result.

2. The target tracking method based on a broad spectrum correlation fusion network according to claim 1 is characterized in that: The step S31 comprises: S311: The coordinate context encoder inputs any one of the last three layers of feature blocks into the residual structure for feature decomposition to obtain feature maps X1 and X2; S312: After the feature map X1 and the feature map X2 are globally pooled, channel cascade, 2D convolution, normalization layer and nonlinear layer operations are performed in sequence to obtain an aggregated feature map : （1）； in, is the normalization layer, is the nonlinear layer transformation, is the output feature of the pooling kernel with height h, is the output feature of the pooling kernel with width w; S313: Split the aggregated feature map into a feature map y1 and a feature map y2 along two directions of the spatial dimension, perform 2D convolution operation and Sigmoid normalization processing on the feature map y1 and the feature map y2 in sequence, and obtain optimization weights in two directions; S314: Fusing the optimization weights in the two directions, and applying the feature map X1 and the feature map X2 to the fused optimization weights to obtain an updated feature map Z; S315: Calculate the feature block input in step S31 by using the Gamma function, and add the feature map Z to the calculation result of the Gamma function, and output the feature map Y by the following formula: （2）； Wherein, y is the feature map Z, and X is the feature block input in step S311.

3. The target tracking method based on broad spectrum correlation fusion network according to claim 1 is characterized in that: The step S33 specifically includes: After the dimension adjustment, the feature blocks of each layer are flattened into sequence information using convolution operations and used as the input of the subsequent Transformer cascade fusion module, wherein the Transformer cascade fusion module includes five pooling strategy encoder ETF modules, the fourth-layer features output by the adjustment layer are used as the input of the query variable Q of the three-way ETF module in parallel, and the third-layer features, fourth-layer features and fifth-layer features output by the adjustment layer are used as the key-value pair inputs of the three-way ETF module in parallel, respectively, the calculation results of the ETF module whose key-value pair input is the fourth-layer features are used as the input of the QKV triplet of the next ETF module, and the output results are input into another ETF module, and multiple self-attention enhancements of the fourth-layer features are completed through the cascaded ETF modules; The calculation formula of the ETF module is: （3）； Among them, Norm is the normalization layer of the ETF module, AvgPool is the average pooling of the ETF module, MLP is the fully connected layer of the ETF module, I is the intermediate variable of the residual structure of the ETF module, It is a multi-head attention mechanism, Q is the query variable Q, K is the key index value in the key-value pair, and V is the corresponding value of the key index value.

4. The target tracking method based on broad spectrum correlation fusion network according to claim 1 is characterized in that: The step S4 comprises: By performing hierarchical convolution on the corresponding layers of the target template and the search area, a deep cross-correlation operation is used to calculate the response map of each level of cross-correlation operation, and the response map R of the deep feature and shallow feature fusion is output by merging and reducing the scale: （4）； in, represents a depth cross-correlation operation, B is the corresponding layer of the target template, and C is the corresponding layer of the search area.

5. The target tracking method based on broad spectrum correlation fusion network according to claim 1 is characterized in that: The spatiotemporal dependency fusion module includes a channel dimension fusion encoder and a mixed space dimension fusion encoder, wherein: The response map is respectively input into the average pooling layer and the maximum pooling layer of the channel dimension fusion encoder for pooling processing, and the feature map after pooling processing is activated by the Mish activation function in the shared MLP submodule to obtain two weights with the same number of channels as the response map, and the two weights are added and normalized by the Sigmoid function to obtain the channel dimension fusion weight; Processing the response map through the Conv1 layer to obtain a feature map N, performing maximum pooling and average pooling on the feature map N in sequence to obtain a feature map N1, and multiplying the vertical tensor group obtained by the dimension change operation of the feature map N with the horizontal tensor group obtained by the dimension change and transposition operation to obtain a feature map N2, performing average pooling on the feature map N2 to obtain a feature map N3, adding the feature map N1 and the feature map N3 and performing normalization processing through the Sigmoid function to obtain a spatial dimension fusion weight; The channel dimension fusion weight and the space dimension fusion weight are multi-dimensionally multiplied and then added to obtain a feature map M, the feature map M and the response map are channel-concatenated to obtain a feature map M1, and the feature map M1 is processed through a Conv3 layer to obtain an optimized feature map M2; The calculation formula of the characteristic graph M2 is: （5）； Wherein, F is the response graph, For the operation of the Conv3 layer, is the channel dimension fusion weight, The weights are fused for the spatial dimensions.

6. The target tracking method based on a broad spectrum correlation fusion network according to claim 5 is characterized in that: The step S5 specifically includes the following steps: S51: inputting the response map into the spatiotemporal dependency fusion module to obtain a feature map M2; S52: Input the feature map M2 into the prediction head to perform classification and regression on the target to obtain the tracking result of the target.

7. The target tracking method based on broad spectrum correlation fusion network according to claim 6 is characterized in that: The step S52 comprises: The prediction head adopts a classification and regression network based on an anchor-free frame mechanism to directly classify and regress the candidate frame at each position in the response map to obtain a target tracking result, wherein the response map represents a linear mapping of the input image of the search area in space with the target template as the weight, and each position in the response map is mapped to a corresponding position in the original image space of the search area.

8. The target tracking method based on broad spectrum correlation fusion network according to claim 1 is characterized in that: The resolutions of the normalized images of the two branches are 127 127 and 255 255, the target number of channels is 256 channels, and the scale of the first feature block is 7 7 256, the scale of the second feature block is 31 31 256.

9. The target tracking method based on broad spectrum correlation fusion network according to claim 1, characterized in that: The adjustment layer adopts a dimension adjustment strategy of a multi-layer convolutional layer to uniformly adjust the number of channels of the three target feature blocks of the two branches after channel cascading to the target channel, including: The number of channels of the three target feature blocks of the two branches after channel cascading is uniformly adjusted from the original [512, 1024, 2048] channels to [256, 256, 256] channels, and the dimensions of the number of channels of the three target feature blocks of the two branches are adjusted to 3 256.

10. A target tracking device based on a broad spectrum correlation fusion network, characterized in that: include: A normalization unit, used to normalize the target template of the twin neural network and the input images of the two branches of the search area to obtain normalized images of the two branches; A feature extraction unit is used to input the normalized images of the two branches into the residual network for feature extraction to obtain the last three layers of feature blocks of the two branches; The Transformer fusion neck unit is used to fuse and enhance the last three layers of feature blocks of the two branches, and obtain the first feature block and the second feature block respectively; The last three layers of feature blocks of the two branches are input into three layers of independent coordinate context encoders to model the long-distance correlation of two spatial dimensions, and three feature blocks with long-distance correlation of two spatial dimensions are obtained accordingly; The three feature blocks of the two branches are respectively input into the adjustment layer for channel adjustment, and three target feature blocks with the target number of channels are correspondingly obtained; The three target feature blocks of the two branches are hierarchically input into the Transformer cascade fusion module for global correlation modeling, and the first feature block and the second feature block with global correlation are obtained accordingly; A similarity matching network unit, configured to perform a deep cross-correlation operation on the first feature block and the second feature block, and obtain a response map by merging and reducing the scale; The prediction unit is used to input the response map into the spatiotemporal dependency fusion module and the prediction head in sequence to perform target classification and regression to obtain the target tracking result.

11. A computer device, characterized in that: include: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the target tracking method based on a wide-spectrum correlation fusion network as described in any one of claims 1 to 9.

12. A non-transitory computer-readable storage medium storing computer instructions, characterized in that: The computer instructions are used to enable the computer to execute the target tracking method based on a wide spectrum correlation fusion network according to any one of claims 1 to 9.

Images