Cross-modal target tracking method based on collaborative strategy and related device

Abstract

The invention belongs to the technical field of computer vision and image processing, and discloses a cross-modal target tracking method based on a collaborative strategy and a related device. The cross-modal target tracking method comprises the following steps: acquiring a template image and a current frame search image; based on the obtained template image and the current frame search image, performing cross-modal target tracking by using the trained cross-modal target tracking model to obtain a cross-modal target tracking result; the cross-modal target tracking model comprises four branch networks and a multi-modal region suggestion network; each of the four branch networks comprises a feature extraction module, a multi-scale channel adaptive enhancement module and a hierarchical progressive attention fusion module. According to the technical scheme, challenges such as scale change, deformation and texture blurring can be effectively handled, and the tracking stability and robustness under the conditions of shielding, modal interference and low illumination are improved.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of computer vision and image processing, and relates to multi-modal perception and target tracking technology, in particular to a cross-modal target tracking method based on a collaborative strategy and related devices. BACKGROUND

[0002] In recent years, target tracking technology is widely used in intelligent security, unmanned driving, space remote sensing and other fields. Its core goal is to locate and track the specified target in continuous frame images in real time. Among them, the tracking method based on twin networks represented by SiamFC, SiamRPN and the like has become one of the mainstream schemes. These methods calculate the feature similarity between the template graph and the search graph, and achieve good tracking performance while maintaining a certain real-time performance.

[0003] In order to further improve the adaptability in extreme environments, the academic community has proposed a cross-modal target tracking scheme. Among them, the visible light and thermal infrared (RGB-Thermal, RGBT) fusion tracking algorithm has attracted widespread attention in recent years. Specifically, the existing RGBT tracking methods can be roughly divided into two categories: one class directly splices or weightedly fuses the features of visible light (Red, Green, Blue, RGB) and thermal infrared (Thermal infrared, TIR) images, and the other class uses attention mechanism or modal adaptive module to improve the fusion effect. However, the above existing cross-modal target tracking methods still face certain challenges in feature modeling and feature fusion. On the one hand, many methods rely on single-scale or coarse-grained features for matching, lack of multi-scale context perception ability, and are difficult to effectively cope with target scaling and deformation. On the other hand, part of the deep network fusion scheme is easy to introduce feature redundancy or semantic shift when integrating multi-source information, which leads to the decline of the discriminant performance of the fused features, thereby limiting the improvement of the overall tracking effect.

[0004] In summary, although the cross-modal tracking technology has certain advantages in environmental adaptability, its fusion efficiency and feature modeling capability still have a lot of room for improvement, especially in complex scenes (such as occlusion, scale transformation, etc.). Therefore, it is necessary to propose a fusion strategy with better structural adaptability and information collaboration capability to fully release the potential of multi-modal perception. SUMMARY

[0005] The present application aims to provide a cross-modal target tracking method based on a collaborative strategy and related devices to solve one or more of the above technical problems. The technical solution disclosed in the present application can effectively cope with challenges such as scale change, deformation and texture blur, and improve the tracking stability and robustness in occlusion, modal interference and low light conditions.

[0006] To achieve the above object, the application adopts the following technical scheme:

[0007] In the first aspect of the application, a cross-modal target tracking method based on a cooperative strategy is provided, comprising the following steps:

[0008] Obtaining a template image and a current frame search image;

[0009] Based on the obtained template image and the current frame search image, a trained cross-modal target tracking model is used for cross-modal target tracking to obtain a cross-modal target tracking result;

[0010] The cross-modal target tracking model comprises four branch networks and a multi-modal region proposal network; the four branch networks are respectively used for inputting a visible light template image, a thermal infrared template image, a visible light search image and a thermal infrared search image, and correspondingly outputting visible light template enhanced features, thermal infrared template enhanced features, visible light search enhanced features and thermal infrared search enhanced features; the multi-modal region proposal network is used for inputting the enhanced features output by the four branch networks, generating target candidate regions through a full convolution network and taking the target candidate regions as the cross-modal target tracking result.

[0011] The four branch networks have the same structure and each comprises a feature extraction module, a multi-scale channel adaptive enhancement module and a hierarchical progressive attention fusion module; the feature extraction module inputs an image and extracts multi-layer features, each layer of the extracted features is subjected to channel attention enhancement and parallel hole convolution processing through each multi-scale channel adaptive enhancement module to obtain each layer of enhanced features fusing features of different scales; the hierarchical progressive attention fusion module is used to fuse the enhanced features by using an attention mechanism and a skip connection to reduce information redundancy and deviation between different levels of features, and obtain global enhanced features containing multi-level cooperative semantics.

[0012] The further improvement of the technical scheme of the application is that the multi-modal region proposal network comprises a visible light branch and a thermal infrared branch; wherein,

[0013] The visible light branch and the thermal infrared branch each comprise a classification branch and a regression branch;

[0014] In the classification branch and the regression branch, the input enhanced features are adjusted through convolution operation, and the adjusted template features are taken as convolution kernels to perform cross-correlation operation with the adjusted search features to generate a visible light classification branch response map, a visible light regression branch response map, a thermal infrared classification branch response map and a thermal infrared regression branch response map; the visible light and thermal infrared corresponding branch response maps are weightedly fused to obtain final classification response maps and regression response maps, and the cross-modal target tracking result is obtained according to the final response maps.

[0015] Further improvement of the technical scheme of the present application is that in the multi-scale channel adaptive enhancement module, the step of performing channel attention enhancement and parallel hole convolution processing to obtain enhanced features of fused different scale features specifically includes:

[0016] First, the selected input layer features are subjected to global maximum pooling and global average pooling to extract global context description at the channel level, and the two global maximum pooling and global average pooling results are spliced to obtain a splicing result.

[0017] Then, the splicing result is input into two fully connected networks, the first fully connected network uses ReLU activation to capture nonlinear channel dependence, and the second fully connected network uses Sigmoid activation to generate normalized channel attention weights.

[0018] Finally, based on the weighted features, multi-path parallel hole convolution branches are used to extract multi-scale information, and the outputs of each branch are added element by element in the channel dimension to be fused into a dynamic perception feature map with multi-scale information as the final enhanced feature; wherein the multi-path parallel hole convolution branches use convolution kernels of the same size and different hole rates to form multiple receptive fields from local fine to global context.

[0019] Further improvement of the technical scheme of the present application is that in the process of extracting global context description at the channel level by global maximum pooling and global average pooling of the input selected layer features, and splicing the two global maximum pooling and global average pooling results to obtain global features, the expression of the global features is:

[0020]

[0021] In the formula, represents the global feature of the corresponding channel; GAP(·) represents the global average pooling operation; GMP(·) represents the global maximum pooling operation; represents element-wise addition; represents an input layer feature;

[0022] In the process of inputting the global features into two fully connected networks, the first fully connected network uses ReLU activation to capture nonlinear channel dependence, and the second fully connected network uses Sigmoid activation to generate normalized channel attention weights, and the channel attention weights are multiplied with the input selected layer features channel by channel to obtain weighted features, the expression of the weighted features is:

[0023]

[0024] wherein, represents a feature with weight; w c represents a channel attention weight; represents element-wise multiplication; epsilon (·) represents a sigmoid activation function; fcl (·) represents a first fully connected layer, fc2 (·) represents a second fully connected layer; delta (·) represents a ReLU activation function.

[0025] Further improvement of the technical scheme of the application is that in the process of inputting an image and extracting multi-layer features by the feature extraction module, the multi-layer features are three-layer features.

[0026] In the hierarchical progressive attention fusion module, the step of using attention mechanism and skip connection to fuse and enhance to reduce information redundancy and deviation between different hierarchical features specifically includes:

[0027] The first layer feature and the second layer feature are element-wise added to form a low-level fusion representation; the low-level fusion representation is input into a convolution attention module to explicitly model the importance of channel and spatial dimensions, and to generate reinforced attention features; the reinforced attention features, the first layer features and the second layer features are element-wise added to form a transition feature representation.

[0028] The transition feature representation and the third layer feature are element-wise added first, then the convolution attention module is used for attention enhancement, and then the attention enhancement result and the first layer feature, the second layer feature, the third layer feature and the transition feature are element-wise added to finally output a global enhanced feature containing multi-level collaborative semantics.

[0029] Further improvement of the technical scheme of the application is that in the process of inputting an image and extracting multi-layer features by the feature extraction module, ResNet-50 network is used to extract features, and the third layer feature map, the fourth layer feature map and the fifth layer feature map are reserved as outputs; wherein the output third layer feature map, the fourth layer feature map and the fifth layer feature map correspond to the first layer feature, the second layer feature and the third layer feature in the multi-layer features in turn respectively.

[0030] Further improvement of the technical scheme of the application is that the specific training steps of the cross-modal target tracking model are as follows:

[0031] A data set is constructed, including: taking the RGB data set GOT10K as the training data set of the benchmark algorithm SiamRPN++, taking the RGBT data set LasHeR as the training set, taking GTOT and RGBT234 as the test set.

[0032] The model is constructed, including: modal expansion of a benchmark network, construction of a multi-scale cooperative module, and construction of a multi-level cooperative module; wherein, when the benchmark network is expanded in mode, the Backbone of the benchmark network is first expanded into four branches suitable for RGBT target tracking, and the four branches are used for feature extraction and correspond to a visible light template branch, a visible light search branch, a thermal infrared template branch and a thermal infrared search branch respectively; in addition, the original region proposal network RPN is expanded into an RGBT modal RPN network: the template features and search features of visible light and thermal infrared are calculated in parallel, the depth separable correlation is calculated, and two classification score maps and regression offset maps are output, and the results are fused through soft weight; when the multi-scale cooperative module is constructed, a plurality of multi-scale channel adaptive enhancement modules are designed; when the multi-level cooperative module is constructed, a level progressive attention fusion module is designed;

[0033] The model is trained, including: first, in the single modal pre-training stage, the model weight of the benchmark algorithm model SiamRPN++ is obtained by using the RGB data set GOT-10K for visible light single modal training; then, the benchmark model is expanded to RGBT multi-modal input, and the multi-scale channel adaptive enhancement module and the level progressive attention fusion module are added, and the LasHeR data set is used as the RGBT data set to train the model in stages: the Backbone parameters are frozen, and only the other parts are trained; when the training round reaches the preset round, the Backbone part layer is unfrozen, and the training is continued until the model loss value converges, and finally the complete model is saved to obtain the trained cross-modal target tracking model.

[0034] In the second aspect of the present application, a cross-modal target tracking system based on a cooperative strategy is provided, comprising:

[0035] An image acquisition module is configured to acquire a template image and a current frame search image.

[0036] A target tracking module is configured to perform cross-modal target tracking based on the acquired template image and current frame search image, using the trained cross-modal target tracking model, and obtain a cross-modal target tracking result.

[0037] The cross-modal target tracking model comprises four branch networks and a multi-modal region proposal network; wherein, the four branch networks are respectively configured to input a visible light template image, a thermal infrared template image, a visible light search image and a thermal infrared search image, and correspondingly output visible light template enhanced features, thermal infrared template enhanced features, visible light search enhanced features and thermal infrared search enhanced features; the multi-modal region proposal network is configured to input the enhanced features output by the four branch networks, generate target candidate regions through a full convolution network, and use the target candidate regions as the cross-modal target tracking result.

[0038] The four branch networks are identical in structure and each comprises a feature extraction module, a multi-scale channel adaptive enhancement module and a hierarchical progressive attention fusion module; the feature extraction module inputs an image and extracts multi-layer features, each layer of the extracted features is subjected to channel attention enhancement and parallel hole convolution processing through each multi-scale channel adaptive enhancement module, and each layer of enhanced features of different scales is obtained; each layer of the enhanced features is fused by the hierarchical progressive attention fusion module using an attention mechanism and a skip connection to reduce information redundancy and deviation between different levels of features, and finally global enhanced features containing multi-level collaborative semantics are obtained.

[0039] In a third aspect, the present application provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the cross-modality target tracking method based on the collaborative strategy according to any one of the first aspect of the present application.

[0040] In a fourth aspect, the present application provides a non-transitory computer readable storage medium having a computer program stored thereon, wherein the computer program is executable by a processor to implement the cross-modality target tracking method based on the collaborative strategy according to any one of the first aspect of the present application.

[0041] Compared with the prior art, the present application has the following beneficial effects:

[0042] The application discloses a cross-modal target tracking method based on a cooperative strategy, which realizes efficient cooperative fusion of visible light and thermal infrared modal features by introducing a multi-scale channel adaptive enhancement module (MCAEM) and a hierarchical progressive attention fusion module (HPAFM), can effectively cope with challenges such as scale change, deformation and texture blur, and improves tracking stability and robustness under conditions such as occlusion, modal interference and low light.

[0043] To sum up, in view of the problem that the existing method lacks a scale modeling mechanism, the application proposes a multi-scale modeling strategy based on a hollow convolution, which can introduce context information under different receptive fields without significantly increasing the amount of calculation, so that the tracker has adaptive ability to target changes such as rapid scaling and local deformation. In view of the problem that the existing deep feature fusion has semantic deviation and redundancy, the application introduces a hierarchical progressive fusion strategy to guide the gradual fusion of deep semantic and shallow details, effectively alleviates the expression inconsistency problem between different layer features, and enhances the feature semantic consistency and structural integrity. BRIEF DESCRIPTION OF DRAWINGS

[0044] In order to more clearly illustrate the technical solutions in the application or the prior art, the drawings needed to be used in the embodiments or prior art description will be briefly introduced below; obviously, the drawings in the following description are some embodiments of the application, and those skilled in the art can also obtain other drawings according to these drawings without creative labor.

[0045] Figure 1 is a flowchart of a cross-modal target tracking method based on a cooperative strategy in an embodiment of the present application;

[0046] Figure 2 is a full flowchart of a cross-modal target tracking method based on a cooperative strategy in a specific embodiment of the present application;

[0047] Figure 3 is a schematic diagram of the principle architecture of a cross-modal target tracking system in a specific embodiment of the present application;

[0048] Figure 4 is a structural schematic diagram of a multi-scale channel adaptive enhancement module in an embodiment of the present application;

[0049] Figure 5 is a structural schematic diagram of a hierarchical progressive attention fusion module in an embodiment of the present application;

[0050] Figure 6 is a schematic diagram of overall performance comparison on the GTOT dataset in an embodiment of the present application;

[0051] Figure 7 is a schematic diagram of overall performance comparison on the RGBT234 dataset in an embodiment of the present application;

[0052] Figure 8 is a schematic diagram of a cross-modal target tracking system based on a cooperative strategy in an embodiment of the present application. DETAILED DESCRIPTION

[0053] In order to make the objectives, technical solutions and advantages of the present application clearer, the technical solutions in the present application will be described clearly and completely below in conjunction with the drawings in the embodiments of the present application; obviously, the described embodiment technical solutions are a part of the embodiments of the present application, and not all the embodiments.

[0054] Based on the technical solutions disclosed in the embodiments of the present application, all other embodiments obtained by those of ordinary skill in the art without making creative efforts fall within the scope of protection of the present application. In addition, the terms “include” and “have” and any variations thereof are intended to cover non-exclusive inclusion, for example, a process, method, system, product or device that includes a series of steps or units does not have to be limited to only those steps or units clearly listed, but can include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0055] Referring to Figure 1 The cross-modal target tracking method based on a cooperative strategy provided in the embodiments of the present application includes the following steps:

[0056] Step 1, obtain a template image and a current frame search image; further illustratively, the current frame search image has a target to be tracked;

[0057] Step 2, based on the template image and the current frame search image obtained in step 1, cross-modal target tracking is performed using a trained cross-modal target tracking model to obtain a cross-modal target tracking result; in a specific example technical solution, the cross-modal target tracking result includes the coordinates (such as the center coordinates or the preset vertex coordinates, etc.) of the target to be tracked, the width and the height.

[0058] In the embodiment of the application, the above-mentioned cross-modal target tracking model includes four branch networks and a multi-modal region proposal network; wherein,

[0059] The four branch networks are respectively used for inputting a visible light template image, a thermal infrared template image, a visible light search image and a thermal infrared search image, and correspondingly outputting visible light template enhanced features, thermal infrared template enhanced features, visible light search enhanced features and thermal infrared search enhanced features; the multi-modal region proposal network is used for inputting the enhanced features output by the four branch networks, and generates high-quality target candidate regions through a full convolution network and as a cross-modal target tracking result;

[0060] The four branch networks have the same structure, and each includes a feature extraction module, a multi-scale channel adaptive enhancement module and a hierarchical progressive attention fusion module; wherein the feature extraction module inputs an image and extracts multi-layer features; each layer of features extracted is subjected to channel attention enhancement and parallel hole convolution processing through each multi-scale channel adaptive enhancement module, to obtain each layer of enhanced features fusing features of different scales; each layer of enhanced features is fused by the hierarchical progressive attention fusion module using an attention mechanism and a skip connection to reduce information redundancy and offset between different levels of features, to obtain a final global reinforced feature containing multi-level collaborative semantics.

[0061] In the embodiment of the application, the multi-modal region proposal network includes a visible light branch and a thermal infrared branch. Each branch includes a classification branch and a regression branch; in the classification branch and the regression branch, the input enhanced features are adjusted through convolution operation, and the adjusted template features are used as a convolution kernel to perform cross-correlation operation with the adjusted search features, to generate corresponding branch response maps (visible light classification branch response map, visible light regression branch response map, thermal infrared classification branch response map, thermal infrared regression branch response map); finally, the final classification response map and the regression response map are obtained by weightedly fusing the corresponding branch response maps of the visible light and the thermal infrared, and the tracking result is obtained according to the final response map.

[0062] The technical scheme disclosed in the embodiment of the present application systematically solves the key problems of low fusion quality, scale insensitivity and semantic shift in the existing cross-modal target tracking method on the basis of maintaining simple structure and efficient operation by introducing a structure synergistic enhancement and information fusion optimization mechanism, and has significant performance improvement and wide application value.

[0063] Please refer to Figure 2 and Figure 3 In the specific embodiment of the present application, a cross-modal target tracking method based on a synergistic strategy is provided, mainly including the following steps:

[0064] (S0), constructing a data set, including: taking the RGB data set GOT10K as the training data set of the benchmark algorithm SiamRPN++, taking the RGBT data set LasHeR as the training set, and taking GTOT and RGBT234 as the test set;

[0065] (S1), constructing a model, including: the construction of the model includes modal expansion of the benchmark network, construction of a multi-scale synergistic module, and construction of a multi-level synergistic module;

[0066] (S2), training the model, including: the training of the model first in a single-modal pre-training stage, using the RGB data set GOT-10K for visible light single-modal training to obtain the model weight of the benchmark algorithm model SiamRPN++; then, expanding the benchmark model to RGBT multi-modal input, adding a multi-scale channel self-adaptive enhancement module and a hierarchical progressive attention fusion module, and using the LasHeR data set as the RGBT data set to train the model in stages: freezing the Backbone parameters first, and only training the other parts; when the training round reaches the preset round, unfreezing the Backbone part layer, and continuing to train until the model loss value converges, and finally saving the complete model;

[0067] (S3), obtaining a tracking result, including: taking the first frame of the visible light and thermal infrared video sequence in the test set as a template image, selecting a target and inputting it into the trained model to generate a final tracking result.

[0068] In the embodiment of the present application, the specific implementation in step (S1) includes:

[0069] (S1-1), modal expansion of the reference network: first, the Backbone of the reference network is expanded into four branches suitable for RGBT target tracking, corresponding to the visible light template branch, the visible light search branch, the thermal infrared template branch and the thermal infrared search branch for feature extraction. And in order to make full use of the complementarity between deep and shallow features in the target tracking task, the network retains the feature maps of the third to fifth layers in the feature extraction stage. On this basis, the original region proposal network (Region Proposal Network, RPN) is expanded into an RGBT modal RPN network: the template features and search features of visible light and infrared are calculated in parallel, and the depth separable correlation is output, and two classification score maps and regression offset maps are output, and then the results are fused through soft weight, so as to take into account the advantages and disadvantages of the two modalities in different scenes;

[0070] (S1-2), constructing a multi-scale cooperative module: the present application designs a multi-scale channel adaptive enhancement module, as shown in Figure 4

[0071] (S1-3), constructing a multi-level cooperative module: the present application designs a hierarchical progressive attention fusion module, as shown in Figure 5

[0072] In the embodiment of the present application, in step (S3), the specific implementation includes:

[0073] (S3-1), obtaining images: the template image is centered on the target selected from the first frame of the visible light and thermal infrared image sequence, and the size of 127x127 pixels is cropped. The subsequent frame is a search image, and the search image is 255x255 pixels in size;

[0074] (S3-2), feature extraction: input the two pairs of images obtained from (S3-1) into the corresponding branch of the backbone network, and obtain image features at different levels. Specifically, the third, fourth and fifth layers of the ResNet-50 network are set as the output layers of the backbone network, which are used to obtain the features of the visible light image and the thermal infrared image;

[0075] (S3-3), feature enhancement: the multi-scale channel adaptive enhancement module and the hierarchical progressive attention fusion module are used to perform multi-scale and multi-level feature enhancement on the features output by (S3-2), and the enhanced template features and search features are output;

[0076] ​​(S3-4), feature matching: the enhanced features of (S3-3) are input into a network with classification branch and regression branch for processing: the classification branch calculates the target confidence of each position, which is used to determine the target position; the regression branch performs bounding box regression on the response result to predict the accurate position and size of the target. In each branch, the algorithm takes the adjusted target feature as the convolution kernel and performs correlation operation with the feature map of the search image to generate the response map of the corresponding branch. The position and size of the target in the current frame are obtained after merging the outputs of the two branches, and the target tracking is completed.

[0077] Please refer to Figure 4 In the embodiment of the application, the multi-scale channel adaptive enhancement module aims to capture and coordinate feature information of different scales using attention mechanism and dilated convolution, mainly including two parts: channel attention enhancement and parallel dilated convolution; wherein,

[0078] First, the multi-layer features extracted from the backbone network are input into a branch respectively, and in each branch, the layer features will first pass through global maximum pooling and global average pooling to extract the global context description at the channel level, and the specific process is as follows:

[0079]

[0080] Among them, represents the global feature of the corresponding channel, GAP(·) represents the global average pooling operation, GMP(·) represents the global maximum pooling operation, represents element-wise addition, represents a layer of features input into the module.

[0081] Then, the result of splicing the two pooling results is input into two fully connected networks, the first layer adopts ReLU activation to capture nonlinear channel dependence, and the second layer adopts Sigmoid activation to generate normalized channel attention weights, and finally the corresponding weights are multiplied with the input features channel by channel to obtain the enhanced features, and the specific process is as follows:

[0082]

[0083] Among them, w c represents the importance weight of the corresponding channel, epsilon(·) represents the sigmoid activation function, fc i (·) represents the i-th fully connected layer, delta(·) represents the ReLU activation function, represents the global feature of the corresponding channel, represents a layer of features extracted by the backbone network, represents element-wise multiplication, represents the feature with weight.

[0084] Interpretively, the channel attention enhancement can automatically identify and amplify the channels that contribute most to the tracking task, while suppressing irrelevant or noise channels, thereby enhancing the expression quality of the feature map.

[0085] After completing the channel weighting, four parallel dilated convolution branches are adopted to realize multi-scale information extraction; wherein the four branches all use the same size of convolution kernel, but are respectively set with different dilated rates, thereby forming multiple receptive fields from local fine to global context; the dilated convolution can effectively expand the receptive field while keeping the computational complexity approximately unchanged, so that the model obtains rich cross-scale features without additional downsampling or pyramid structure. The branch outputs are added element by element in the channel dimension, and are fused into a dynamic perception feature map U c , for subsequent similarity matching and bounding box regression, the specific process is:

[0086]

[0087] Wherein, U c represents the dynamic perception feature map output by the module, represents element-by-element addition, represents a feature with weight, y i (·) represents the i-th dilated convolution layer, i = 1, 2, 3, 4.

[0088] Please refer to Figure 5 In the embodiment of the application, the hierarchical progressive attention fusion module aims to fuse and enhance the feature information of different levels by using attention mechanism and skip connection, and reduce the information redundancy and deviation between different levels of features.

[0089] Specifically, the three-layer feature maps enhanced by the pre-sequence module are sequentially executed in the following processing process: the first layer and the second layer features are added element by element to fuse and build a preliminary low-level fusion representation. The fusion result is then input into the convolution attention module CBAM to explicitly model the importance of the channel and spatial dimension, and generate the enhanced attention feature. Then, the attention feature is added element by element again with the first layer and the second layer original features to form a transition feature representation. On this basis, the module performs the same operation again on the transition feature and the third layer feature: first element-by-element fusion, then attention enhancement through the CBAM module, and finally fusion of the enhanced result with the first layer, the second layer, the third layer original features and the transition feature, and finally output the global enhanced feature map containing multi-level collaborative semantics. The specific process is:

[0090]

[0091] Wherein, represents the i-th layer feature input, i = 1, 2, 3, represent a fusion feature after corresponding features are fused, represent a reinforced attention feature, CBAM(·) represents a channel-spatial joint attention mechanism, and T represents a transition feature, represent a global reinforced feature map output by the module.

[0092] Explanatorily, the above-mentioned parts are through multi-layer feature flow in a layer-by-layer stacked attention fusion manner, highlight key areas, and suppress redundant interference.

[0093] In the above specific embodiments of the present application, ResNet-50 backbone network is used to extract multi-modal features, parallel hole convolution is used to realize multi-scale feature extraction enhancement, and layer-by-layer attention mechanism is used to optimize deep and shallow feature fusion. This method significantly improves the complementary expression ability between modalities and the target tracking accuracy, enhances the robustness to complex scenes such as occlusion and scale transformation, and is suitable for intelligent security, unmanned driving and other fields.

[0094] Please refer to Figure 6 and Figure 7 In the specific embodiments of the present application, two commonly used data sets GTOT and RGBT234 in the field of RGBT target tracking are compared with other mainstream RGBT target tracking algorithms to illustrate the superiority of the present application. In the GTOT data set, the method of the embodiment of the present application is significantly better than the existing comparative algorithm in terms of precision and success rate, the accuracy is 89.4%, which is 3.3% higher than the second place, and the success rate is 72.1%, which is 3.9% higher than the second place; the above results show that the new scheme of the embodiment of the present application has excellent overall adaptability in tracking tasks with multiple complex factors. In the RGBT234 data set, although the challenge is more difficult and the scene is more complex, the method of the embodiment of the present application still leads in both precision and success rate, the precision is 77.3%, which is 2.3% higher than the second place, and the success rate is 53.6%, which is 0.3% higher than the second place. In summary, thanks to the enhancement of the multi-scale and multi-level cooperative mechanism to the target representation, the model used in the embodiment of the present application can maintain the continuous perception and discrimination ability of the target state under interference conditions.

[0095] The following is an apparatus embodiment of the present application, which can be used to execute the method embodiments of the present application. For details not disclosed in the apparatus embodiment, please refer to the method embodiments of the present application.

[0096] Please refer to Figure 8 In the embodiment of the present application, a cross-modal target tracking system based on a cooperative strategy is provided, which comprises:

[0097] An image acquisition module is used to acquire a template image and a current frame search image.

[0098] The target tracking module is configured to search images based on the acquired template image and the current frame, perform cross-modal target tracking by using a trained cross-modal target tracking model, and obtain a cross-modal target tracking result.

[0099] The cross-modal target tracking model comprises four branch networks and a multi-modal region proposal network.

[0100] The four branch networks have the same structure and each comprises a feature extraction module, a multi-scale channel adaptive enhancement module and a hierarchical progressive attention fusion module.

[0101] In an embodiment of the present application, a computer device is provided, which comprises a processor and a memory. The memory is configured to store a computer program, and the computer program comprises program instructions. The processor is configured to execute the program instructions stored in the computer storage medium. The processor can be a central processing unit (CPU), and can also be other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc. The processor is the computing core and control core of the terminal, and is suitable for implementing one or more instructions. Specifically, the processor is suitable for loading and executing one or more instructions in the computer storage medium to implement a corresponding method flow or a corresponding function. The processor in the embodiments of the present application can be used to perform the operation of the cross-modal target tracking method based on the cooperative strategy.

[0102] In an embodiment of the present application, a storage medium, specifically a computer readable storage medium (Memory) is provided, which is a memory device in a computer device, for storing programs and data. It can be understood that the computer readable storage medium herein can include an internal storage medium in the computer device, and of course can include an extended storage medium supported by the computer device. The computer readable storage medium provides a storage space, which stores an operating system of the terminal. Moreover, one or more instructions suitable for being loaded and executed by the processor are also stored in the storage space, and the instructions can be one or more computer programs (including program codes). It should be noted that the computer readable storage medium herein can be a high-speed RAM (Random Access Memory) memory, or a non-volatile memory such as at least one disk memory. The one or more instructions stored in the computer readable storage medium can be loaded and executed by the processor to implement the corresponding steps of the above-mentioned embodiment of the cross-modal target tracking method based on the cooperative strategy.

[0103] Those skilled in the art should understand that the embodiments of the present application can be provided as a method, a system, or a computer program product. Therefore, the present application can take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present application can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, optical storage, etc.) containing computer-usable program code.

[0104] The present application is described with reference to the flowcharts and / or block diagrams according to the embodiments of the present application. It should be understood that each flow and / or block in the flowcharts and / or block diagrams, and the combination of the flows and / or blocks in the flowcharts and / or block diagrams can be implemented by computer program instructions. These computer program instructions can be provided to a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing devices to produce a machine, so that the instructions executed by the computer or other programmable data processing devices generate a device for implementing the functions specified in the flowcharts and / or block diagrams. Figure 1 The functions specified in one or more flows and / or blocks Figure 1 The devices for implementing the functions specified in one or more flows and / or blocks.

[0105] These computer program instructions can also be stored in a computer readable storage medium, which can guide the computer or other programmable data processing devices to work in a specific manner, so that the instructions stored in the computer readable storage medium produce a product including instruction devices, which implement the functions specified in the flowcharts and / or block diagrams. Figure 1one or more processes and / or blocks Figure 1 the function specified in the one or more blocks.

[0106] These computer program instructions can also be loaded into computer or other programmable data processing devices, so that a series of operation steps are performed on the computer or other programmable data processing devices to generate computer-implemented processes, so that the instructions executed on the computer or other programmable data processing devices provide processes for implementing the flow Figure 1 one or more processes and / or blocks Figure 1 the function specified in the one or more blocks.

[0107] Finally, it should be noted that: the above examples are only used to illustrate the technical solutions of the present application, but not to limit it, although the above embodiments of the present application have been described in detail, those skilled in the art should understand: the specific embodiments of the present application can be modified or replaced by the same, without departing from the spirit and scope of the present application, any modification or equivalent replacement, which should be covered within the scope of protection of the claims of the present application.

Claims

1. A method for cross-modality object tracking based on a cooperative strategy, characterized in that, The method comprises the following steps: obtaining a template image and a current frame search image; based on the obtained template image and the current frame search image, performing cross-modal target tracking by using a trained cross-modal target tracking model to obtain a cross-modal target tracking result; The cross-modal target tracking model comprises four branch networks and a multi-modal region proposal network; the four branch networks are respectively used for inputting a visible light template image, a thermal infrared template image, a visible light search image and a thermal infrared search image, and correspondingly outputting visible light template enhanced features, thermal infrared template enhanced features, visible light search enhanced features and thermal infrared search enhanced features; the multi-modal region proposal network is used for inputting the enhanced features output by the four branch networks, generating target candidate regions through a full convolution network and taking the target candidate regions as the cross-modal target tracking result; The four branch networks have the same structure and each comprise a feature extraction module, a multi-scale channel adaptive enhancement module and a hierarchical progressive attention fusion module; the feature extraction module inputs an image and extracts multi-layer features, each layer of the extracted features is subjected to channel attention enhancement and parallel hole convolution processing through each multi-scale channel adaptive enhancement module, and each layer of enhanced features of different scales is obtained; the enhanced features of each layer are fused by the hierarchical progressive attention fusion module through an attention mechanism and a skip connection to reduce information redundancy and deviation between different levels of features, and a final global enhanced feature containing multi-level collaborative semantics is obtained.

2. The cross-modal target tracking method based on a collaborative strategy according to claim 1, characterized in that: The multi-modal region proposal network comprises a visible light branch and a thermal infrared branch; wherein The visible light branch and the thermal infrared branch each comprise a classification branch and a regression branch; In the classification branch and the regression branch, the input enhanced features are adjusted through convolution operation, and the adjusted template features are taken as convolution kernels to perform cross-correlation operation with the adjusted search features to generate a visible light classification branch response map, a visible light regression branch response map, a thermal infrared classification branch response map and a thermal infrared regression branch response map; the corresponding branch response maps of the visible light and the thermal infrared are weightedly fused to obtain a final classification response map and a regression response map, and the cross-modal target tracking result is obtained according to the final response map.

3. The cross-modal target tracking method based on a collaborative strategy according to claim 1, characterized in that: In the multi-scale channel adaptive enhancement module, the step of performing channel attention enhancement and parallel hole convolution processing to obtain enhanced features of fused different scale features comprises: First, the selected layer features are subjected to global maximum pooling and global average pooling to extract a channel-level global context description, and the two-pooling results of the global maximum pooling and the global average pooling are spliced to obtain a splicing result; Then, the spliced result is input into two layers of fully connected networks, the first layer of fully connected networks adopts ReLU activation to capture nonlinear channel dependence, and the second layer of fully connected networks adopts Sigmoid activation to generate normalized channel attention weights, the channel attention weights are multiplied with the input selected layer features channel by channel to obtain weighted features; Finally, based on the weighted features, multi-scale information is extracted by using multi-parallel dilated convolution branches, the outputs of the branches are added element by element in the channel dimension to be fused into a dynamic perception feature map with multi-scale information as the final enhanced feature; wherein the multi-parallel dilated convolution branches use the same size of convolution kernel and are respectively set with different hole rates to form multiple receptive fields from local fine to global context.

4. The cross-modal target tracking method based on a cooperative strategy according to claim 3, characterized in that, In the process of splicing the global maximum pooling and the global average pooling to extract the global context description at the channel level and obtaining the global feature, the expression of the global feature is: In the formula, represent the global features corresponding to the channel; GAP(·) represents a global average pooling operation; GMP(·) represents a global max pooling operation; and represents element-wise addition. represents an input layer of features; In the process of inputting the global feature into two layers of fully connected networks, the first layer of fully connected networks adopts ReLU activation to capture nonlinear channel dependence, and the second layer of fully connected networks adopts Sigmoid activation to generate normalized channel attention weights, the channel attention weights are multiplied with the input selected layer features channel by channel to obtain weighted features, the expression of the weighted features is: wherein represent features with weights; w c representing channel attention weights; representing element-wise multiplication; ε(·) represents a sigmoid activation function; fc1(·) represents a first fully connected layer, fc2(·) represents a second fully connected layer; and δ(·) represents a ReLU activation function.

5. The cross-modal target tracking method based on a cooperative strategy according to claim 1, characterized in that, In the process of inputting the image into the feature extraction module and extracting multi-layer features, the multi-layer features are specifically three-layer features; In the hierarchical progressive attention fusion module, the steps of using attention mechanism and skip connection to fuse and enhance to reduce the information redundancy and deviation between different hierarchical features specifically include: The first layer feature and the second layer feature are added element by element to fuse and construct a low-level fusion representation; the low-level fusion representation is input into a convolution attention module to explicitly model the importance of the channel and spatial dimensions and generate reinforced attention features; and the reinforced attention features, the first layer feature and the second layer feature are added element by element to form a transition feature representation. The transition feature representation and the third layer feature are first added element by element to fuse, then the convolution attention module is used for attention enhancement, and then the attention enhancement result and the first layer feature, the second layer feature, the third layer feature and the transition feature are added element by element to finally output the global enhanced feature containing multi-level cooperative semantics.

6. The cross-modal target tracking method based on a cooperative strategy according to claim 5, characterized in that, In the process of inputting an image and extracting multi-layer features by the feature extraction module, a ResNet-50 network is adopted to extract features, and a third layer feature map, a fourth layer feature map and a fifth layer feature map are reserved as outputs; wherein the third layer feature map, the fourth layer feature map and the fifth layer feature map correspond to a first layer feature, a second layer feature and a third layer feature in the multi-layer features respectively.

7. The cross-modal target tracking method based on a cooperative strategy according to claim 1, characterized in that, The specific training steps of the cross-modal target tracking model are as follows: A data set is constructed, including: taking the RGB data set GOT10K as the training data set of the benchmark algorithm SiamRPN++, taking the RGBT data set LasHeR as the training set, and taking GTOT and RGBT234 as the test set; A model is constructed, including: modal expansion of the benchmark network, construction of a multi-scale cooperative module, and construction of a multi-level cooperative module; wherein, when the benchmark network is expanded, the Backbone of the benchmark network is expanded into four branches suitable for RGBT target tracking, and the four branches are used for feature extraction and correspond to the visible light template branch, the visible light search branch, the thermal infrared template branch and the thermal infrared search branch respectively; in addition, the original region proposal network RPN is expanded into an RGBT modal RPN network: the template features and search features of visible light and thermal infrared are calculated in parallel, the depth separable correlation is calculated, and two classification score maps and regression offset maps are output; the results are fused through soft weight; when the multi-scale cooperative module is constructed, multiple multi-scale channel adaptive enhancement modules are designed; when the multi-level cooperative module is constructed, a level-by-level attention fusion module is designed; The model is trained, including: first, in the single-modal pre-training stage, the RGB data set GOT-10K is used for visible light single-modal training to obtain the model weight of the benchmark algorithm model SiamRPN++; then, the benchmark model is expanded to RGBT multi-modal input, and the multi-scale channel adaptive enhancement module and the level-by-level attention fusion module are added, and the LasHeR data set is used as the RGBT data set to train the model in stages: first, freeze the Backbone parameters, and only train the other parts; when the training round reaches the preset round, unfreeze the Backbone part layer, continue to train until the model loss value converges, and finally save the complete model to obtain the trained cross-modal target tracking model.

8. A cross-modality object tracking system based on a cooperative strategy, characterized in that, including: An image acquisition module is configured to acquire a template image and a current frame search image; A target tracking module is configured to perform cross-modal target tracking based on the acquired template image and current frame search image, using the trained cross-modal target tracking model, and obtain a cross-modal target tracking result. The cross-modality target tracking model comprises four branch networks and a multi-modality region proposal network; the four branch networks are respectively used for inputting a visible light template image, a thermal infrared template image, a visible light search image and a thermal infrared search image, and correspondingly outputting visible light template enhanced features, thermal infrared template enhanced features, visible light search enhanced features and thermal infrared search enhanced features; the multi-modality region proposal network is used for inputting the enhanced features output by the four branch networks, and generates target candidate regions through a full convolution network and takes the target candidate regions as cross-modality target tracking results. The four branch networks have the same structure and each comprise a feature extraction module, a multi-scale channel adaptive enhancement module and a hierarchical progressive attention fusion module; the feature extraction module inputs an image and extracts multi-layer features, each layer of the extracted features is subjected to channel attention enhancement and parallel hole convolution processing through each multi-scale channel adaptive enhancement module, and each layer of enhanced features of different scales is obtained; each layer of the enhanced features is fused through the hierarchical progressive attention fusion module by using an attention mechanism and a skip connection to reduce information redundancy and deviation between different levels of features, and finally global enhanced features containing multi-level collaborative semantics are obtained.

9. An electronic device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, The processor executes the program to implement the cross-modality target tracking method based on a collaborative strategy according to any one of claims 1 to 7.

10. A non-transitory computer-readable storage medium having stored thereon a computer program, characterized in that, The computer program is executed by the processor to implement the cross-modality target tracking method based on a collaborative strategy according to any one of claims 1 to 7.

Images