Cross-modal cross-view-angle dynamic target detection tracking method for complex scene sequence image

Abstract

The invention relates to a cross-modal cross-view-angle dynamic target detection tracking method for a complex scene sequence image. The method comprises the following steps: constructing a cross-modal cross-view-angle dynamic target detection tracking data set; constructing a cross-modal cross-view-angle dynamic target detection tracking basic network; constructing a multi-scale cavity convolution module used for view angle and scale alignment and a feature alignment module used for modal alignment, and adding the multi-scale cavity convolution module and the feature alignment module to the basic network to form a cross-modal cross-view angle dynamic target detection tracking model; training a cross-modal cross-view-angle dynamic target detection tracking model based on the constructed training set to obtain a trained detection tracking model; and inputting to-be-detected tracking data into the trained detection tracking model to obtain a cross-modal cross-view-angle detection and tracking result of the dynamic target. According to the method, the problem that dynamic target features are difficult to correspond due to view angle change and attitude difference can be effectively solved.

Description

Technical Field

[0001] This invention relates to the fields of artificial intelligence and target detection technology, specifically to a cross-modal and cross-view dynamic target detection and tracking method for complex scene sequence images. Background Technology

[0002] Target detection and tracking is a crucial research area in computer vision, with broad application prospects, such as intelligent surveillance, autonomous driving, and drone navigation. Traditional target detection and tracking methods are typically based on a single modality (e.g., visible light images) or a fixed viewpoint, making it difficult to handle the challenges posed by multimodal data fusion and viewpoint changes in complex scenes. With the rapid development of sensor technology, acquiring multimodal data (e.g., visible light, infrared, radar, etc.) has become more convenient, providing richer information sources for target detection and tracking. However, the heterogeneity of multimodal data, noise interference, and issues such as target deformation and occlusion across viewpoints make accurate detection and tracking of dynamic targets particularly difficult. Existing methods often perform well in single-modal or fixed-viewpoint scenarios, but their robustness and accuracy significantly decrease in cross-modal and cross-viewpoint scenarios, failing to meet the needs of practical applications.

[0003] In recent years, the rise of deep learning technology has brought new breakthroughs to the field of object detection and tracking. Convolutional Neural Network (CNN)-based models have performed well in feature extraction and object association, but they still have limitations in cross-modal data fusion and cross-viewpoint object matching. In addition, dynamic objects in complex scenes often have characteristics such as rapid movement and frequent changes in appearance, which further increases the difficulty of detection and tracking.

[0004] Therefore, designing a method that can effectively integrate multimodal information, adapt to changes in perspective, and achieve accurate detection and tracking of dynamic targets has become the focus of current research. Summary of the Invention

[0005] In view of the above-mentioned technical problems, this invention proposes a cross-modal and cross-view dynamic target detection and tracking method for complex scene sequence images. It can effectively address the problem that dynamic target features are difficult to correspond due to changes in viewpoint and pose differences, thereby enhancing the network's ability to model multi-scale structural information and deep semantic representation, and comprehensively improving the cross-modal and cross-view dynamic target detection and tracking capabilities.

[0006] The technical solution to the technical problem of this invention is: a method for cross-modal and cross-view dynamic target detection and tracking of complex scene sequence images, comprising the following steps:

[0007] Step S1: Construct a cross-modal, cross-view dynamic target detection and tracking dataset;

[0008] Step S2: Construct a cross-modal, cross-view dynamic target detection and tracking basic network; the basic network includes a visible light detection branch and an infrared detection branch, used to output detection bounding boxes frame by frame, and to obtain target slices based on the detection bounding boxes; the basic network also includes a Kalman filter and a data association module, which sequentially complete trajectory prediction, trajectory matching and trajectory update; wherein, the data association module combines the feature matching results of the Hungarian algorithm and the ReID module to realize the association between the target and the trajectory, and finally outputs the cross-modal, cross-view dynamic target detection and tracking results;

[0009] Step S3: Construct a multi-scale dilated convolution module for viewpoint and scale alignment and a feature alignment module for modality alignment, and embed the multi-scale dilated convolution module and the feature alignment module into the ReID module of the basic network to form a cross-modality and cross-viewpoint dynamic target detection and tracking model;

[0010] Step S4: Train the cross-modal and cross-view dynamic target detection and tracking model based on the training set constructed in step S1 to obtain the trained detection and tracking model;

[0011] Step S5: Input the data to be detected and tracked into the trained detection and tracking model to obtain the cross-modal and cross-view detection and tracking results of dynamic targets.

[0012] According to one technical solution of the present invention, step S1 includes:

[0013] Step S11: Based on the category system and sample annotation specifications of dynamic targets, collect a cross-modal and cross-view dynamic target detection dataset with horizontal bounding boxes annotating the target locations and categories in the images;

[0014] Step S12: Slice the images in the collected cross-modal and cross-view dynamic target detection dataset to form a cross-modal and cross-view dynamic target detection and tracking dataset;

[0015] Step S13: Divide the cross-modal and cross-view dynamic target detection and tracking dataset into a training set, a validation set, and a test set according to a preset ratio.

[0016] According to one technical solution of the present invention, in step S2, constructing a cross-modal, cross-view dynamic target detection and tracking basic network includes:

[0017] A high-performance target detection model is used as the basic detector. The input visible light image sequence and infrared image sequence are processed respectively to extract the detection results containing target bounding boxes, category information and confidence scores in their respective modalities.

[0018] Based on the results of basic detection, a target state modeling and prediction mechanism based on Kalman filter is introduced to predict the target's position and velocity in the next frame by using the target's historical position information and motion trajectory.

[0019] A data association module is introduced, and the prediction results are input into the data association module to match the detection results of the current frame with the established tracking targets; the data association module integrates the optimal matching strategy based on motion prediction and the feature similarity measure based on the ReID module;

[0020] The ReID module employs a non-local attention mechanism to capture global information and long-distance dependencies, and combines generalized average pooling to extract fine-grained discriminative features to improve cross-modal and cross-view feature differentiation capabilities. The data association module outputs the cross-modal and cross-view dynamic target detection and tracking results after multi-source information fusion.

[0021] According to one technical solution of the present invention, the ReID module adopts a dual-stream network as the basic structure and uses a dual-stream ResNet-50 as the backbone network to extract feature maps of visible light and infrared target slices respectively; and during the training phase, a weighted regularized triplet loss is introduced for optimization.

[0022] According to one technical solution of the present invention, in step S3, constructing a multi-scale dilated convolution module for viewpoint and scale alignment includes:

[0023] Construct a module containing three parallel dilated convolution branches with dilation rates of 1, 2, and 3, respectively;

[0024] Each of the dilated convolutional branches includes a channel reduction layer, a dilated convolutional layer, and an activation function layer connected in sequence; the channel reduction layer is used to reduce the number of channels in the input feature map to a certain proportion of the original number of channels;

[0025] The feature maps output by each branch are spliced ​​and fused along the channel dimension to form a multi-scale joint representation.

[0026] Finally, a 1×1 convolutional layer is used to restore the original input feature size.

[0027] According to one technical solution of the present invention, in step S3, constructing the feature alignment module for modal alignment includes:

[0028] The feature alignment module receives high-level feature maps from visible light and infrared modes;

[0029] By calculating the difference between the visible light modal high-level feature map and the infrared modal high-level feature map in the feature space, a feature alignment loss is constructed;

[0030] During model training, the feature alignment loss is incorporated into the total loss function to constrain the consistency of visible light and infrared modes in high-level semantic features.

[0031] According to one technical solution of the present invention, the feature alignment loss adopts a feature alignment loss based on mean square error, the expression of which is:

[0032]

[0033] in, These are high-level feature maps for the visible light mode and the infrared mode, respectively. Their feature dimensions are both B×C×H×W, where B is the batch size, C is the number of channels, and H and W are the height and width of the feature map, respectively.

[0034] According to one technical solution of the present invention, step S4 includes:

[0035] The cross-modal, cross-view dynamic target detection and tracking model is trained using the training set from step S1;

[0036] During training, cross-entropy loss is used as classification supervision, and weighted regularized triplet loss and feature alignment loss are introduced to jointly optimize model parameters.

[0037] The network performance is evaluated using the validation set in step S1. The stochastic gradient descent optimization algorithm is adopted, the learning rate decay strategy is set, and the network parameters are iteratively updated until the model converges, thus obtaining the trained cross-modal and cross-view dynamic target detection and tracking model.

[0038] Input the test set from step S1 into the trained detection and tracking model to obtain the detection and tracking results on the test set, thus completing the model validation.

[0039] Compared with the prior art, the present invention has the following beneficial effects:

[0040] This invention provides a cross-modal and cross-view dynamic target detection and tracking method for complex scene image sequences. Addressing the challenge of matching cross-modal features due to different viewpoints and poses, this method introduces a multi-scale dilated convolution module to acquire richer contextual and structural information. Furthermore, it utilizes a feature alignment module to measure and constrain the feature differences between visible light and infrared modalities at higher levels, achieving effective alignment of the two modalities. Through joint optimization of multi-scale representation enhancement and cross-modal alignment, this invention significantly improves the accuracy and robustness of the network in multi-level feature extraction, embedding consistency modeling, and cross-modal and cross-view dynamic target detection and tracking. Attached Figure Description

[0041] Figure 1This diagram illustrates a cross-modal, cross-view dynamic target detection and tracking network for sequential images in a complex scene, according to one embodiment of the present invention.

[0042] Figure 2 A schematic diagram illustrating a ReID module network architecture according to an embodiment of the present invention;

[0043] Figure 3 A schematic diagram illustrating a multi-scale dilated convolution module according to an embodiment of the present invention;

[0044] Figure 4 A schematic diagram illustrating a non-local attention module according to an embodiment of the present invention;

[0045] Figure 5 This schematic diagram illustrates a process flow of a dynamic target detection and tracking method for sequential images across modalities and viewpoints in complex scenarios according to an embodiment of the present invention. Detailed Implementation

[0046] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the accompanying drawings used in the embodiments will be briefly described below. Obviously, the drawings described below are merely some embodiments of the present invention, and those skilled in the art can obtain other drawings based on these drawings without any creative effort.

[0047] The present invention will now be described in detail with reference to the accompanying drawings and specific embodiments. The embodiments cannot be described in detail here, but the embodiments of the present invention are not limited to the following embodiments.

[0048] like Figures 1 to 5 As shown, this invention provides a method for dynamic target detection and tracking across modalities and viewpoints in sequence images under complex scenes. By constructing a framework of "independent detection first, then enhanced association", and innovatively introducing a multi-scale dilated convolution module and a feature alignment module, this invention can achieve higher tracking accuracy and robustness on publicly available cross-modal tracking datasets compared to traditional fusion detection methods or simple post-detection association methods. In particular, its advantages are more obvious in scenes with drastic viewpoint changes and complex lighting conditions.

[0049] like Figure 5 As shown, the present invention provides a method for cross-modal and cross-view dynamic target detection and tracking of complex scene sequence images, comprising the following steps:

[0050] Step S1: Construct a cross-modal, cross-view dynamic target detection and tracking dataset, including:

[0051] Step S11: Based on the category system and sample annotation specifications of dynamic targets, collect a cross-modal and cross-view dynamic target detection dataset with horizontal bounding boxes annotating the target locations and categories in the images;

[0052] Based on a predefined dynamic target category system and detailed sample annotation specifications, labeled image or video data is collected from multiple sources. Annotation information includes at least the target's horizontal bounding box in the image and its category label. The collected data should cover different scenes, different times, different weather conditions, and the same scene captured from different perspectives, thus forming a qualified initial dataset for cross-modal and cross-view dynamic target detection.

[0053] Step S12: Slice the images in the collected cross-modal and cross-view dynamic target detection dataset to form a cross-modal and cross-view dynamic target detection and tracking dataset;

[0054] Step S13: Divide the cross-modal and cross-view dynamic target detection and tracking dataset into a training set, a validation set, and a test set according to a preset ratio;

[0055] For example, it can be randomly divided into a training set, a validation set, and a test set in a 7:1:2 ratio. The training set is used to learn the model parameters, the validation set is used to monitor model performance and adjust hyperparameters during training, and the test set is used to finally evaluate the model's generalization ability.

[0056] Step S2: Construct a cross-modal, cross-view dynamic target detection and tracking basic network; the basic network includes a visible light detection branch and an infrared detection branch, used to output detection bounding boxes frame by frame, and to obtain target slices based on the detection bounding boxes; the basic network also includes a Kalman filter and a data association module, which sequentially complete trajectory prediction, trajectory matching and trajectory update; wherein, the data association module combines the feature matching results of the Hungarian algorithm and the ReID module to realize the association between the target and the trajectory, and finally outputs the cross-modal, cross-view dynamic target detection and tracking results;

[0057] like Figure 1 As shown, the high-performance target detection model YOLO is first used as the base detector to extract the target's bounding box, category information, and confidence score from the input video sequence. Based on this, a Kalman filter is introduced to model and predict the target's state, using the target's historical position and motion information to predict its position and velocity in the next frame. The prediction results are then input together with the detection results of the current frame into the data association module.

[0058] The data association module combines the feature matching results of the Hungarian algorithm and the ReID module to associate the target with the trajectory. The ReID module employs a non-local attention mechanism to capture global information and long-distance dependencies, while using generalized average pooling to obtain fine-grained discriminative features specific to the domain, thereby enhancing feature representation capabilities. The data association module ultimately outputs cross-modal, cross-view dynamic target detection and tracking results.

[0059] like Figure 2 As shown, the ReID module uses a two-stream network as its basic structure, employing a two-stream ResNet-50 as the backbone network to extract feature maps of two different modalities from the input image, and combines a non-local attention module (such as...) Figure 3 (As shown) To capture global information and long-range dependencies, during the training phase, all features before and after the batch normalization layer are input into different loss functions, including weighted regularized triplet loss, to jointly optimize the cross-modal and cross-view target detection and tracking network. Subsequently, a similarity measurement network is applied to the feature maps to output the cross-modal and cross-view dynamic target detection and tracking results.

[0060] Step S3: Construct a multi-scale dilated convolution module for viewpoint and scale alignment and a feature alignment module for modality alignment, and add the multi-scale dilated convolution module and the feature alignment module to the base network to form a cross-modality and cross-viewpoint dynamic target detection and tracking model;

[0061] Enhancements were made to the ReID module of the base network by adding two key sub-modules to improve the alignment and discriminative power of the extracted features under cross-viewpoint and cross-modal conditions.

[0062] Step S3 is the core step of this invention, and specifically includes:

[0063] Step S31: Construct a multi-scale dilated convolution module: such as Figure 3 As shown, this module (Multi-Scale Dilated Convolution Module, MSDC) is designed to enhance the robustness of features to changes in target scale and viewpoint deformation.

[0064] The specific construction method includes building multiple parallel dilated convolutional branches. Each branch uses the same kernel size (e.g., 3×3) but sets a different dilation rate. For example, three branches can be built with dilation rates of 1, 2, and 3 respectively. These dilated convolutional layers can effectively expand the receptive field while maintaining the feature map spatial resolution, thereby capturing a wider range of local and contextual information without introducing additional downsampling operations.

[0065] To control computational overhead and improve training efficiency, each dilated convolutional branch in the module reduces the channel dimension of the input feature map to a certain proportion of the original dimension (e.g., 1 / 4 or 1 / 3) to reduce subsequent fusion costs. After convolution, the features are introduced with non-linearity through the ReLU activation function, enhancing the model's ability to express complex structures and semantic patterns. Subsequently, the feature maps output by each branch are concatenated and fused along the channel dimension to form a multi-scale joint representation. To maintain consistency with the backbone network interface, the fused feature map is also passed through a standard convolutional layer (e.g., a 1×1 convolution) to restore its channel count to match the original input, facilitating subsequent module processing and feature propagation. After this step, each branch of MSDC can generate an independent embedding representation, thereby providing richer feature information at multiple scales. The embedding generated by each branch can be written in the following form:

[0066]

[0067] Where f represents the original embedding, This represents the embedding generated by the multi-scale dilated convolution module. This represents a ReLU activation layer, and Concat means concatenating the features from each branch along the channel dimension. This represents the j-th deformable convolutional layer with a deformable kernel size of i. This indicates a convolutional layer with a kernel size of 1×1.

[0068] The introduction of multi-scale dilated convolutional modules not only enables the model to adapt to the diversity of target scales but also improves the consistency and discriminative ability of cross-modal features, thereby enhancing overall matching performance. By capturing features from different receptive fields in parallel, the multi-scale dilated convolutional modules allow the network to simultaneously perceive both local details and global context of the target. This helps the model understand target scale scaling and shape changes caused by different camera perspectives, and the extracted features are more invariant to such geometric transformations, thus achieving "scale and perspective alignment" across different viewpoints.

[0069] Step S32: Construct the Feature Alignment Module: This module aims to reduce the semantic differences between visible light and infrared modes in the feature space.

[0070] The specific construction method includes: this module operates in the backbone feature extraction stage of the network (e.g., the third and fourth stages of ResNet-50). Let... These represent feature maps output from the same batch of data from the visible light branch and the infrared branch at the same high-level network (e.g., Stage 3 or Stage 4), with each feature having a dimension of B×C×H×W (B is the batch size, C is the number of channels, and H and W are the height and width).

[0071] The core of the feature alignment module is a loss function based on a distance metric. In this implementation, a feature alignment loss based on mean squared error (MSE) is used. The loss function calculates the sum of squared differences in pixel values ​​across all locations and channels on the feature maps corresponding to the two modalities, and then averages these differences.

[0072]

[0073] This loss function is added to the network's total loss during training. Through backpropagation optimization, this loss term forces the network to learn feature representations that make the high-level semantic features from the same target but different modalities as similar in numerical distribution as possible.

[0074] Through the aforementioned feature alignment module, this invention can guide the network to form a consistent cross-modal embedding representation in the high-level semantic space while maintaining the differences in features across modalities. This module effectively reduces cross-modal feature shifts caused by viewpoint differences, pose variations, and different imaging mechanisms by constraining feature differences pixel-by-pixel. This significantly improves the robustness and accuracy of cross-modal and cross-viewpoint dynamic target detection and tracking.

[0075] The constructed multi-scale dilated convolutional module is inserted into the feature extraction path of the ReID module (e.g., after the backbone network outputs features), and the feature alignment module is integrated into the loss calculation during the training phase. This results in an enhanced cross-modal, cross-view dynamic object detection and tracking model.

[0076] Step S4: Train the cross-modal, cross-view dynamic target detection and tracking model based on the training set constructed in step S1 to obtain the trained detection and tracking model, including:

[0077] The cross-modal, cross-view dynamic target detection and tracking model is trained using the training set from step S1;

[0078] During training, cross-entropy loss is used as classification supervision, and weighted regularized triplet loss and feature alignment loss are introduced to jointly optimize model parameters.

[0079] The network performance is evaluated using the validation set in step S1. The stochastic gradient descent optimization algorithm is adopted, the learning rate decay strategy is set, and the network parameters are iteratively updated until the model converges, thus obtaining the trained cross-modal and cross-view dynamic target detection and tracking model.

[0080] The cross-modal, cross-view dynamic object detection and tracking network is trained using the training set constructed in step S1, and its matching performance is evaluated using the validation set defined in step S1. In the ReID module, cross-entropy loss is used as classification supervision, and weighted regularized triplet loss and the aforementioned feature alignment loss are introduced to jointly optimize the discriminability and consistency of the cross-modal embedding space. The backbone network is initialized using a ResNet-50 pre-trained on the ImageNet dataset and iteratively optimized using stochastic gradient descent (SGD). The training epochs are set to 100, the batch size to 4, and the initial learning rate to 0.01. As the iteration process proceeds, the parameters of the cross-modal, cross-view dynamic object detection and tracking network are continuously updated until the model performance stabilizes and converges, ultimately obtaining the cross-modal, cross-view dynamic object detection and tracking model.

[0081] Input the test set from step S1 into the trained detection and tracking model to obtain the detection and tracking results on the test set, thus completing the model validation.

[0082] The test set from step S1 is input into the trained cross-modal, cross-view dynamic target detection and tracking model to obtain the detection and tracking results. On the one hand, Rank-1 accuracy and mean average precision (mAP) are used as performance evaluation metrics to compare the quantitative performance of the base network and the cross-modal, cross-view dynamic target detection and tracking method proposed in this invention on the test set. On the other hand, by comparing the detection and tracking visualization results of the two methods, the effectiveness of the proposed method in target recognition, matching, and tracking in complex scenes is further verified.

[0083] Step S5: Input the data to be detected and tracked into the trained detection and tracking model to obtain the cross-modal and cross-view detection and tracking results of dynamic targets.

[0084] In some embodiments of the present invention, step S3, constructing a multi-scale dilated convolution module for viewpoint and scale alignment, includes:

[0085] Construct a network structure containing multiple parallel dilated convolution branches, with each branch using dilated convolution with different dilation rates for feature extraction;

[0086] The feature maps output by each branch are spliced ​​and fused along the channel dimension to form a multi-scale joint representation.

[0087] The number of channels in the fused feature map is adjusted to generate an enhanced target slice feature embedding.

[0088] In some embodiments of the present invention, step S3, constructing a feature alignment module for modality alignment, includes:

[0089] The feature alignment module receives high-level feature maps from visible light and infrared modes;

[0090] By calculating the difference between the visible light modal high-level feature map and the infrared modal high-level feature map in the feature space, a feature alignment loss is constructed;

[0091] During model training, the feature alignment loss is incorporated into the total loss function to constrain the consistency of visible light and infrared modes in high-level semantic features.

[0092] According to one aspect of the present invention, an electronic device is provided, comprising: one or more processors, one or more memories, and one or more computer programs; wherein the processor is connected to the memory, and the one or more computer programs are stored in the memory; when the electronic device is running, the processor executes the one or more computer programs stored in the memory to cause the electronic device to perform a cross-modal and cross-view dynamic target detection and tracking method for complex scene sequence images as described in any of the above technical solutions.

[0093] The processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor can be a microprocessor or any conventional processor.

[0094] The memory can be an internal storage unit of the terminal device, such as a hard drive or RAM. Alternatively, it can be an external storage device, such as a plug-in hard drive, Smart Media Card (SMC), Secure Digital (SD) card, or Flash Card. Furthermore, the memory can include both internal and external storage units. The memory is used to store the computer program and other programs and data required by the terminal device. It can also be used to temporarily store data that has been output or will be output.

[0095] According to one aspect of the present invention, a computer-readable storage medium is provided for storing computer instructions, which, when executed by a processor, implement a cross-modal, cross-view dynamic target detection and tracking method for complex scene sequence images as described in any of the above technical solutions.

[0096] For example, computer-readable storage media can be read-only memory (ROM), random access memory (RAM), read-only optical disc (CD-ROM), magnetic tape, floppy disk, and optical data storage devices. They can be implemented using computer-executable program code, thus allowing them to be stored in a storage device for execution by a computing device, or they can be fabricated as separate integrated circuit modules, or multiple modules or steps can be fabricated as a single integrated circuit module. Therefore, this invention is not limited to any particular hardware and software combination.

[0097] In summary, this invention proposes a cross-modal and cross-view dynamic target detection and tracking method for complex scene sequence images, and puts forward an innovative framework of "detect first, then align and associate": unlike traditional early fusion detection methods, this invention first performs target detection independently in each modality, avoiding contamination of the overall detector by low-quality modal data. Subsequently, for the independent detection results, a dedicated module is designed to solve the core challenges in the association process, making the technical approach clearer and more robust.

[0098] This invention, by introducing a multi-scale dilated convolution module and utilizing parallel branches with different dilation rates, effectively expands the receptive field of features and captures contextual information of the target at different scales without downsampling or loss of resolution. This makes the embedded features generated by the module more adaptable to changes in the target's scale and geometric deformation caused by different viewpoints, achieving alignment between viewpoint and scale, and effectively solving the problem of scale and deformation alignment across viewpoints.

[0099] This invention effectively solves the problem of semantic feature alignment across modalities. By introducing a feature alignment module and explicitly using a mean squared error-based feature alignment loss during the training phase, it constrains high-level semantic features from visible and infrared modalities to be as close as possible in the embedding space. This explicit alignment supervision signal guides the network to learn a consistent deep semantic representation that is strongly correlated with the target identity across modalities, thereby reducing the interference of modal differences on feature matching and achieving modality alignment.

[0100] This invention improves the overall performance of target tracking in complex scenarios. Through the synergistic effect of viewpoint / scale alignment and modality alignment, the method of this invention can provide the data association module with target features that are more discriminative and more consistent. This enables the system to more accurately associate the same target appearing in different modalities and different frames in complex scenarios such as target occlusion, drastic changes in lighting, severe weather, and multi-view observation, forming a continuous and stable tracking trajectory, significantly improving the tracking accuracy (such as Rank-1 accuracy) and robustness (such as average precision mAP).

[0101] Furthermore, it should be noted that the embodiments of the present invention may take the form of a computer program product implemented on one or more computer-usable storage media containing computer-usable program code.

[0102] Embodiments of the present invention are described with reference to flowchart illustrations and / or block diagrams of methods, terminal devices (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, embedded processor, or other programmable data processing terminal device to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing terminal device, generate instructions for implementing the flowchart illustrations. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0103] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing terminal device to operate in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The functions specified in one or more boxes. These computer program instructions may also be loaded onto a computer or other programmable data processing terminal equipment to cause a series of operational steps to be performed on the computer or other programmable terminal equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable terminal equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0104] It should also be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or terminal device that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or terminal device. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or terminal device that includes said element.

[0105] Finally, it should be noted that the above description represents a preferred embodiment of the present invention. It should be pointed out that although preferred embodiments have been described, those skilled in the art, once they understand the basic inventive concept of the present invention, can make various improvements and modifications without departing from the principles described herein. These improvements and modifications should also be considered within the scope of protection of the present invention. Therefore, the appended claims are intended to be interpreted as including both the preferred embodiments and all changes and modifications falling within the scope of the embodiments of the present invention.

Claims

1. A method for cross-modal and cross-view dynamic target detection and tracking of complex scene sequence images, characterized in that, Includes the following steps: Step S1: Construct a cross-modal, cross-view dynamic target detection and tracking dataset; Step S2: Construct a cross-modal, cross-view dynamic target detection and tracking basic network; the basic network includes a visible light detection branch and an infrared detection branch, used to output detection bounding boxes frame by frame, and to obtain target slices based on the detection bounding boxes; the basic network also includes a Kalman filter and a data association module, which sequentially complete trajectory prediction, trajectory matching and trajectory update; wherein, the data association module combines the feature matching results of the Hungarian algorithm and the ReID module to realize the association between the target and the trajectory, and finally outputs the cross-modal, cross-view dynamic target detection and tracking results; Step S3: Construct a multi-scale dilated convolution module for viewpoint and scale alignment and a feature alignment module for modality alignment, and embed the multi-scale dilated convolution module and the feature alignment module into the ReID module of the basic network to form a cross-modality and cross-viewpoint dynamic target detection and tracking model; Step S4: Train the cross-modal and cross-view dynamic target detection and tracking model based on the training set constructed in step S1 to obtain the trained detection and tracking model; Step S5: Input the data to be detected and tracked into the trained detection and tracking model to obtain the cross-modal and cross-view detection and tracking results of dynamic targets.

2. The method according to claim 1, characterized in that, Step S1 includes: Step S11: Based on the category system and sample annotation specifications of dynamic targets, collect a cross-modal and cross-view dynamic target detection dataset with horizontal bounding boxes annotating the target locations and categories in the images; Step S12: Slice the images in the collected cross-modal and cross-view dynamic target detection dataset to form a cross-modal and cross-view dynamic target detection and tracking dataset; Step S13: Divide the cross-modal and cross-view dynamic target detection and tracking dataset into a training set, a validation set, and a test set according to a preset ratio.

3. The method according to claim 1, characterized in that, In step S2, constructing the basic network for cross-modal and cross-view dynamic target detection and tracking includes: A high-performance target detection model is used as the basic detector. The input visible light image sequence and infrared image sequence are processed respectively to extract the detection results containing target bounding boxes, category information and confidence scores in their respective modalities. Based on the results of basic detection, a target state modeling and prediction mechanism based on Kalman filter is introduced to predict the target's position and velocity in the next frame by using the target's historical position information and motion trajectory. A data association module is introduced, and the prediction results are input into the data association module to match the detection results of the current frame with the established tracking targets; the data association module integrates the optimal matching strategy based on motion prediction and the feature similarity measure based on the ReID module; The ReID module employs a non-local attention mechanism to capture global information and long-distance dependencies, and combines generalized average pooling to extract fine-grained discriminative features to improve cross-modal and cross-view feature differentiation capabilities. The data association module outputs the cross-modal and cross-view dynamic target detection and tracking results after multi-source information fusion.

4. The method according to claim 3, characterized in that, The ReID module uses a dual-stream network as its basic structure and a dual-stream ResNet-50 as its backbone network to extract feature maps of visible light and infrared target slices respectively; and during the training phase, a weighted regularized triplet loss is introduced for optimization.

5. The method according to claim 1, characterized in that, In step S3, constructing a multi-scale dilated convolutional module for viewpoint and scale alignment includes: Construct a module containing three parallel dilated convolution branches with dilation rates of 1, 2, and 3, respectively; Each of the dilated convolutional branches includes a channel reduction layer, a dilated convolutional layer, and an activation function layer connected in sequence; the channel reduction layer is used to reduce the number of channels in the input feature map to a certain proportion of the original number of channels; The feature maps output by each branch are spliced ​​and fused along the channel dimension to form a multi-scale joint representation. Finally, a 1×1 convolutional layer is used to restore the original input feature size.

6. The method according to claim 1, characterized in that, In step S3, constructing the feature alignment module for modality alignment includes: The feature alignment module receives high-level feature maps from visible light and infrared modes; By calculating the difference between the visible light modal high-level feature map and the infrared modal high-level feature map in the feature space, a feature alignment loss is constructed; During model training, the feature alignment loss is incorporated into the total loss function to constrain the consistency of visible light and infrared modes in high-level semantic features.

7. The method according to claim 6, characterized in that, The feature alignment loss is based on mean squared error, and its expression is as follows: in, These are high-level feature maps for the visible light mode and the infrared mode, respectively. Their feature dimensions are both B×C×H×W, where B is the batch size, C is the number of channels, and H and W are the height and width of the feature map, respectively.

8. The method according to claim 1, characterized in that, Step S4 includes: The cross-modal, cross-view dynamic target detection and tracking model is trained using the training set from step S1; During training, cross-entropy loss is used as classification supervision, and weighted regularized triplet loss and feature alignment loss are introduced to jointly optimize model parameters. The network performance is evaluated using the validation set in step S1. The stochastic gradient descent optimization algorithm is adopted, the learning rate decay strategy is set, and the network parameters are iteratively updated until the model converges, thus obtaining the trained cross-modal and cross-view dynamic target detection and tracking model. Input the test set from step S1 into the trained detection and tracking model to obtain the detection and tracking results on the test set, thus completing the model validation.

Images