Multi-modal collaborative online evolutionary target detection method based on feature cross enhancement

Abstract

The invention provides a multi-mode collaborative online evolution target detection method based on feature cross enhancement, and belongs to the technical field of target detection. According to the method, a multi-modal data set of a ship target is obtained from different sensors, normalization preprocessing is carried out, a fusion strategy of each modal feature map is dynamically adjusted by using an adaptive evolution mechanism, data of different modals are fused in real time and feature information between the modals is enhanced through an online fusion and feature cross enhancement module, and the multi-modal data set of the ship target is obtained. And finally, carrying out real-time detection on the ship target by using a regional convolutional neural network (RCNN) and outputting a detection result. According to the multi-modal collaborative online evolution target detection method based on feature cross enhancement, the problems of data missing, noise interference and real-time performance in multi-modal data fusion are effectively solved, and the method is suitable for a complex dynamic environment and has high application value and practicability.

Description

Technical Field

[0001] The present invention relates to the technical field of target detection, and in particular to a multimodal collaborative online evolution target detection method based on feature cross-enhancement. Background Art

[0002] Multimodality in the target detection process refers to combining data sources from multiple different perception channels, such as optical, radar, infrared, sonar and other information, to enhance detection accuracy and reliability. Compared with single-modality detection methods, multimodal detection methods have significant characteristics and advantages. Single modality is often affected by factors such as environment, weather, and lighting, resulting in reduced accuracy and reliability of target recognition. Multimodal methods, by comprehensively utilizing data from different modalities, can still rely on other modalities for effective detection when a certain modality fails. For example, when optical imaging is affected by factors such as haze and strong light reflection, radar or infrared images can make up for this deficiency, thereby avoiding the problem of misidentification and missed detection caused by a single modality. In addition, by fusing information features of different modalities and using appropriate algorithm models for data fusion and processing. Multimodal detection can make full use of the characteristics of different perception channels to achieve more comprehensive and accurate target detection. This method can give full play to the advantages of various sensors and make up for the shortcomings of a single modality, thereby improving the overall ship target detection capability and having higher robustness and adaptability.

[0003] However, the multimodal collaborative ship target detection method also faces a series of challenges. First, the heterogeneity of data sources makes the fusion of different modal data complicated, especially in terms of data format, resolution, and spatiotemporal synchronization. In addition, in the multimodal collaborative process, the fusion of data from different modalities may be affected by noise interference, thereby affecting the detection accuracy. More importantly, the acquisition and processing of multimodal data usually requires efficient algorithm support, and these algorithms need to have strong real-time processing capabilities to adapt to the dynamic battlefield environment. The high real-time and low latency requirements of the algorithm are also a major challenge faced by multimodal collaborative ship target detection.

[0004] At present, researchers have proposed a variety of ship target detection methods based on multimodal data.

[0005] For example, in Example 1 (the invention patent of the Institute of Space Information Innovation of the Chinese Academy of Sciences in 2024, "Fine-grained ship target identification method and device based on heterogeneous data sharing semantics"), the multimodal data of ship targets, including at least two different data sources such as optical data, SAR data and radar radiation data, is combined to enhance the recognition ability of fine-grained targets. Multimodal data can complement each other and effectively improve the recognition accuracy. The shortcoming of this method is that it does not consider the real-time problem of the algorithm. For high-value targets such as ships, real-time detection is very important.

[0006] Example 2, (the invention patent of the 14th Institute of China Electronics Technology Group Corporation in 2023, "A method for intelligent comprehensive identification of maritime ship targets based on cross-domain multi-features"), improves the recognition confidence of ship targets by fusing three types of data: radar broadband one-dimensional image data, radiation source parameters, and image intelligence data. However, the limitation of this method is that it fails to fully consider the impact on the recognition results when a certain data source is missing or damaged in some cases. Therefore, in practical applications, if a certain data source is lost or damaged, it may lead to a significant decrease in recognition accuracy, thereby affecting the reliability of ship target detection.

[0007] Although the technical means proposed in these studies take into account the advantages of multimodal data fusion, there are still some problems that need to be solved, such as how to deal with missing data and how to improve the real-time performance of the fusion algorithm. Summary of the invention

[0008] The purpose of the present invention is to provide a multimodal collaborative online evolving target detection method based on feature cross-enhancement, which effectively solves the problems of data missing, noise interference and real-time performance in multimodal data fusion, is suitable for complex dynamic environments, and has high application value and practicality.

[0009] To achieve the above object, the present invention provides a multimodal collaborative online evolving target detection method based on feature cross-enhancement, comprising the following steps: Step S1, acquiring a multimodal data set of a ship target from different sensors and preprocessing it to obtain a preprocessed data set; Step S2: dynamically adjust the fusion strategy of each modal feature map according to the validity of each modal data through an adaptive evolution mechanism; Step S3: When the multimodal data source is continuously updated, the data of different modalities are fused in real time through the online fusion and feature cross-enhancement module, and the feature information between the modalities is enhanced to obtain an enhanced feature map; Step S4: Based on the enhanced feature map, the regional convolutional neural network RCNN is used to perform real-time detection of ship targets and output the detection results.

[0010] Preferably, in step S1, multimodal data of the ship target is obtained from different sensors, specifically: Optical image data captured by drones ; Acquiring radar data using airborne radar systems ; Using radiation source detectors to obtain radiation source data .

[0011] Preferably, in step S1, the preprocessing is a normalization operation, and the specific formula is as follows: ; in, represents the preprocessed dataset, represents a multimodal dataset, Representing multimodal datasets The mean of Representing multimodal datasets The standard deviation of .

[0012] Preferably, in step S2, the adaptive evolution mechanism is as follows: ; in, represents the feature fusion weight after adaptive adjustment, Represents the preprocessed dataset Is it valid? If it is valid, it is 1; otherwise, it is 0.

[0013] Preferably, in step S3, the online fusion and feature cross-enhancement module includes an input layer, a feature extraction submodule, a feature cross-enhancement submodule and an output layer.

[0014] Preferably, the input layer is used to input optical image data, radar image data and radiation source data; the dimension of the optical image data is ,in, , , Represent the height, width and number of channels of the optical image respectively; the dimension of the radar image data is ,in, , , Respectively represent the height, width and number of channels of radar image data; the dimension of radiation source data is ,in, , , Respectively represent the height, width and number of channels of the radiation source data.

[0015] Preferably, the feature extraction submodule includes an optical image feature extraction network, a radar image feature extraction network and a radiation source data feature extraction network; The optical image feature extraction network includes a convolutional layer 1, a maximum pooling layer 1, and four residual units connected in sequence, and each residual unit contains two convolutional layers 2, two maximum pooling layers 2, and a skip connection; The radar image feature extraction network includes convolution layer 3, convolution layer 4, maximum pooling layer 3, convolution layer 5, convolution layer 6 and maximum pooling layer 4 connected in sequence; The radiation source data feature extraction network adopts a multi-layer perceptron structure, which includes three fully connected layers. The number of neurons in the fully connected layer 1 is 128, the number of neurons in the fully connected layer 2 is 256, and the number of neurons in the fully connected layer 3 is 256. After the optical image feature extraction network, radar image feature extraction network and radiation source data feature extraction network, the optical image data, radar data and radiation source data obtain feature maps respectively. , , .

[0016] Preferably, the feature cross enhancement submodule receives feature maps of different modalities from the feature extraction submodule , Representing the feature maps of optical image data, radar data, and radiation source data respectively, the feature maps are first pooled by global average pooling. Compress to Vector , calculated as: ; in, , are the height and width of the feature map, respectively. ; Then through the fully connected layer , the weight matrix is , the bias is , and get the output : ; After ReLU activation function After that, the output is : ; After the fully connected layer , the weight matrix is , the bias is , and through the Sigmoid activation function Get the attention weight vector : ; in, Size ; Next, the attention weight vector With feature map Multiply to get the weighted feature map : ; The three weighted feature maps , , Splice and get , after a 1×1 convolutional layer To perform fusion, the convolution kernel is , the bias is , get the fused feature map : .

[0017] Preferably, the output layer is a fused feature map Perform further convolution and activation operations, using a 3×3 convolution kernel , the bias is , the step size is 1, the padding is 1, and the activation function is the ReLU activation function , and obtain the final enhanced feature map : .

[0018] Therefore, the present invention adopts the above-mentioned multimodal collaborative online evolving target detection method based on feature cross-enhancement, and the beneficial technical effects are as follows: (1) Adaptive evolution: Different modal data may be affected by environmental changes, sensor failure or interference, resulting in partial data loss or quality degradation. The present invention dynamically adjusts the fusion strategy of different modal data to ensure the stability and detection accuracy of the system even when data is missing. Specifically, the system automatically adjusts the fusion weight of each modality according to the effectiveness of each modality to ensure that in the case of incomplete data, other valid data can compensate for the missing part, thereby achieving efficient target detection. This adaptive evolution mechanism enables the system to perform well in a changing and uncertain environment.

[0019] (2) Online: The system can receive and fuse the latest data from different sensors in real time while the data is constantly updated. Through online learning and data processing mechanisms, the model can be updated at any time to quickly adapt to changes in new data. Compared with traditional offline data processing methods, the present invention can more effectively cope with dynamically changing environments and ensure the continuity and reliability of target detection. In particular, in practical applications, sensor data often changes continuously, and real-time online processing significantly improves the system's adaptability. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 This is a flow chart of a multimodal collaborative online evolving target detection method based on feature cross-enhancement of the present invention. DETAILED DESCRIPTION

[0021] The technical solution of the present invention is further described below through the accompanying drawings and embodiments.

[0022] Unless otherwise defined, technical or scientific terms used in the present invention shall have the common meanings understood by one having ordinary skills in the field to which the present invention belongs.

[0023] Embodiment 1 like Figure 1 As shown, it is a flow chart of a multi-modal collaborative online evolving target detection method based on feature cross-enhancement of the present invention, which specifically includes the following steps: 1. Data input and preprocessing.

[0024] First, obtain multimodal data of ship targets from different sensors. Assume that the input data set is ,in, is the optical image data, is the radar data, is the radiation source data. The collection process of each data is as follows: Optical image data: Optical image data is obtained by drone photography and is a two-dimensional image matrix with a size of ,in, , , Represent the height, width and number of channels of the optical image respectively.

[0025] Radar data: Radar data is obtained through the airborne radar system, usually the data after the echo signal is processed, in the form of a two-dimensional radar image, with a size of ,in, , , Respectively represent the height, width and number of channels of radar image data.

[0026] Radiation source data: Radiation source data is obtained through radiation source detectors, containing radiation information of different bands, in the form of a two-dimensional matrix with a size of ,in, , , Respectively represent the height, width and number of channels of the radiation source data. After the input data is acquired, in order to ensure that the data can be efficiently processed in the neural network, each type of data is normalized. Assume that the optical image data The pixel value range is [0,255], radar data The intensity value range is [0,1], the radiation source data The amplitude range is [0,100]. They are uniformly normalized to obtain the standardized data set: ; in, represents the preprocessed dataset, represents a multimodal dataset, Representing multimodal datasets The mean of Representing multimodal datasets The standard deviation of .

[0027] 2. Adaptive evolution.

[0028] An important innovation of the present invention is the adaptive evolution mechanism. In practical applications, different modal data may be affected by various factors, resulting in partial data loss. For example, radar data may be lost in severe weather conditions, and optical images may be affected by occlusion or lighting problems. In this case, traditional target detection methods may not be able to cope with it, but the present invention overcomes this problem through an adaptive evolution mechanism, and adjusts the fusion strategy of each modal feature map according to the validity of each modal data. For example, when a certain modal data is missing, the system can automatically increase the feature contribution of other modalities through a weighted mechanism, thereby compensating for the impact of the missing modality.

[0029] The specific implementation process is that in the data fusion process, an adaptive evolution adjustment strategy based on weighted average is used to dynamically adjust the weight according to the missing data of each modality. The feature fusion weight after adaptive adjustment is: ; in, represents the feature fusion weight after adaptive adjustment, Represents the preprocessed dataset Is it valid? If it is valid, it is 1, otherwise it is 0. Through this adaptive mechanism, the system can automatically adjust the contribution of each modality in the target detection network according to the availability of each modality data, thereby ensuring that the target detection task can be completed even when some modality data is missing.

[0030] 3. Online fusion and feature cross enhancement.

[0031] Online fusion refers to the real-time fusion of data from different modalities while multimodal data sources are continuously updated, ensuring that the system can receive and process the latest data from different sensors at any time. Specifically, the present invention flexibly processes the fusion process of optical images, radar data, and radiation source data through online learning and real-time data processing mechanisms to ensure that the system can cope with the ever-changing environment and data sources.

[0032] Feature cross-enhancement is to improve the accuracy and robustness of target detection by cross-processing and enhancing the features of different modalities in the process of multimodal data fusion, mining the complementary information between them. Specifically, optical images, radar data, and radiation source data are used as input modalities, and the corresponding feature maps are obtained through their respective feature extraction networks. Then, the feature cross-enhancement submodule fuses these feature maps, and enhances the mutual information between the modalities through weighting, convolution and other operations, thereby extracting more discriminative features.

[0033] The online fusion and feature cross-enhancement module is the core part of the present invention. In order to effectively fuse multimodal data, we designed a cross-feature enhancement module (CFEM). This module performs deep feature extraction on the data of each modality based on the regional convolutional neural network (RCNN), and organically fuses the features of different modalities through a cross-enhancement mechanism. Unlike simple splicing or addition of feature maps, the feature cross-enhancement module uses multi-layer convolution and attention mechanisms to specifically enhance the relationship between modalities and highlight the most important features for target detection. During the fusion process, the system can adaptively select the best feature combination to achieve more accurate and robust target detection. Through this feature cross-enhancement method, the system can effectively cope with complex environments such as data missing and noise interference, and improve the overall performance of multimodal target detection. This module is mainly composed of the following parts: 3.1 Input layer: The input layer is used to input optical image data, radar image data, and radiation source data; the dimension of the optical image data is ,in, , , Represent the height, width and number of channels of the optical image respectively; the dimension of the radar image data is ,in, , , Respectively represent the height, width and number of channels of radar image data; the dimension of radiation source data is ,in, , , Respectively represent the height, width and number of channels of the radiation source data.

[0034] 3.2 Feature extraction submodule: 3.2.1 Optical image feature extraction network: The network uses a residual neural network. The first layer is the convolution layer 1, the second layer is the maximum pooling layer 1, and the third to sixth layers, the seventh to tenth layers, the eleventh to fourteenth layers, and the fifteenth to eighteenth layers are four residual units.

[0035] Each residual unit contains two convolutional layers, two maximum pooling layers and a skip connection. Assume that the input feature map is , the convolution kernel of the first convolution layer Conv1 is , the bias is , then the output for: . After the ReLU activation function After that, we get The convolution kernel of the second convolution layer Conv2 is , the bias is , output . Finally, with the input Perform residual connection to obtain the output of the residual unit Multiple such residual units are connected in series to complete the feature extraction of the optical image.

[0036] 3.2.2 Radar image feature extraction network: The network consists of six layers, including four convolutional layers and two max pooling layers.

[0037] Convolutional layer 3: Preliminary extraction of distance, angle and other features in the radar image. Assume that the input is , output .

[0038] Convolutional layer 4: Enhanced feature extraction, output .

[0039] Max pooling layer 3: pooling kernel 2×2, stride 2, output .

[0040] Convolutional layer 5: mining higher-level features and outputting .

[0041] Convolutional layer 6: Continue to deepen feature extraction and output .

[0042] Max pooling layer 4: pooling kernel 2×2, step size 2, to obtain the final radar image feature map .

[0043] 3.2.3 Radiation source data feature extraction: A multi-layer perceptron (MLP) structure is adopted, which includes three fully connected layers. The number of neurons in the fully connected layer one is 128, the number of neurons in the fully connected layer two is 256, and the number of neurons in the fully connected layer three is 256.

[0044] Assume that the input radiation source data is , the weight matrix of the fully connected layer 1 is , the bias is , then the output for: , after the ReLU activation function After getting The following two fully connected layers are analogous to this, and finally the feature extraction of the radiation source data is completed.

[0045] After the optical image feature extraction network, radar image feature extraction network and radiation source data feature extraction network, the optical image data, radar data and radiation source data obtain feature maps respectively. , , , whose dimensions are ,in, is the number of channels, , are the height and width of the feature map, respectively.

[0046] 3.2.4 Feature Cross Enhancer Module: In order to perform effective feature fusion, a feature cross-enhancement module is introduced to fuse feature maps of different modalities through convolution operations and weighting mechanisms.

[0047] 1. Attention weight calculation: The feature cross enhancement submodule receives feature maps of different modalities from the feature extraction submodule. , Representing the feature maps of optical image data, radar data, and radiation source data respectively, the feature maps are first pooled by global average pooling. Compress to Vector , calculated as: ; in, , are the height and width of the feature map, respectively. .

[0048] Then through the fully connected layer , the weight matrix is , the bias is , and get the output : ; After ReLU activation function After that, the output is : ; After the second fully connected layer , the weight matrix is , the bias is , and through the Sigmoid activation function Get the attention weight vector : ; in, Size ; Next, the attention weight vector With feature map Multiply to get the weighted feature map : ; 2. Feature cross fusion: The three weighted feature maps , , Splice and get , after a 1×1 convolutional layer To perform fusion, the convolution kernel is , the bias is , get the fused feature map : ; 4. Output layer: The fused feature map Perform further convolution and activation operations, using a 3×3 convolution kernel , the bias is , the step size is 1, the padding is 1, and the activation function is the ReLU activation function. Get the final enhanced feature map : ; It contains complementary information of multimodal data for subsequent target detection tasks.

[0049] 4. Real-time target detection and feedback.

[0050] Through the enhanced feature map , the system enters the ship target detection stage. Target detection uses regional convolutional neural network (RCNN) for target classification and positioning. Assume that after being processed by the target detection module, a detection result set is obtained ,in Indicates A detected target, including the target's location, category and confidence.

[0051] It is worth noting that the contents not elaborated in detail in the present invention are all prior art and are well known to those skilled in the art.

[0052] Therefore, the present invention adopts the above-mentioned multimodal collaborative online evolving target detection method based on feature cross-enhancement, which effectively solves the problems of data missing, noise interference and real-time performance in multimodal data fusion, is suitable for complex dynamic environments, and has high application value and practicality.

[0053] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that they can still modify or replace the technical solution of the present invention with equivalents, and these modifications or equivalent replacements cannot cause the modified technical solution to deviate from the spirit and scope of the technical solution of the present invention.

Claims

1. A multi-modal collaborative online evolution target detection method based on feature cross-enhancement, characterized in that: The following steps are involved: Step S1, acquiring a multimodal data set of a ship target from different sensors and preprocessing it to obtain a preprocessed data set; Step S2: dynamically adjust the fusion strategy of each modal feature map according to the validity of each modal data through an adaptive evolution mechanism; Step S3: When the multimodal data source is continuously updated, the data of different modalities are fused in real time through the online fusion and feature cross-enhancement module, and the feature information between the modalities is enhanced to obtain an enhanced feature map; Step S4: Based on the enhanced feature map, the regional convolutional neural network RCNN is used to perform real-time detection of ship targets and output the detection results.

2. The multimodal collaborative online evolving target detection method based on feature cross-enhancement according to claim 1 is characterized in that: In step S1, multimodal data of the ship target is obtained from different sensors, specifically: Optical image data captured by drones ; Acquiring radar data using airborne radar systems ; Using radiation source detectors to obtain radiation source data .

3. The multimodal collaborative online evolving target detection method based on feature cross-enhancement according to claim 1 is characterized in that: In step S1, the preprocessing is a normalization operation, and the specific formula is as follows: ； in, represents the preprocessed dataset, represents a multimodal dataset, Representing multimodal datasets The mean of Representing multimodal datasets The standard deviation of .

4. The multimodal collaborative online evolving target detection method based on feature cross-enhancement according to claim 1 is characterized in that: In step S2, the adaptive evolution mechanism is as follows: ； in, represents the feature fusion weight after adaptive adjustment, Represents the preprocessed dataset Is it valid? If it is valid, it is 1; otherwise, it is 0.

5. The multimodal collaborative online evolving target detection method based on feature cross-enhancement according to claim 1 is characterized in that: In step S3, the online fusion and feature cross-enhancement module includes an input layer, a feature extraction submodule, a feature cross-enhancement submodule and an output layer.

6. The multimodal collaborative online evolving target detection method based on feature cross-enhancement according to claim 5 is characterized in that: The input layer is used to input optical image data, radar image data, and radiation source data; the dimension of the optical image data is ,in, , , Represent the height, width and number of channels of the optical image respectively; the dimension of the radar image data is ,in, , , Respectively represent the height, width and number of channels of radar image data; the dimension of radiation source data is ,in, , , Respectively represent the height, width and number of channels of the radiation source data.

7. The multimodal collaborative online evolving target detection method based on feature cross-enhancement according to claim 6 is characterized in that: The feature extraction submodule includes an optical image feature extraction network, a radar image feature extraction network, and a radiation source data feature extraction network; The optical image feature extraction network includes a convolutional layer 1, a maximum pooling layer 1, and four residual units connected in sequence, and each residual unit includes two convolutional layers 2, two maximum pooling layers 2, and a skip connection; The radar image feature extraction network includes convolution layer 3, convolution layer 4, maximum pooling layer 3, convolution layer 5, convolution layer 6 and maximum pooling layer 4 connected in sequence; The radiation source data feature extraction network adopts a multi-layer perceptron structure, which includes three fully connected layers. The number of neurons in the fully connected layer 1 is 128, the number of neurons in the fully connected layer 2 is 256, and the number of neurons in the fully connected layer 3 is 256. After the optical image feature extraction network, radar image feature extraction network and radiation source data feature extraction network, the optical image data, radar data and radiation source data obtain feature maps respectively. , , .

8. The multimodal collaborative online evolving target detection method based on feature cross-enhancement according to claim 7 is characterized in that: The feature cross-enhancement submodule receives feature maps of different modalities from the feature extraction submodule. , Representing the feature maps of optical image data, radar data, and radiation source data respectively, the feature maps are first pooled by global average pooling. Compressed to Vector , calculated as: ； in, , are the height and width of the feature map, respectively. ; Then through the fully connected layer , the weight matrix is , the bias is , and get the output : ； After ReLU activation function After that, the output is : ； Then through the fully connected layer , the weight matrix is , the bias is , and through the Sigmoid activation function Get the attention weight vector : ； in, Size ; Next, the attention weight vector With feature map Multiply to get the weighted feature map : ； The three weighted feature maps , , Splice and get , after a 1×1 convolutional layer To perform fusion, the convolution kernel is , the bias is , get the fused feature map : 。 9. The multimodal collaborative online evolving target detection method based on feature cross-enhancement according to claim 8, characterized in that: The output layer is the fused feature map Perform further convolution and activation operations, using a 3×3 convolution kernel , the bias is , the step size is 1, the padding is 1, and the activation function is the ReLU activation function , and obtain the final enhanced feature map : 。

Images