A target robust detection method based on multi-modal data fusion in a small sample scene

Abstract

The application belongs to the technical field of target detection, and discloses a target robust detection method based on multi-modal data fusion in a small sample scene. The method first performs conditional adaptive diversity enhancement on limited photoelectric, radar and detection modal data to expand the data. Then, the photoelectric data is processed by a multi-dimensional feature extraction backbone network and a lightweight decoupling detection head to output category and position prediction; the radar data is processed by a multi-layer perception network and a position prediction network to output position prediction; and the detection data is processed by a multi-layer perception network and a category prediction network to output category prediction. Finally, all the prediction results are input into an adaptive multi-modal alignment expert fusion system, and through cross-modal weighted alignment and expert decision fusion, the final high-precision target category and position information are output. The application effectively improves the precision and robustness of multi-modal fusion detection in a small sample.

Description

Technical Field

[0001] This application belongs to the field of target detection technology, specifically relating to a robust target detection method based on multimodal data fusion in small sample scenarios. Background Technology

[0002] In recent years, with the rapid development and widespread application of target platforms such as drones, achieving accurate and robust automatic detection and recognition has become a core task in the field of computer vision and information fusion. Especially considering their "low, slow, and small" characteristics, and their concealment in complex environments, traditional target detection methods relying on single visible light sensors face severe challenges. To improve the reliability of systems in complex scenarios such as adverse weather and optical obstruction, cross-modal target detection technology that integrates heterogeneous information from multiple sources, including optoelectronics, radar, and detection, has become an important research direction.

[0003] In the evolution of multimodal fusion object detection, deep learning-based fusion methods have demonstrated significant advantages. Early research focused on designing complex network architectures to fuse intermediate or decision-level features from different sensors through direct concatenation or weighted averaging. However, these methods typically assume sufficient and well-labeled multimodal training data. In practical engineering applications, especially when dealing with new or rare targets, the number of pairwise multimodal samples available is often extremely limited, resulting in small-sample learning scenarios. This scarcity of samples makes deep learning models prone to overfitting, hindering their ability to fully learn the effective features and intrinsic relationships of each modality, severely limiting their generalization ability and recognition accuracy in real-world open environments.

[0004] Existing technologies suffer from several inherent limitations when dealing with the problem of multimodal fusion with small sample sizes. First, at the data level, traditional data augmentation methods such as simple geometric transformations or noise additions are insufficient to effectively generate high-quality multimodal data with sufficient diversity based on limited samples, failing to fundamentally alleviate the problem of insufficient representation learning caused by small sample sizes. Second, at the feature fusion level, most methods fail to fully consider the essential differences in physical meaning and statistical properties between different modal information. For example, radar modalities typically provide precise distance and velocity information, optoelectronic modalities contain rich texture and shape visual features, while detection information includes specific category attributes. This heterogeneity leads to significant semantic gaps and spatial misalignments between modal features. Direct fusion introduces a large amount of noise and conflicting information rather than complementary information. Furthermore, existing fusion strategies typically employ fixed or simply learned weights, lacking an adaptive mechanism that can dynamically evaluate the predictive reliability of each modality based on specific sample content and intelligently resolve conflicts, resulting in a sharp performance degradation when some modalities are disturbed or fail.

[0005] Therefore, existing small-sample multimodal target detection methods suffer from semantic and spatial misalignment when processing heterogeneous modal information such as photoelectric, radar, and detection methods. Furthermore, they lack a fusion mechanism that can adaptively balance the reliability of each modality and intelligently resolve conflicts, resulting in unstable detection performance in complex real-world environments. Summary of the Invention

[0006] To address the shortcomings of existing technologies in multimodal target detection under conditions of scarce samples, namely insufficient accuracy and weak environmental adaptability, this application proposes a robust target detection method based on multimodal data fusion for small sample scenarios. The method expands data through conditional sample augmentation, designs a dedicated network to extract and optimize features of each modality, and introduces an adaptive cross-modal alignment and expert decision fusion mechanism to achieve high-precision and robust target recognition in complex scenarios.

[0007] To achieve the above technical objectives, this application specifically adopts the following technical solution:

[0008] In one aspect of this application, a robust target detection method based on multimodal data fusion is provided for small sample scenarios, comprising the following steps:

[0009] S1: Input a limited number of paired photoelectric mode data, radar mode data, and detection mode data;

[0010] S2: The conditionally adaptive diversity enhancement module is used to augment the paired photoelectric modal data, radar modal data, and detection modal data.

[0011] S3: Input the expanded photoelectric modal data into the multidimensional feature extraction backbone network for photoelectric feature extraction, and then input it into the lightweight decoupled detection head module to obtain the category prediction result based on photoelectric information. and location prediction results ;

[0012] S4: Input the expanded radar modal data into a multilayer perceptron network for radar feature extraction, and then input it into a position prediction network module to obtain a position prediction result based on radar information. ;

[0013] S5: Input the expanded detection modality data into a multilayer perceptron network for detection feature extraction, and then input it into a category prediction network module to obtain category prediction results based on the detection information. ;

[0014] S6: Calculate the category prediction results. Location prediction results Location prediction results and category prediction results Input the data into the adaptive multimodal alignment expert fusion system to obtain the final category prediction result. and location prediction results .

[0015] In one implementation, step S2 achieves data augmentation through a conditionally adaptive diversity enhancement module, which is a generative adversarial network including a generator and a discriminator.

[0016] The generator receives sample vectors. With random noise The initial input features are obtained through embedding and concatenation operations. ;

[0017] Initial input features After including Multi-scale residual module group processing with residual modules of different scales yields multi-level texture features. ;

[0018] Multi-layer texture features Then, after combining category conditions The adaptive instance normalization module performs style modulation to obtain style modulation features. ;

[0019] Finally, the decoding layer of the generator produces synthetic samples for data augmentation. .

[0020] In one implementation, the discriminator employs a dual-head structure. For an input sample, the output of the discriminator is composed of the global branch discrimination result and the local branch discrimination result; that is, for a real sample... Or generate samples Its output is or The global branch is used to evaluate the overall realism of the image, while the local branch is used to evaluate the local texture realism of the image. Output the overall probability of truth. Output the local block discrimination result. This represents the discrimination result of the discriminator.

[0021] In one implementation, in step S3, the multidimensional feature extraction backbone network processes the input optoelectronic modal data using a multi-scale receptive field convolution operator. Specifically, it includes:

[0022] Features are obtained by channel interaction through 1×1 convolution. ;

[0023] Features It is divided into four equal parts in terms of channel dimension. ;

[0024] right Depth-separable convolutions with kernel sizes of 3×3, 5×5, and 7×7 were performed respectively to obtain... ;

[0025] Will Features are obtained by splicing. ;

[0026] Through residual connection The output of the multi-scale receptive field convolution operator is obtained.

[0027] In one implementation, the multidimensional feature extraction backbone network further includes a feature reuse unit for aggregating multi-layer information during feature transfer; the feature reuse unit processes the features output by the multi-scale receptive field convolution operator. Or, the intermediate features obtained through further processing can be used as input features. and perform the following operations:

[0028] Input features Divided into two parts along the channel dimension and ;

[0029] Will Passing in sequence The multi-scale receptive field convolution operator is used for processing, and the output after each processing is compared with the original. splicing, through Enhanced features are obtained after the second iteration. ;

[0030] Enhance features and By concatenating the features, we obtain the aggregated features. ;

[0031] Aggregation features The number of channels is adjusted by the convolution-normalization-activation module to obtain the output features of the feature reuse unit. .

[0032] In one implementation, in step S3, the lightweight decoupling detection head module processes the output features of the feature multiplexing unit. Processing is performed to decouple the prediction task, specifically as follows:

[0033] The feature Enhanced features are obtained through a 3×3 convolutional layer. ;

[0034] The enhanced features Input a classification branch consisting of a 1×1 convolutional layer to obtain the category prediction result. ;

[0035] The enhanced features Inputting a localization branch consisting of a 3×3 convolutional layer yields the location prediction result. .

[0036] In one implementation, in step S4, the multilayer perceptron network includes Layer, for radar mode data , its first The layer transformation is as follows:

[0037]

[0038] in, For input data, For activation function, For the first Layer weight matrix, For the first Layer bias; output of the last layer of the network As extracted depth features .

[0039] In one implementation, in step S5, the multilayer perceptron network includes Layers, for detecting modal data , its first The layer transformation is as follows:

[0040]

[0041] in, For input data, For activation function, For the first Layer weight matrix, For the first Layer bias; output of the last layer of the network As extracted depth features .

[0042] In one implementation, in step S6, the adaptive multimodal alignment expert fusion system operates according to the following steps:

[0043] a) Adaptive cross-modal alignment: for photoelectric position prediction results and radar position prediction results Assign weights and ,satisfy And calculate the weighted position results. ;Prediction results for photoelectric categories and detection category prediction results Assign weights and ,satisfy And calculate the weighted category results. ;

[0044] b) Expert fusion decision-making: The weighted position results are then... and weighted category results The input is integrated and decided by an expert network, and the final category prediction result is output. and location prediction results .

[0045] The beneficial effects of this application are as follows:

[0046] This application effectively expands the multimodal training data under small sample conditions by introducing a conditionally adaptive diversity enhancement module, thereby enhancing the model's generalization ability. The designed multidimensional feature extraction backbone network and lightweight decoupled detection head improve the representation quality and detection efficiency of photoelectric modal features. Dedicated multilayer perceptron networks constructed for radar and detection modalities strengthen the extraction capability of key information in their respective modalities. Finally, through an adaptive multimodal alignment expert fusion system, intelligent alignment and conflict resolution of cross-modal information are achieved, significantly improving the accuracy and robustness of target detection in complex scenarios. Attached Figure Description

[0047] Figure 1 This is the multimodal target recognition network architecture of the embodiments of this application;

[0048] Figure 2 This is a schematic diagram of the condition-adaptive diversity enhancement module in an embodiment of this application;

[0049] Figure 3 This is a schematic diagram of the multidimensional feature extraction backbone network module in an embodiment of this application;

[0050] Figure 4 This is a schematic diagram of the adaptive multimodal alignment expert fusion system module in an embodiment of this application. Detailed Implementation

[0051] The technical solution of this application will be clearly and completely described below with reference to specific embodiments. However, those skilled in the art will understand that the embodiments described below are only some embodiments of this application, not all embodiments, and are only used to illustrate this application, and should not be regarded as limiting the scope of this application. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0052] To address the three core challenges of multimodal target detection under small sample conditions—data scarcity, modal heterogeneity, and rigid fusion decision-making mechanisms—a hierarchical, adaptive, and highly complementary collaborative processing framework is constructed. First, a data augmentation module based on conditional generative adversarial networks (GANs) injects controllable intra-class diversity into limited samples, mitigating model overfitting caused by data scarcity at its source. Second, dedicated feature extraction paths are designed for the different physical characteristics and information advantages of three types of heterogeneous data: photoelectric, radar, and detection data. For photoelectric images, a multi-scale receptive field and feature reuse backbone network combined with a decoupled detection head are used to enhance spatial detail perception and task specificity. For radar point clouds and detection parameters, a multilayer perceptron network is used to efficiently extract their inherent positional and category attributes. Finally, an adaptive multimodal alignment and expert fusion decision-making system is designed. This system does not perform simple feature splicing or voting; instead, it explicitly models and bridges the spatial and semantic differences between different modalities through a dynamic weight allocation mechanism. Based on this, expert knowledge is introduced to intelligently arbitrate and integrate potentially conflicting intermediate prediction results. The technical solution of this application enables the system to fully explore and reliably integrate the complementary value of multi-source heterogeneous information under the condition of limited samples, thereby achieving high-precision, high-robust and stable detection of targets in complex environments.

[0053] In one specific implementation, a robust target detection method based on multimodal data fusion is provided for small sample scenarios, including the following steps:

[0054] S1: Input a limited number of paired photoelectric mode data, radar mode data, and detection mode data.

[0055] In this application, the term "limited quantity" refers to the fact that the number of effective multimodal training samples belonging to the same target category that can be obtained during the training phase is relatively small, which cannot meet the needs of sufficient training of conventional data-driven models and belongs to the small sample learning scenario.

[0056] In this application, "paired" refers to a set of multimodal data that are strictly aligned in time, space and semantics, and are collected synchronously or quasi-synchronously by different sensors deployed in a coordinated manner for the same observed target instance.

[0057] In some embodiments, each pair of data includes data from three different sensors: a photoelectric sensor, a radar sensor, and a detection device.

[0058] The photoelectric modal data is acquired by photoelectric sensors (such as visible light cameras, infrared thermal imagers, etc.). The photoelectric modal data is usually in the form of images or video sequences, providing rich spatial and morphological feature information such as the visual appearance, texture, and shape of the target.

[0059] The radar modal data is acquired by radar sensors. This radar modal data typically includes physical characteristics such as the target's point cloud information, range, radial velocity, azimuth, elevation angle, and radar cross-section, effectively reflecting the target's motion state and some of its physical attributes.

[0060] The detection modal data is acquired by specific detection equipment (such as radio detection systems, acoustic sensors, etc.). The detection modal data typically contains electromagnetic signal features, acoustic features, or other identifiable non-visual physical features related to the target's identity, used to assist in the determination of the target category.

[0061] In some embodiments, the acquired raw multi-source sensor data first requires preprocessing and alignment. Preprocessing includes, but is not limited to: denoising and normalizing photoelectric images; filtering and coordinate system transformation of radar point clouds; and initial feature extraction and formatting of detection signals. Alignment ensures that for the same target event, the data from the three modalities are synchronized in timestamps and point to the same target area in the observation space, thus forming an effective "paired" data sample. Finally, the preprocessed and aligned sets of multimodal data samples are used as the data input for the entire detection method, so that subsequent steps can perform feature extraction, prediction, and fusion.

[0062] S2: The conditionally adaptive diversity enhancement module is used to augment the paired photoelectric modal data, radar modal data, and detection modal data.

[0063] To address the issues of overfitting and weak generalization ability caused by insufficient training data in small-sample scenarios, a dedicated data augmentation unit for a conditionally adaptive diversity enhancement module is constructed. This module is a generative adversarial network (GAN) used to generate high-quality, diverse synthetic samples under given category semantics, thereby effectively expanding the training dataset.

[0064] The conditional adaptive diversity enhancement module includes a generator and a discriminator.

[0065] In some embodiments, the generator is used to generate synthetic samples based on class conditions and random noise. The processing includes the following steps:

[0066] First, the sample vector representing the real sample... With a randomly sampled noise vector The features are fused to form the initial input features of the generator. Specifically, after processing the sample vector through embedding operations, it is concatenated with the noise vector, i.e.:

[0067]

[0068] in, Represents the sample vector Perform embedding operations. This indicates a vector concatenation operation.

[0069] Next, the initial input features A multi-scale residual module group is fed in. The multi-scale residual module group includes... Each residual block has a different receptive field. These residual blocks perform convolutional transformations on the input features in parallel to capture multi-level texture information, ranging from fine to macroscopic. The outputs of all residual blocks are summed and aggregated to obtain a multi-level texture feature rich in texture details. The formula for its calculation is:

[0070]

[0071] in, Indicates the first Each residual module, This represents the number of residual blocks.

[0072] Then, in order to inject specific category information into the generation process and to controllably modulate the "style" of the generated samples, the multi-layered texture features are... Input an adaptive instance normalization module. The adaptive instance normalization module utilizes given class conditions. The provided statistics (mean and standard deviation) are applicable to... Normalization and affine transformation are performed to obtain style modulation features. Specifically, it is expressed as:

[0073]

[0074] in, This indicates an adaptive instance normalization operation. and These represent the mean and standard deviation of the calculated feature, respectively.

[0075] Finally, style modulation features The decoding layer of the input generator. The decoding layer typically consists of a series of upsampling and convolutional layers, responsible for reconstructing the feature maps into synthetic samples with the same size as the real samples. ,Right now:

[0076]

[0077] in, It summarizes the entire decoding process from features to the final generated sample.

[0078] In some embodiments, the discriminator is used to distinguish whether the input sample is real or synthesized by the generator. To more rigorously evaluate the generation quality, the discriminator employs a two-branch structure with global and local branches.

[0079] The global branch is used to evaluate the authenticity and reasonableness of the input features as a whole. The local branch is used to divide the image into multiple local blocks and evaluate the authenticity of the texture details of each local block separately.

[0080] For any input sample (whether it is a real sample) Or generate samples The final output of the discriminator is the concatenation of the global branch discrimination result and the local branch discrimination result, that is:

[0081]

[0082] in, This indicates the discriminant's judgment result. Output the overall probability of truth. Output the local block discrimination result.

[0083] In some embodiments, the generator and discriminator are jointly optimized through an adversarial training process based on the discrimination results. The total loss function upon which training is based... It consists of three weighted factors:

[0084]

[0085] in, These are the weight coefficients for adversarial loss, perceptual loss, and diversity regularization loss, respectively.

[0086] To combat the loss, a generator is constructed based on the discriminator's output, driving the generator to produce realistic samples sufficient to "fool" the discriminator, while simultaneously training the discriminator to accurately distinguish between real and fake samples. Its form is as follows:

[0087]

[0088] in, This represents a real image sampled from the real data distribution. This represents random noise sampled from a noise distribution. E It represents the mathematical expectation.

[0089] Perceptual loss is used to constrain the generation of samples. Compared with real samples The distance in the feature space of the pre-trained deep network ensures consistency at the high-level semantic level. It is calculated as follows:

[0090]

[0091] in, Denotes the square of the L2 norm. This indicates that the pre-trained network is in the first... Feature mapping of layers.

[0092] The diversity regularization loss is used to encourage the generator to apply the same category conditions. Different random noise inputs and This produces differentiated outputs, thereby increasing the diversity of generated samples. It is defined as:

[0093] ;

[0094] in, Describing the L1 norm, Indicates the generator under category conditions The following is an example of random noise input. The generated samples, Indicates the generator in category condition The following is an example of random noise input. The generated sample.

[0095] By minimizing the total loss function By optimizing the generator and alternately optimizing the discriminator, the resulting conditionally adaptive diversity enhancement module can generate a large amount of accurate and diverse augmented data for each target class.

[0096] S3: Input the expanded photoelectric modal data into the multidimensional feature extraction backbone network for photoelectric feature extraction, and then input it into the lightweight decoupled detection head module to obtain the category prediction result based on photoelectric information. and location prediction results .

[0097] To better address the issues of poor generalization ability of photoelectric modal data features, weak small target detection capability, and coupling of classification and localization tasks, a multi-dimensional feature extraction backbone network and a lightweight decoupled detection head module are employed. The multi-dimensional feature extraction backbone network is used to extract robust multi-scale features, while the lightweight decoupled detection head module decouples the classification and localization tasks to improve detection efficiency and accuracy. Through this dual design of "multi-dimensional feature extraction backbone + task decoupling," efficient extraction and fusion of photoelectric modal data features and enhanced discriminative power are achieved, laying the foundation for real-time, rapid, and accurate detection.

[0098] In some embodiments, the multidimensional feature extraction backbone network includes multiple feature reuse units stacked sequentially. Each feature reuse unit performs feature extraction using a multi-scale receptive field convolution operator (MF-Conv).

[0099] Specifically, given the input photoelectric mode data ,in, H Indicates the height of the feature map. W Indicates the width of the feature map. C This represents the number of channels in the feature map. The MF-Conv operator performs the following transformation:

[0100] a) Through Convolutional layers exchange information between channels to obtain features. .

[0101] b) Features It is evenly divided into four parts along the channel dimension, namely: Each of the parts .

[0102] c) Apply depthwise separable convolution (DW-Conv) with different kernel sizes to the last three parts to capture multi-scale spatial information:

[0103] .

[0104] d) The three features after processing Features of Part One By splicing the data, the fused features are obtained. .

[0105] e) Finally, the fused features are added to the original input features via residual connections to obtain the output of the MF-Conv operator: .

[0106] In some embodiments, the structure of each feature multiplexing unit is as follows:

[0107] a) The input features of the feature multiplexing unit Divided into two parts along the channel dimension: .

[0108] b) Features obtained from the partitioning Perform iterative processing. Let Then pass it through continuously The MF-Conv operator is cascaded and compared with the original one after each iteration. The process of adding features is represented as follows:

[0109]

[0110] go through Enhanced features are obtained after the second iteration. .

[0111] c) Enhanced features With another part of the features Perform channel splicing: .

[0112] d) The CBS module is used to adjust the number of channels and perform non-linear activation on the concatenated features. The CBS module sequentially performs convolution, batch normalization, and activation function operations, and the output is:

[0113] .

[0114] In some embodiments, the multidimensional features are extracted from the features ultimately output by the backbone network. Input the lightweight decoupled detection head module (LD-Head) to decouple category prediction from location prediction.

[0115] First, use Convolutional layers for features Enhancement is performed to obtain enhanced features. :

[0116] .

[0117] Secondly, enhance features Two independent convolutional branches are input in parallel:

[0118] a) Category branches: using Convolutional layers output the predicted category of the target. .

[0119] b) Locating a branch: using Convolutional layer, outputting the predicted location of the target. .

[0120] S4: Input the expanded radar modal data into a multilayer perceptron network for radar feature extraction, and then input it into a position prediction network module to obtain a position prediction result based on radar information. .

[0121] In some embodiments, the multilayer sensor network is a network containing A fully connected feedforward neural network is used to process the input radar mode data. Perform layer-by-layer nonlinear transformation and feature abstraction.

[0122] Specifically, radar mode data As the input to the multilayer perceptron network, that is, let the features of layer 0... ;

[0123] For the network's first layer( The calculation process is expressed as follows:

[0124]

[0125] in, For the first The weight matrix of the layer, These are the feature dimensions of its input and output, respectively;

[0126] For the first Layer bias;

[0127] This is the preset activation function;

[0128] The output features of the previous layer are used as the input of this layer;

[0129] After all The layers are processed sequentially to obtain the deep feature representation of the radar modal data, which is the output of the last layer:

[0130]

[0131] in, For the output dimension of the last layer, N This represents the number of samples.

[0132] In some embodiments, the location prediction network consists of one or more fully connected layers. The deep features extracted by the multilayer perceptron network... Input the location prediction network module to regress the target's specific location.

[0133] Specifically, the location prediction network uses the depth features As input, it undergoes transformation and mapping through its internal fully connected layers, ultimately outputting the predicted coordinates of the target in space. The process can be summarized as follows:

[0134]

[0135] in, This indicates a fully connected operation.

[0136] S5: Input the expanded detection modality data into a multilayer perceptron network for detection feature extraction, and then input it into a category prediction network module to obtain category prediction results based on the detection information. .

[0137] In some embodiments, the multilayer sensor network is a network containing A fully connected feedforward neural network is used to process the input detection modality data. Perform layer-by-layer nonlinear transformation and feature abstraction.

[0138] Specifically, the detection modal data As the input to the multilayer perceptron network, that is, let the features of layer 0... ;

[0139] For the network's first layer( The calculation process is expressed as follows:

[0140]

[0141] in, For the first The weight matrix of the layer, These are the feature dimensions of its input and output, respectively;

[0142] For the first Layer bias;

[0143] This is the preset activation function;

[0144] The output features of the previous layer are used as the input of this layer;

[0145] After all The layers are processed sequentially to obtain the deep feature representation of the detected modality data, which is the output of the last layer:

[0146]

[0147] in, For the output dimension of the last layer, N This represents the number of samples.

[0148] In some embodiments, the category prediction network module consists of one or more fully connected layers. The deep features extracted by the multilayer perceptron network... Input the category prediction network module to determine the category to which the target belongs.

[0149] Specifically, the category prediction network module uses the deep features As input, it undergoes transformation and mapping through its internal fully connected layers, ultimately outputting the target's category prediction result. The process can be summarized as follows:

[0150]

[0151] in, This indicates a fully connected operation.

[0152] S6: Calculate the category prediction results. Location prediction results Location prediction results and category prediction results Input the data into the adaptive multimodal alignment expert fusion system to obtain the final category prediction result. and location prediction results .

[0153] The adaptive multimodal alignment expert fusion system aims to effectively integrate three heterogeneous modal data—photoelectric, radar, and detection information—to improve target detection performance in complex scenarios. The system receives prediction results from three modal processing branches: category and location predictions from the photoelectric modality, location predictions from the radar modality, and category predictions from the detection modality. Through an adaptive cross-modal alignment mechanism, it coordinates the potential differences and complementarities between the spatial and semantic levels of the prediction information. Furthermore, it utilizes expert knowledge to intelligently integrate and make decisions regarding redundant and conflicting information, ultimately outputting robust and accurate unified target category determination and location information.

[0154] The adaptive cross-modal alignment mechanism performs weighted fusion of similar prediction information from different sensors to balance the contribution of each modality. Specifically, it includes location information alignment and category information alignment.

[0155] In some embodiments, the location information alignment includes: the adaptive multimodal alignment expert fusion system providing the location prediction results of the photoelectric modality. Position prediction results of radar modes Each is assigned an adaptive weight coefficient. and Satisfying the constraints Its values ​​are all within the range Internal. Weighted fusion location prediction results. The following formula is used to calculate:

[0156] .

[0157] In some embodiments, the category information alignment includes: the adaptive multimodal alignment expert fusion system providing category prediction results for the photoelectric modality. And the category prediction results of the detection mode Each is assigned an adaptive weight coefficient. and Satisfying the constraints And its values ​​are all within the range Internal. Weighted fusion category prediction results. The following formula is used to calculate:

[0158] .

[0159] In some embodiments, the adaptive multimodal alignment expert fusion system is a trained decision module that can comprehensively evaluate the credibility and contextual relationship of each modality prediction based on the input multimodal fusion information.

[0160] The adaptive multimodal alignment expert fusion system compresses redundant information and arbitrates and selects conflicting information to intelligently generate the optimal final decision. This process is represented as follows:

[0161]

[0162] in, Let be the decision function of the adaptive multimodal alignment expert fusion system. This is the final target detection result output.

[0163] After adaptive alignment and expert decision-making, a unified and high-precision prediction result based on the fusion of three modal information is output, which is the final target category prediction. and final target location prediction .

[0164] Example

[0165] This embodiment provides a robust target detection method based on multimodal data fusion in small sample scenarios, referring to... Figure 1 As shown, this is achieved through the following steps:

[0166] Step 1: Multimodal data input and preprocessing.

[0167] Acquire a limited number of paired multimodal training data sets. Each set contains three modalities of information acquired simultaneously: photoelectric images (visible light or infrared), radar point cloud data, and detection signal feature data. Preprocess the raw data, including: normalizing the size of the photoelectric images to a fixed resolution and normalizing their brightness; filtering noise from the radar point clouds and converting them into feature vectors containing information such as range, azimuth, elevation, and radial velocity; and filtering and initial feature extraction of the detection signals, converting them into fixed-dimensional feature vectors. Ensure that the three sets of data are strictly aligned in time and space to form a set of sample pairs usable for training.

[0168] Step 2: Conditional Adaptive Diversity Enhancement (Data Augmentation), refer to... Figure 2 As shown.

[0169] 1. Convert the sample vector With random noise Embedding and fusion are performed to obtain the initial input of the generator:

[0170]

[0171] in, This represents a vector concatenation operation. Output: Features are used as input to the generator.

[0172] 2. Obtain the initial input features Input a multi-scale residual module group and perform convolutional transformations on the input features with different receptive fields to capture multi-level texture features:

[0173]

[0174] in, Indicates the first Residual modules at various scales Number of modules. Output As input features for subsequent style modulation.

[0175] 3. Input the Adaptive Instance Normalization (AdaIN) module and combine it with category conditions. Style control is achieved by adjusting the mean and variance:

[0176]

[0177] in, and These represent the feature mean and standard deviation, respectively. Output As input to the generator's final decoding stage.

[0178] 4. Modulate style features Input to the decoding layer to generate synthetic samples of the target category:

[0179]

[0180] in, This represents the generated image sample. This result serves as the input to the discriminator.

[0181] 5. Generate samples Input the real sample into the discriminator The discriminator consists of global branches and local branches:

[0182]

[0183] in, Output the overall probability of truth. Output the local block discrimination result. The discriminator output is used as input for calculating the loss function.

[0184] 6. Based on the discriminator's output, jointly optimize the generator and discriminator. The loss function includes adversarial loss, perceptual loss, and diversity regularization loss. The total loss function is:

[0185]

[0186] in, This represents a real image sampled from the real data distribution. This represents random noise sampled from a noise distribution. This indicates that the pre-trained network is in the first... Feature mapping of layers, These represent the weight parameters for the three types of loss, This is the total loss function for joint optimization.

[0187] Step 3: Photoelectric modal feature extraction and decoupling prediction, refer to... Figure 3 As shown.

[0188] 1. To improve the spatial feature extraction capability, the multi-scale receptive field convolution operator MF-Conv is employed. Specifically, let the input photoelectric modal data be:

[0189]

[0190] First, through Convolution achieves channel interaction, resulting in:

[0191]

[0192] Then, in the channel dimension Divided into four equal parts:

[0193]

[0194] Subsequently, depthwise separable convolution (DW-Conv) was performed on each of the three parts:

[0195]

[0196] Then compare the above results with splicing:

[0197]

[0198] Finally, the output of MF-Conv is obtained by combining the residual connections:

[0199]

[0200] 2. To avoid information redundancy and enhance the utilization of intermediate layer features, repeated stacking is performed in the backbone network. Each CP-Conv module constructs a feature reuse unit. Specifically, the input feature map is first processed... Divided into two parts along the direction of the passage:

[0201]

[0202] Then, go through After the MF-Conv step, the enhanced features are obtained by sequential aggregation. :

[0203]

[0204] Then and splicing:

[0205]

[0206] Finally, to keep the number of channels constant, the CBS module (convolution + normalization + activation) is used to adjust the channels:

[0207] 3. To balance detection accuracy and computational efficiency, a lightweight, decoupled detection head module, LD-Head, is used to decouple category prediction from location prediction. First, the features... Introduction Convolution yields enhanced features :

[0208]

[0209] Then, enhance the features enter Convolutional class branching and The convolutional localization branch yields the category and location prediction results:

[0210]

[0211] Finally, the prediction results based on photoelectric modes can be expressed as follows: .

[0212] Step 4: Radar modal feature extraction and location prediction.

[0213] 1. Transfer radar mode data Feature extraction is performed on the input from a multilayer perceptron (MLP) network. Assume the MLP network contains... Layers, each layer transforms as follows:

[0214]

[0215] in, For input features, For the first The weight matrix of the layer, For the first Layer bias, This is the activation function. Ultimately, it undergoes... Layer processing yields a depth feature representation of the radar data:

[0216]

[0217] in, This is the output dimension of the last layer.

[0218] 2. Obtain the radar signature The input is used in the location prediction network to predict the target location. The location prediction network consists of several fully connected layers (FC), and the output is the predicted target location.

[0219]

[0220] in, Indicates a fully connected operation. The final predicted location includes the target's coordinates in space.

[0221] Step 5: Detection modality feature extraction and category prediction.

[0222] 1. Detect modal data Feature extraction is performed on the input from a multilayer perceptron (MLP) network. Assume the MLP network contains... Layers, each layer transforms as follows:

[0223]

[0224] in, For input features, For the first The weight matrix of the layer, For the first Layer bias, This is the activation function. Ultimately, it undergoes... Layer processing yields a depth feature representation of the radar data:

[0225]

[0226] in, This is the output dimension of the last layer.

[0227] 2. Obtain the detected features The input is fed into a category prediction network to predict the target category. The category prediction network consists of fully connected layers (FC), and its output is the predicted target location.

[0228]

[0229] in, Indicates a fully connected operation. This is the final prediction category.

[0230] Step Six: Adaptive Multimodal Alignment Expert Fusion, refer to Figure 4 As shown.

[0231] 1. Adaptive cross-modal alignment mechanism

[0232] The core of the adaptive multimodal alignment expert fusion system is the cross-modal alignment mechanism, which can handle the differences and complementarities between different modalities at the spatial and semantic levels. First, the system aligns the prediction results of the three modalities through a cross-modal alignment network. Assume the input to this alignment network is:

[0233]

[0234] Then, the cross-modal alignment network will calculate the alignment weights for each modality using an adaptive weighting method:

[0235]

[0236] in, These are the prediction weights for photoelectric location, radar location, photoelectric category, and detection category, respectively.

[0237] The weighted prediction results for each modality are obtained:

[0238]

[0239] in, These represent the weighted position and category prediction results under photoelectric, radar, and detection modes, respectively.

[0240] 2. Expert Integration and Decision-Making Mechanism

[0241] Then, the fusion system intelligently integrates the aligned results through an expert network. The expert network further optimizes information fusion based on the characteristics of each modality and the importance of the prediction results.

[0242]

[0243] in, The final target detection result, after expert fusion, includes category prediction and location prediction. During decision-making, the network can automatically process redundant and conflicting information, selecting the optimal target determination and location information.

[0244] 3. Output high-precision prediction results

[0245] After the alignment and fusion steps described above, the final output is a high-precision category and location prediction result based on trimodal information:

[0246]

[0247] in, For the final target category prediction, Predicting the final target location.

[0248] Although the embodiments of this application have been described above in conjunction with the accompanying drawings, this application is not limited to the specific embodiments and application fields described above. The specific embodiments described above are merely illustrative and instructive, not restrictive. Those skilled in the art can make many other forms based on the guidance of this specification and without departing from the scope of protection of the claims of this application, and these are all within the scope of protection of this application.

Claims

1. A robust target detection method based on multimodal data fusion in small sample scenarios, characterized in that, Includes the following steps: S1: Input a limited number of paired photoelectric mode data, radar mode data, and detection mode data; The photoelectric modal data exists in the form of images or video sequences. The radar modal data includes physical characteristic information such as point cloud information, range, radial velocity, azimuth angle, elevation angle and radar cross section of the target. The detection modal data includes electromagnetic signal characteristics, acoustic characteristics or other identifiable non-visual physical characteristic information related to the target's identity. S2: The conditionally adaptive diversity enhancement module is used to augment the pairs of photoelectric modal data, radar modal data, and detection modal data; The conditionally adaptive diversity enhancement module is a generative adversarial network, comprising a generator and a discriminator; The generator receives sample vectors. With random noise The initial input features are obtained through embedding and concatenation operations. ,Right now: in, Represents the sample vector Perform the embedding operation. This represents a vector concatenation operation; Initial input features After including Multi-scale residual module group processing with residual modules of different scales yields multi-level texture features. The formula for its calculation is: in, Indicates the first Each residual module, This represents the number of residual blocks; Multi-layer texture features Then, after combining category conditions The adaptive instance normalization module performs style modulation to obtain style modulation features. ; Finally, the decoding layer of the generator produces synthetic samples for data augmentation. ; S3: Input the expanded photoelectric modal data into the multidimensional feature extraction backbone network for photoelectric feature extraction, and then input it into the lightweight decoupled detection head module to obtain the category prediction result based on photoelectric information. and location prediction results The multidimensional feature extraction backbone network uses a multi-scale receptive field convolution operator to process the input photoelectric modal data. Specifically, it includes: Features are obtained by channel interaction through 1×1 convolution. ; Features It is divided into four equal parts in terms of channel dimension. ; right Depth-separable convolutions with kernel sizes of 3×3, 5×5, and 7×7 were performed respectively to obtain... ; Will Features are obtained by splicing. ; Through residual connection The output of the multi-scale receptive field convolution operator is obtained; S4: Input the expanded radar modal data into a multilayer perceptron network for radar feature extraction, and then input it into a position prediction network module to obtain a position prediction result based on radar information. ; S5: Input the expanded detection modality data into a multilayer perceptron network for detection feature extraction, and then input it into a category prediction network module to obtain category prediction results based on the detection information. ; S6: Calculate the category prediction results. Location prediction results Location prediction results and category prediction results Input the data into the adaptive multimodal alignment expert fusion system to obtain the final category prediction result. and location prediction results .

2. The robust target detection method based on multimodal data fusion in small sample scenarios according to claim 1, characterized in that, The discriminator employs a dual-head structure. For an input sample, the output of the discriminator is composed of the global branch discrimination result and the local branch discrimination result. That is, for a real sample... Or generate samples Its output is or The global branch is used to evaluate the overall realism of the image, while the local branch is used to evaluate the local texture realism of the image. Output the overall probability of truth. Output the local block discrimination result. This represents the discrimination result of the discriminator.

3. The robust target detection method based on multimodal data fusion in small sample scenarios according to claim 1, characterized in that, The multidimensional feature extraction backbone network also includes a feature reuse unit, used to aggregate multi-layer information during feature transfer; the feature reuse unit converts the features output by the multi-scale receptive field convolution operator into a single feature. Or, the intermediate features obtained through further processing can be used as input features. and perform the following operations: Input features Divided into two parts along the channel dimension and ; Will Passing in sequence The multi-scale receptive field convolution operator is used for processing, and the output after each processing is compared with the original. splicing, through Enhanced features are obtained after the second iteration. ; Enhance features and By splicing, aggregated features are obtained. ; Aggregation features The number of channels is adjusted by the convolution-normalization-activation module to obtain the output features of the feature reuse unit. .

4. The robust target detection method based on multimodal data fusion in small sample scenarios according to claim 3, characterized in that, The lightweight decoupling detection head module outputs features from the feature multiplexing unit. Processing is performed to decouple the prediction task, specifically: The feature Enhanced features are obtained through a 3×3 convolutional layer. ; The enhanced features Input a classification branch consisting of a 1×1 convolutional layer to obtain the category prediction result. ; The enhanced features Inputting a localization branch consisting of a 3×3 convolutional layer yields the location prediction result. .

5. The robust target detection method based on multimodal data fusion in small sample scenarios according to claim 1, characterized in that, In step S4, the multilayer perceptron network includes Layer, for radar mode data , its first The layer transformation is as follows: in, For input data, For activation function, For the first Layer weight matrix, For the first Layer bias; output of the last layer of the network As extracted depth features .

6. The robust target detection method based on multimodal data fusion in small sample scenarios according to claim 1, characterized in that, In step S5, the multilayer perceptron network includes Layers, for detecting modal data , its first The layer transformation is as follows: in, For input data, For activation function, For the first Layer weight matrix, For the first Layer bias; output of the last layer of the network As extracted depth features .

7. The robust target detection method based on multimodal data fusion in small sample scenarios according to claim 1, characterized in that, In step S6, the adaptive multimodal alignment expert fusion system operates according to the following steps: a) Adaptive cross-modal alignment: for photoelectric position prediction results and radar position prediction results Assign weights and ,satisfy And calculate the weighted position results. ;Prediction results for photoelectric categories and detection category prediction results Assign weights and ,satisfy And calculate the weighted category results. ; b) Expert fusion decision-making: The weighted position results are then... and weighted category results The input is integrated and decided by an expert network, and the final category prediction result is output. and location prediction results .

Images