A method for individual identification based on cross-modal cross-attention fusion of visible light, infrared and IQ signals

Abstract

This invention discloses an individual recognition method based on cross-modal cross-attention, which fuses visible light, infrared, and IQ signals, belonging to the fields of artificial intelligence and multimodal image processing technology. Addressing the problems of insufficient multimodal heterogeneous feature fusion capability, imbalanced modal semantic expression, and unstable classification accuracy in existing technologies, this invention acquires visible light images, infrared images, and raw IQ signals. After preprocessing, image features are extracted using convolutional neural networks combined with spatial attention modules, and signal spectral features are extracted using a convolutional hybrid network. Then, a cross-modal bidirectional cross-attention mechanism is employed to construct three sets of modal pairs for bidirectional semantic interaction and feature fusion. Finally, the fused features are input into a classifier to obtain the recognition result. This invention enables deep semantic fusion, dynamically adapts to changes in modal quality, and improves the accuracy and robustness of target recognition in complex environments.

Description

Technical Field

[0001] This invention belongs to the field of artificial intelligence and multimodal image processing technology, and particularly relates to an individual recognition method based on cross-modal cross-attention fusion of visible light, infrared and IQ signals. Background Technology

[0002] With the rapid development of artificial intelligence technology and intelligent sensors, multimodal perception and fusion has become an important research direction in the fields of image recognition and target detection. In traditional image recognition tasks, visible light images are widely used due to their rich information and ease of acquisition. However, in special scenarios such as uneven lighting, severe occlusion, nighttime, or smoke, relying solely on visible light images cannot guarantee the robustness of the recognition system. Infrared thermal imaging images can perceive the thermal radiation characteristics of targets in complex environments, providing valuable supplementary information. Meanwhile, IQ signals acquired by communication radar and other devices, as raw complex signal data, contain the target's motion characteristics, frequency characteristics, and reflection characteristics, and have unique value in target recognition. However, there are significant differences between visible light images, infrared images, and IQ signals: in terms of data structure, images are two-dimensional pixel matrices, while IQ signals are one-dimensional or complex sequences; in terms of semantic granularity, images reflect the target's texture and shape, while signals emphasize physical layer features; the inconsistency between spatial and frequency dimensions leads to a semantic gap in feature representation. Current mainstream multimodal fusion methods mainly focus on combinations such as image and text, RGB and depth maps, or hyperspectral and LiDAR. Research on fusion methods involving three information sources—electromagnetic signals, infrared images, and visible light images—is still in its early stages. Existing fusion methods, such as early-stage fusion, mid-stage fusion, and late-stage fusion, have significant shortcomings when handling such highly heterogeneous modalities. Early-stage fusion uses feature concatenation or weighted addition, directly connecting modal features at a low level, making it difficult to achieve effective semantic alignment. Mid-stage fusion uses shared convolution or attention mechanisms after extracting modal features, but the fusion method is relatively coarse and fails to introduce deep semantic interactions between modalities, resulting in limited representational capabilities after fusion. Late-stage fusion performs weighted voting after independent prediction of each modality, failing to adaptively allocate attention weights based on the quality, completeness, or task relevance of the input modality. Especially in complex scenarios such as military security and low-altitude target monitoring, different modalities contribute differently to the discrimination results, making it difficult for traditional methods to balance real-time performance, accuracy, and robustness.

[0003] Furthermore, current research lacks modular structures and optimization mechanisms for joint learning of images and IQ signals, failing to effectively capture the deep semantic relationships and interdependencies among modalities. Therefore, existing technologies struggle to address issues such as the high difficulty of fusion due to strong modal heterogeneity, the inability to dynamically adjust modal importance, and the lack of a unified framework, thus hindering the practical application of high-precision, robust multimodal recognition in complex environments. Summary of the Invention

[0004] To address the aforementioned technical problems, this invention proposes an individual identification method based on cross-modal cross-attention fusion of visible light, infrared, and IQ signals, thereby resolving the issues present in the prior art.

[0005] In a first aspect, to achieve the above objectives, the present invention provides an individual identification method based on cross-modal cross-attention fusion of visible light, infrared, and IQ signals, comprising the following steps:

[0006] Acquire visible light images, infrared images, and raw IQ signals of the target;

[0007] The visible light image and infrared image are subjected to spatial resolution unification and standard normalization processing, and the IQ signal is filtered and spectral transformed to obtain the amplitude spectrum and phase spectrum;

[0008] The texture and thermal imaging features of the visible light and infrared images are extracted using a convolutional neural network combined with a spatial attention module, and the spectral features of the IQ signal are extracted using a convolutional hybrid network.

[0009] A cross-modal bidirectional cross-attention mechanism is adopted to construct three sets of modal pairs among the visible light image, infrared image and IQ signal features. For each modal pair, the features of one modality are used as the query vector and the features of the other modality are used as the key vector. Cross-attention calculation is performed to extract complementary features and the outputs are fused.

[0010] The features fused by the cross-modal bidirectional cross-attention mechanism are aggregated to obtain a unified fused feature representation.

[0011] The unified fusion feature representation is input into the classifier to obtain the image classification prediction result.

[0012] Optionally, the process of acquiring the visible light image, infrared image, and raw IQ signal of the target includes:

[0013] The system acquires visible light images, infrared images, and raw IQ signals of the target area. The visible light and infrared images are spatially aligned two-dimensional image modes and are uniformly sized. The raw IQ signals are in complex sampling form, with the sampling length of each signal segment set to a set number of points. The IQ signals include two channels: in-phase and quadrature.

[0014] Optionally, the process of performing spatial resolution unification and standard normalization on visible light and infrared images, and filtering and spectral transformation on the IQ signal includes:

[0015] The visible light image and the infrared image are scaled to the same size and then standardized using the image channel mean and standard deviation.

[0016] The IQ signal is bandpass filtered to remove high-frequency noise, and the amplitude spectrum and phase spectrum are extracted by Fourier transform to form a frequency domain pseudo-image representation.

[0017] Optionally, the process of using a convolutional neural network combined with a spatial attention module to extract texture and thermal imaging features of visible light and infrared images, and using a convolutional hybrid network to extract spectral features of the IQ signal includes:

[0018] For visible light and infrared images, ResNet network is used as the convolutional backbone network for feature extraction, and a spatial attention mechanism is combined. The spatial attention mechanism generates a spatial attention map by performing max pooling and average pooling operations on the input feature map, concatenating it along the channel dimension, and inputting it into the convolutional network. The original feature map is then weighted element-wise to enhance the feature response of salient regions.

[0019] For IQ signals, the Convmixer network is used to perform convolutional mixing operations on the amplitude spectrum and phase spectrum to extract high-dimensional representation features of the signal.

[0020] Optionally, the process of employing a cross-modal bidirectional cross-attention mechanism to construct three sets of modal pairs among visible light images, infrared images, and IQ signal features, and performing cross-attention calculation for each modal pair to extract complementary features, includes:

[0021] Three sets of mode pairs were constructed, including visible light image and infrared image mode pairs, visible light image and IQ signal mode pairs, and infrared image and IQ signal mode pairs;

[0022] For each modality pair, bidirectional cross-attention computation is performed, where the first direction uses the first modality feature as the query vector and the second modality feature as the key and value vector; the second direction uses the second modality feature as the query vector and the first modality feature as the key and value vector.

[0023] The bidirectional attention outputs of each modality pair are fused to obtain modal interaction features;

[0024] Learnable modality-guided weight parameters are introduced to adaptively adjust the attention output contribution based on the modality input confidence, and the interaction features of all modality pairs are fused into a unified representation through linear mapping and concatenation operations.

[0025] Optionally, the process of aggregating the features fused through the cross-modal bidirectional cross-attention mechanism to obtain a unified fused feature representation, and inputting the unified fused feature representation into the classifier, includes:

[0026] The updated trimodal features are merged into a unified classification vector through vector concatenation or weighted averaging.

[0027] The unified classification vector is input into a classifier composed of a multilayer perceptron, and the class prediction result is output.

[0028] Construct a total loss function, including the primary classification cross-entropy loss and the auxiliary consistency loss, and jointly optimize the parameters of the feature extractor, fusion module and classifier through the backpropagation algorithm.

[0029] Secondly, the present invention also provides an individual identification system based on cross-modal cross-attention fusion of visible light, infrared, and IQ signals, used to implement an individual identification method based on cross-modal cross-attention fusion of visible light, infrared, and IQ signals, the system comprising:

[0030] The data acquisition module is used to acquire visible light images, infrared images, and raw IQ signals of the target;

[0031] The preprocessing module is used to perform spatial resolution unification and standard normalization on the visible light image and infrared image, and to filter and perform spectral transformation on the IQ signal to obtain the amplitude spectrum and phase spectrum.

[0032] The feature extraction module is used to extract the texture and thermal imaging features of the visible light image and infrared image using a convolutional neural network combined with a spatial attention module, and to extract the spectral features of the IQ signal using a convolutional hybrid network.

[0033] The feature fusion module is used to construct three modal pairs between the visible light image, infrared image and IQ signal features using a cross-modal bidirectional cross-attention mechanism. For each modal pair, the features of one modality are used as the query vector and the features of the other modality are used as the key vector. Cross-attention calculation is performed to extract complementary features and the outputs are fused.

[0034] The classification module is used to aggregate the features fused by the feature fusion module to obtain a unified fused feature representation, and input the unified fused feature representation into the classifier to obtain the image classification prediction result.

[0035] Thirdly, the present invention also provides a computer terminal device, comprising:

[0036] One or more processors;

[0037] A memory, coupled to the processor, for storing one or more programs;

[0038] When the one or more programs are executed by the one or more processors, the one or more processors implement the steps of the individual identification method based on cross-modal cross-attention fusion of visible light, infrared and IQ signals in the first aspect described above.

[0039] Fourthly, the present invention also provides a computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, it implements the steps of the individual identification method based on cross-modal cross-attention fusion of visible light, infrared and IQ signals in the first aspect described above.

[0040] Fifthly, the present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the individual identification method based on cross-modal cross-attention fusion of visible light, infrared and IQ signals in the first aspect described above.

[0041] Compared with the prior art, the present invention has the following advantages and technical effects:

[0042] This invention provides an individual recognition method based on cross-modal cross-attention fusion of visible light, infrared, and IQ signals. Through a cross-modal bidirectional cross-attention mechanism, this invention achieves deep semantic interaction and complementary feature extraction between visible light images, infrared images, and IQ signals, effectively overcoming the semantic gap between heterogeneous modalities. The introduced learnable modality-guided weights can dynamically adjust the fusion strategy according to the input data quality, enhancing the system's robustness in cases of partial modality loss or degradation. This method provides a unified framework for image and electromagnetic signal fusion, significantly improving the accuracy and adaptability of target recognition in complex environments. Attached Figure Description

[0043] The accompanying drawings, which form part of this invention, are used to provide a further understanding of the invention. The illustrative embodiments of the invention and their descriptions are used to explain the invention and do not constitute an undue limitation of the invention. In the drawings:

[0044] Figure 1 This is a flowchart illustrating the overall method of an embodiment of the present invention;

[0045] Figure 2 This is a diagram illustrating the model training framework of an embodiment of the present invention. Detailed Implementation

[0046] It should be noted that, unless otherwise specified, the embodiments and features described in the present invention can be combined with each other. The present invention will now be described in detail with reference to the accompanying drawings and embodiments.

[0047] It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases the steps shown or described may be executed in a different order than that shown here.

[0048] Example 1

[0049] like Figure 1 As shown, this embodiment provides an individual identification method based on cross-modal cross-attention fusion of visible light, infrared, and IQ signals, including:

[0050] Acquire visible light images, infrared images, and raw IQ signals of the target;

[0051] The visible light image and infrared image are subjected to spatial resolution unification and standard normalization processing, and the IQ signal is filtered and spectral transformed to obtain the amplitude spectrum and phase spectrum;

[0052] The texture and thermal imaging features of the visible light and infrared images are extracted using a convolutional neural network combined with a spatial attention module, and the spectral features of the IQ signal are extracted using a convolutional hybrid network.

[0053] A cross-modal bidirectional cross-attention mechanism is adopted to construct three sets of modal pairs among the visible light image, infrared image and IQ signal features. For each modal pair, the features of one modality are used as the query vector and the features of the other modality are used as the key vector. Cross-attention calculation is performed to extract complementary features and the outputs are fused.

[0054] The features fused by the cross-modal bidirectional cross-attention mechanism are aggregated to obtain a unified fused feature representation.

[0055] The unified fusion feature representation is input into the classifier to obtain the image classification prediction result.

[0056] Furthermore, the specific process described above includes:

[0057] S1: Acquire visible light and infrared images of the target data and the raw IQ signal mode as input.

[0058] S2: Spatial resolution is unified and standardization is performed on infrared and visible light images respectively; low-pass filtering and noise reduction are performed on IQ signals and Fourier transform is performed to extract amplitude and phase spectra.

[0059] S3: Visible light (RGB) image modal and infrared (IR) image modal use ResNet and spatial attention module to extract texture and thermal imaging information; the amplitude spectrum and phase spectrum of the IQ signal are convolved using Convmixer to extract signal spatial features.

[0060] S4: The fusion stage adopts a cross-modal bidirectional cross-attention mechanism to build an interaction channel between the three modalities. Each pair is fused after extracting complementary features using cross-attention.

[0061] S5: The features fused from the three modalities through a bidirectional cross-attention mechanism are concatenated or weighted to obtain a unified fused feature representation. This fused feature is then input into a classification head composed of a multilayer perceptron (MLP) to output the final image classification prediction result.

[0062] S6: Construct a total loss function for supervised training. Optimize all learnable parameters in the feature extractor, fusion module, and classifier together using the backpropagation algorithm. After training, the model can perform image recognition tasks during the inference phase.

[0063] As one implementation method in this embodiment, the process of acquiring the visible light image, infrared image, and raw IQ signal of the target includes:

[0064] The system acquires visible light images, infrared images, and raw IQ signals of the target area. The visible light and infrared images are spatially aligned two-dimensional image modes and are uniformly sized. The raw IQ signals are in complex sampling form, with the sampling length of each signal segment set to a set number of points. The IQ signals include two channels: in-phase and quadrature.

[0065] Furthermore, the process of acquiring the visible light image, infrared image, and raw IQ signal of the target is the step in S1, which includes the following:

[0066] S1.1: Obtain multi-source modal input for individual targets:

[0067] Three types of input data were collected from the target area:

[0068] Visible light images:

[0069] (1)

[0070] Infrared image:

[0071] (2)

[0072] Raw IQ signal:

[0073] (3)

[0074] in This indicates two channels for IQ. This represents the height and width of the frequency domain pseudo-image.

[0075] The specific operating procedure is as follows: (Refer to the attached document) Figure 1 , Figure 2In this embodiment, the multimodal data originates from the AIS dataset and includes three modalities: RGB visible light images, infrared thermal imaging images, and complex IQ signals. The IQ signals are represented in complex sampling form, with each signal segment having a sampling length of 256 points. The RGB and infrared images are spatially aligned two-dimensional image modalities with uniform dimensions. Spatial size unification and normalization are performed on RGB and infrared images to ensure scale consistency in subsequent feature extraction. The IQ signal is first bandpass filtered to remove high-frequency noise, and then the amplitude spectrum and phase spectrum are obtained through Fast Fourier Transform (FFT) to form a pseudo-image input to adapt to the convolutional neural network processing structure.

[0076] As one implementation method in this embodiment, the process of performing spatial resolution unification and standard normalization on visible light and infrared images, and filtering and spectral transformation on the IQ signal includes:

[0077] The visible light image and the infrared image are scaled to the same size and then standardized using the image channel mean and standard deviation.

[0078] The IQ signal is bandpass filtered to remove high-frequency noise, and the amplitude spectrum and phase spectrum are extracted by Fourier transform to form a frequency domain pseudo-image representation.

[0079] Furthermore, the process of unifying and standardizing the spatial resolution of the visible light image and the infrared image, and filtering and transforming the IQ signal, is the step in S2, which includes the following:

[0080] S2.1: Image modality normalization and spatial alignment:

[0081] Both visible light and infrared images are scaled down to the same size. And adopt standard normalization:

[0082] (4)

[0083] in and These represent the mean and standard deviation of the image channels, respectively.

[0084] S2.2: IQ signal preprocessing:

[0085] For IQ signals Bandpass filtering and spectral transformation are performed to obtain the amplitude spectrum. Phase spectrum :

[0086] (5)

[0087] (6)

[0088] in This indicates the Fourier transform operation.

[0089] In step 2, the specific operation process is as follows: (Refer to the appendix) Figure 1 , Figure 2 Image modalities (RGB and infrared) are input into a convolutional neural network with a ResNet backbone, respectively, and combined with a spatial attention module to extract their texture and thermal imaging features. The IQ signal modality is input into a Convmixer network for deep convolution and channel mixing operations to extract its spectral spatial features. All three modalities output feature vectors of the same dimension, denoted as . , and .

[0090] As one implementation method in this embodiment, the process of using a convolutional neural network combined with a spatial attention module to extract texture and thermal imaging features of visible light and infrared images, and using a convolutional hybrid network to extract spectral features of the IQ signal includes:

[0091] For visible light and infrared images, ResNet network is used as the convolutional backbone network for feature extraction, and a spatial attention mechanism is combined. The spatial attention mechanism generates a spatial attention map by performing max pooling and average pooling operations on the input feature map, concatenating it along the channel dimension, and inputting it into the convolutional network. The original feature map is then weighted element-wise to enhance the feature response of salient regions.

[0092] For IQ signals, the Convmixer network is used to perform convolutional mixing operations on the amplitude spectrum and phase spectrum to extract high-dimensional representation features of the signal.

[0093] Furthermore, the process of using a convolutional neural network combined with a spatial attention module to extract texture and thermal imaging features from visible light and infrared images, and using a convolutional hybrid network to extract spectral features of the IQ signal, is the step in S3, which includes the following:

[0094] S3.1: Image Modal Feature Extraction:

[0095] Feature extraction is performed on RGB and infrared images separately using shared or independent convolutional backbone networks (such as ResNet), while spatial attention mechanisms are combined to enhance region response:

[0096] (7)

[0097] (8)

[0098] The spatial attention mechanism involves performing max pooling and average pooling on the input feature map, concatenating them along the channel dimension, and then inputting them into a small convolutional network to generate a spatial attention map. This process enhances the feature response of salient regions by weighting the original feature map element by element.

[0099] S3.2: IQ signal modal feature extraction:

[0100] The IQ spectrogram is convolutionally mixed using the Convmixer network to extract its high-dimensional representation.

[0101] (9)

[0102] In step 3, the specific operation process is as follows: (Refer to the appendix) Figure 1 , Figure 2 The feature vectors of the three modalities are cross-fused pairwise to construct three sets of modal interaction relationships: RGB ↔ IR, IR ↔ IQ, and RGB ↔ IQ. Each set achieves complementary semantic extraction between features through a cross-modal bidirectional cross-attention module. Specifically, one modality is used as the query, and the other as the key and value for multi-head attention computation. The resulting interaction features are concatenated with the residuals of the original features and then fed into a unified fusion mapping layer, outputting a unified feature representation vector.

[0103] As one implementation method in this embodiment, the process of constructing three sets of modal pairs among visible light images, infrared images, and IQ signal features using a cross-modal bidirectional cross-attention mechanism, and performing cross-attention calculation for each modal pair to extract complementary features, includes:

[0104] Three sets of mode pairs were constructed, including visible light image and infrared image mode pairs, visible light image and IQ signal mode pairs, and infrared image and IQ signal mode pairs;

[0105] For each modality pair, bidirectional cross-attention computation is performed, where the first direction uses the first modality feature as the query vector and the second modality feature as the key and value vector; the second direction uses the second modality feature as the query vector and the first modality feature as the key and value vector.

[0106] The bidirectional attention outputs of each modality pair are fused to obtain modal interaction features;

[0107] Learnable modality-guided weight parameters are introduced to adaptively adjust the attention output contribution based on the modality input confidence, and the interaction features of all modality pairs are fused into a unified representation through linear mapping and concatenation operations.

[0108] Furthermore, the process of employing a cross-modal bidirectional cross-attention mechanism to construct three sets of modal pairs among visible light images, infrared images, and IQ signal features, and performing cross-attention calculation for each modal pair to extract complementary features, includes the steps in S4, including the following:

[0109] S4.1: Modal Pair Construction and Cross-Attention Mechanism:

[0110] Three modality pairs are constructed: RGB ↔ IR, RGB ↔ IQ, and IR ↔ IQ. A bidirectional cross-attention interaction structure is built between each modality pair. A bidirectional attention computation path is set for each modality pair, with one modality serving as the query vector and the other as the key-value vector, to achieve the extraction and fusion of semantically complementary information.

[0111] Taking RGB and infrared modes as examples, let's assume their feature dimensions are unified as follows: The following attention relationships are constructed respectively:

[0112] (10)

[0113] (11)

[0114] The query matrix and key matrix are obtained by linear transformation of the original modal features respectively:

[0115] (12)

[0116] The bidirectional cross-attention module features two modalities. and Given the input, construct the query matrix, key matrix, and value matrix through linear transformations. The input feature dimension is... ,in Indicates the time step. This represents the embedding dimension at each position. All query, key, and value matrices are generated through linear mapping, maintaining consistent feature dimensions to facilitate attention weight calculation and feature alignment.

[0117] By fusing the bidirectional attention outputs of this modality pair, we obtain the modal interaction output:

[0118] (13)

[0119] Using the same strategy, cross-attention paths for RGB–IQ and IR–IQ mode pairs are constructed respectively to obtain the interaction features between all mode pairs.

[0120] S4.2: Modal Feature Fusion Output:

[0121] The cross-attention outputs between modal pairs are then fused to form a final unified multimodal representation. Specifically:

[0122] (14)

[0123] in, It is a learnable linear mapping matrix. This represents a vector-level concatenation operation. The resulting fused representation contains semantically complementary features between all modality pairs, serving as input for subsequent image classification or recognition tasks.

[0124] To further enhance the robustness of fusion, this invention introduces modality-guided weight parameters into the attention path. An improved fusion strategy is defined to adaptively adjust the attention output contribution based on modal input confidence or task weights:

[0125] (15)

[0126] in, During training, updates are performed through a learnable mechanism to suppress interference from noisy modalities while enhancing the output capability of modal paths with reliable information sources. This multimodal fusion strategy improves the complementarity and semantic consistency between modalities by explicitly modeling the bidirectional attention relationship between them, thereby enhancing the discriminative ability of the final classifier while maintaining model interpretability.

[0127] In step 4, the specific operation process is as follows: (Refer to the appendix) Figure 1 , Figure 2 The fused feature representation input is a classification head consisting of two multilayer perceptrons (MLPs). The first MLP layer is used for dimensionality compression, and the second MLP layer is connected to a softmax activation function, outputting a class probability distribution used for image classification tasks. This output is compared with the ground truth labels to calculate the cross-entropy loss, which is used for training supervision.

[0128] As one implementation method in this embodiment, the process of aggregating the features fused through the cross-modal bidirectional cross-attention mechanism to obtain a unified fused feature representation, and inputting the unified fused feature representation into the classifier, includes:

[0129] The updated trimodal features are merged into a unified classification vector through vector concatenation or weighted averaging.

[0130] The unified classification vector is input into a classifier composed of a multilayer perceptron, and the class prediction result is output.

[0131] Construct a total loss function, including the primary classification cross-entropy loss and the auxiliary consistency loss, and jointly optimize the parameters of the feature extractor, fusion module and classifier through the backpropagation algorithm.

[0132] Furthermore, the process of aggregating the features fused through the cross-modal bidirectional cross-attention mechanism to obtain a unified fused feature representation, and inputting the unified fused feature representation into the classifier, belongs to step S5 and includes the following:

[0133] S5.1: Multimodal fusion feature aggregation:

[0134] Updated trimodal features By concatenating or weighting the data, a unified classification vector can be formed:

[0135] (16)

[0136] S5.2 MLP Classification Prediction:

[0137] Fusion features Input to a multilayer perceptron (MLP) classifier, output class prediction results:

[0138] (17)

[0139] In step 5, the specific operation process is as follows: (Refer to the appendix) Figure 1 , Figure 2 A multi-task loss function consisting of three parts is constructed: main classification loss (cross-entropy), used to supervise the final output; modality consistency loss, constraining the semantic alignment of modal features and improving fusion robustness; and attention regularization loss, used to prevent redundant channel activation. The weighted sum of each loss term forms the total loss function. The Adam optimizer and cosine annealing learning rate scheduling strategy are used to jointly optimize the parameters of each module in the feature extractor, attention module, and classifier through backpropagation. The training hyperparameters are set as follows: the initial learning rate is set to... The training run consisted of 100 epochs, with a batch size of 128 and 16 data loading threads (num_workers). To avoid overfitting, an early stopping mechanism was introduced during training. The dataset was divided into training, validation, and test sets in a 6:2:2 ratio.

[0140] Furthermore, step S6 includes the following:

[0141] S6.1 Multi-task loss function design:

[0142] Using cross-entropy loss for supervision of the main classification task:

[0143] (18)

[0144] Auxiliary modal contrast loss can also be used. Loss of attentional consistency, etc.

[0145] (19)

[0146] S6.2 Joint Training and Inference Deployment:

[0147] Backpropagation is used to optimize the parameters of each module in the network. After training, the model can be deployed in image recognition tasks, exhibiting stable classification capabilities regardless of whether multimodal information is complete or partially missing.

[0148] In step 6, the specific operation process is as follows: (Refer to the appendix) Figure 1 , Figure 2 After training, the model is deployed on embedded platforms or edge terminals. During the inference phase, any combination of available modalities from the three modalities is input. The system uses a trained cross-attention mechanism to achieve semantic completion and discrimination between modalities, enabling image recognition and classification prediction of the target individual. Even in scenarios where some modalities are missing, the system still exhibits strong robustness and discrimination performance.

[0149] Based on this, the present invention provides an individual recognition method based on cross-modal cross-attention fusion of visible light, infrared and IQ signals. It adopts a cross-modal bidirectional cross-attention mechanism to achieve complementary semantic extraction between modalities. The unified feature representation after fusion is trained by multi-task supervision and can achieve high-precision recognition of target individuals in multi-modal input scenarios. It is applicable to various practical scenarios such as security monitoring, target detection and target recognition.

[0150] Example 2

[0151] In this embodiment, a computer terminal device is provided, including:

[0152] One or more processors;

[0153] A memory, coupled to the processor, for storing one or more programs;

[0154] When the one or more programs are executed by the one or more processors, the one or more processors implement the steps of the above-described individual identification method based on cross-modal cross-attention fusion of visible light, infrared and IQ signals.

[0155] In this embodiment, a computer-readable storage medium is also provided, on which a computer program is stored. When the computer program is executed by a processor, it implements the steps of the above-described individual identification method based on cross-modal cross-attention fusion of visible light, infrared and IQ signals.

[0156] In this embodiment, an electronic device is also provided, including a memory and a processor. The memory stores a computer program, and the processor is configured to run the computer program to perform the steps of the above-described individual identification method based on cross-modal cross-attention fusion of visible light, infrared, and IQ signals.

[0157] In this embodiment, a computer program product is also provided, including a computer program that, when executed by a processor, implements the steps of the above-described individual identification method based on cross-modal cross-attention fusion of visible light, infrared, and IQ signals.

[0158] The aforementioned program can run on a processor or be stored in memory (or a computer-readable medium). Computer-readable media includes both permanent and non-permanent, removable and non-removable media, and information storage can be achieved by any method or technology. Information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random-access memory (SRAM), dynamic random-access memory (DRAM), other types of random-access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transfer medium that can be used to store information accessible by a computing device.

[0159] These computer programs may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes can be implemented by different modules for different steps.

[0160] This embodiment provides such a device or system. The system, referred to as a cross-modal cross-attention-based visible light, infrared, and IQ signal fusion individual identification system, includes:

[0161] The data acquisition module is used to acquire visible light images, infrared images, and raw IQ signals of the target;

[0162] The preprocessing module is used to perform spatial resolution unification and standard normalization on the visible light image and infrared image, and to filter and perform spectral transformation on the IQ signal to obtain the amplitude spectrum and phase spectrum.

[0163] The feature extraction module is used to extract the texture and thermal imaging features of the visible light image and infrared image using a convolutional neural network combined with a spatial attention module, and to extract the spectral features of the IQ signal using a convolutional hybrid network.

[0164] The feature fusion module is used to construct three modal pairs between the visible light image, infrared image and IQ signal features using a cross-modal bidirectional cross-attention mechanism. For each modal pair, the features of one modality are used as the query vector and the features of the other modality are used as the key vector. Cross-attention calculation is performed to extract complementary features and the outputs are fused.

[0165] The classification module is used to aggregate the features fused by the feature fusion module to obtain a unified fused feature representation, and input the unified fused feature representation into the classifier to obtain the image classification prediction result.

[0166] As one implementation method in this embodiment, the data acquisition module includes:

[0167] An image acquisition unit is used to acquire visible light and infrared images of a target area, wherein the visible light and infrared images are spatially aligned two-dimensional image modalities and are uniformly sized.

[0168] The signal acquisition unit is used to acquire the raw IQ signal of the target area, wherein the raw IQ signal is in complex sampling form, the sampling length of each signal segment is set to a set number of points, and the IQ signal includes two channels: in-phase and quadrature.

[0169] As one implementation method in this embodiment, the preprocessing module includes:

[0170] An image preprocessing unit is used to scale the visible light image and the infrared image to the same size and perform standardization processing using the image channel mean and standard deviation;

[0171] The signal preprocessing unit is used to perform bandpass filtering on the IQ signal to remove high-frequency noise, and to extract the amplitude spectrum and phase spectrum through Fourier transform operation to form a frequency domain pseudo-image representation.

[0172] As one implementation method in this embodiment, the feature extraction module includes:

[0173] The image feature extraction unit is used to extract features from visible light images and infrared images by using ResNet network as the convolutional backbone network, and combining it with a spatial attention mechanism. The spatial attention mechanism performs max pooling and average pooling operations on the input feature map, concatenates it along the channel dimension, and inputs it into the convolutional network to generate a spatial attention map. The original feature map is then weighted element-wise to enhance the feature response of salient regions.

[0174] The signal feature extraction unit is used to extract high-dimensional representation features of IQ signals by performing convolutional mixing operations on the amplitude spectrum and phase spectrum using the Convmixer network.

[0175] As one implementation method in this embodiment, the feature fusion module includes:

[0176] The mode pair construction unit is used to construct three sets of mode pairs, including visible light image and infrared image mode pairs, visible light image and IQ signal mode pairs, and infrared image and IQ signal mode pairs;

[0177] The cross-attention calculation unit is used to perform bidirectional cross-attention calculation for each modality pair, wherein the first direction uses the first modality feature as the query vector and the second modality feature as the key and value vector, and the second direction uses the second modality feature as the query vector and the first modality feature as the key and value vector;

[0178] The feature fusion unit is used to fuse the bidirectional attention outputs of each modality pair to obtain modal interaction features;

[0179] The adaptive weighting unit is used to introduce learnable modality-guided weight parameters, adaptively adjust the attention output contribution based on the modality input confidence, and fuse the interaction features of all modality pairs into a unified representation through linear mapping and concatenation operations.

[0180] As one implementation method in this embodiment, the classification module includes:

[0181] The feature aggregation unit is used to merge the updated trimodal features into a unified classification vector through vector concatenation or weighted averaging.

[0182] The classifier unit is used to input the unified classification vector into a classifier composed of a multilayer perceptron and output the category prediction result;

[0183] The loss calculation unit is used to construct the total loss function, which includes the main classification cross-entropy loss and the auxiliary consistency loss. It jointly optimizes the parameters of the feature extraction module, the feature fusion module and the classifier unit through the backpropagation algorithm.

[0184] The system or apparatus is used to implement the functions of the methods in the above embodiments. Each module in the system or apparatus corresponds to each step in the method, as has been described in the method and will not be repeated here.

[0185] The above implementation method solves the problem of individual identification based on the fusion of visible light, infrared and IQ signals in cross-modal cross-attention in related technologies, thereby ensuring that the problems existing in the prior art are resolved.

[0186] The above are merely preferred embodiments of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A visible, infrared and IQ signal fusion individual identification method based on cross-modal cross-attention, characterized in that, The method comprises the following steps: obtaining visible light images, infrared images and original IQ signals of a target; performing spatial resolution unification and standard normalization processing on the visible light images and infrared images, and performing filtering and spectral transformation on the IQ signals to obtain amplitude spectrum and phase spectrum; extracting texture and thermal imaging features of the visible light images and infrared images using a convolutional neural network combined with a spatial attention module, and extracting spectral features of the IQ signals using a convolutional mixing network; adopting a cross-modal bidirectional cross-attention mechanism to construct three groups of modal pairs between the visible light image, infrared image and IQ signal features, and for each modal pair, taking the features of one modal as a query vector and the features of the other modal as a key and value vector to perform cross-attention calculation to extract complementary features and fuse the output; the process of adopting the cross-modal bidirectional cross-attention mechanism to construct the three groups of modal pairs between the visible light image, infrared image and IQ signal features, and for each modal pair, performing cross-attention calculation to extract complementary features comprises: constructing three groups of modal pairs, including a visible light image and infrared image modal pair, a visible light image and IQ signal modal pair, and an infrared image and IQ signal modal pair; for each modal pair, performing bidirectional cross-attention calculation respectively, wherein in the first direction, the features of the first modal are taken as the query vector and the features of the second modal are taken as the key and value vectors; in the second direction, the features of the second modal are taken as the query vector and the features of the first modal are taken as the key and value vectors; fusing the bidirectional attention output of each modal pair to obtain modal interaction features; introducing a learnable modal guidance weight parameter for adaptively adjusting the attention output contribution according to the modal input confidence, and fusing the interaction features of all modal pairs into a unified representation through linear mapping and concatenation operation; aggregating the features fused through the cross-modal bidirectional cross-attention mechanism to obtain a unified fusion feature representation; inputting the unified fusion feature representation into a classifier to obtain an image classification prediction result.

2. The method of claim 1, wherein, The process of obtaining the visible light images, infrared images and original IQ signals of the target comprises: collecting visible light images, infrared images and original IQ signals of a target region, wherein the visible light images and infrared images are spatially aligned two-dimensional image modalities and are unified to a set size; the original IQ signal is in complex sampling form, and the sampling length of each signal is set to a set number of points, and the IQ signal includes in-phase and quadrature channels.

3. The method of claim 1, wherein, The process of performing spatial resolution unification and standard normalization processing on the visible light images and infrared images, and performing filtering and spectral transformation on the IQ signals comprises: scaling the visible light images and infrared images to the same size and performing standardization processing using image channel mean and standard deviation; performing band-pass filtering on the IQ signal to remove high-frequency noise, and extracting amplitude spectrum and phase spectrum through Fourier transform operation to form a frequency domain pseudo-image representation.

4. The method of claim 1, wherein, The process of extracting texture and thermal imaging features of the visible light image and the infrared image using a convolutional neural network combined with a spatial attention module, and extracting spectral features of the IQ signal using a convolutional mixing network includes: For the visible light image and the infrared image, a ResNet network is used as a convolutional backbone network for feature extraction, and a spatial attention mechanism is combined, which generates a spatial attention map by inputting the convolutional network after maximum and average pooling operations on the input feature map along the channel dimension, and element-wise weights the original feature map to enhance the feature response of the significant area; For the IQ signal, a Convmixer network is used to perform convolutional mixing operations on the amplitude spectrum and the phase spectrum to extract high-dimensional representation features of the signal.

5. The method of claim 1, wherein, The process of aggregating the features fused by the cross-modal bidirectional cross-attention mechanism to obtain a unified fusion feature representation, and inputting the unified fusion feature representation into a classifier includes: The updated three-modal features are combined into a unified classification vector through vector splicing or weighted average operation; The unified classification vector is input into a classifier composed of multiple layers of perceptron to output a class prediction result; A total loss function is constructed, including a main classification cross-entropy loss and an auxiliary consistency loss, and the parameters of the feature extractor, the fusion module and the classifier are jointly optimized by the back propagation algorithm.

6. A visible, infrared and IQ signal fusion individual identification system based on cross-modal cross-attention, characterized in that, The system for implementing the method of any one of claims 1-5 includes: A data acquisition module for acquiring visible light images, infrared images and original IQ signals of a target; A preprocessing module for performing spatial resolution unification and standard normalization on the visible light images and the infrared images, and performing filtering and spectral transformation on the IQ signal to obtain an amplitude spectrum and a phase spectrum; A feature extraction module for extracting texture and thermal imaging features of the visible light image and the infrared image using a convolutional neural network combined with a spatial attention module, and extracting spectral features of the IQ signal using a convolutional mixing network; A feature fusion module for adopting a cross-modal bidirectional cross-attention mechanism to construct three sets of modal pairs between the visible light image, the infrared image and the IQ signal features, for each modal pair, taking the features of one modal as the query vector and the features of the other modal as the key-value vector, performing cross-attention calculation to extract complementary features, and fusing the output; The feature fusion module includes: A modal pair construction unit for constructing three sets of modal pairs, including a visible light image and an infrared image modal pair, a visible light image and an IQ signal modal pair, and an infrared image and an IQ signal modal pair; A cross-attention calculation unit for each modal pair, respectively performing bidirectional cross-attention calculation, where the first direction takes the first modal features as the query vector and the second modal features as the key and value vectors, and the second direction takes the second modal features as the query vector and the first modal features as the key and value vectors; A feature fusion unit for fusing the bidirectional attention outputs of each modal pair to obtain modal interaction features; An adaptive weight unit is configured to introduce learnable modal guidance weight parameters, adaptively adjust attention output contributions according to modal input confidences, and fuse interaction features of all modal pairs into a unified representation through linear mapping and concatenation operations. A classification module is configured to aggregate the features fused by the feature fusion module to obtain a unified fused feature representation, input the unified fused feature representation into a classifier, and obtain an image classification prediction result.

7. A computer terminal device, characterized by comprising: one or more processors; a memory coupled to the processors for storing one or more programs; when the one or more programs are executed by the one or more processors, the one or more processors implement steps of the method according to any one of claims 1-5.

8. A computer-readable storage medium having stored thereon a computer program, characterized in that The computer program is executed by the processor to implement the steps of the method according to any one of claims 1-5.

9. A computer program product comprising a computer program, characterized in that, The computer program is executed by the processor to implement the steps of the method according to any one of claims 1-5.

Images