Target association method based on multi-modal data feature alignment

Abstract

The invention discloses a target association method based on multi-modal data feature alignment, and relates to the technical field of multi-modal information processing and intelligent decision making. The multi-modal information of the synthetic aperture radar, the high-resolution range profile, the track, the visible light, the infrared, the text, the voice and the like is covered. And through cooperative work of a plurality of intelligent association sub-models, feature extraction and alignment association are performed on different modal data by using deep learning technologies such as a two-dimensional convolutional neural network, a one-dimensional convolutional neural network, a bidirectional long-short term memory network, an attention mechanism and Transform, association similarity is calculated, and pairing mode data is output. The method is applied to the fields of target intelligent identification, resource environment monitoring and the like, has obvious advantages in complex sea conditions, low visibility and multi-interference scenes, can improve the target detection, identification, tracking and early warning capabilities, and provides accurate decision support for safety management and control, emergency disposal, resource monitoring and the like in the fields of shore, sea, air and sky.

Description

Technical Field

[0001] This invention relates to the field of multimodal information processing and intelligent decision-making technology, and specifically proposes a target association method based on multimodal data feature alignment. Background Technology

[0002] Current maritime target monitoring mainly relies on single sensors or limited information sources, which presents significant technical bottlenecks: First, traditional methods depend on single-modal analysis of visible / infrared images or track data, making it difficult to meet the target identification needs under complex sea conditions. For example, visible light images fail at night or in smoke, SAR images suffer from speckle noise interference, and single HRRP data lacks spatial context information. Second, multi-source sensor data suffers from problems such as spatiotemporal alignment difficulties and feature spatial heterogeneity. For instance, there are resolution differences between space-based satellite and airborne UAV detection images, and the spectral characteristics of visible light and infrared images lead to feature alignment failures. The semantic gap between text / speech and images is difficult to bridge. Third, existing methods do not fully mine the time-series characteristics of track data and lack the ability to dynamically correlate target motion trajectories with image / text features. Traditional track association algorithms cannot effectively combine target micro-motion features and motion patterns. Fourth, in complex environments, the rapid maneuvering of targets requires association algorithms to have high real-time performance and strong anti-interference capabilities. Traditional rule-based association methods are difficult to adapt to dynamic scenarios, and methods based on shallow models have insufficient feature representation capabilities. Summary of the Invention

[0003] In view of this, this invention proposes a target association method based on multimodal data feature alignment. It integrates heterogeneous data from multiple sources, including synthetic aperture radar (SAR), inverse synthetic aperture radar (ISAR), high-resolution range profiles (HRRP), visible light and infrared images, tracks, text, and speech, by constructing a deep neural network model to achieve cross-modal feature alignment and intelligent association. This method eliminates noise interference through data preprocessing, extracts image and HRRP features using two-dimensional or one-dimensional convolutional neural networks, and mines the spatiotemporal characteristics of tracks using a bidirectional long short-term memory network (Bi-LSTM). It employs a contrastive learning framework and a Transformer architecture to achieve multimodal feature fusion and semantic alignment, effectively addressing the technical bottlenecks of traditional methods in multi-sensor collaboration, spatiotemporal feature fusion, and adaptability to complex scenarios. The method significantly improves the accuracy of maritime target identification and the confidence of track association, and can be widely applied in fields such as marine monitoring and intelligent transportation, providing an efficient solution for multi-source information fusion decision-making.

[0004] This association method utilizes multiple intelligent association sub-models working collaboratively to associate multimodal data information of a target centered on flight track data. These sub-models include: SAR, ISAR, and flight track intelligent association sub-models; flight track and HRRP intelligent association sub-models; flight track and visible light intelligent association sub-models; flight track and infrared intelligent association sub-models; and flight track, text, and voice intelligent association sub-models.

[0005] This invention can be achieved through the following technical means:

[0006] A target association method based on multimodal data feature alignment includes the following steps:

[0007] Step 1, Multimodal data preprocessing: Preprocess the input synthetic aperture radar image, inverse synthetic aperture radar image, visible light image, infrared image, high-resolution range image, track data, text and speech data respectively;

[0008] Step 2, Deep Feature Extraction: Construct a hierarchical or fused network structure to extract deep feature vectors for each modality;

[0009] Step 3, Cross-modal feature alignment: This involves aligning features from different modalities within a shared semantic space using a contrastive learning framework. Specifically:

[0010] Calculate the triplet loss function value:

[0011] L_triplet = max(0, ||F a - F p ||² - ||F a - F n ||² + margin)

[0012] In the formula F a F is the feature vector of the trajectory. p For other modal feature vectors of the target corresponding to this trajectory feature, F n The feature vectors of other targets are defined, and the margin is a preset boundary value.

[0013] Train the hierarchical or fused network structure based on the triplet loss function value until the triplet loss function value converges, and save the current hierarchical or fused network structure.

[0014] Step 4, Multimodal Association Decision: In the aligned shared feature space, a hierarchical strategy is used to achieve feature-level association of data. Specifically:

[0015] Calculate the cosine similarity between the depth feature vectors of two different modalities:

[0016]

[0017] Where · represents the vector dot product, and ||x|| and ||y|| represent the L2 norm of the vector;

[0018] Step 5, output the association confidence between targets, including:

[0019] The cosine similarity correlation metric is converted into a standardized confidence score: 0-100%;

[0020] The method clearly displays the associated targets and their identification types in a pairing pattern, thus completing the target association method based on multimodal data feature alignment.

[0021] Furthermore, the specific method of step 1 is as follows:

[0022] For synthetic aperture radar images, inverse synthetic aperture radar images, visible light images, and infrared images: the preprocessing includes grayscale processing, basic noise smoothing, and contrast adjustment;

[0023] For high-resolution range image and track data: the preprocessing includes extracting the latitude, longitude, speed, and heading spatiotemporal features of the track, and uniformly using a dynamic time warping algorithm for sequence alignment; amplitude normalization and noise suppression are performed on the high-resolution range image data;

[0024] For text and speech data: The preprocessing includes processing speech data using speech-to-text technology, followed by semantic segmentation and feature representation of the text data.

[0025] Furthermore, the preprocessing of synthetic aperture radar images includes grayscale conversion based on weighted averaging and speckle noise suppression using Gaussian smoothing filtering.

[0026] Preprocessing for inverse synthetic aperture radar images includes using a phase gradient autofocus algorithm for motion compensation to correct phase errors, and contrast stretching to enhance target features.

[0027] Preprocessing for visible light images includes noise suppression using Gaussian filtering, contrast enhancement using histogram equalization, and precise target localization and region of interest cropping using deep learning target detection algorithms.

[0028] Preprocessing for infrared images includes radiometric calibration and correction based on Planck's law, noise reduction using median filtering, contrast enhancement using histogram equalization, and preliminary target segmentation based on adaptive thresholding or edge detection.

[0029] Preprocessing of track data includes spatiotemporal alignment using a dynamic time warping algorithm, with the distance metric being Euclidean distance based on multidimensional features and the window constraint being Sakoe-Chiba.

[0030] Preprocessing for text data includes context-aware word segmentation and input embedding sequence construction for the feature extraction model to fully utilize the model's semantic representation capabilities.

[0031] Furthermore, the specific method for step 2 is as follows:

[0032] For synthetic aperture radar (SAR) images, inverse synthetic aperture radar (ISAR) images, visible light images, and infrared images: Modality: A two-dimensional convolutional neural network architecture based on residual network and Transformer is adopted to extract the spatial structure and global dependency features of the images;

[0033] For high-resolution range image modalities: a sequence model combining a one-dimensional convolutional neural network with a long short-term memory network or its variants is used to capture the time-series features of the range image;

[0034] For trajectory modes: a model combining a bidirectional long short-term memory network and a spatiotemporal Transformer is used to extract deep spatiotemporal dynamic features of the target trajectory;

[0035] For text / speech modalities: a Transformer-based pre-trained language model is used to extract deep contextual semantic features from the text and fuse prosodic features extracted from the speech.

[0036] Furthermore, step 3 is performed as follows:

[0037] Image modal unification adopts a two-stream network design: the ResNet-50 network extracts local fine features, and the VisionTransformer network captures global context-dependent features;

[0038] The deep feature extraction of text and speech modalities adopts a hierarchical attention mechanism, which realizes the aggregation and representation of word-level, sentence-level and paragraph-level information based on the BERT model.

[0039] Furthermore, cross-modal feature alignment incorporates one or more of the following mechanisms to enhance the representation and alignment of image features:

[0040] (1) Spatial global relation attention module: used to capture the global spatial context relations within image features;

[0041] (2) Channel Global Attention Module: Used to dynamically adjust the weight of each channel of image features and enhance key channel information.

[0042] Furthermore, multimodal association decision-making supports one or more of the following strategies:

[0043] (1) Multi-scale feature fusion: Effectively splicing or fusing the global or regional features output by ResNet from the image modality dual-stream network with the global block features output by ViT;

[0044] (2) Dynamic temporal attention mechanism: used to align and focus on key time segments in modal data with different timestamps or asynchronous sampling during association decision-making. The beneficial effects of this invention are:

[0045] 1. This invention employs a deep learning-driven multimodal feature extraction and a contrastive learning-guided cross-modal feature alignment method to convert multi-source heterogeneous observation information such as target images, time-series signals, and text / speech into unified semantic features. By calculating and comparing the feature similarity in the shared semantic space, the intrinsic correlation between different observation data is determined.

[0046] 2. The method of the present invention can be used to solve the challenges of heterogeneous data fusion, cross-modal semantic gap, and insufficient target association accuracy and robustness in complex scenarios, providing support and basis for multi-sensor information fusion, intelligent recognition system, and automated situation understanding platform. Attached Figure Description

[0047] Figure 1 This is a flowchart illustrating the implementation method of the intelligent correlation sub-model of SAR, ISAR, and target trajectory in this embodiment of the invention.

[0048] Figure 2 This is a flowchart illustrating the implementation method of the intelligent association sub-model of track, text, and voice in an embodiment of the present invention. Detailed Implementation

[0049] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be described in further detail below.

[0050] A target association method based on multimodal data feature alignment includes the following steps:

[0051] I. For example Figure 1 As shown, the SAR, ISAR, and intelligent trajectory correlation sub-models

[0052] Step 1: Data Preprocessing Stage

[0053] 1. SAR Image Data Preprocessing

[0054] (1) The multi-channel SAR image is converted to grayscale using a weighted average method, and the formula is as follows:

[0055] Gray = 0.299R + 0.587G + 0.114B

[0056] Gray represents the pixel value of the calculated single-channel grayscale image; R, G, and B represent the intensity values ​​of the red, green, and blue channels of the corresponding pixels in the original multi-channel image, respectively; 0.299, 0.587, and 0.114 are commonly used weighting coefficients.

[0057] The grayscale image Gray obtained in the previous step is convolved with a Gaussian filter to suppress speckle noise. The weights of the Gaussian filter at position (x, y) are calculated using the following formula:

[0058]

[0059] A Gaussian filter is defined as a 5×5 matrix with its center at (0,0), and coordinates (x,y) representing the offset relative to the center point. σ is the standard deviation of the Gaussian function, controlling the smoothness, and is set to 1.0. The weights at each position of the convolution kernel are calculated based on the Gaussian function G(x,y).

[0060] 2. ISAR Image Data Preprocessing

[0061] (1) During ISAR imaging, target motion may cause the image to defocus, so motion compensation is required. Phase gradient autofocus (PGA) is a commonly used motion compensation method. It improves image quality by iteratively estimating and correcting phase error (compensating the original phase φ(t) to φcomp(t)).

[0062] The PGA algorithm is explained as achieving self-focusing through steps such as iterative selection of strong scattering points, window function processing and FFT, phase gradient estimation, phase error integration, and phase compensation.

[0063] (2) The contrast between the target and the background in the ISAR image is enhanced by using a contrast stretching method to highlight the target features. The image pixel values ​​are stretched from the interval [a,b] to [c,d], and the calculation formula is as follows:

[0064]

[0065] Where Inew(x,y) is the new pixel value after stretching; I(x,y) is the original pixel value; [a,b] is the original range of interest; and [c,d] is the target output range. The pixel value is linearly stretched from the original dynamic range [a,b] to [0, 255].

[0066] 3. Track data preprocessing

[0067] (1) The ship trajectory data composed of AIS records of variable length is organized in chronological order to form a time series data in a unified format, including spatiotemporal characteristics such as latitude and longitude, speed, and heading.

[0068] (2) To eliminate the dimensional differences between different features, standardization is performed using the following formula:

[0069]

[0070] Where x_norm is the standardized eigenvalue; x is the original eigenvalue; μ is the mean of the eigenvalue; and σ is the standard deviation of the eigenvalue.

[0071] (3) Dynamic Time Warping (DTW) algorithm is used to align track sequences from different sources or with time offsets. Specific parameters: the distance metric is Euclidean distance based on normalized multidimensional features (longitude, latitude, speed, heading); the window constraint is Sakoe-Chiba band, and the window width w is set to 15% of the average length of the sequence to be aligned.

[0072] Step 2: SAR Image Target Feature Extraction Stage

[0073] 1. Construction of a 2D Convolutional Neural Network (ResNet-ViT Two-Stream Network)

[0074] This embodiment employs a dual-stream parallel network architecture, with the ResNet-50 stream and the ViT stream processing the input image independently and extracting feature information at different levels. The ResNet-50 stream focuses on extracting local structural features of the image, capturing detailed information such as the edges and textures of the target through hierarchical convolution operations; while the ViT stream models the global spatial relationships of the image through a self-attention mechanism, excelling at capturing the overall shape and spatial layout of the target.

[0075] (1) ResNet stream: The ResNet-50 architecture is adopted.

[0076] Input image size 224x224x1 (grayscale SAR image). First convolutional layer (Conv1): 64 7×7 convolutional kernels, stride 2, padding size 3, ReLU activation function, followed by batch normalization (BatchNorm). Max pooling layer: 3×3 window, stride 2. ResNet modules (from conv2_x to conv5_x): standard bottleneck structure, with 256, 512, 1024, and 2048 output channels respectively at each stage. Processing method: Each convolutional layer is followed by batch normalization and ReLU activation. Output: 2048-dimensional feature vector after global average pooling (GAP).

[0077] (2) ViT stream: adopts ViT-Base / 16 architecture.

[0078] Input image size: 224x224x1 (can be copied from a single channel to a three-channel input, or use specific pre-trained weights for a single-channel image). Image patch size: 16×16, embedding dimension: 768, number of transformer encoder layers: 12, number of attention heads: 12, multilayer perceptron size (middle layer size of feedforward network): 3072, dropout rate: 0.1, output: 768-dimensional feature vector corresponding to the [CLS] label output by the last transformer encoder layer.

[0079] (3) Enhancement mechanism:

[0080] The explanation of the inherent characteristics of the ViT architecture (spatial global relationships and local partitioning), and the enhancement component actually added to the ResNet-50 architecture: the channel global attention module (SE Block).

[0081] Spatial Global Relationship Attention Module: The self-attention mechanism of ViT flow itself realizes the capture of global spatial relationships.

[0082] Channel Global Attention Module: A Squeeze-and-Excitation (SE) Block is embedded after the output feature map of each residual stage of ResNet-50. The SE Block obtains channel descriptors through a Squeeze operation (global average pooling), then learns channel weights through an Excitation operation (two fully connected layers: the first FC reduces dimensionality with a reduction ratio r of 16 and ReLU activation; the second FC increases dimensionality with Sigmoid activation), and finally rescales the original feature map.

[0083] Local partitioning strategy: ViT's patch partitioning is a type of local partitioning.

[0084] The reason this invention employs a dual-stream network architecture instead of a single-stream network is that SAR images have unique scattering characteristics, making both local detail features (hull structure, deck details) and overall spatial relationships (target orientation, size ratio) crucial for target recognition. ResNet-50 effectively extracts local detail features through hierarchical convolution, but its ability to model long-range spatial relationships is limited. ViT, on the other hand, effectively captures global contextual information through its self-attention mechanism, but its extraction of local details is less precise than that of convolutional networks. By combining the advantages of both through a dual-stream design, high-quality local and global features can be obtained simultaneously, significantly improving the accuracy and robustness of SAR image target recognition.

[0085] 2. Feature extraction operation

[0086] (1) The ResNet-50 stream and the ViT stream are parallel dual-stream network architectures, rather than serial pipelines. The same SAR image after preprocessing will be simultaneously input into two independent network branches for feature extraction, and the outputs of the two will be combined in the subsequent feature fusion stage. This dual-stream design is to simultaneously capture the local fine features of the image (handled by the ResNet-50 stream) and the global contextual relationships (handled by the ViT stream). The two functions are complementary and cannot be substituted for each other.

[0087] The preprocessed SAR images are input into the ResNet-50 stream and the ViT stream, respectively. Two-dimensional convolutional layers perform convolution operations with the images using kernels to extract spatial features. The calculation formula is as follows:

[0088]

[0089] Where i,j: spatial coordinates of the output feature map, representing the activation value at that location. For a 224×224 input image, the output feature map size after the first convolution layer is 112×112, and the values ​​of i,j range from 0 to 111. m,n: relative offset coordinates relative to the center of the convolution kernel. For a 7×7 convolution kernel, the values ​​of m,n range from -3 to 3, where (0,0) represents the center position of the convolution kernel. i+m,j+n: pixel values ​​at the corresponding positions in the input image. In the SAR image processing of this invention, these values ​​are preprocessed grayscale values ​​(0-255). m,n: convolution kernel weights, parameters learned by the network. In the first convolution layer of the ResNet-50 stream, there are 64 7×7 convolution kernels, each containing 49 weight parameters.

[0090] Taking SAR image processing as an example, when calculating the activation value at position (5,7) of the output feature map, the convolution operation aligns the center of the 7×7 convolution kernel with the position (5,7) of the input image, and then iterates through all positions of the convolution kernel (-3≤m≤3, -3≤n≤3). For each relative position (m,n), the product of the pixel value at position (5+m,7+n) of the input image and the weight wm,n of the convolution kernel is calculated. Finally, all products are summed and the bias b is added. After passing through the activation function, the output value y5,7 is obtained.

[0091] (2) Reduce the feature dimension through max pooling operation. The max pooling formula is:

[0092]

[0093] Where y is the pooling output value; S is the pooling window (3×3); and xm,n are the pixel values ​​within the pooling window S in the input feature map. This operation is independent of the convolution step and is a downsampling process performed after the convolution operation. m,n within the pooling window represents the relative coordinates with respect to the top-left corner of the pooling window, ranging from 0 to S-1 (for a 3×3 window, m,n∈{0,1,2}), which differs from the meaning of m,n representing the offset relative to the center of the convolution kernel in the convolution operation. The main purpose of this step is to reduce the spatial dimensionality of the feature map, reduce computational complexity, expand the receptive field, and enhance the translation invariance of the features.

[0094] (3) Feature fusion and projection: The 2048-dimensional vector output from the ResNet-50 stream is concatenated with the 768-dimensional vector output from the ViT stream to obtain a 2816-dimensional fused feature. This fused feature is then passed through a fully connected layer FC(2816, D_shared) (i.e., FC(2816, 512)) and ReLU activation to obtain the final SAR image target feature vector (512-dimensional).

[0095] Taking SAR image processing in this embodiment as an example: the preprocessed 224×224×1 grayscale SAR image is simultaneously input into a ResNet-50 stream and a ViT stream. The ResNet-50 stream extracts local structural features through a 50-layer deep network, ultimately outputting a 2048-dimensional feature vector; the ViT stream extracts global spatial relationship features through a 12-layer Transformer encoder, ultimately outputting a 768-dimensional feature vector. The outputs of both are fused into a 2816-dimensional feature vector through a concatenation operation, and then projected onto a 512-dimensional shared feature space through a fully connected layer, finally obtaining a high-quality feature representation that can simultaneously reflect the local details and global spatial relationships of the target.

[0096] Step 3: ISAR Image Target Feature Extraction Stage

[0097] 1. Construction of a 2D Convolutional Neural Network (ResNet-ViT Two-Stream Network)

[0098] It adopts the ResNet-ViT dual-stream network architecture, which is completely consistent with SAR image feature extraction, including:

[0099] (1) ResNet stream: The ResNet-50 architecture is used with the same parameters, including the input image size of 224×224×1, convolutional layer parameters (64 7×7 convolutional kernels, stride of 2, padding of 3), max pooling parameters (3×3 window, stride of 2), and the number of output channels in each stage (256, 512, 1024, 2048), etc.

[0100] (2) ViT stream: uses the ViT-Base / 16 architecture with identical parameters, including input image size 224×224×1, block size 16×16, embedding dimension 768, encoder layer number 12, attention head number 12, MLP size 3072, etc.

[0101] (3) Enhancement mechanism: The same SE Block structure with completely identical parameters is embedded after the output of each residual stage of ResNet-50.

[0102] The reason this invention uses the same network structure to process SAR and ISAR images is that, although their imaging principles differ, both are converted to standard grayscale image format after preprocessing, and their target feature extraction requirements are highly similar. This design also reflects the modularity and reusability advantages of this invention, reducing system complexity while ensuring consistency in feature extraction.

[0103] Step 4: Target Feature Extraction from Track Data

[0104] 1. Construction of Bidirectional Long Short-Term Memory Network (Bi-LSTM) and Spatiotemporal Transformer

[0105] A hybrid model consisting of a Bi-LSTM layer and a subsequent temporal Transformer encoder layer is established.

[0106] (1) Bi-LSTM: 1 layer Bi-LSTM, with 128 hidden units per layer (therefore the Bi-LSTM output is 256-dimensional). The activation function is tanh, and the recurrent activation function is sigmoid. The recurrent dropout is set to 0.2.

[0107] (2) Spatiotemporal Transformer: 2 Transformer Encoder Layers, model hidden dimension (d_model) is 256 (matching the output dimension of Bi-LSTM), number of multi-head self-attention heads (num_heads) is 4, feedforward network (FFN) intermediate layer dimension (dim_feedforward) is 512, and dropout rate is 0.1.

[0108] 2. Feature extraction operation

[0109] (1) Input the preprocessed track data into the model in time series order, and pass the LSTM unit through the forget gate. Input gate Output gate The memory unit Ct is used to capture long-term and short-term dependencies in track data, and the calculation formula is as follows:

[0110] Forgotten Gate:

[0111] in The output of the forget gate; σ is the Sigmoid function; As weight; It is a concatenation of the previous hidden state and the current input; For bias.

[0112] Input Gate:

[0113] in For input gate output; As weight; For bias.

[0114] Candidate memory unit: C_t

[0115] Memory unit: * C_t

[0116] in This represents the current cell state. C represents the previous cell state; C_t represents the candidate memory.

[0117] Output gate:

[0118] in Output gate output; As weight; For bias.

[0119] Hidden state:

[0120] in Currently hidden; The cell state is activated by tanh.

[0121] (2) The outputs of the forward LSTM layer and the backward LSTM layer are concatenated at each time step to form the output sequence of Bi-LSTM (feature fusion and projection). This sequence is then input into the spatiotemporal Transformer encoder. Finally, by performing average pooling on the representations of all time steps of the Transformer output sequence, a 256-dimensional track sequence representation is obtained. This representation is then passed through a fully connected layer FC(256, D_shared) (i.e., FC(256, 512)) and ReLU activation to obtain the final track data target feature vector (512 dimensions).

[0122] Step 5: SAR, ISAR, and target track multi-source data correlation stage

[0123] 1. Classifier construction (an auxiliary task for intra-modal contrastive learning)

[0124] Classifiers for SAR, ISAR, and target track data classification are constructed based on fully connected layers. Each classifier is an FC(512, 128) -> ReLU -> FC(128, K) -> Softmax structure, where K is a predefined common ship type (K=6).

[0125] 2. Intramodal contrastive learning (optional auxiliary loss, used to improve the quality of single-modal features)

[0126] (1) Using the D_shared (512-dimensional) features of the three modal data obtained from the encoder in steps two, three, and four as the input to their respective classifiers, perform a softmax operation on the classifier outputs to obtain the predicted probabilities of belonging to each category. Let the classifier output be z, then the softmax calculation formula is:

[0127]

[0128] Where σ(z)j is the probability that the input z belongs to the j-th class; zj is the j-th element of z; and exp is the exponential function. Each element zj represents the raw score of the input sample belonging to the j-th class. For example, when z3=2.5, it means that the raw score of the sample belonging to the 3rd class is relatively high.

[0129] Taking SAR image classification as an example: Suppose we have an input SAR image. After preprocessing and feature extraction, the feature vector is fed into the classifier. The classifier output z=[1.2, 0.8, 2.5, 0.1, -0.5, 0.3, 1.1, 0.7, 0.4, -0.2], which, after softmax, yields the probability distribution σ(z)=[0.08, 0.05, 0.45, 0.01, 0.005, 0.02, 0.15, 0.09, 0.06, 0.005], indicating that the image has a 45% probability of belonging to class 3.

[0130] (2) Estimate the posterior distribution of the samples and measure the distance between it and the true posterior distribution of the samples using the cross-entropy loss function or KL divergence, the formula of which is:

[0131]

[0132] Where DKL(P || Q) is the KL divergence from distribution Q to P; P(i) is the probability of event i in the true distribution; Q(i) is the probability of event i in the model-estimated distribution. P: True label distribution (One-hot encoding). In this invention, each training sample has a definite class label. For example, if a SAR image is indeed of class 3, then P=[0, 0, 1, 0, 0, 0, 0, 0, 0, 0]. Q: The probability distribution predicted by the model (i.e., the softmax output). For example, in the above example, Q=σ(z)=[0.08,0.05, 0.45, 0.01, 0.005, 0.02, 0.15, 0.09, 0.06, 0.005], i: Class index, i=1,2,...,6, representing the i-th of the 6 target classes. In this embodiment, the calculated KL divergence is 0.798, indicating that there is a significant difference between the current classifier's prediction and the true label, and further optimization is needed.

[0133] The significance of KL divergence: It measures the difference between the model's predicted distribution Q and the true distribution P. In this invention, it is used to evaluate the accuracy of the classifier in classifying the target; the smaller the value, the more accurate the prediction.

[0134] Intramodal classification is used only as an auxiliary training task to improve feature extraction quality, and is not the ultimate goal of this invention. In practical applications, the classifier layer will be removed, and only the feature extraction part will be used for cross-modal association.

[0135] 3. Intermodal data association

[0136] (1) Cross-modal feature alignment: The D_shared (512-dimensional) features of SAR, ISAR, and track are trained using a triplet loss function to learn a shared embedding space, as shown in the formula:

[0137] L_triplet = max(0, ||F a - F p ||² - ||F a - F n ||² + margin)

[0138] In this embodiment, F a Specifically refers to the 512-dimensional feature vector of track data, F p F refers to the 512-dimensional feature vector of a SAR or ISAR image belonging to the same target as the track data. nThis refers to the 512-dimensional feature vector of the SAR or ISAR image that does not belong to the same target as the track data. This triplet loss is used during model training to map features from different modalities to a shared semantic space, ensuring that features from different modalities of the same target are close to each other in space, while features from different targets are far apart. After training, this loss function is no longer used; instead, the learned feature mappings are used for actual target association. The margin is set to 0.5. Training strategy: The AdamW optimizer (learning rate 1e-5, weight decay 0.01) is used, along with an online hard sample mining strategy.

[0139] (2) Initial Relevance Calculation: In the aligned shared embedding space, calculate the cosine similarity between feature vectors of different modal pairs. Let the feature vectors of the two modal data be x and y (both 512-dimensional), and the cosine similarity calculation formula is:

[0140]

[0141] To measure the strength of their association, · represents the vector dot product, and ||x|| and ||y|| represent the L2 norm of the vectors.

[0142] (3) Dynamic association optimization:

[0143] Multi-source feature fusion calculation of comprehensive score: The multi-source feature fusion mechanism in this invention strictly distinguishes different associated scenarios and adopts corresponding metrics and weight allocations: For track-track associated scenarios (tracks of the same target captured by different sensors), only DTW distance is used as the metric, at which time w_sim=0 and w_dtw=1.0, and the normalized DTW distance Norm_DTW_dist is directly calculated, and the comprehensive score Score_comprehensive = 1 - Norm_DTW_dist; For track-image associated scenarios (SAR / ISAR / visible light / infrared images and tracks), only cosine similarity is used as the metric, at which time w_sim=1.0 and w_dtw=0, and the comprehensive score Score_comprehensive = Sim(V_SAR, V_Track); In special mixed scenarios (where track sequences and multiple observation sources need to be considered simultaneously and there is a time synchronization relationship), a weighted fusion strategy is adopted, which is only used in specific application scenarios. This scenario-based weight allocation mechanism ensures the rigor and applicability of the technical solution, avoiding the misapplication of time series algorithms to non-time series data. It also highlights the core technology of this invention in handling the correlation between time series and discrete observations—the dynamic time attention mechanism rather than the DTW algorithm. When the correlated object is two track sequences (the same target track captured by different sensors), it is necessary to fuse the dynamic time warping (DTW) distance. First, the dynamic time warping (DTW) algorithm is used to calculate the distance between the two track sequences. The DTW distance is then normalized to Norm_DTW_dist (the smaller the value, the more similar the data). The comprehensive score is then calculated as Score_Comprehensive = w_sim * Sim(V_SAR,V_Track) + w_dtw * (1 - Norm_DTW_dist), where w_sim and w_dtw have weights of w_sim = 0.6 and w_dtw = 0.4.

[0144] (4) Final confidence level assessment:

[0145] Confidence = Sigmoid(Score_comprehensive - threshold) * 100%

[0146] Where Sigmoid(x) = 1 / (1 + exp(-x)), and threshold is a learnable or preset bias / threshold parameter of 0.5. If the association involves track sequences and discrete observations (images), a dynamic temporal attention mechanism (additive attention, attention hidden dimension of 128) can be used to calculate the track context vector C_track_context, and then the cosine similarity between C_track_context and the discrete observation features is calculated as part or all of the Score_synthesis.

[0147] (5) Output of association results: Based on the calculated association confidence (if it is greater than the preset threshold of 70%, the association is considered successful), output the association pair and the association confidence.

[0148] This invention employs a complete technical chain of "preprocessing-feature extraction-feature alignment-association decision," with each step forming a tightly closed-loop system. The data preprocessing stage lays the foundation for subsequent feature extraction, transforming heterogeneous data into standard-format input through targeted processing of different modalities. The feature extraction stage uses a deep neural network architecture to map the preprocessed data into high-quality D-shared-dimensional feature vectors, achieving a transformation from raw observations to a high-dimensional feature space. The feature alignment stage uses a contrastive learning mechanism to map features from different modalities to a unified shared semantic space, ensuring comparability of data from different modalities on the same feature dimension. The association decision stage calculates similarity based on the aligned features and ultimately outputs the target association confidence score.

[0149] Specifically, this invention constructs the entire system around track data, with track features serving as the fundamental element throughout all steps, ensuring consistency in multimodal associations. During the training phase, intramodal contrastive learning and cross-modal feature alignment tasks are collaboratively optimized to enhance feature representation capabilities. In the inference phase, feature fusion and confidence assessment mechanisms achieve a mapping from the feature space to practical applications. This end-to-end design ensures that each step maintains functional independence while forming an organically unified whole.

[0150] II. Track and HRRP Intelligent Correlation Sub-model

[0151] The preprocessing of track data and the extraction of target features from track data are the same as those in the SAR, ISAR, and intelligent track association sub-models, and will not be repeated below.

[0152] Step 1: Data Preprocessing Stage

[0153] 1. HRRP Data Preprocessing

[0154] (1) Format Standardization: HRRP data exists in the form of feature vectors with a strict temporal context semantic structure. To adapt to subsequent model processing, its data format needs to be standardized, converting HRRP data into a unified [time step, 256] two-dimensional tensor structure to ensure time series consistency. Specifically, this includes: ① resampling to the standard time interval through linear interpolation; ② zero-padding frames with fewer than 256 units and truncating frames with more than 256 units.

[0155] (2) Feature Dimension Preservation Transformation: To preserve the contextual semantic structure of HRRP data, a one-dimensional convolutional kernel is used for feature enhancement while maintaining the feature dimension unchanged. Let the input HRRP data sequence (the vector at each time step) be x_t, where t represents the time step index, and the one-dimensional convolutional kernel weight vector be w_conv1d_pre (the weight value of the convolutional kernel at position j).

[0156] y_t_k = Σ_j (x_t_k+j * w_conv1d_pre_j) + b_pre

[0157] Where: j∈{-1,0,1} (because the kernel size m=3), k is the feature dimension index (k∈{0,1,...,255}). This preprocessing convolution is directly applied to the original HRRP data, providing preliminary enhanced input for the 1D-CNN feature extraction in the subsequent step two. It forms a hierarchical relationship with the convolution operations in the subsequent construction of the one-dimensional convolutional neural network (1D-CNN) and bidirectional long short-term memory network (Bi-LSTM), ensuring that the network can more effectively learn the temporal features of the HRRP data.

[0158] (3) Amplitude normalization: The amplitude value of each HRRP distance image is linearly scaled to the interval [0, 1].

[0159] (4) Noise suppression: A one-dimensional moving average filter is used, and the window size is set to 3.

[0160] Step 2: HRRP Target Feature Extraction Stage

[0161] 1. Construction of a One-Dimensional Convolutional Neural Network (1D-CNN) and a Bidirectional Long Short-Term Memory Network (Bi-LSTM)

[0162] (1) Network architecture design: Build a sequence model consisting of multiple one-dimensional convolutional layers, pooling layers and subsequent Bi-LSTM layers.

[0163] (2) 1D-CNN parameters:

[0164] Input: Preprocessed HRRP sequence (N time steps, each time step has a feature dimension of 256).

[0165] One-dimensional convolutional layer 1: 64 convolutional kernels, kernel size 9, stride 1, padding method 'same', activation function is ReLU;

[0166] One-dimensional max pooling layer 1: pooling window size is 2, stride is 2;

[0167] One-dimensional convolutional layer 2: 128 convolutional kernels, kernel size 7, stride 1, padding method 'same', activation function is ReLU;

[0168] One-dimensional max pooling layer 2: pooling window size is 2, stride is 2.

[0169] (3) Bi-LSTM parameters: 1 layer Bi-LSTM, with 128 hidden units per layer (128 forward, 128 backward, and 256 dimensions after splicing).

[0170] 2. Feature extraction operation

[0171] (1) Multi-layer convolution operation: The preprocessed HRRP data is input into a one-dimensional convolutional neural network, and convolution operations are performed using two convolutional layers and K filters. Let the output of the first convolutional layer be y_ik^(1) (the i-th position, the k-th filter), then:

[0172]

[0173] i: represents the position index in the convolution output sequence, with a value range from 0 to N-1 (N is the length of the output sequence), j: represents the relative position within the convolution kernel, with a value range from -4 to 4 (because the convolution kernel size is 9), where j=0 represents the center position of the convolution kernel, x_i+j: represents the feature value of the input HRRP sequence at position i+j, that is, the specific dimension value of the 256-dimensional distance image vector at time step i+j, w_jk^(1): represents the weight value of the k-th convolution kernel in the first layer at relative position j, b_k^(1): represents the weight value of the k-th convolution kernel in the first layer at relative position j. The bias terms of k filters, where K is the number of filters in the first layer, which is 64. Specific example: When calculating the activation value of the filter at position i=10 and the k=5th filter in the output sequence, the center of the convolution kernel is aligned with position 10 of the input sequence. Then, all relative positions j∈{-4,-3,...,0,...,3,4} of the convolution kernel are traversed. For each position, the product of the input feature x_10+j and the corresponding weight w_j5^(1) is calculated. All products are accumulated and then the bias b_5^(1) is added to obtain the output value y_10,5^(1).

[0174] (2) Downsampling and Feature Map Stretching: The feature map (sequence) output from the first convolutional layer is pooled (downsampled) and then input into the second convolutional network. The output of the second convolutional layer is y^(2). Finally, the convolutional feature sequence is input into the Bi-LSTM layer. The output sequence of Bi-LSTM is averaged and pooled across all time steps to obtain a 256-dimensional HRRP sequence representation. This representation is then passed through a fully connected layer FC(256, D_shared) (i.e., FC(256, 512)) and ReLU activation to obtain a high-dimensional feature vector (512-dimensional) containing the target semantic information.

[0175] Step 3: Track and HRRP Multi-Source Data Correlation Stage

[0176] 1. Classifier Construction

[0177] Fully connected layer classifier construction: Construct FC(512,K) -> Softmax classifiers for HRRP features (512-dimensional) and track features (512-dimensional) respectively. K is the predefined number of classes, the same as above.

[0178] 2. Intramodal contrastive learning

[0179] (1) Feature Input and Probability Calculation: The features of the HRRP and track data obtained from the encoder are used as input to the classifier, and a softmax operation is performed on the output of the classifier. Let the classifier output be z, then the softmax calculation formula is:

[0180]

[0181] The formula is explained as above.

[0182] (2) Sample differentiation: Single-modal features are optimized using metrics such as KL divergence. The formula for calculating KL divergence is:

[0183]

[0184] The formula is explained as above.

[0185] 3. Intermodal data association

[0186] (1) Cross-modal feature alignment: The 512-dimensional features of HRRP and track are aligned using a triplet loss function (margin=0.5).

[0187] (2) Initial correlation calculation: Calculate the cosine similarity between the aligned HRRP features and the track features. Let the two modal data feature vectors be x and y, and the cosine similarity calculation formula is:

[0188]

[0189] The formula is explained as above.

[0190] (3) Association Decision and Confidence Output: Based on the cosine similarity results (association is considered if the result is greater than the threshold of 0.70), the results are converted into a confidence score of 0-100%, and the final association confidence score is calculated as follows:

[0191] Confidence = Sigmoid(Score_comprehensive - threshold) * 100%

[0192] Where Sigmoid(x) = 1 / (1 + exp(-x)), and threshold is a learnable or preset bias / threshold parameter of 0.5. If the association involves track sequences and discrete observations (images), a dynamic temporal attention mechanism (additive attention, attention hiding dimension 128 layers) is applied here to calculate the context-weighted representation of the track sequence for the current HRRP image, and then cosine similarity calculation and confidence evaluation are performed.

[0193] III. Intelligent Correlation Sub-model of Flight Track and Visible Light

[0194] The preprocessing of track data and the extraction of target features from track data are the same as those in the SAR, ISAR, and intelligent track association sub-models, and will not be repeated below.

[0195] Step 1: Data Preprocessing Stage

[0196] 1. Visible light image data preprocessing

[0197] (1) Image quality improvement: Denoising the acquired visible light image can be performed using Gaussian filtering, with the following formula:

[0198]

[0199] Where G(x, y) is the weight of the Gaussian filter at a position offset (x, y) relative to the center point; σ is the standard deviation of the Gaussian function, controlling the smoothness. The standard deviation σ is set to 1.0, and the kernel size is set to 5x5.

[0200] Meanwhile, CLAHE (contrast-limited adaptive histogram equalization) enhances image contrast, adjusts the grayscale distribution of the image, and highlights target features, especially in poor lighting conditions or when shadows are present.

[0201] Clip Limit (Contrast Limit Threshold): Used to limit the degree of contrast amplification during histogram equalization. If the count value of a gray level exceeds this limit, the excess will be evenly redistributed to other gray levels. Typical values ​​range from 2.0 to 4.0. In this invention, Clip Limit is set to 3.0. Lower values ​​provide stronger control and prevent excessive noise amplification; higher values ​​allow for greater contrast enhancement.

[0202] Tile Grid Size: The image is divided into multiple non-overlapping rectangular tiles, and histogram equalization is performed independently within each tile. The grid size determines the extent of the local region. Typical values ​​are 8x8 or 16x16. In this invention, the Tile Grid Size is set to (8, 8), meaning the image is divided into 8x8 tiles. Smaller tile sizes better adapt to local contrast variations but may introduce block artifacts; larger tile sizes are closer to global equalization.

[0203] (2) Target detection and cropping: The YOLOv8s (small version) algorithm for target detection is used to locate target objects in the image.

[0204] Training / Fine-tuning: A YOLOv8s model pre-trained on large datasets such as COCO is used, and then fine-tuned on a specific dataset containing ships on the sea surface. The specific steps are as follows:

[0205] 1) Data preparation: Construct a labeled dataset containing 5000+ ships of different categories. Each image contains bounding box annotations of the ship targets. The dataset is divided into training set (80%), validation set (15%) and test set (5%).

[0206] 2) Model initialization: Load COCO pre-trained weights as initial parameters;

[0207] 3) Training configuration: The transfer learning strategy is adopted, freezing the first half of the model layers and only fine-tuning the second half of the layers; the learning rate is set to 1e-4, the batch size to 16, and the number of training epochs to 50.

[0208] 4) Data augmentation: Apply augmentation strategies such as random rotation (±15°), horizontal flipping, and brightness and contrast adjustment to improve the model's generalization ability;

[0209] 5) Performance optimization: The confidence threshold (finally set to 0.4) and NMS IoU threshold (finally set to 0.5) were dynamically adjusted based on the performance of the validation set.

[0210] 6) The input image size was uniformly scaled to 640x640. After fine-tuning, the model achieved an mAP@0.5 of 92.3% on the ship detection task, which is 18.5 percentage points higher than the original COCO pre-trained model.

[0211] After fine-tuning the YOLOv8s model, the input image size was uniformly scaled to 640x640, the confidence threshold was set to 0.4, and the IoU (Intersection over Union) threshold for NMS (Non-Maximum Suppression) was set to 0.5.

[0212] Cropping: Based on the target bounding box coordinates output by YOLOv8, the smallest bounding rectangle region containing the target is cropped from the original image, and the boundary can be extended outward by 10% as context information. The cropped region is then uniformly scaled to the input size of 224x224 pixels for the subsequent CNN model (ResNet-50).

[0213] Step 2: Visible Light Image Target Feature Extraction Stage

[0214] 1. Convolutional Neural Network Construction (ResNet-ViT Two-Stream Network)

[0215] (1) Model architecture design: The same ResNet-ViT dual-stream network architecture as SAR / ISAR image feature extraction is adopted, and the specific structure is as follows.

[0216] (2) ResNet Stream: ResNet-50 is used as the backbone network. Input image size is 224×224×3 (color visible light image). First convolutional layer: 64 7×7 convolutional kernels, stride 2, padding 3, ReLU activation function, followed by batch normalization (BatchNorm). Max pooling layer: 3×3 window, stride 2. ResNet modules (from conv2_x to conv5_x): standard Bottleneck structure, with 256, 512, 1024, and 2048 output channels at each stage, respectively. Each convolutional layer is followed by batch normalization and ReLU activation function. Output is a 2048-dimensional feature vector after global average pooling (GAP).

[0217] (3) ViT Stream: The ViT-Base / 16 architecture is adopted. The input image size is 224×224×3. The image patch size is 16×16, the embedding dimension is 768, the number of transformer encoder layers is 12, the number of attention heads is 12, the size of the multilayer perceptron (the size of the intermediate layer of the feedforward network) is 3072, and the dropout rate is 0.1. The output [CLS] is a 768-dimensional feature vector labeled in the output of the last transformer encoder layer. (3) Enhancement mechanism:

[0218] Spatial Global Relationship Attention Module: The self-attention mechanism of ViT flow itself realizes the capture of global spatial relationships.

[0219] Channel Global Attention Module: A Squeeze-and-Excitation (SE) Block is embedded after the output feature map of each residual stage of ResNet-50. The SE Block obtains channel descriptors through a Squeeze operation (global average pooling), then learns channel weights through an Excitation operation (two fully connected layers: the first FC reduces dimensionality with a reduction ratio r of 16 and ReLU activation; the second FC increases dimensionality with Sigmoid activation), and finally rescales the original feature map.

[0220] Local partitioning strategy: ViT's patch partitioning is a type of local partitioning.

[0221] 2. Feature extraction operation

[0222] (1) Input the preprocessed visible light image into the ResNet-50 stream and the ViT stream, respectively. In the convolutional layer, the image features are extracted through convolution operations, using the following formula:

[0223]

[0224] The formula explanation is consistent with that in the SAR, ISAR, and intelligent trajectory association sub-models.

[0225] (2) Reduce the feature dimension through max pooling operation. The max pooling formula is:

[0226]

[0227] The formula explanation is consistent with that in the SAR, ISAR, and intelligent trajectory association sub-models.

[0228] (2) Feature fusion: The 2048-dimensional vector output from the ResNet-50 stream is concatenated with the 768-dimensional vector output from the ViT stream to obtain a 2816-dimensional fused feature. This fused feature is then passed through a fully connected layer FC(2816, D_shared) (i.e., FC(2816, 512)) and ReLU activation to obtain the final visible light image target feature vector (512-dimensional).

[0229] Step 3: Correlation of Track and Visible Light Multi-Source Data

[0230] 1. Classifier Construction

[0231] (1) Construction of fully connected layer classifiers: Construct FC(512, K) -> Softmax classifiers for visible light image features (512-dimensional) and track features (512-dimensional) respectively. K is the total number of predefined categories. The formula interpretation is the same as in the SAR, ISAR, and track intelligent association sub-models.

[0232] 2. Intramodal contrastive learning (with optional auxiliary loss)

[0233] (1) Feature Input and Probability Calculation: The features of the visible light image and track data obtained from the encoder are used as the input to their respective classifiers, and a softmax operation is performed on the output of the classifiers. Let the classifier output be z, then the softmax calculation formula is:

[0234] (for k=1~K)

[0235] The formula explanation is consistent with that in the SAR, ISAR, and intelligent trajectory association sub-models.

[0236] (2) Estimate the posterior distribution of the samples and measure the distance between it and the true posterior distribution of the samples using the cross-entropy loss function or KL divergence, the formula of which is:

[0237]

[0238] The formula explanation is consistent with that in the SAR, ISAR, and intelligent trajectory association sub-models.

[0239] 3. Intermodal data association

[0240] (1) Cross-modal feature alignment: The 512-dimensional features of the visible light image and the track are aligned using the triple loss function (margin=0.5).

[0241] (2) Initial correlation calculation: Calculate the cosine similarity between the aligned visible light image features and the track features, and use the cosine similarity:

[0242]

[0243] The formula explanation is consistent with that in the SAR, ISAR, and intelligent trajectory association sub-models.

[0244] (3) Association Decision and Confidence Output: Based on the cosine similarity results (association is indicated if the similarity is greater than the threshold of 0.70), the results are converted into a confidence score of 0-100%. The final association confidence score is calculated (simplified version):

[0245] Confidence = Sigmoid(Score_comprehensive - threshold) * 100%

[0246] Where Sigmoid(x) = 1 / (1 + exp(-x)), and threshold is a learnable or preset bias / threshold parameter of 0.5. If the association involves track sequences and discrete observations (images), a dynamic temporal attention mechanism (additive attention, attention hidden dimension 128) is applied here to calculate the context-weighted representation of the track sequence for the current visible light image, and then cosine similarity calculation and confidence evaluation are performed.

[0247] IV. Track and Infrared Intelligent Correlation Sub-model

[0248] The preprocessing of track data and the extraction of target features from track data are the same as those in the SAR, ISAR, and intelligent track association sub-models, and will not be repeated below.

[0249] Step 1: Data Preprocessing Stage

[0250] 1. Infrared image data preprocessing

[0251] (1) Radiometric Correction: Since infrared images are easily affected by factors such as ambient temperature and detector performance, radiometric correction is required to convert the digital signal output by the detector into the true radiance value and eliminate the interference of these factors. A blackbody-based radiometric correction method is adopted, based on Planck's law, with the following formula:

[0252]

[0253] Where B(λ,T) is the radiance of the blackbody at wavelength λ and temperature T; h is Planck's constant (6.626 x 10⁻³). 4 J·s); c is the speed of light (3.0 x 10⁻⁶). 8 (m / s); k is the Boltzmann constant (1.38 x 10⁻²³ J / K). This step relies on accurate sensor calibration parameters.

[0254] (2) Noise Reduction and Enhancement: Median filtering is used to denoise the infrared image, removing salt-and-pepper noise and other noise. The median filter window size is set to 5x5. Simultaneously, CLAHE is used to enhance the image, expanding its dynamic range, improving the contrast between the target and background, and highlighting target features. In this invention, the Clip Limit is 2.0 (the noise in infrared images may differ from that in visible light; a lower clip limit is more suitable to avoid amplifying thermal noise). The Tile Grid Size is (8,8).

[0255] (3) Target segmentation: Otsu's adaptive threshold segmentation algorithm is used to automatically determine the optimal segmentation threshold based on the grayscale (temperature) difference between the target and the background in the infrared image, thus segmenting the target from the background. The cropped target area is then uniformly scaled to the input size of 224x224 pixels for the subsequent CNN model.

[0256] Step 2: Infrared Image Target Feature Extraction Stage

[0257] 1. Convolutional Neural Network Construction (ResNet-ViT Two-Stream Network)

[0258] (1) Construct a ResNet-ViT dual-stream network specifically designed for the characteristics of infrared images. Infrared images are usually single-channel.

[0259] (2) ResNet stream: ResNet-50 is used. The input image size is 224x224x1, which is different from the previous one. The other parameters are the same as the ResNet-50 part of the SAR, ISAR and track intelligent association sub-model. The output is a 2048-dimensional vector after GAP.

[0260] (3) ViT stream: Using ViT-Base / 16, the input image size is 224x224x1. The processing method is: ① Copy the single-channel infrared image into a three-channel image (the three channels have the same value); ② Or use a ViT-Base / 16 model pre-trained on a single-channel infrared image dataset. "Specific pre-training" refers to model pre-training on a dedicated dataset containing a large number of infrared images to adapt to the spectral and noise characteristics of infrared images. The remaining parameters are the same as the ViT-Base / 16 part of the SAR, ISAR, and track intelligent association sub-models, and the output is a 768-dimensional vector of [CLS]token.

[0261] (4) Enhancement mechanism:

[0262] Spatial Global Relationship Attention Module: The self-attention mechanism of ViT flow itself realizes the capture of global spatial relationships.

[0263] Channel Global Attention Module: A Squeeze-and-Excitation (SE) Block is embedded after the output feature map of each residual stage of ResNet-50. The SE Block obtains channel descriptors through a Squeeze operation (global average pooling), then learns channel weights through an Excitation operation (two fully connected layers: the first FC reduces dimensionality with a reduction ratio r of 16 and ReLU activation; the second FC increases dimensionality with Sigmoid activation), and finally rescales the original feature map.

[0264] Local partitioning strategy: ViT's patch partitioning is a type of local partitioning.

[0265] 2. Feature extraction operation

[0266] (1) Input the preprocessed infrared images into the ResNet-50 stream and the ViT stream, respectively. The convolutional layer extracts image features through convolution operations, using the following formula:

[0267]

[0268] The formula is explained as above.

[0269] (2) Max pooling through the pooling layer:

[0270]

[0271] The formula is explained as above.

[0272] (3) Feature fusion: The 2048-dimensional vector output from the ResNet-50 stream is concatenated with the 768-dimensional vector output from the ViT stream to obtain a 2816-dimensional fused feature. Then, it is passed through a fully connected layer FC(2816, D_shared) (i.e., FC(2816, 512)) and connected to ReLU to obtain the final infrared image target feature vector (512-dimensional).

[0273] Step 3: Correlation of Track and Infrared Multi-Source Data

[0274] 1. Classifier Construction

[0275] Fully connected layer classifier construction: Construct FC(512, K) -> Softmax classifiers for infrared image features (512-dimensional) and track features (512-dimensional) respectively. K is the predefined total number of categories.

[0276] 2. Intramodal contrastive learning (with optional auxiliary loss)

[0277] (1) Feature Input and Probability Calculation: The features of the infrared image and track data obtained from the encoder are used as the input to their respective classifiers, and a softmax operation is performed on the output of the classifiers. Let the classifier output be z, then the softmax calculation formula is:

[0278] (for k=1~K)

[0279] The formula is explained as above.

[0280] (2) Sample Differentiation: Single-modal features are optimized using metrics such as KL divergence. KL divergence is utilized as follows:

[0281]

[0282] The formula is explained as above.

[0283] 3. Intermodal data association

[0284] (1) Cross-modal feature alignment: The 512-dimensional features of infrared images and tracks are aligned using a triplet loss function (margin=0.5).

[0285] (2) Initial correlation calculation: Calculate the cosine similarity between the aligned infrared image features and the track features. Let the two modal data feature vectors be x and y (both 512-dimensional), and the cosine similarity calculation formula is:

[0286]

[0287] To measure the strength of their correlation, · represents the vector dot product, and ||x|| and ||y|| represent the L2 norm of the vectors, where x is the infrared image feature vector and y is the track feature vector.

[0288] (3) Association Decision and Confidence Output: Based on the cosine similarity results (association is indicated if the similarity is greater than the threshold of 0.70), the results are converted into a confidence score of 0-100%. The final association confidence score is calculated (simplified version):

[0289] Confidence = Sigmoid(Score_comprehensive - threshold) * 100%

[0290] Where Sigmoid(x) = 1 / (1 + exp(-x)), and threshold is a learnable or preset bias / threshold parameter of 0.5.

[0291] If the association involves track sequences and discrete observations (images), a dynamic temporal attention mechanism (additive attention, attention hiding dimension 128) is applied here to calculate the context-weighted representation of the track sequence for the current infrared image, and then cosine similarity calculation and confidence assessment are performed.

[0292] V. For example Figure 2 As shown, the intelligent association sub-model of trajectory, text, and voice

[0293] Step 1: Data Collection

[0294] The system collects track data, descriptive text, and descriptive audio data of maritime targets. Track data can be obtained from equipment such as AIS (Automatic Identification System) and radar tracking systems. Text data consists of written descriptions of the maritime target's status, behavior, intentions, and location. Audio data comprises relevant voice broadcasts or dialogue records.

[0295] Step 2: Track, text, and speech preprocessing

[0296] 1. Track data preprocessing

[0297] (1) Data cleaning: Remove outliers from the track data (by setting reasonable thresholds for speed and acceleration for detection and elimination) and missing values ​​(by filling in missing values ​​through linear interpolation). If the speed at a certain moment in the track data is obviously unreasonable and extremely high, it can be regarded as an outlier and processed.

[0298] (2) Normalization: The cleaned track data is converted into features with good numerical characteristics that are suitable for input to deep learning models.

[0299] a. Latitude and longitude feature processing:

[0300] The original latitude and longitude coordinates are absolute geographical locations, with a large numerical range, and direct input into the model may not yield good results. Therefore, they are converted to relative coordinates: the starting point of the track segment is selected as the reference point (lon0, lat0), and the latitude and longitude (lonᵢ, latᵢ) of all subsequent points are converted to offsets relative to this reference point: Δlonᵢ = lonᵢ - lon0, Δlatᵢ = latᵢ - lat0. This conversion helps the model learn relative motion patterns and reduces the numerical range.

[0301] b. Speed ​​and heading characteristics:

[0302] Speed ​​Over Ground (SOG): Used directly, in units of knots or meters per second (m / s).

[0303] Course Over Ground (COG): The raw course over ground is typically an angle value between 0 and 360 degrees. Directly using angle values ​​may result in numerical discontinuities. Therefore, the course over ground is converted to its cosine and sine components.

[0304] cos_cog = cos(COG * π / 180)

[0305] sin_cog = sin(COG * π / 180)

[0306] At this point, an angle value is converted into two consecutive values ​​in the range of [-1, 1], which better preserves the periodicity and continuity of the angle.

[0307] (3) Data augmentation: Increase the diversity of track data by randomly adding a small amount of Gaussian noise (standard deviation is 2% of the original feature standard deviation) to the track point coordinates or features, and slightly changing the timestamp of the track point (time jitter).

[0308] 2. Preprocessing of Text Data

[0309] (1)Word Segmentation: Use natural language processing tools to perform word segmentation on the text. For Chinese text, the exact mode of the Jieba segmentation library is used; for English text, the Tokenizer of spaCy is used.

[0310] (2)Removing Stop Words: Remove common stop words in the text (such as "de", "shi", "zai" in Chinese, and "a", "the", "is" in English), and use a predefined standard stop word list.

[0311] (3)Transformer Input Serialization:

[0312] a. Tokenization: Use the WordPieceTokenizer that is exactly the same as the BERT model in the subsequent text encoder.

[0313] b. Adding Special Tokens: Add the [CLS] token at the beginning of the sequence and the [SEP] token at the end of the sequence (or between sentences).

[0314] c. Converting to ID Sequence: Pad all sequences to a unified maximum length of 128 tokens, use the padding token ID specified by the model (usually 0), and truncate the sequences that exceed the maximum length at the tail.

[0315] d. Generating Attention Mask and Segment IDs (for single-sentence input, Segment IDs are usually all 0).

[0316] 3. Preprocessing of Speech Data

[0317] (1)Speech Recognition: Use the Whisper model of OpenAI's speech recognition technology (select the medium version) to convert the original speech waveform data into text.

[0318] (2)Perform exactly the same operations as the above text data preprocessing on the converted text, including word segmentation, removing stop words, and Transformer input serialization.

[0319] Step Three: Build the Model

[0320] Build a large pre-trained model for contrastive language trajectory, which consists of a trajectory encoder and a text encoder, both of which output features with D_shared = 512 dimensions.

[0321] (1) Trajectory Encoder: A model that combines Bi-LSTM and spatio-temporal Transformer

[0322] a. Bi-LSTM: A 1-layer Bi-LSTM with 256 hidden units per layer (Bi-LSTM output is 512-dimensional). The activation function is tanh, the recurrent activation function is sigmoid, and the recurrent dropout is set to 0.2.

[0323] b. Spatiotemporal Transformer: 2 Transformer Encoder Layers, model hidden dimension (d_model) is 512 (matching the output dimension of Bi-LSTM), number of multi-head self-attention heads (num_heads) is 8 (512 / 8=6 dimensions per head), feedforward network (FFN) intermediate layer dimension (dim_feedforward) is 1024 (2 * d_model), dropout rate is 0.1, and learnable 1D temporal location encoding is added before the input.

[0324] c. Output: Average pooling is performed on the representations of all time steps of the Transformer encoder output sequence to obtain a 512-dimensional track sequence representation. This is the D-shared dimensional track feature.

[0325] (2) Text Encoder: A Transformer-based Pre-trained Language Model

[0326] a. Model selection: The BERT-Base (bert-base-chinese) model was selected.

[0327] b. BERT-Base parameters: 12-layer Transformer Encoder, hidden layer dimension (d_model): 768, number of multi-head self-attention heads (num_heads): 12, feedforward network (FFN) intermediate layer dimension (dim_feedforward): 3072.

[0328] c. Output: Take the 768-dimensional vector corresponding to the [CLS] token output by the last Transformer Encoder layer. Then pass it through a fully connected layer FC(768, D_shared) (i.e., FC(768, 512)) and follow with ReLU activation to obtain the final text / speech features (512 dimensions).

[0329] d. Hierarchical attention mechanism: For more detailed text representation, on the token sequence output by BERT, the token representation of each sentence (or identified by [SEP] if the text contains multiple sentences) is first averaged to obtain sentence vectors. Then, these sentence vectors are aggregated through a 1-layer, 4-head self-attention layer with the same hidden dimension as the sentence vector dimension. The output is then projected to the D_shared dimension.

[0330] Step 4: Comparative Learning

[0331] The contrastive language trajectory pre-training large model trains the representations of trajectories and text through a contrastive learning mechanism. This is the core step in the model training phase. After training, the model parameters are fixed for subsequent practical association tasks. This phase specifically includes:

[0332] (1) Training Sample Construction: Each training batch (batch_size is 32) contains a batch of track samples and a batch of text samples. For each track sample A in the batch, its corresponding original text description B_text is its positive sample. All other text samples in the batch (or those selected through an online hard negative sample mining strategy) can be regarded as its negative samples. The reverse is also true.

[0333] (2) Similarity score calculation: In each training step, the cosine similarity between the track feature vector and the text feature vector after being projected into the D_shared dimension shared embedding space is calculated. This cosine similarity is completely consistent with the sim calculation method used in the subsequent association stage, but in this training stage, the similarity value is used to optimize the model parameters rather than directly determining the association result.

[0334] (3) Distinction between training and application phases: This contrastive learning phase is only used during model training. Its goal is to learn an effective feature mapping function so that related multimodal data are close to each other in the shared feature space. After training is completed, this contrastive learning mechanism is no longer used. Instead, the pre-trained track encoder and text encoder are used for feature extraction and association judgment.

[0335] Step 5: Loss Function

[0336] (1) Training objective: For each training sample (anchor), the model calculates its similarity score with positive samples (another modality of data that is associated with it) and negative samples (another modality of data that is not associated with it), with the goal of making the similarity score of positive samples higher than that of negative samples.

[0337] (2) The goal of the model is to make the similarity score of positive samples higher than that of negative samples. A triplet loss function is used (assuming sim is cosine similarity):

[0338] L = max(0, margin – sim(A, B_pos) + sim(A, B_neg))

[0339] Where A: is the feature vector of the anchor point (e.g., track) in the D_shared dimension shared embedding space. B_pos: is the feature vector of the positive sample (e.g., text related to A) in the D_shared dimension shared embedding space. B_neg: is the feature vector of the negative sample (e.g., text unrelated to A) in the D_shared dimension shared embedding space. sim(X, Y): represents the cosine similarity between X and Y. margin: a preset boundary value set to 0.2.

[0340] Meanwhile, this triplet loss function is equivalent to the above statement, except that it uses a different similarity metric. The above is in Euclidean distance form, while this formula is in cosine similarity form.

[0341] Two-way loss calculation: Calculate the loss when the track is the anchor point and the text is the positive / negative sample, and calculate the loss when the text is the anchor point and the track is the positive / negative sample. The two losses are added together or averaged to obtain the total loss. This loss is only used for model parameter optimization during the training phase. After training, the model parameters are fixed and this loss is no longer calculated.

[0342] Step Six: Backpropagation and Optimization

[0343] This training method is applicable to all intelligent association sub-models in this invention, including SAR, ISAR, and track intelligent association sub-models; track and HRRP intelligent association sub-models; track and visible light intelligent association sub-models; track and infrared intelligent association sub-models; and track, text, and speech intelligent association sub-models. The sub-models differ only in their specific network architecture and data preprocessing, but the training framework and optimization strategy remain consistent.

[0344] (1) By calculating the gradient of the loss function, the gradient is propagated back to the parameters of the model using the backpropagation algorithm.

[0345] (2) The AdamW optimizer is used to update the model parameters. The parameters of AdamW are: learning rate: 2e-5, β1: 0.9, β2: 0.999, ε: 1e-8, weight decay: 0.01. A linear learning rate scheduling strategy is adopted, which involves linear warm-up (accounting for 10% of the total training steps) followed by linear decay.

[0346] (3) During training, all sub-models use a contrastive learning framework to optimize feature alignment capabilities. For image-related sub-models (SAR, ISAR, visible light, infrared), triplet loss function is used to align image features and track features; for HRRP sub-models, triplet loss is used to align HRRP features and track features; for text / speech sub-models, triplet loss is used to align text / speech features and track features.

[0347] (4) This training process will iterate through multiple training steps (5-10 epochs, depending on the dataset size and convergence speed) until the model's contrastive loss on the validation set converges or a predefined stopping condition is met (early stopping). Each sub-model may be fine-tuned in hyperparameters such as learning rate and number of training epochs according to the characteristics of its own task, but the overall training framework remains consistent.

[0348] Step 7: Correlation of Track, Text / Speech Data from Multiple Sources

[0349] After the models (track encoder and text encoder) have converged through step six, this step describes how to use the trained models to correlate actual tracks with text / speech data.

[0350] 1. Feature extraction and projection:

[0351] (1) For new track data to be associated, after the same track data preprocessing in step two, it is input into the trained track encoder (a model combining Bi-LSTM and spatiotemporal Transformer) to extract its feature vector V_track_aligned in the D_shared dimension shared semantic space.

[0352] (2) For new text data to be associated and speech data after speech recognition and text preprocessing, input them into the trained text encoder and extract their feature vector V_text_aligned in the D_shared dimension shared semantic space.

[0353] 2. Intermodal data association:

[0354] (1) Initial correlation calculation: In the aligned shared embedding space, data correlation is performed by measuring the similarity between features of different modalities. The cosine similarity between the track feature vector V_track_aligned and the text / speech feature vector V_text_aligned obtained in step one is calculated:

[0355] Sim(V_track_aligned,V_text_aligned)=(V_track_aligned·V_text_aligned) / (||V_track_aligned|| * ||V_text_aligned||)

[0356] This similarity score is the initial measure of association strength, where · represents the vector dot product, and ||V_track_aligned|| and ||V_text_aligned|| represent the L2 norm of the vectors.

[0357] 3. Association Decision Making and Confidence Output:

[0358] (1) Calculation of final association confidence: The cosine similarity Sim(V_track_aligned, V_text_aligned) calculated in the previous step (as a simplified form of Score_fused before fusion) is converted into a standardized final association confidence of 0-100% using the Sigmoid function:

[0359] Confidence_final = Sigmoid( Sim(V_track_aligned, V_text_aligned) -threshold ) * 100%

[0360] Where Sigmoid(x) = 1 / (1 + exp(-x)), and threshold is a learnable or preset bias / threshold parameter set to 0.5.

[0361] If the input track is a sequence, and the text / speech features are single descriptions of the entire sequence, the associations here have already been implicitly considered for the entire sequence through the encoder (specifically, the pooling of the track encoder). If more detailed alignment of specific phrases in the text with specific segments of the track is required, then more complex attention mechanisms need to be introduced in the model design or subsequent processing. For the target-level associations of this sub-model, the above methods are sufficient.

[0362] (2) Output of association results: Based on the calculated association confidence (if it is greater than the preset threshold of 0.70, the association is considered successful), output the associated track and text / speech pair and the association confidence.

[0363] This invention proposes a deep learning-based method for multimodal data feature alignment and target association. This method designs customized neural network encodings for various sensor data (including images, time-series signals, text, and speech) to effectively map and align heterogeneous features from SAR, ISAR, visible / infrared images, HRRP, tracks, and text / speech sources into a unified shared semantic space using a contrastive learning framework (such as optimized triplet loss). Finally, the inherent correlation and confidence levels between different observation data are determined by calculating the cosine similarity between aligned feature vectors and using optional dynamic matching strategies. This effectively solves the challenge of multi-source heterogeneous information fusion in complex scenarios, significantly improving the accuracy and intelligence level of target association. Compared to traditional methods, this invention innovatively adopts the latest fine-tuned image-text large model technology, with an autonomous and controllable algorithm. It can achieve retrieval based on target attributes, needs, and speech semantics, effectively improving the accuracy of multimodal information fusion and recognition, and has broad application prospects in fields such as civilian marine monitoring.

[0364] Those skilled in the art will recognize that the described embodiments are intended to help readers understand the principles of the invention and should be understood as not limiting the scope of protection of the invention to the described embodiments. Various modifications and variations can be made to the invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the invention should be included within the scope of the claims of the invention.

Claims

1. A target association method based on multi-modal data feature alignment, characterized in that, The method comprises the following steps: Step 1, multi-modal data preprocessing: the input synthetic aperture radar image, inverse synthetic aperture radar image, visible light image, infrared image, high-resolution range image, track data, text and speech data are respectively preprocessed; Step 2, deep feature extraction: a hierarchical or fusion network structure is constructed to extract deep feature vectors of each mode; Step 3, cross-modal feature alignment: different modal features are aligned in a shared semantic space through a contrastive learning framework, specifically: Calculate the value of the triplet loss function: L_triplet = max(0, ||F a - F p ||² - ||F a - F n ||² + margin) In the formula, F a is a track feature vector, F p is a target other modal feature vector corresponding to the track feature, F n is a feature vector of other targets, and margin is a preset boundary value. According to the value of the triplet loss function, the hierarchical or fusion network structure is trained until the value of the triplet loss function converges, and the current hierarchical or fusion network structure is saved; Step 4, multi-modal correlation decision: in the aligned shared feature space, the hierarchical strategy is used to realize the correlation of data features at the level, specifically: Calculate the cosine similarity between the deep feature vectors of the two different modes: Where, · represents vector dot product, ||x|| and ||y|| represent L2 norm of vector; Step 5, output the correlation confidence between the targets, including: Convert the correlation measure of the cosine similarity into a standardized confidence score: 0-100%; In a paired mode, clearly display the correlated targets and their identification types, and complete the target correlation method based on multi-modal data feature alignment. 2.The target association method based on multi-modal data feature alignment according to claim 1, characterized in that, The specific way of step 1 is: For synthetic aperture radar images, inverse synthetic aperture radar images, visible light images, and infrared images, the preprocessing includes grayscale processing, basic noise smoothing, and contrast adjustment; For high-resolution range images and track data, the preprocessing includes extracting the latitude, longitude, speed, and heading space-time features of the track, and uniformly using dynamic time warping algorithm for sequence alignment; the high-resolution range image data is amplitude normalized and noise suppressed; For text and speech data, the preprocessing includes processing speech data through speech-to-text technology, and then performing semantic word segmentation and feature representation on the text data.

3. The target correlation method based on multi-modal data feature alignment according to claim 2, characterized in that: The preprocessing of the synthetic aperture radar image includes grayscale based on the weighted average method and speckle noise suppression using Gaussian smoothing filter; The preprocessing of the inverse synthetic aperture radar image includes motion compensation using phase gradient autofocus algorithm to correct phase error, and contrast stretching to enhance target features; The preprocessing of the visible light image includes noise suppression using Gaussian filter, contrast enhancement using histogram equalization, and target accurate positioning and region of interest cropping using deep learning target detection algorithm; The preprocessing of the infrared image includes radiation calibration and correction according to Planck's law, noise reduction using median filter, contrast enhancement using histogram equalization, and initial target segmentation based on adaptive threshold method or edge detection; The preprocessing of the track data includes time and space alignment using dynamic time warping algorithm, wherein the distance measurement method is the Euclidean distance based on multi-dimensional features, and the window constraint condition is Sakoe-Chiba; Preprocessing of text data: including context-aware word segmentation and input embedding sequence construction for feature extraction model to fully utilize the semantic representation capability of the model. 4.The target association method based on multi-modal data feature alignment according to claim 1, characterized in that, The specific way of step 2 is: For synthetic aperture radar images, inverse synthetic aperture radar images, visible light images, and infrared images: modalities: a two-dimensional convolutional neural network architecture based on the combination of residual network and Transformer is used to extract the spatial structure and global dependence features of the image; For high-resolution range image modalities: a sequence model combining one-dimensional convolutional neural network and long short-term memory network or its variants is used to capture the time series features of the range image; For track modalities: a model combining bidirectional long short-term memory network and spatio-temporal Transformer is used to extract deep spatio-temporal dynamic features of the target motion trajectory; For text / speech modalities: a pre-trained language model based on Transformer is used to extract deep context semantic features of the text, and the prosodic features extracted from the speech are fused. 5.The target association method based on multi-modal data feature alignment according to claim 1, characterized in that, The specific way of step 3 is: The image modalities uniformly adopt a double-flow network design: the ResNet-50 network extracts local fine features, and the VisionTransformer network captures global context dependence features; The deep feature extraction of text and speech modalities adopts a hierarchical attention mechanism to realize the aggregation and representation of word-level, sentence-level, and paragraph-level information based on the BERT model. 6.The target association method based on multi-modal data feature alignment according to claim 1, characterized in that, Cross-modal feature alignment includes one or more of the following mechanisms to enhance the expression and alignment effect of image features: (1) Spatial global relationship attention module: used to capture the global spatial context relationship within the image features; (2) Channel global attention module: used to dynamically adjust the weights of each channel of the image features to enhance the key channel information. 7.The target association method based on multi-modal data feature alignment according to claim 1, characterized in that, Multi-modal correlation decision supports one or more of the following strategies: (1) Multi-scale feature fusion: effectively concatenating or fusing the global or regional features output from the ResNet of the image modalities double-flow network and the global block features output from the ViT; (2) Dynamic temporal attention mechanism: used to align and focus on the key time segments in the modal data with different timestamps or asynchronous sampling during correlation decision.

Images