A multimodal semantic encoding method based on inter-modal correlation strength adjustment

Abstract

This invention provides a multimodal semantic coding method based on modal correlation strength adjustment, belonging to the field of semantic communication technology. This invention utilizes radar and a camera to detect and capture targets, acquiring radar echo signals and RGB images of the targets; a multi-task prediction model is constructed to predict the target's motion state. This invention calculates a correlation strength tensor and classifies targets into high and low correlation categories based on the correlation strength tensor. In low-correlation scenarios with insufficient multimodal consistency or uneven modal quality, a restricted fusion strategy is implemented to suppress interference from unreliable modalities on the fusion results. Furthermore, the correlation strength tensor is introduced into the semantic coding stage of a large language model, participating in the coding process as a semantic embedding modulation signal. Amplitude modulation of the fused semantic embedding using the correlation strength tensor enhances the multimodal semantics of high-correlation samples during coding, while the semantics of low-correlation samples are conservatively processed.

Description

Technical Field

[0001] This invention belongs to the field of semantic communication technology, and in particular relates to a multimodal semantic coding method based on the adjustment of intermodal correlation strength. Background Technology

[0002] With the rapid development of autonomous driving, intelligent monitoring, and drone collaboration, single sensors can no longer meet the perception needs of complex scenarios, making multi-sensor fusion perception the mainstream technology. This typically requires encoding and transmitting multimodal perception data in wireless networks or bandwidth-constrained environments to support downstream tasks. Therefore, semantic encoding technology for multimodal data has become an important technical means to achieve perception and communication collaboration. Its core lies in reducing transmission overhead while maintaining the integrity and consistency of multimodal semantic information as much as possible.

[0003] In existing multimodal semantic communication systems, typical schemes employ fixed-structure multimodal fusion and encoding strategies. In these schemes, data from different modalities are first processed separately by feature extraction networks to obtain semantic features. These features are then fused using feature concatenation and a static attention mechanism to form a unified multimodal semantic representation. Finally, a fixed-parameter encoder encodes and transmits the fused semantic features. While this approach works well under conditions of high modal semantic consistency and weak environmental interference, in complex real-world scenarios, when the semantic quality of some modalities deteriorates due to factors such as occlusion, lighting changes, or electromagnetic interference, continuing to fuse and encode using a fixed method can easily introduce noise interference from low-quality modalities, thus affecting overall communication efficiency and downstream task performance. Summary of the Invention

[0004] To address the common problems in existing multimodal semantic coding methods, such as difficulty in perceiving changes in multimodal semantic consistency, fixed coding strategies, and unreasonable allocation of coding resources, this invention aims to provide a multimodal semantic coding method based on the adjustment of intermodal correlation strength. By introducing the characterization of the degree of multimodal semantic consistency and its changes during the coding process, the coding process is adjusted according to the differences in the effectiveness of multimodal semantic information and the characteristics of information overlap in different scenarios. This improves semantic coding efficiency and enhances the robustness of the system in complex environments while ensuring the reliable transmission of key semantic information.

[0005] This invention provides a multimodal semantic coding method based on intermodal correlation strength adjustment, comprising the following steps:

[0006] The radar and camera are used to detect and photograph the target, respectively, to obtain radar echo signals and RGB images of the target;

[0007] A multi-task prediction model is constructed, including a multimodal semantic feature extraction module, an association strength quantification module, an association-aware multimodal semantic fusion module, an association-aware LLM semantic encoding module, and a multi-task collaborative semantic decoding module.

[0008] The radar echo signal and the target's RGB image are input into a multi-task prediction model to predict the target's motion state; the target motion state includes at least the target distance, target speed, and target motion angle.

[0009] Among them, the multimodal semantic feature extraction module is used to extract semantic features from radar echo signals and RGB images to obtain radar semantic feature vectors and visual semantic feature vectors;

[0010] The correlation strength quantization module is used to quantize the intensity of radar semantic feature vectors and visual semantic feature vectors to obtain the correlation strength tensor, and to divide the radar semantic feature vectors and visual semantic feature vectors into different correlation scenarios.

[0011] The association-aware multimodal semantic fusion module is used to perform weighted fusion of radar semantic feature vectors and visual semantic feature vectors according to the associated scene to obtain fused semantic features;

[0012] The association-aware LLM semantic coding module is used to perform semantic mapping and encoding on fused semantic features to generate semantic coding results;

[0013] The multi-task collaborative semantic decoding module is used to decode the semantic encoding results and generate target motion state prediction results.

[0014] Furthermore, the multimodal semantic feature extraction module includes a radar signal semantic extractor and an image semantic extractor;

[0015] The semantic feature vector of radar echo signal is obtained by extracting semantic features from radar signal semantic extractor. The specific method is as follows:

[0016] The radar signal semantic extractor includes a complex-valued convolutional network, which consists of multiple complex-valued convolutional layers.

[0017] The radar echo signal input to the radar signal semantic extractor is split into real and imaginary parts. The split real and imaginary parts are input into a complex-valued convolutional network, and after passing through multiple complex-valued convolutional layers, a complex-valued feature representation is obtained. The complex-valued feature representation is downsampled to obtain a complex-valued temporal semantic representation of the radar echo signal. The real and imaginary parts of the complex-valued temporal semantic representation are concatenated along the feature dimension to obtain a real-valued semantic feature vector. The real-valued semantic feature vector is mapped to a unified low-dimensional semantic space through a learnable fully connected layer and normalized to obtain the radar semantic feature vector output by the radar signal semantic extractor.

[0018] The image semantic extractor is used to extract semantic features from RGB images to obtain visual semantic feature vectors. The specific method is as follows:

[0019] The RGB image is scaled and pixel normalized and then input into the image semantic extractor. The RGB image input into the image semantic extractor is divided into several non-overlapping image blocks according to a preset size. Each image block is unfolded into a one-dimensional vector along the spatial and channel dimensions. The unfolded one-dimensional vector is linearly mapped through a learnable fully connected layer to obtain the semantic feature vector of the image block. Based on the semantic feature vectors of all image blocks, an image semantic sequence is generated.

[0020] A regional semantic representation is constructed and a first-level route filtering is performed to form a regional route index set. Based on the regional route index set, a second-level image semantic sequence is calculated to obtain the semantic features of the RGB image. The semantic features of the RGB image are mapped to a visual semantic feature vector with the same dimension as the radar semantic feature vector through a linear projection layer.

[0021] Furthermore, the specific method for constructing a region-level semantic representation and performing first-level route filtering is as follows:

[0022] Based on the spatial location of the image semantic sequence in the RGB image, the image semantic sequence is divided into multiple regions, each region containing several adjacent semantic feature vectors; the semantic feature vectors in each region are averaged and aggregated to obtain the region-level semantic representation of each region.

[0023] The region-level semantic representations of all regions are combined into a region-level semantic sequence. Based on this sequence, region-level query vectors and region-level key vectors are generated, and a region relevance matrix is ​​calculated. Each element in the region relevance matrix represents the relevance between two regions. Elements are then selected from each row of the region relevance matrix. The elements with the highest relevance are used to form a regional routing index set;

[0024] The specific method for calculating the second-level image semantic sequence based on the regional-level routing index set is as follows:

[0025] Query vectors, key vectors, and value vectors are generated based on the image semantic sequence; a subset of keys and a subset of values ​​are selected from the key vectors and value vectors based on the regional-level routing index set; the image semantic sequence is updated based on the query vector, the subset of keys, and the subset of values ​​to obtain the second-level image semantic sequence; the second-level image semantic sequence is aggregated by average pooling to obtain the semantic features of the RGB image.

[0026] Furthermore, the specific method for intensity quantization of radar semantic feature vectors and visual semantic feature vectors is as follows:

[0027] Feature-level association strength quantization is performed on radar semantic feature vectors and visual semantic feature vectors, and the feature-level association strength is calculated. Sample-level association strength quantization is also performed on radar semantic feature vectors and visual semantic feature vectors, and the sample-level association strength is calculated. Image modal quality factors and radar signal modal quality factors are extracted for single-modal quality assessment. Based on the image modal quality factors and radar signal modal quality factors, the feature-level association strength and sample-level association strength are weighted and fused to obtain the association strength tensor. .

[0028] Furthermore, the specific method for calculating the feature-level correlation strength is as follows:

[0029] For radar semantic feature vectors and visual semantic feature vectors, semantic similarity is calculated using the vector inner product form, and the semantic similarity is linearly mapped to obtain the linearly mapped semantic similarity.

[0030] The radar semantic feature vector and the visual semantic feature vector are subjected to Softmax transformation to obtain the radar semantic weight and the image semantic weight.

[0031] The Kullback-Leibler divergence is used to measure the difference between radar semantic weights and image semantic weights, and symmetric KL divergence is calculated. The symmetric KL divergence is normalized and mapped to semantic complementarity score. Feature-level association strength is calculated based on semantic complementarity score.

[0032] Furthermore, the specific method for calculating the sample-level association strength is as follows:

[0033] Based on visual semantic features and radar semantic feature vectors, the target's position estimate in three-dimensional space and the target's spatial position in the radar coordinate system are extracted respectively. The difference between the target's spatial position in three-dimensional space and the radar coordinate system is calculated and mapped to spatial matching degree.

[0034] The motion vectors of the target in three-dimensional space and the target in the radar coordinate system are extracted separately. The similarity between the motion vectors of the target in three-dimensional space and the target in the radar coordinate system is calculated using cosine similarity and normalized to the motion matching degree.

[0035] By combining spatial matching degree and motion matching degree, the sample-level association strength is calculated.

[0036] Furthermore, the specific methods for extracting image modal quality factors and radar signal modal quality factors are as follows:

[0037] The proportion of the target in the RGB image is calculated as the image modal quality factor; the stability of the radar echo signal is calculated as the radar signal modal quality factor.

[0038] The specific method for classifying into different related scenarios is as follows:

[0039] Set the correlation strength threshold When the correlation strength tensor At that time, radar semantic feature vectors and visual semantic feature vectors are classified into highly correlated scenes, and the correlation strength tensor is... At that time, radar semantic feature vectors and visual semantic feature vectors are classified into low-association scenarios.

[0040] Furthermore, the specific method for weighted fusion of radar semantic feature vectors and visual semantic feature vectors based on the associated scenario is as follows:

[0041] In highly correlated scenarios, visual fusion weights are set based on the correlation strength tensor. Radar fusion weights In low-association scenarios, visual fusion weights are set based on the association strength tensor. Radar fusion weights The visual semantic features and radar semantic feature vectors are fused and L2 normalized to obtain fused semantic features.

[0042] Furthermore, the specific methods for semantic mapping and encoding of fused semantic features are as follows:

[0043] The fused semantic features are projected onto the LLM semantic embedding space through a linear mapping to construct the initial semantic embedding; a correlation strength tensor is introduced. Amplitude modulation is applied to the initial semantic embedding to form an association-aware semantic embedding; the association-aware semantic embedding is then... The input is fed into a large language model encoder consisting of a multi-layered Transformer structure to obtain the semantic encoding result.

[0044] Furthermore, the multi-task collaborative semantic decoding module includes a shared semantic decoding backbone network and multiple task-specific decoding heads;

[0045] The specific method for decoding the semantic encoding result and generating the target motion state prediction result is as follows:

[0046] Through linear mapping and reconstruction operations, the semantic encoding results are converted into initial decoding features, which are then input into a shared semantic decoding backbone network to extract shared features.

[0047] By inputting shared features into different task-specific decoding heads, the predicted target motion state can be obtained.

[0048] The beneficial effects of adopting the above technical solution are as follows: This invention provides a multimodal semantic encoding method based on the adjustment of intermodal correlation strength. By calculating the correlation strength tensor, a normalized scalar signal that can be directly used for system regulation is formed. Based on this, this invention classifies high and low correlation according to the correlation strength tensor: in high-correlation scenarios with consistent multimodal semantics, matching physical attributes, and high modal quality, enhanced multimodal semantic fusion is performed to fully utilize the complementary advantages of different modalities; in low-correlation scenarios with insufficient multimodal consistency or uneven modal quality, a restricted fusion strategy is implemented to suppress the interference of unreliable modalities on the fusion result. This makes the multimodal fusion process have clear scenario adaptability, rather than static fusion. Unlike existing multimodal methods that only adjust at the feature fusion stage, this invention further introduces the correlation strength tensor into the semantic encoding stage of the large language model, using it as a semantic embedding modulation signal to participate in the encoding process. By modulating the amplitude of the fused semantic embedding through the correlation strength tensor, the semantic aggregation behavior of the self-attention within the LLM is indirectly affected, so that the multimodal semantics of high-correlation samples are enhanced during the encoding process, while the semantics of low-correlation samples are conservatively processed. This enables the LLM coding process to have the ability to perceive the reliability of multimodal collaboration. Attached Figure Description

[0049] Figure 1 A schematic diagram of the multi-task prediction model structure provided in Embodiment 1 of the present invention;

[0050] Figure 2 Flowchart of the multimodal semantic feature extraction module provided in Embodiment 1 of the present invention;

[0051] Figure 3 Flowchart of the correlation strength quantification module provided in Embodiment 1 of the present invention;

[0052] Figure 4 Flowchart of the multimodal semantic fusion module based on association perception provided in Embodiment 1 of the present invention;

[0053] Figure 5 Flowchart of the association-aware LLM semantic coding module provided in Embodiment 1 of the present invention;

[0054] Figure 6 Flowchart of the multi-task collaborative semantic decoding module provided in Embodiment 1 of the present invention;

[0055] Figure 7 The results of predicting target distance in different scenarios provided in Embodiment 1 of the present invention include (a) the results of predicting target distance using different methods in the whole dataset, (b) the results of predicting target distance using different methods in a highly correlated scenario, and (c) the results of predicting target distance using different methods in a low-correlation scenario.

[0056] Figure 8 The results of predicting the target's motion speed in different scenarios provided in Embodiment 1 of the present invention are as follows: (a) is the result of predicting the target's motion speed using different methods in the whole dataset; (b) is the result of predicting the target's motion speed using different methods in a highly correlated scenario; and (c) is the result of predicting the target's motion speed using different methods in a low-correlation scenario.

[0057] Figure 9 The results of predicting the target motion direction angle in different scenarios provided in Embodiment 1 of the present invention are as follows: (a) is the result of predicting the target motion direction angle using different methods in the whole dataset; (b) is the result of predicting the target motion direction angle using different methods in a highly correlated scenario; and (c) is the result of predicting the target motion direction angle using different methods in a low-correlation scenario. Detailed Implementation

[0058] The specific implementation methods of this application will be further described in detail below with reference to the accompanying drawings and embodiments.

[0059] Example 1:

[0060] Although existing multimodal semantic coding methods have reduced the transmission overhead of multi-source sensing data to some extent, they still have the following shortcomings in real-world complex application scenarios when combined with their specific technical solutions.

[0061] First, existing multimodal semantic encoding methods generally assume a relatively stable consistency between different modal semantic information, making it difficult to adapt to differences in semantic reliability caused by environmental changes. In existing solutions, multimodal semantic features are typically integrated in the fusion stage using preset weights and static structures, and encoded using an encoder with fixed parameters. These methods are designed with the assumption that different modalities have essentially the same ability to describe the target semantics in different scenarios. In practical applications, due to factors such as occlusion, lighting changes, and electromagnetic interference, the semantic expressive ability of different modalities for the same target or scene can change significantly, with some modalities containing more noise and irrelevant information. Because the encoding process does not differentiate between these changes in semantic consistency, reliable semantic information is easily weakened during encoding, affecting the overall quality of semantic expression. Second, existing methods do not distinguish the differences in the effectiveness of different modal semantic information in the current scenario during the encoding stage, leading to an uneven allocation of encoding resources. In the background solutions, the fused multimodal semantic features are usually used as a whole input encoder, processed using uniform encoding rules. This approach does not differentiate between the accuracy, stability, and redundancy of different modal semantic information in the current scenario. When some modal semantic information has low effectiveness, encoding it together with other modalities may not only introduce noisy semantics that consume limited coding resources, but also lose high-quality semantic information that is more critical to the current task, thereby reducing the ability of the encoded semantic representation to depict the real scene. Finally, existing multimodal semantic coding methods lack a unified quantitative basis that can reflect the degree of consistency of multimodal semantics and participate in coding decisions. The coding process in the background technology usually relies on fixed structure or static parameter configuration, without introducing a quantitative description of the consistency, complementarity or conflict of multimodal semantic expressions. The encoder cannot adjust the coding method according to the changes in multimodal semantic relationships in different scenarios, thus limiting the overall performance improvement of multimodal semantic communication systems in complex and dynamic environments.

[0062] To address the common problems in existing multimodal semantic coding methods, such as difficulty in perceiving changes in multimodal semantic consistency, fixed coding strategies, and unreasonable allocation of coding resources, this embodiment aims to provide a multimodal semantic coding method based on the adjustment of intermodal correlation strength. By introducing the characterization of the degree of multimodal semantic consistency and its changes during the coding process, the coding process is adjusted according to the differences in the effectiveness of multimodal semantic information and the characteristics of information overlap in different scenarios. This improves semantic coding efficiency and enhances the robustness of the system in complex environments while ensuring the reliable transmission of key semantic information.

[0063] This embodiment presents a multimodal semantic coding method based on intermodal correlation strength adjustment, such as... Figure 1As shown, a multi-task prediction model is constructed, including a multimodal semantic feature extraction module, a correlation strength quantization module, a correlation-aware multimodal semantic fusion module, a correlation-aware LLM semantic encoding module, and a multi-task collaborative semantic decoding module; the steps include:

[0064] Step 1: Use radar and camera to detect and photograph the target, and obtain radar echo signals and RGB images of the target;

[0065] Step 2: Construct a multimodal semantic feature extraction module to extract semantic features from radar echo signals and RGB images;

[0066] like Figure 2 As shown, the multimodal semantic feature extraction module includes a parallel radar signal semantic extractor and an image semantic extractor, which are used to extract semantic features from radar echo signals and RGB images, respectively. Through unified feature mapping and standardization operations, the semantic features of the two modalities are aligned to the same embedding space, thereby providing a consistent and comparable feature representation for subsequent cross-modal correlation strength calculation and semantic fusion processing.

[0067] Step 2.1: Input the radar echo signal into the radar signal semantic extractor to extract the radar semantic feature vector;

[0068] A radar signal semantic extractor is used to perform semantic modeling on radar echo signals, extracting semantic information from the radar echo signals that can characterize the target's existence state, range changes, motion trends, and signal structure features. Since radar echo signals are essentially complex-valued time-series signals containing key information such as amplitude and phase, and the target's range and motion state are often reflected in the time structure and phase changes of the radar echo signals, this embodiment directly models the radar echo signals in the signal domain. It employs a complex-valued convolutional network to extract the semantic features of the radar signals step by step, thereby avoiding the strong dependence of traditional manual feature extraction methods on the physical model of the signal and improving the generalization ability of semantic modeling.

[0069] For the radar echo signal input to the radar signal semantic extractor ,in, Represents the set of complex numbers. Indicates the radar receiving channel. This represents the number of time sampling points corresponding to each radar receiving channel. To facilitate neural network processing, the complex-valued radar echo signal is decomposed into real and imaginary parts: ,in, Let be the real part of the radar echo signal. This represents the imaginary part of the radar echo signal.

[0070] The radar signal semantic extractor includes a complex-valued convolutional network, which includes... A complex-valued convolutional layer; based on the complex-valued representation, the radar echo signal is input into multiple complex-valued convolutional layers to perform complex-valued one-dimensional convolution operations to extract the local semantic pattern of the radar echo signal in the time dimension. Let the th... The complex convolution kernel parameters of the layer are:

[0071] ;

[0072] in, For the first Complex convolution kernel parameters of the layer, For the first The real part weight matrix of the complex-valued convolution kernel. For the first The imaginary part weight matrix of the complex-valued convolution kernel;

[0073] For input number Intermediate features of complex-valued convolutional layers , No. The real part of the intermediate features output by each complex convolutional layer With the imaginary part Calculated separately as follows:

[0074] ;

[0075] ;

[0076] Where * denotes a one-dimensional convolution operation, For input number Intermediate features of complex-valued convolutional layers The real part, For input number Intermediate features of complex-valued convolutional layers The imaginary part, .

[0077] Through the above complex-valued convolution calculation, the amplitude and phase changes of radar signals can be modeled simultaneously, thereby capturing semantic information related to target distance and velocity.

[0078] Apply nonlinear activation functions to the real and imaginary parts of the intermediate features output by each complex convolutional layer to obtain the intermediate features input to the next complex convolutional layer;

[0079] radar echo signal go through After multiple complex-valued convolutional layers and nonlinear activation functions, the complex-valued feature representation is obtained:

[0080] ;

[0081] in, It is a non-linear activation function;

[0082] Subsequently, in the time dimension, the first... Complex feature representation of the output of a complex convolutional layer Downsampling is performed to reduce redundant sampling points and integrate signal variation information across time intervals. Through cascaded processing of the above-mentioned multiple layers, the variation features scattered at different time locations in the radar echo signal are gradually converged to form a complex-valued temporal semantic representation that reflects the overall time structure.

[0083] Next, after completing the multi-layer temporal semantic modeling, the complex-valued temporal semantic representation of the radar echo signal is obtained. The real and imaginary parts of the complex-valued temporal semantic representation are concatenated along the feature dimension to form a real-valued semantic feature vector. :

[0084] ;

[0085] in, Let be the real part of the complex-valued temporal semantic representation. For the imaginary part of the complex-valued temporal semantic representation, represents the feature dimension of the real-valued semantic feature vector. This real-to-virtual concatenation operation can convert complex-valued radar semantic features into standard real-valued feature representations without losing phase information.

[0086] Finally, a learnable fully connected layer is used to process the real-valued semantic feature vectors. The radar signal semantic feature vector is obtained by mapping to a unified low-dimensional semantic space and performing normalization. :

[0087] ;

[0088] ;

[0089] in, The semantic embedding vector is obtained after mapping through a learnable fully connected layer. The radar semantic feature vector output by the radar signal semantic extractor. As trainable parameters, this embodiment sets , .

[0090] Step 2.2: Input the RGB image into the image semantic extractor to extract the visual semantic feature vector;

[0091] The image semantic extractor takes the input RGB image as the processing object. Its goal is to extract high-level semantic features from the RGB image that can represent the target category, structural relationship and scene semantics, while retaining the representation that can distinguish the instance location and context information, so as to provide a reliable image-side semantic foundation for subsequent cross-modal semantic consistency evaluation.

[0092] The RGB image of the input image semantic extractor is scaled and pixel normalized to meet the input requirements of the image semantic extractor and reduce the influence of factors such as lighting and contrast in different scenes on the feature distribution. The RGB image... According to fixed dimensions The image is divided into several non-overlapping image patches. Each image patch is unfolded into a one-dimensional vector along the spatial and channel dimensions. A learnable fully connected layer is used to linearly map the unfolded one-dimensional vector of the image patch onto the image. The semantic feature space of dimension, in this embodiment, is defined as dimension. The corresponding image semantic sequence is obtained. As shown in the formula below:

[0093] ;

[0094] in, For the first The semantic feature vector corresponding to each image patch The total number of image patches, The height of the RGB image. The width of the RGB image;

[0095] Constructing a region-level semantic representation and performing first-level route filtering includes:

[0096] Image semantic sequence Based on their spatial location in the RGB image, they are divided into Each region contains several adjacent semantic feature vectors. For the ... The semantic feature vectors within the regions are averaged and aggregated to obtain the th region. Regional-level semantic representation of each region As shown in the formula below:

[0097] ;

[0098] in, Indicates the first The set of token indexes contained in each region, as set in this embodiment. .

[0099] Combine all region-level semantic representations into a region-level semantic sequence Generate region-level query vectors and regional key vectors , and These are learnable linear mapping parameters used to project region-level semantic representations onto the query space and key space;

[0100] Calculate the regional correlation matrix In the regional correlation matrix, each element represents the correlation between two regions; elements are selected from each row of the regional correlation matrix. The elements with the highest relevance form a regional-level routing index set. .

[0101] Based on regional-level routing index set Calculate the second-level image semantic sequence The specific method is as follows:

[0102] Based on image semantic sequence Generate query vectors Key vector Sum value vector Based on a regional-level routing index set From the key vector Sum value vector Selecting a subset of keys Sum value subset Based on query vector Key subset Sum value subset Update image semantic sequence The second-level image semantic sequence is obtained. As shown in the formula below:

[0103] ;

[0104] Next, the second-level image semantic sequence Average pooling is used to aggregate the data, forming the semantic features of the RGB image. As shown in the formula below:

[0105] ;

[0106] Semantic features of RGB images Semantic space alignment and normalization are performed through a linear projection layer. semantic features of RGB images The data is mapped to a visual semantic representation consistent with the semantic dimension of the radar signal and then normalized to obtain the visual semantic feature vector output by the visual semantic extractor. As shown in the formula below:

[0107] ;

[0108] ;

[0109] in, The semantic embedding vector is obtained after mapping through a learnable fully connected layer; The weights representing the visual alignment projection are trainable parameters; This is the visual semantic feature vector output by the visual semantic extractor.

[0110] Step 3: Construct a correlation strength quantization module to perform intensity quantization on radar semantic feature vectors and visual semantic feature vectors to obtain correlation strength tensors, and divide radar semantic feature vectors and visual semantic feature vectors into different correlation scenarios;

[0111] The association strength quantification module is used to collaboratively analyze the radar semantic feature vectors and visual semantic feature vectors output by the multimodal semantic feature extraction module. It quantifies the consistency and matching degree of different modalities in the current sample from both semantic and instance levels, generating association strength values ​​to characterize cross-modal effectiveness differences. This is because multimodal semantic consistency is reflected both in the similarity and complementarity of high-level semantic representations and in the matching degree of specific instances in physical space and motion states. Figure 3 As shown, the association strength quantification module adopts a two-level evaluation method that combines feature-level quantization and sample-level quantization. At the same time, in order to avoid low-quality modes from misleading the association evaluation results, a modal quality weighting mechanism is introduced to correct the association strength, thereby obtaining stable and reliable association strength quantification results.

[0112] Step 3.1: Quantize the feature-level association strength of the radar semantic feature vector and the visual semantic feature vector, and calculate the feature-level association strength;

[0113] This embodiment evaluates the representational relationship between image modalities and radar signal modalities in a unified semantic space through feature-level semantic consistency calculation and semantic complementarity calculation, focusing on characterizing the consistency and potential complementarity of the two modalities in the direction of semantic expression;

[0114] For radar semantic feature vectors and visual semantic feature vectors Semantic similarity is calculated using the vector dot product method. and semantic similarity Perform a linear mapping to the [0,1] interval to obtain the semantic similarity after the linear mapping. :

[0115] ;

[0116] ;

[0117] Among them, semantic similarity ;

[0118] radar semantic feature vector and visual semantic feature vectors This is converted into a probability distribution form to characterize the attention weight distribution of image modalities and radar signal modalities on each semantic dimension in a unified semantic space. Specifically, this involves the radar semantic feature vector... and visual semantic feature vectors Applying a softmax transform yields the radar semantic weights. Image semantic weights As shown in the formula below:

[0119] ;

[0120] in, , , , Visual semantic feature vector In the unified semantic space The attention weight of each semantic dimension Radar semantic feature vector In the unified semantic space The attention weight of each semantic dimension;

[0121] The Kullback-Leibler divergence measure is used to measure the difference in semantic feature distributions between the two modalities; the greater the difference, the stronger the complementarity. Radar semantic weights... Image semantic weights The difference is shown in the following formula:

[0122] ;

[0123] To ensure that the complementarity measure is insensitive to the modal order, the symmetric KL divergence is calculated. As shown in the formula below:

[0124] ;

[0125] After normalization, the symmetric KL divergence is mapped to a semantic complementarity score in the interval [0,1]. As shown in the formula below:

[0126] ;

[0127] Based on semantic complementarity scoring Calculate feature-level correlation strength As shown in the formula below:

[0128] ;

[0129] in, To use fixed weighting coefficients, this embodiment sets... .

[0130] Step 3.2: Quantize the sample-level association strength of the radar semantic feature vector and the visual semantic feature vector, and calculate the sample-level association strength;

[0131] First, the spatial location of the target in the image modality is determined. In the image modality, a pre-trained target detection head is used to detect visual semantic features. Extracting the target's position allows for the estimation of the target's position in three-dimensional space. As shown in the formula below:

[0132] ;

[0133] in, For target depth, for The x-coordinate of the center point of the target bounding box at any given time. for The ordinate of the center point of the target bounding box at any given time. This is the camera intrinsic parameter matrix;

[0134] Then, the target's spatial location under the radar signal mode is determined. Within the radar signal mode, the radar semantic feature vector... The target distance is directly output via the linear regression head. With azimuth .

[0135] Calculate the spatial position of the target in the radar coordinate system As shown in the formula below:

[0136] ;

[0137] Finally, the target's position estimate in three-dimensional space is calculated. Spatial position of the target in the radar coordinate system Spatial location differences And mapped to spatial matching degree As shown in the formula below:

[0138] ;

[0139] ;

[0140] in, This is an adjustment coefficient used to control the sensitivity of distance to matching degree; in this embodiment, it is set to... =1.1.

[0141] The motion vector in three-dimensional space is determined by the difference in the target's three-dimensional spatial position between two adjacent frames. As shown in the formula below:

[0142] ;

[0143] in, for Position estimation of the target in 3D space. , indicates 1 frame.

[0144] Next, the target's motion vector in the radar coordinate system is determined. As shown in the formula below:

[0145] ;

[0146] in, The radial velocity of the target is output by the fully connected layer;

[0147] The motion vector of the target in three-dimensional space is calculated using cosine similarity. The motion vector of the target in the radar coordinate system similarity between The motion matching degree is normalized to the interval [0,1]. As shown in the formula below:

[0148] ;

[0149] ;

[0150] By combining spatial consistency and motion consistency, the sample-level correlation strength is calculated. As shown in the formula below:

[0151] ;

[0152] in As a balance coefficient, this embodiment sets .

[0153] Step 3.3: Extract image modal quality factors and radar signal modal quality factors for single-modal quality assessment; based on image modal quality factors and radar signal modal quality factors, perform weighted fusion of feature-level correlation strength and sample-level correlation strength to obtain the correlation strength tensor;

[0154] Image modal quality factor Based on the proportion of the target in the RGB image The modal quality factor is calculated to measure the visibility of a target in an image. It is derived from the target bounding box size output by the target detection head and the total area of ​​the input image. As shown in the formula below:

[0155] ;

[0156] in, The target bounding box width, The height of the target bounding box. The total area of ​​the image;

[0157] Radar signal modal quality factor Stability of radar echo signals The calculation yields the following formula:

[0158] ;

[0159] in, For radar signals in continuous Time and The amount of semantic feature change at time step. The stability coefficient;

[0160] Based on image modal quality factors and radar signal modal quality factors, a weighted fusion of feature-level correlation strength and sample-level correlation strength is performed to obtain the correlation strength tensor. :

[0161] ;

[0162] ;

[0163] in, For feature-level association strength weights, For sample-level association strength weights, >0 is a balance factor to prevent extremely low modal quality. Too large;

[0164] Step 3.4: Set the correlation strength threshold for the correlation strength tensor. Divided into different related scenarios;

[0165] Set the correlation strength threshold In this embodiment, a threshold is set. The correlation strength tensor Divided into different related scenarios:

[0166] when At that time, the radar semantic feature vector and the visual semantic feature vector are divided into highly correlated scenes, indicating that the input image and radar signal have consistent multimodal semantics, matching physical attributes, and high modal quality;

[0167] when When radar semantic feature vectors and visual semantic feature vectors are classified into low-association scenarios, it indicates that the multimodal semantic consistency between the input image and radar signal is low, the degree of physical attribute matching is insufficient, and low-quality modal interference exists.

[0168] Step 4: Construct a multimodal semantic fusion module based on association perception. In different association scenarios, use different fusion weights to perform weighted fusion of visual semantic feature vectors and radar semantic feature vectors to obtain fused motion semantic features, and perform normalization processing to obtain fused semantic features.

[0169] like Figure 4 By clearly distinguishing between highly correlated and low-correlation scenarios, this embodiment can dynamically adjust the multimodal semantic fusion strategy based on the degree of collaborative effectiveness of multimodal information in the current sample. This ensures prediction stability while making full use of high-quality, multimodal consistent information, thereby improving the overall performance of speed and angle prediction tasks.

[0170] In highly correlated scenarios, while ensuring the semantic consistency of multimodal motion, the system simultaneously leverages the image modality's ability to perceive changes in motion direction and the radar modality's sensitivity to velocity information. Visual fusion weights are set based on the correlation strength tensor. Radar fusion weights Visual semantic features and radar semantic feature vector By fusing the data, preliminary semantic features are obtained. As shown in the formula below:

[0171] ;

[0172] ;

[0173] To ensure the numerical stability of the fused features in subsequent predictions, the initial semantic features... L2 normalization is performed to obtain fused semantic features. As shown in the formula below:

[0174] ;

[0175] in, For L2 normalization;

[0176] In low-association scenarios, significant differences exist between different modalities, and direct deep fusion may lead to semantic conflicts. Therefore, a conservative fusion strategy is adopted to reduce cross-modal interference. The modality with higher contribution to sample-level association is selected as the dominant modality. Since the target task is to predict motion parameters such as velocity and angle, the radar mode is defaulted as the dominant mode when the association strength is low.

[0177] The fusion strategy aims at maintaining motion stability and limits the participation of non-dominant modalities. Visual fusion weights are set. Radar fusion weights for:

[0178] ;

[0179] Among them, when hour, Limited to a small range of values. (Regarding visual semantic features) and radar semantic feature vector Fusion is performed to obtain preliminary semantic features. :

[0180] ;

[0181] Preliminary semantic features L2 normalization is performed to obtain fused semantic features. As shown in the formula below:

[0182] ;

[0183] Step 5: Construct an association-aware LLM semantic encoding module

[0184] Figure 5 This is a flowchart of association-aware LLM semantic encoding. Since large language models use fixed-dimensional embedding vectors as input, this module first performs semantic mapping on the fused semantic features to achieve unified alignment of different modal semantics to the LLM input space.

[0185] The fused semantic features are projected onto the LLM semantic embedding space through a linear mapping to construct the initial semantic embedding without introducing an association-aware mechanism. :

[0186] ;

[0187] in, and These are trainable parameters;

[0188] This embodiment controls the initial semantic embedding through association strength. The expression strength in the LLM encoding process. When the multimodal semantics are highly consistent, the physical attributes match, and the modal quality is high, the fused semantics have high reliability and are more fully encoded in LLM; when the multimodal semantic consistency is low, the semantics should be prevented from having an excessive influence on the encoding result.

[0189] Introducing association strength to the initial semantic embedding Amplitude modulation is performed to form an association-aware semantic embedding. :

[0190] ;

[0191] in, The modulation coefficient is fixed.

[0192] When the correlation strength tensor Greater than the association strength threshold When the semantic embedding amplitude is amplified, the overall magnitude of the semantic embedding is magnified; when the association strength tensor... Less than or equal to the association strength threshold At this time, the semantic embedding amplitude remains at a low level. Without changing the semantic embedding direction, LLM enables the ability to distinguish the reliability of multimodal semantics in different samples during subsequent encoding. In the Transformer encoding process of LLM, the self-attention mechanism calculates the query, key, and value vectors through the input embedding and determines the semantic aggregation weights based on the vector inner product. Due to the modulated association-aware semantic embedding... Its contribution to self-attention calculation varies depending on the magnitude of the association. When the association strength is greater than the association strength threshold... When the association-aware semantic embedding is more likely to obtain a larger attention weight in attention computation, it will be repeatedly aggregated and strengthened in the multi-layer encoding process; when the association strength is less than or equal to the association strength threshold... At this time, the attention contribution of association-aware semantic embedding is suppressed to avoid low-reliability semantics from interfering with the encoding results.

[0193] Embed the modulated association-aware semantics The input is fed into a large language model encoder consisting of multiple Transformer layers. Each encoding unit updates the semantics through self-attention and feedforward transformation, extracting high-level semantic relations layer by layer. The layer encoding process is represented as follows:

[0194] ;

[0195] After multi-layer semantic modeling, the final semantic encoding result is obtained. As shown in the formula below:

[0196] ;

[0197] Step 6: Construct a multi-task collaborative semantic decoding module to decode the semantic encoding results and generate target motion state prediction results;

[0198] like Figure 6 The multi-task collaborative semantic decoding module performs structured decoding of the high-level semantic embeddings output by the association-aware LLM semantic encoding module. Based on shared semantic representations, it collaboratively estimates the parameters of target distance, velocity, and motion angle. This module adopts a structure combining a shared backbone network and task-specific decoding heads, which improves the accuracy and stability of parameter estimation for each task while ensuring semantic consistency across multiple tasks.

[0199] The multi-task collaborative semantic decoding module includes a shared semantic decoding backbone network and multiple task-specific decoding heads, which include at least a distance estimation decoding head, a velocity estimation decoding head, and an angle estimation decoding head.

[0200] First, the semantic encoding result is obtained through linear mapping and reconstruction operations. Convert to initial decoding features As shown in the formula below:

[0201] ;

[0202] in, , These are the mapping parameters, and these are the trainable parameters. This is the input feature tensor for the decoding stage.

[0203] The shared semantic decoding backbone network consists of three convolutional layers for processing the initial decoded features. Perform step-by-step decoding to extract shared features suitable for multiple tasks. , No. The layer convolution decoding process is represented as:

[0204] ;

[0205] in, Indicates the first Two-dimensional convolution operation, For non-linear activation functions, when hour, The shared backbone output features are used to unify the semantics of the spatial and motion states of the target in modeling, providing a common feature foundation for subsequent multi-task decoding.

[0206] In shared features Based on this, multiple task-specific decoding heads are set up to perform accurate regression of different physical parameters. These task-specific decoding heads include a distance estimation decoding head, a velocity estimation decoding head, and an angle estimation decoding head; the distance estimation decoding head shares features. As input, the predicted target distance is output through convolution and linear mapping. As shown in the formula below:

[0207] ;

[0208] in, This indicates a feature aggregation operation; and All of these are trainable distance regression parameters.

[0209] The velocity estimation decoder head structure is independent of the range decoder head and is used to output the predicted target motion velocity. As shown in the formula below:

[0210] ;

[0211] in, and All of these are trainable velocity regression parameters;

[0212] Angle estimation decoding head is used to output the predicted target motion direction angle. As shown in the formula below:

[0213] ;

[0214] in, and All of these are trainable angle regression parameters;

[0215] Each task has its own dedicated decoding head parameters that are not shared, thus ensuring the independence and accuracy of the estimation of different physical quantities.

[0216] Step 7: Construct a multi-task prediction model based on the multimodal semantic feature extraction module, the association strength quantization module, the association-aware multimodal semantic fusion module, the association-aware LLM semantic encoding module, and the multi-task collaborative semantic decoding module. Train the multi-task prediction model with the goal of multi-task collaborative optimization, so that the multi-task prediction model learns the mapping between task loss and decoding strategy.

[0217] During the training phase, distance, velocity, and angle prediction tasks are co-optimized using a joint loss function to establish the loss function for the multi-task prediction model. As shown in the formula below:

[0218] ;

[0219] in, Let distance prediction loss function be used. The loss function for velocity prediction is... The loss function for predicting motion angles is... and All are weighting coefficients. The true value of the target distance. For the true value of the target's velocity, This represents the true value of the target's direction angle of motion.

[0220] By co-training a shared backbone and a dedicated regression head, the decoded features can simultaneously serve multiple physical parameter estimation tasks.

[0221] This embodiment proposes a correlation strength quantization mechanism to characterize the degree of collaborative effectiveness between image modalities and radar signal modalities under the current sample. The correlation strength tensor in this embodiment is not an abstract weight, but rather a normalized scalar signal that can be directly used for system regulation, obtained through joint calculation of multi-dimensional information including feature-level semantic consistency, semantic complementarity, sample-level physical attribute matching, and single-modal quality assessment. This correlation strength serves as the core control variable throughout the entire process and forms the basis for subsequent multimodal fusion and semantic encoding adaptive adjustment.

[0222] Building upon this, this embodiment does not simply concatenate or weight multimodal features in a fixed manner. Instead, it divides multimodal features into high and low correlations based on the correlation strength tensor: In high-correlation scenarios with consistent multimodal semantics, matching physical attributes, and high modal quality, enhanced multimodal semantic fusion is performed to fully utilize the complementary advantages of different modalities; in low-correlation scenarios with insufficient multimodal consistency or uneven modal quality, a restricted fusion strategy is implemented to suppress the interference of unreliable modalities on the fusion result. This makes the multimodal fusion process have clear scenario adaptability, rather than static fusion.

[0223] Unlike existing multimodal methods that only adjust the features during the feature fusion stage, this embodiment further introduces the association strength tensor into the semantic encoding stage of the Large Language Model (LLM), using it as a semantic embedding modulation signal to participate in the encoding process. By modulating the amplitude of the fused semantic embedding through the association strength tensor, the semantic aggregation behavior of the self-attention mechanism within the LLM is indirectly affected. This ensures that the multimodal semantics of highly associated samples are enhanced during encoding, while the semantics of low-association samples are conservatively processed. This gives the LLM encoding process the ability to perceive the reliability of multimodal collaboration.

[0224] This embodiment introduces a task-co-training mechanism to jointly optimize distance prediction, velocity prediction, and angle prediction tasks, thereby further improving the overall prediction accuracy of the multi-task prediction model. This mechanism allows the distance, velocity, and angle prediction tasks to share a single high-level semantic representation. Through joint optimization, the loss functions of the three tasks work together, avoiding redundancy and conflicts between tasks that occur during individual training, thus further enhancing the stability and accuracy of the predictions.

[0225] The prediction tasks in this embodiment include predicting target distance, target velocity, and target direction angle. To evaluate the performance of the multi-task prediction model, this embodiment selects different baseline schemes for comparative experiments and adopts the following standard evaluation metrics for regression tasks: distance prediction accuracy, velocity prediction accuracy, and angle prediction accuracy. Root mean square error (RMSE) is used to measure the difference between the predicted distance and the true value. Furthermore, during the evaluation process, the robustness of the model to multimodal data is further examined by analyzing error changes under different scenarios (such as high-correlation and low-correlation scenarios).

[0226] The baseline scheme selected in this embodiment includes:

[0227] CV-S (Image-Based Sensing): This method uses only image modalities (i.e., RGB image sequences) to perform target motion state prediction. Specifically, the image modalities extract features through a convolutional neural network (CNN) and predict the target's distance, velocity, and angle using a regression head.

[0228] RS-S (Radar Signal-Based Sensing): This method predicts the target motion state using only radar signal modes (i.e., signal data obtained from radar sensors). The radar signals are preprocessed, and then a CNN network is used for regression prediction.

[0229] SIMAC: It uses multimodal feature fusion technology to jointly encode the features of images and radar signals through a large language model, and then uses a regression head to predict the target's distance, speed and angle.

[0230] Comparative experimental results are as follows Figure 7 , Figure 8 and Figure 9 As shown, where, Figure 7 (a) shows the results of predicting target distance using different methods on the entire dataset. Figure 7 (b) shows the results of predicting target distance using different methods in a highly correlated scenario. Figure 7 (c) shows the results of predicting target distance using different methods in low-association scenarios; Figure 8 (a) shows the results of predicting the target's velocity using different methods on the entire dataset. Figure 8 (b) shows the results of predicting the target's motion velocity using different methods in a highly correlated scenario. Figure 8 (c) shows the results of predicting the target's motion velocity using different methods in low-association scenarios; Figure 9 (a) shows the results of predicting the target motion direction angle using different methods on the entire dataset. Figure 9 (b) shows the results of predicting the target motion direction angle using different methods in a highly correlated scenario. Figure 9 (c) shows the results of predicting the target motion direction angle using different methods in low-association scenarios;

[0231] from Figure 7-9 The distance, velocity, and angle prediction results on the full dataset show a significant performance gap between the single-modal methods CV-S and RS-S and the multimodal methods. In the three prediction tasks, CV-S (visual only) consistently achieved the highest RMSE, while RS-S (radar signal only) generally outperformed CV-S, indicating that radar signals have a more stable advantage in estimating physical quantities (distance, velocity, and angle), but their prediction accuracy remains significantly limited. In contrast, this embodiment consistently achieved the lowest RMSE on the full dataset and maintained a stable decreasing trend under different SNR conditions. This result demonstrates that the visual modality and the radar signal modality are significantly complementary in terms of perceptual information. A single modality can only capture a portion of the scene information, while multimodal semantic modeling and fusion can simultaneously utilize the structural semantic information of vision and the physical measurement characteristics of radar signals, thereby significantly improving prediction accuracy and verifying the necessity and effectiveness of the multimodal complementarity mechanism in perceptual tasks.

[0232] Further comparison of prediction results across the entire dataset, high-association scenarios, and low-association scenarios clearly reveals the impact of intermodal correlation strength on model performance. In high-association scenarios, the overall RMSE for all three tasks is significantly lower than the average level of the entire dataset. This embodiment demonstrates the most significant advantage in high-association scenarios, consistently maintaining the lowest RMSE. This indicates that when image modalities and radar signal modalities are highly consistent at both semantic and physical levels, multimodal information can be fully utilized collaboratively, significantly reducing prediction uncertainty. In low-association scenarios, single-modal methods (CV-S, RS-S) show the most significant performance degradation; SIMAC's RMSE also shows a significant increase. In contrast, this embodiment exhibits the smallest performance degradation, maintaining a low RMSE. This suggests that when the degree of intermodal correlation decreases, multimodal methods that directly fuse or employ fixed strategies are more susceptible to misinterpretation, while methods with correlation perception capabilities can effectively suppress interference from unreliable modalities. In conclusion, modal correlation strength is a key factor affecting multimodal perception performance. Adopting a unified fusion strategy under different correlation states is not optimal; instead, the information utilization method needs to be dynamically adjusted based on modal consistency.

[0233] In the three types of tasks and three scenarios, both SIMAC and this embodiment belong to multimodal methods, but there is still a stable performance gap between them. SIMAC has significantly reduced RMSE compared to single-modal methods, indicating that the semantic-driven multimodal perception framework itself is effective; however, in the full dataset, high-association scenarios, and low-association scenarios, the RMSE of this embodiment is significantly lower and consistently better than SIMAC. This fully demonstrates that the intermodal association strength modeling mechanism introduced in this embodiment, which quantifies and models the intermodal association strength and introduces it as a control signal into the multimodal semantic processing flow, can significantly improve the perception robustness and prediction accuracy in complex scenarios.

[0234] Example 2:

[0235] This embodiment proposes an electronic device, including: one or more processors, and a memory, wherein the memory is used to store instructions, and when the instructions are executed by the one or more processors, the one or more processors execute the multimodal semantic coding method based on intermodal correlation strength adjustment.

[0236] The electronic device may be a mobile phone, computer, or tablet computer, etc., and includes a memory and a processor. The memory stores a computer program, which, when executed by the processor, implements the multimodal semantic coding method based on intermodal correlation strength adjustment as described in the embodiments. It is understood that the electronic device may also include an input / output (I / O) interface and communication components.

[0237] The processor is used to execute all or part of the steps in the multimodal semantic coding method based on intermodal correlation strength adjustment as described in the above embodiments. The memory is used to store various types of data, which may include, for example, instructions for any application or method in the electronic device, as well as application-related data.

[0238] The processor can be implemented as an Application Specific Integrated Circuit (ASIC), Digital Signal Processor (DSP), Programmable Logic Device (PLD), Field Programmable Gate Array (FPGA), controller, microcontroller, microprocessor, or other electronic components, and is used to execute the multimodal semantic coding method based on intermodal correlation strength adjustment described in the above embodiments.

[0239] Example 3:

[0240] This embodiment proposes a computer-readable storage medium that stores executable instructions. When these instructions are executed, if they are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium.

[0241] The computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the multimodal semantic coding method based on intermodal correlation strength adjustment described in the various embodiments of this application.

[0242] The aforementioned storage media include: flash memory, hard disks, multimedia cards, card-type memory (e.g., SD (Secure Digital Memory Card) or DX (Memory Data Register, MDR) memory), random access memory (RAM), static random-access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic storage, disks, optical discs, servers, APP (Application) app stores, and other media capable of storing program verification codes. These media store computer programs, which, when executed by a processor, can implement the various steps of the multimodal semantic coding method based on intermodal correlation strength adjustment described above.

[0243] Example 4:

[0244] This embodiment proposes a computer program product, including a computer program or instructions, which, when executed by a processor, implements the multimodal semantic coding method based on intermodal correlation strength adjustment.

[0245] Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or part of the technical solution, can be embodied in the form of a computer program product.

[0246] The various embodiments in this application are described in a progressive manner. The same or similar parts between the various embodiments can be referred to each other. Each embodiment focuses on describing the differences from other embodiments.

[0247] The scope of protection of this application is not limited to the embodiments described above. Obviously, those skilled in the art can make various modifications and variations to this disclosure without departing from the scope and spirit of this disclosure. If such modifications and variations fall within the scope of this disclosure and its equivalents, then the intent of this disclosure also includes these modifications and variations.

Claims

1. A multi-modal semantic encoding method based on inter-modal correlation strength regulation, characterized in that, Includes the following steps: The radar and camera are used to detect and photograph the target, respectively, to obtain radar echo signals and RGB images of the target; A multi-task prediction model is constructed, including a multimodal semantic feature extraction module, an association strength quantification module, an association-aware multimodal semantic fusion module, an association-aware LLM semantic encoding module, and a multi-task collaborative semantic decoding module. The radar echo signal and the target's RGB image are input into a multi-task prediction model to predict the target's motion state; the target motion state includes at least the target distance, target speed, and target motion angle. Among them, the multimodal semantic feature extraction module is used to extract semantic features from radar echo signals and RGB images to obtain radar semantic feature vectors and visual semantic feature vectors; The correlation strength quantization module is used to quantize the intensity of radar semantic feature vectors and visual semantic feature vectors to obtain the correlation strength tensor, and to divide the radar semantic feature vectors and visual semantic feature vectors into different correlation scenarios. The association-aware multimodal semantic fusion module is used to perform weighted fusion of radar semantic feature vectors and visual semantic feature vectors according to the associated scene to obtain fused semantic features; The association-aware LLM semantic coding module is used to perform semantic mapping and encoding on fused semantic features to generate semantic coding results; The multi-task collaborative semantic decoding module is used to decode the semantic encoding results and generate target motion state prediction results.

2. The multi-modal semantic coding method based on inter-modal correlation strength adjustment according to claim 1, characterized in that, The multimodal semantic feature extraction module includes a radar signal semantic extractor and an image semantic extractor; The semantic feature vector of radar echo signal is obtained by extracting semantic features from radar signal semantic extractor. The specific method is as follows: The radar signal semantic extractor includes a complex-valued convolutional network, which consists of multiple complex-valued convolutional layers. The radar echo signal input to the radar signal semantic extractor is split into real and imaginary parts. The split real and imaginary parts are input into a complex-valued convolutional network, and after passing through multiple complex-valued convolutional layers, a complex-valued feature representation is obtained. The complex-valued feature representation is downsampled to obtain a complex-valued temporal semantic representation of the radar echo signal. The real and imaginary parts of the complex-valued temporal semantic representation are concatenated along the feature dimension to obtain a real-valued semantic feature vector. The real-valued semantic feature vector is mapped to a unified low-dimensional semantic space through a learnable fully connected layer and normalized to obtain the radar semantic feature vector output by the radar signal semantic extractor. The image semantic extractor is used to extract semantic features from RGB images to obtain visual semantic feature vectors. The specific method is as follows: The RGB image is scaled and pixel normalized and then input into the image semantic extractor. The RGB image input into the image semantic extractor is divided into several non-overlapping image blocks according to a preset size. Each image block is unfolded into a one-dimensional vector along the spatial and channel dimensions. The unfolded one-dimensional vector is linearly mapped through a learnable fully connected layer to obtain the semantic feature vector of the image block. Based on the semantic feature vectors of all image blocks, an image semantic sequence is generated. A regional semantic representation is constructed and a first-level route filtering is performed to form a regional route index set. Based on the regional route index set, a second-level image semantic sequence is calculated to obtain the semantic features of the RGB image. The semantic features of the RGB image are mapped to a visual semantic feature vector with the same dimension as the radar semantic feature vector through a linear projection layer.

3. The multimodal semantic coding method based on intermodal correlation strength adjustment according to claim 2, characterized in that, The specific method for constructing a region-level semantic representation and performing first-level route filtering is as follows: Based on the spatial location of the image semantic sequence in the RGB image, the image semantic sequence is divided into multiple regions, each region containing several adjacent semantic feature vectors; the semantic feature vectors in each region are averaged and aggregated to obtain the region-level semantic representation of each region. The region-level semantic representations of all regions are combined into a region-level semantic sequence. Based on the region-level semantic sequence, region-level query vectors and region-level key vectors are generated, and a region relevance matrix is ​​calculated. Each element in the region relevance matrix represents the relevance between two regions. Select from each row of the regional correlation matrix The elements with the highest relevance are used to form a regional routing index set; The specific method for calculating the second-level image semantic sequence based on the regional-level routing index set is as follows: Query vectors, key vectors, and value vectors are generated based on the image semantic sequence; a subset of keys and a subset of values ​​are selected from the key vectors and value vectors based on the regional-level routing index set; the image semantic sequence is updated based on the query vector, the subset of keys, and the subset of values ​​to obtain the second-level image semantic sequence; the second-level image semantic sequence is aggregated by average pooling to obtain the semantic features of the RGB image.

4. The multimodal semantic coding method based on intermodal correlation strength adjustment according to claim 1, characterized in that, The specific method for intensity quantization of radar semantic feature vectors and visual semantic feature vectors is as follows: The feature-level association strength of radar semantic feature vectors and visual semantic feature vectors is quantized, and the feature-level association strength is calculated. The radar semantic feature vector and the visual semantic feature vector are quantized at the sample level, and the sample-level association strength is calculated; the image modal quality factor and the radar signal modal quality factor are extracted for single-modal quality assessment. Based on image modal quality factors and radar signal modal quality factors, a weighted fusion of feature-level correlation strength and sample-level correlation strength is performed to obtain the correlation strength tensor. .

5. A multimodal semantic coding method based on intermodal correlation strength adjustment according to claim 4, characterized in that, The specific method for calculating the feature-level correlation strength is as follows: For radar semantic feature vectors and visual semantic feature vectors, semantic similarity is calculated using the vector inner product form, and the semantic similarity is linearly mapped to obtain the linearly mapped semantic similarity. The radar semantic feature vector and the visual semantic feature vector are subjected to Softmax transformation to obtain the radar semantic weight and the image semantic weight. The Kullback-Leibler divergence is used to measure the difference between radar semantic weights and image semantic weights, and symmetric KL divergence is calculated. The symmetric KL divergence is normalized and mapped to semantic complementarity score. Feature-level association strength is calculated based on semantic complementarity score.

6. A multimodal semantic coding method based on intermodal correlation strength adjustment according to claim 5, characterized in that, The specific method for calculating sample-level association strength is as follows: Based on visual semantic features and radar semantic feature vectors, the target's position estimate in three-dimensional space and the target's spatial position in the radar coordinate system are extracted respectively. The difference between the target's spatial position in three-dimensional space and the radar coordinate system is calculated and mapped to spatial matching degree. The motion vectors of the target in three-dimensional space and the target in the radar coordinate system are extracted separately. The similarity between the motion vectors of the target in three-dimensional space and the target in the radar coordinate system is calculated using cosine similarity and normalized to the motion matching degree. By combining spatial matching degree and motion matching degree, the sample-level association strength is calculated.

7. A multimodal semantic coding method based on intermodal correlation strength adjustment according to claim 6, characterized in that, The specific methods for extracting image modal quality factors and radar signal modal quality factors are as follows: The proportion of the target in the RGB image is calculated as the image modal quality factor; the stability of the radar echo signal is calculated as the radar signal modal quality factor. The specific method for classifying into different related scenarios is as follows: Set the correlation strength threshold When the correlation strength tensor At that time, radar semantic feature vectors and visual semantic feature vectors are classified into highly correlated scenes, and the correlation strength tensor is... At that time, radar semantic feature vectors and visual semantic feature vectors are classified into low-association scenarios.

8. A multimodal semantic coding method based on intermodal correlation strength adjustment according to claim 1, characterized in that, The specific method for weighted fusion of radar semantic feature vectors and visual semantic feature vectors based on the associated scenario is as follows: In highly correlated scenarios, visual fusion weights are set based on the correlation strength tensor. Radar fusion weights In low-association scenarios, visual fusion weights are set based on the association strength tensor. Radar fusion weights The visual semantic features and radar semantic feature vectors are fused and L2 normalized to obtain fused semantic features.

9. A multimodal semantic coding method based on intermodal correlation strength adjustment according to claim 1, characterized in that, The specific method for semantic mapping and encoding of fused semantic features is as follows: The fused semantic features are projected onto the LLM semantic embedding space through a linear mapping to construct the initial semantic embedding; a correlation strength tensor is introduced. Amplitude modulation is applied to the initial semantic embedding to form an association-aware semantic embedding; the association-aware semantic embedding is then... The input is fed into a large language model encoder consisting of a multi-layered Transformer structure to obtain the semantic encoding result.

10. A multimodal semantic coding method based on intermodal correlation strength adjustment according to claim 1, characterized in that, The multi-task collaborative semantic decoding module includes a shared semantic decoding backbone network and multiple task-specific decoding heads; The specific method for decoding the semantic encoding result and generating the target motion state prediction result is as follows: Through linear mapping and reconstruction operations, the semantic encoding results are converted into initial decoding features, which are then input into a shared semantic decoding backbone network to extract shared features. By inputting shared features into different task-specific decoding heads, the predicted target motion state can be obtained.

Images