Method for generating multi-modal features and computer device

Abstract

Embodiments of the present application disclose a multi-modal feature generation method and a computer device. Embodiments of the present application first obtain a target single-modal feature in an actual scene and send it to an encoder of a pre-trained public hidden space model to convert it into a structured single-modal sparse code; then, the single-modal sparse code is input into a pre-trained target prediction model, which infers and completes missing information according to learned mapping rules to generate complete multi-modal sparse code; finally, the multi-modal sparse code is input into a shared weight decoder for decoding and restoration, and multi-modal features containing rich cross-modal information are output for use by downstream tasks. Even if a sensor in a multi-modal sensor fails, the single-modal feature of the failed sensor can be supplemented.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to a method for generating multimodal features and a computer device. Background Technology

[0002] Existing environmental perception systems heavily rely on multiple sensors (such as cameras and radar) operating simultaneously online, resulting in high hardware costs and complex joint calibration. Such systems have poor robustness; if a relevant sensor fails in severe weather (such as radar failure due to heavy rain), the system will directly lose that modality information, leading to a sharp drop in perception performance. Summary of the Invention

[0003] One objective of this application is to provide a method and computer device for generating multimodal features, thereby improving the related technology's inability to complete the single-modal features of a failed sensor when a sensor in a multimodal sensor fails.

[0004] In a first aspect, embodiments of this application provide a method for generating multimodal features, comprising: acquiring target unimodal features; inputting the target unimodal features into a pre-trained common hidden space model to obtain a unimodal sparse code, wherein the common hidden space model includes a pre-trained encoder and a shared weight decoder; inputting the unimodal sparse code into a pre-trained target prediction model, so that the target prediction model performs multimodal prediction processing on the unimodal sparse code to obtain a multimodal sparse code; and inputting the multimodal sparse code into the shared weight decoder, so that the shared weight decoder performs decoding processing on the multimodal sparse code to obtain multimodal features.

[0005] Optionally, the step of inputting the target single-modal features into a pre-trained common hidden space model to obtain single-modal sparse coding includes: obtaining a set of modal features, the set of modal features including sample features of multiple single modalities; training the common hidden space model to be trained based on the set of modal features to obtain a pre-trained common hidden space model; and encoding the target single-modal features based on the encoder of the pre-trained common hidden space model to obtain single-modal sparse coding.

[0006] Optionally, the step of training the common hidden space model to be trained based on the modal feature set to obtain a pre-trained common hidden space model includes: encoding the sample features using an encoder to be trained to obtain sample sparse encodings in the common hidden space, wherein the sample sparse encodings include independent activation regions, common activation regions, and non-activation regions; generating a set of target loss functions based on the sample sparse encodings of each sample feature, wherein the set of target loss functions is used to constrain the independent activation regions of each sample sparse encoding to form a mutually exclusive relationship, and also to constrain the feature distribution of the common activation regions of each sample sparse encoding to remain similar; decoding the sample sparse encodings using a shared weight decoder to be trained to obtain reconstructed features; generating a reconstruction loss function based on the reconstructed features and the sample features; and performing model training operations based on the set of target loss functions and the reconstruction loss function to obtain a pre-trained common hidden space model.

[0007] Optionally, the target loss function set includes a sparse loss function, a mutual exclusion loss function, and a similarity loss function. The step of generating the target loss function set based on the sample sparse coding of each sample feature includes: generating a sparse loss function based on each sample sparse coding, wherein the sparse loss function constrains the sparsity of the sample sparse coding to increase the sparsity of the sample sparse coding; generating a mutual exclusion loss function based on each sample sparse coding, wherein the mutual exclusion loss function constrains any two sample features to remain mutually exclusive in the independent activation region; and generating a similarity loss function based on each sample sparse coding, wherein the similarity loss function constrains any two sample features to maintain similar feature distributions in the common activation region.

[0008] Optionally, generating a sparse loss function based on each of the sample sparse codes includes: calculating the L1 norm of the sample sparse codes, and generating a sparse loss function based on the feature channel size of the common hidden space and the L1 norm of the sample sparse codes; generating a mutual exclusion loss function based on each of the sample sparse codes includes: calculating the element-wise product of any two sample sparse codes, summing all element-wise products to obtain a total product, determining the number of features in the sample feature set, and generating a mutual exclusion loss function based on the number of features and the total product; generating a similarity loss function based on each of the sample sparse codes includes: calculating the similarity of any two sample sparse codes to obtain a code similarity, summing all code similarities to obtain a total code similarity, determining the number of features in the sample feature set, and generating a similarity loss function based on the number of features and the total code similarity.

[0009] Optionally, generating a reconstruction loss function based on the reconstructed features and the sample features includes: calculating the feature difference between the reconstructed features and the sample features; determining the feature channel size of the common hidden space and the number of features in the modal feature set; and generating a reconstruction loss function based on the feature channel size of the common hidden space, the number of features in the modal feature set, and the feature difference.

[0010] Optionally, the step of inputting the unimodal sparse code into a pre-trained target prediction model, so that the target prediction model performs multimodal prediction processing on the unimodal sparse code to obtain a multimodal sparse code, includes: obtaining a modal feature set, the modal feature set including sample features of multiple unimodals; training the target prediction model to be trained based on the modal feature set to obtain a pre-trained target prediction model; and performing multimodal prediction processing on the unimodal sparse code based on the pre-trained target prediction model to obtain a multimodal sparse code.

[0011] Optionally, the step of training the target prediction model to be trained based on the modal feature set to obtain a pre-trained target prediction model includes: inputting the sample features into the encoder of a pre-trained common hidden space model to obtain target sparse coding; performing feature fusion processing on the target sparse coding of all sample features to obtain fused sparse coding; inputting the target sparse coding into the target prediction model to be trained for multimodal prediction processing to obtain multimodal prediction coding; and training the target prediction model to be trained based on the fused sparse coding and the multimodal prediction coding to obtain a pre-trained target prediction model.

[0012] Optionally, the step of training the target prediction model to be trained based on the fused sparse coding and the multimodal predictive coding to obtain a pre-trained target prediction model includes: generating a loss optimization function based on the fused sparse coding and the multimodal predictive coding; and training the target prediction model to be trained based on the loss optimization function to obtain a pre-trained target prediction model.

[0013] In a second aspect, embodiments of this application provide a computer device including a memory and a processor. The memory is connected to the processor, and the processor is configured to execute one or more computer programs stored in the memory. When the processor executes the one or more computer programs, it causes the computer device to implement the above-described method for generating multimodal features.

[0014] The embodiments of this application can achieve the following technical effects: First, the single-modal features of the target in the actual scene are obtained and fed into the encoder of the pre-trained public hidden space model, which is then transformed into a structured single-modal sparse code. Subsequently, the single-modal sparse code is input into the pre-trained target prediction model, which infers and fills in the missing information based on the learned mapping rules to generate a complete multimodal sparse code. Finally, the multimodal sparse code is input into the shared weight decoder for decoding and restoration, and outputs multimodal features containing rich cross-modal information for use by downstream tasks. Even if a sensor fails in a multimodal sensor, the single-modal features of the failed sensor can still be filled in. Attached Figure Description

[0015] To more clearly illustrate the technical solutions of the embodiments of this application, the drawings used in the description of the embodiments of this application will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0016] Figure 1 A flowchart illustrating a method for generating multimodal features according to an embodiment of this application; Figure 2 This is a schematic diagram of the structure of a computer device provided in an embodiment of this application. Detailed Implementation

[0017] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are for illustrative purposes only and are not intended to limit the scope of this application. All other embodiments obtained by those skilled in the art based on the embodiments in this application without inventive effort are within the scope of protection of this application.

[0018] It should be noted that, unless there is a conflict, the various features in the embodiments of this application can be combined with each other, all of which are within the protection scope of this application. Furthermore, although functional modules are divided in the device schematic diagram and a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in a different order than the module division in the device or the order in the flowchart. Moreover, the terms "first," "second," and "third" used in this application do not limit the data or execution order, but only distinguish identical or similar items with essentially the same function and effect.

[0019] Existing autonomous robots, self-driving cars, and other environmental perception and interaction equipment often employ 2D optical imaging sensors and 3D radar sensors for environmental detection, combined with text, audio, and other modalities for human-computer interaction. For data fusion from these multimodal sensors, current methods commonly employ addition, weighted addition, feature concatenation, or Transformer cross-attention. These methods essentially limit themselves to exploring shallow interactions between multimodal features and emphasize the relative importance of each modality's information in the fused features using weights, lacking explicit modeling of the deep structural relationships between modalities.

[0020] While multi-sensor devices offer some redundancy in terms of safety and performance, the high hardware costs, complex multi-sensor calibration, and immense computational burden of multimodal data have led the market to gradually shift towards a design paradigm primarily based on single-modal sensors (especially optical sensors), such as in purely vision-based autonomous vehicles. However, single-modal inputs inevitably face the challenge of missing environmental information; for example, purely optical imaging lacks distance information, and purely radar imaging lacks texture information.

[0021] To compensate for the lack of information in a single modality, current technologies often require deep learning models to learn and generate information from other modalities based on single-modal inputs. The main methods include: a) Mapping method based on Multi-Layer Perceptron (MLP): This method establishes a direct mapping between input modal features and missing modal features during the training phase, and generates approximate missing modal features during the inference phase. While attempting to establish a shared space for multimodal features, this method lacks explicit constraints on the spatial structure. Due to the significant differences between modal features (e.g., involving image-to-3D discrete point cloud re-representation, dimensionality compression from 3D point cloud to 2D projection, and matrix-to-text serialization), direct mapping easily leads to feature confusion and unsatisfactory generation results.

[0022] b) Generative Adversarial Networks (GANs) based methods: These methods utilize generative models to generate outputs from single-modal inputs in other modalities, using discriminators or ground truth information as supervisory constraints. However, this process inevitably faces the inherent flaw of contradictory gradient directions between the generator and discriminator, leading to highly unstable model training.

[0023] In summary, existing technologies for generating multimodal modes from single modalities are all dedicated to exploring the mapping problem from a single modality to other modalities within a common space. However, their core drawback is that the construction of the common space lacks reasonable structural constraints, making it difficult to guarantee the accuracy of the mapping between modalities. The generated features are often noisy and deviate from the true distribution, failing to meet the stringent requirements of security and robustness in practical applications.

[0024] In view of this, this application proposes a method for generating multimodal features, which can be seamlessly embedded as a plug-in into deep learning feature extraction networks that take single-modal (such as 2D images, 3D point clouds, text, etc.) information as input. It aims to improve the environmental perception (such as semantic segmentation, depth information prediction, target recognition, etc.) performance of existing autonomous driving and autonomous robots in extreme or single sensor failure scenarios.

[0025] Specifically, this embodiment explicitly constructs a common hidden space, dividing features from different modalities into independent activation regions, common activation regions, and inactive regions, and introduces a multilayer perceptron (MLP) to achieve cross-modal generation. During the training phase, multimodal sample features are jointly trained to obtain a pre-trained common hidden space model (including an encoder and a shared weight decoder) with structured sparse coding capabilities, as well as a pre-trained target prediction model responsible for cross-modal completion. During the inference phase, this embodiment can stably generate high-quality multimodal features using only any single-modal feature input through encoding, MLP prediction completion, and shared weight decoding.

[0026] The overall implementation process is as follows: First, feature vectors of multimodal samples are extracted using existing deep learning feature extraction networks. Based on these paired features, a common hidden space model and a target prediction model are trained offline, enabling the models to grasp the mutual exclusion and consensus rules between modalities. In the real-world deployment phase of machine learning-based mobile equipment (such as autonomous vehicles), if encountering single-modal input (e.g., only camera images during heavy rain), this embodiment encodes and performs multimodal prediction processing on the single-modal features, generating predicted multimodal fusion sparse codes, and then decodes them back to multimodal features. These completed multimodal features can be directly input into downstream task networks, effectively compensating for the information loss of the original single-modal input, thereby significantly improving the accuracy and robustness of perception tasks such as segmentation, detection, and depth prediction.

[0027] For details, please refer to Figure 1 In this embodiment of the application, multimodal features are generated through steps S11 to S14, as detailed below: S11, Obtain the target single-modal features.

[0028] The target single-modal feature refers to the feature data actually received by the system during the inference phase, containing information of one modality. Specifically, the target single-modal feature is obtained by a feature extraction model extracting features from sensor information acquired by sensors. Sensor information includes image information, laser point cloud information, radar point cloud information, text information, video information, or audio information, etc. For example, if the sensor information is image information, and the feature extraction model is a deep learning feature extraction network, the target single-modal feature is obtained by the deep learning feature extraction network extracting features from the image information acquired by the camera device. The target single-modal feature can be the contour features of a human body, the contour features of other objects, or other texture features. As another example, if the sensor information is point cloud information, the target single-modal feature is obtained by the deep learning feature extraction network extracting features from the point cloud information acquired by the LiDAR. As yet another example, if the sensor information is speech information, the target single-modal feature is obtained by the deep learning feature extraction network extracting features from the speech information acquired by the speech acquisition device.

[0029] For example, an autonomous vehicle is equipped with cameras and LiDAR. In heavy rain, when the LiDAR fails, the vehicle relies solely on the cameras behind the windshield. The cameras capture real-time images of the road ahead, generating continuous RGB image information. These RGB images are 1920×1080 color images. This RGB image information is input into a pre-trained deep learning feature extraction network. Through feature extraction processing by the deep learning network, high-dimensional target single-mode features are obtained. The deep learning feature extraction network can be a ResNet-50, MobileNet, or VisionTransformer model.

[0030] S12, input the target unimodal features into the pre-trained public hidden space model to obtain unimodal sparse coding.

[0031] The common hidden space model is a pre-trained network containing a pre-trained encoder and a shared-weight decoder, responsible for converting different target unimodal features into a unified intermediate language. The encoder is a network module that maps and compresses the input target unimodal features into sparse codes within the common hidden space, transforming the target unimodal features into specific activation signals within the common hidden space. The shared-weight decoder is a network module corresponding to the encoder and using the same set of network parameters to restore features, ensuring that all codes within the common hidden space can be restored to the same standard. It is used to translate the finally completed multimodal sparse codes back into multimodal features that the system can directly use. Shared weights mean that during the training phase, the decoder uses the same set of network weight parameters to restore data from all modalities, such as image features and point cloud features. This ensures that the codes within the common hidden space have unified and standardized decryption rules.

[0032] Unimodal sparse encoding is a minimal representation of the target unimodal features in the common hidden space, with most values ​​being 0. The target unimodal feature is typically a high-dimensional feature tensor, for example, a feature map of [256, 64, 192]. In this embodiment, the target unimodal feature is input into a fully connected layer (or a global pooling layer plus a fully connected layer) in a pre-trained encoder. The fully connected layer stores a pre-trained weight matrix. The target unimodal feature is multiplied by the weight matrix and a bias term is added, thereby compressing and flattening the high-dimensional feature into a fixed-length low-dimensional vector. This allows complex pixel-level information to be refined into a more compact semantic feature vector. The dimensionality-reduced semantic feature vector is then processed by a non-linear activation function in the encoder (such as ReLU, Sigmoid, or tanh). The rule of the ReLU function is: for each element in the semantic feature vector, if the value is greater than 0, the original value is retained; if the value is less than or equal to 0, it is forcibly changed to 0. Because the pre-trained encoder has undergone strict constraints through a sparse loss function during the early training phase, it has learned to map a large amount of irrelevant redundant information to the negative range. Therefore, after ReLU activation, the values ​​of most elements in the semantic feature vector become 0, with only a small number of elements remaining non-zero, thus obtaining a single-modal sparse encoding. Single-modal sparse encoding preserves the most essential feature information of the modality in a minimalist way.

[0033] For example, the image information captured by the camera device is extracted by a CNN network to obtain target monomodal features containing a large amount of texture, color, and edge information. In this embodiment, the target monomodal features are input into a pre-trained encoder, which performs the following operations: Mapping: The encoder's weight matrix translates and compresses the target single-modal features to obtain a digit sequence of length 256.

[0034] Truncation: The ReLU activation function examines the numerical sequence. The encoder detects that the object has red and octagonal features, and the corresponding positive values ​​in the dimensions are preserved (e.g., 2.3 for the 5th dimension and 1.5 for the 18th dimension); while the values ​​of the background sky, irrelevant noise, etc., are deliberately suppressed to negative values ​​by the encoder, and the ReLU activation function directly turns them to 0.

[0035] Result: The final output unimodal sparse code is [0,0,2.3,0,...,1.5,0], with fewer than 30 non-zero numbers out of 256. The unimodal sparse code only fills in the visual information that the camera device can see, while the dimensions related to distance information are all 0 (blank), awaiting completion by the subsequent target prediction model.

[0036] S13. Input the single-modal sparse code into the pre-trained target prediction model so that the target prediction model can perform multimodal prediction processing on the single-modal sparse code to obtain the multimodal sparse code.

[0037] The target prediction model is a pre-trained model used for unimodal sparse coding cues to infer and generate information about missing modalities, enabling information completion within the common latent space. The target prediction model can be a multilayer perceptron, a linear prediction model, or other types of models.

[0038] Multimodal sparse coding is a hidden space coding that contains all modal information after prediction by the target prediction model. It belongs to the completed multimodal information. For example, multimodal sparse coding is the result of single-modal sparse coding completion, which includes comprehensive information such as image texture and point cloud depth.

[0039] The system takes the unimodal sparse code generated by the encoder in the previous step as input data and passes it to the input layer of the pre-trained target prediction model. The pre-trained target prediction model has already learned the mapping rules from the unimodal latent space to the multimodal fusion latent space during the training phase, and its internal weight parameters are fixed. The pre-trained target prediction model performs a weighted summation operation on the fully connected layers. The unimodal sparse code is multiplied by the weight matrix of the first hidden layer of the target prediction model, and a bias term is added to obtain the weighted summation result. The pre-trained target prediction model performs a nonlinear activation operation, inputting the weighted summation result into a nonlinear activation function (such as ReLU). The nonlinear activation function can capture the complex nonlinear mapping relationships between modalities. The pre-trained target prediction model performs deep network iterative operations. Through continuous iterative calculations of multiple fully connected layers and activation functions, the target prediction model infers and activates those missing modal dimensions (such as distance and normal vectors) that were originally 0, based on cues from known modalities (such as the activation values ​​of color and shape in an image). The pre-trained target prediction model performs multimodal sparse coding output operations. After computational operations in the output layer of the target prediction model, the target prediction model generates multimodal sparse codes. Multimodal sparse coding not only preserves the feature information of the original input modalities but also completes the feature information of other missing modalities.

[0040] For example, the single-modal sparse code generated in the previous step resembles an incomplete table, with non-zero numbers only filled in the dimensions corresponding to color (red) and shape (octagon), while the cell corresponding to the distance "15 meters" is 0. In this embodiment, the single-modal sparse code is used to pre-train a target prediction model. The target prediction model internally performs fast matrix multiplication and activation calculations for multimodal prediction processing. The target prediction model detects that the current input lacks distance information, so it generates a new non-zero value in the "distance" dimension, which was originally 0, through weight calculation. Finally, the target neural network model outputs a multimodal sparse code. At this point, the multimodal sparse code not only contains visual information but also inferred 3D spatial depth information, preparing the subsequent decoder to reconstruct the complete multimodal features.

[0041] S14, input the multimodal sparse code into the shared weight decoder so that the shared weight decoder can decode the multimodal sparse code to obtain multimodal features.

[0042] Multimodal features are high-dimensional feature vectors that fuse multimodal information as the final output. They can be directly used for downstream tasks such as object detection and semantic segmentation. Multimodal features can provide autonomous driving systems with richer and more accurate environmental descriptions than single-modal features, directly improving the accuracy and robustness of perception tasks (such as ranging and recognition).

[0043] In the inference phase, this embodiment first acquires the target's single-modal features in the actual scene and feeds them into the encoder of a pre-trained public hidden space model, transforming them into structured single-modal sparse coding. Subsequently, the single-modal sparse coding is input into a pre-trained target prediction model, which infers and fills in the missing information based on the learned mapping rules, generating a complete multimodal sparse coding. Finally, the multimodal sparse coding is input into a shared weight decoder for decoding and reconstruction, outputting multimodal features containing rich cross-modal information for use by downstream tasks. Even if a sensor fails in a multimodal sensor, the single-modal features of the failed sensor can still be filled in.

[0044] In this embodiment, the multimodal sparse code (e.g., a 256-dimensional vector containing visual and depth information) generated by the target prediction model in the previous step is passed to the shared weight decoder in the pre-trained public hidden space model. This embodiment performs matrix dimensionality upscaling, multiplying the 256-dimensional multimodal sparse code with the weight matrix of the fully connected layer inside the decoder and adding a bias term. Through iterative computation of multiple layers, the low-dimensional and compact vector is gradually expanded and upscaled, restoring a high-dimensional tensor structure. Nonlinear activation functions are interspersed during the dimensionality upscaling process to help the network reconstruct complex feature distributions and spatial structure information, ensuring that the restored features are not overly smoothed or distorted. After multiple layers of inverse mapping computation, the decoder finally outputs a high-dimensional feature tensor, thus obtaining the multimodal features. These multimodal features contain information from the original input modality (e.g., texture and color of a camera image) and also incorporate information from other modalities predicted and completed by the model (e.g., depth and normal vectors of radar), and can be directly input into downstream task networks (e.g., target detection heads and semantic segmentation heads).

[0045] For example, after the previous step, the target prediction model outputs a complete multimodal sparse code. In the 256-dimensional vector of the multimodal sparse code, activation values ​​such as "red," "octagon," and the predicted "distance 15 meters" are recorded. The target prediction model transmits the multimodal sparse code to the shared weight decoder. The shared weight decoder uses the same set of weight parameters as when processing real radar data to perform matrix multiplication and deconvolution calculations on the 256-dimensional vector, gradually unfolding the compressed numbers and restoring them into a high-dimensional tensor with spatial dimensions. Finally, the decoder outputs the multimodal features. At this point, the multimodal features possess both the clear texture seen by the camera and the precise three-dimensional spatial depth, similar to radar scanning. After obtaining the multimodal features, the downstream target detection network can not only recognize the stop sign but also accurately calculate that it is 15 meters away, thereby controlling the autonomous vehicle to brake smoothly. Thus, the closed loop of single-modal to multimodal generation and restoration is completed.

[0046] To provide a detailed understanding of the embodiments of this application, the following example is provided for further description: An autonomous vehicle is driving in a rainstorm. During the rainstorm, the autonomous vehicle's LiDAR malfunctions, and the camera captures a "red stop sign." The system acquires the target's single-modal features corresponding to the "red stop sign," which, after processing by an encoder, yields a single-modal sparse code (containing only color and shape, without distance information). The single-modal sparse code is input into a target prediction model, which completes the distance information based on the single-modal sparse code to obtain a multimodal sparse code. Finally, the multimodal sparse code is translated by a shared-weight decoder to obtain multimodal features. These multimodal features help the autonomous vehicle accurately determine the sign and brake.

[0047] This application's embodiments achieve the goal of generating complete multimodal features using only the target single-modal features as input, breaking the hardware limitation that multiple sensors must be online simultaneously, reducing costs, and providing modal information redundancy in extreme scenarios, significantly improving perception security.

[0048] In this embodiment of the application, through steps S121 to S123, the target unimodal features are input into a pre-trained common hidden space model to obtain unimodal sparse coding, as detailed below: S121, Obtain the modal feature set, which includes sample features of multiple single modalities.

[0049] The modal feature set is a feature library collected during the training phase, containing features from various modalities, including image modalities, point cloud modalities, and text modalities. The modal feature set provides rich, multi-dimensional learning materials for training the common hidden space model, enabling the model to learn the diversity of information from different modalities and thus summarize the mutual exclusion and consensus relationships between various modalities.

[0050] Sample features are feature vectors extracted from the original modality data. Assume the modality feature set contains sample features from N single modalities. Where F is the modal feature set, C is the feature channel size, and N is the number of sample features contained in the modal feature set. Let S be the feature of the i-th sample, and R be the set of real numbers. Define the common hidden space of the multimodal features as S, where S is the hidden space for any sample feature. An encoder with a given parameter-driven common hidden space Each sample feature is mapped to a common hidden space S, and a shared weight decoder is used. Reconstruct the features of each sample from the public hidden space S. ,in, For encoder, For a shared weight decoder, d is the channel size of the common hidden space.

[0051] For example, before the vehicle is officially put on the road, the mobile equipment will simultaneously activate all types of sensors and drive under normal operating conditions (such as sunny days) to collect multimodal raw data with strictly aligned timestamps and spatial coordinates. The collected raw data of each modality will be input into the deep learning feature extraction backbone network of the corresponding modality to perform high-dimensional feature extraction, thereby obtaining sample features. The sample features of the extracted multiple single modalities will be combined and stored in the form of sample pairs to obtain the modal feature set.

[0052] For example, when an autonomous driving test vehicle is driving on a sunny day, the lidar on the roof and the camera behind the windshield work simultaneously to capture one frame of raw RGB image data and one frame of 3D laser point cloud data for the same "red stop sign". The raw RGB image is input into a visual backbone network (such as ResNet-50) to extract the image feature tensor. At the same time, the 3D laser point cloud data is input into a point cloud processing network (such as PointNet or VoxelNet) to extract the point cloud feature tensor. The extracted image feature vector and the corresponding point cloud feature vector are bound together as a sample pair. By collecting tens of thousands of sample pairs, a modal feature set containing 2D image sample features and 3D point cloud sample features is formed.

[0053] S122, based on the modal feature set, train the public hidden space model to be trained to obtain a pre-trained public hidden space model.

[0054] The public hidden space model to be trained is a model whose network parameters are not yet fixed and is in a learning state. Before practical application, the system first collects a modal feature set containing multiple sample features, and uses the modal feature set to jointly train the public hidden space model to be trained until the model parameters converge to obtain the pre-trained public hidden space model. During inference, the encoder in the pre-trained model is directly called to encode the target single modal features input in real time, thereby extracting the corresponding single modal sparse code.

[0055] In this embodiment, steps A1 to A5 are performed to train the common hidden space model to be trained based on the modal feature set, resulting in a pre-trained common hidden space model, as detailed below: Step A1: Use the encoder to be trained to encode the sample features to obtain the sparse coding of the samples in the common hidden space.

[0056] Sample sparse coding is a coding method that takes sample features from the modality feature set, inputs them into the encoder to be trained, and generates codes after random initialization or continuous updating of the weight matrix and truncation by the activation function.

[0057] Sparse coding of samples includes independent activation regions, common activation regions, and inactive regions. Independent activation regions are areas that are activated only by a specific modality (with a value greater than 0) and store information unique to that modality (such as image color). For example, independent activation regions are specifically used to store the RGB color information of an image, and the point cloud does not respond in this region.

[0058] A common activation region is a region that is jointly activated by multiple modalities and stores cross-modal shared information. For example, a common activation region stores the geometric contour information of an object, and images and point clouds respond in this region and express similar information.

[0059] The inactive region is the region where all modes are inactive, filtering out redundancy and noise.

[0060] Encoders for public hidden spaces There are sparse codings of samples corresponding to any sample features. Among them, sample sparse coding The expression is Arbitrary sample features In the public hidden space S, it can be decoupled into independently activated areas. Public Activation Area Inactive area Composition. Among them... Represents only sample features It has an activation response in public hidden spaces, that is Non-negative sparse coding; common activation region Representative sample features and all other modal feature sets They all possess non-negative sparse coding, and at the same time, the corresponding and Because they exhibit similar sparse coding distribution characteristics within the common activation region, i.e. Inactive area Representative sample features The non-positive sparse coding result is represented in the common hidden space S. Indicates sample characteristics In the remaining modal feature set of This manifests as an activation response, requiring that the independent activation regions of each modality be mutually exclusive: .

[0061] For example, in the encoder's output layer, a fixed-length output vector (e.g., 256 dimensions) is pre-defined. During network design, these 256 dimensions are logically divided into three continuous or discrete regions: independent activation regions, common activation regions, and inactive regions. For example, the independent activation regions are the first 100 dimensions of the sample sparse encoding, specifically used to store features unique to each modality (such as color / texture in 2D images, or spatial distance / normal vectors in 3D point clouds). The common activation regions are the middle 100 dimensions (dimensions 101-200) of the sample sparse encoding, specifically used to store features common to all modalities (such as the geometric contours and spatial positions of objects). The inactive regions are the last 56 dimensions (dimensions 201-256) of the sample sparse encoding, serving as information isolation areas, with their output values ​​forced to 0.

[0062] This embodiment extracts sample features (such as high-dimensional image feature tensors) corresponding to a single modality from the modal feature set and inputs them into the encoder to be trained. The fully connected layer inside the encoder uses the current weight matrix to be trained to perform matrix multiplication and bias addition operations on the sample features, compressing and reducing their dimensionality to the aforementioned 256-dimensional vector. This embodiment applies specific activation and masking strategies to the dimensionality-reduced vector to generate the final sample sparse encoding, as detailed below: 1) Zeroing out inactive areas: In this embodiment of the application, the last 56 dimensions are set to 0 directly through masking operations to ensure that there is absolutely no information leakage in this area.

[0063] 2) Sparse activation of independent and common regions: A non-linear activation function (such as ReLU) is applied to the first 200 dimensions (independent and common activation regions), where values ​​greater than 0 are retained and values ​​less than 0 are truncated to 0. Under the constraints of subsequent multiple loss functions (such as sparse loss function, mutually exclusive loss function, and similarity loss function) during the training phase, the model gradually learns to adjust the weights: mapping image features to their independent activation regions as much as possible, mapping common contour features to common activation regions, while irrelevant information is compressed to negative values ​​and becomes 0.

[0064] In other embodiments, the present application embodiments can also utilize the encoder to be trained to encode sample features, obtaining sample sparse coding in the common hidden space. The sample sparse coding includes independent activation regions, common activation regions, and inactive regions; wherein, the independent activation regions, common activation regions, and inactive regions are generated based on an energy-based masking mechanism. Specifically, the present application embodiments regard the encoded feature activation values ​​as energy distributions. By comparing the energy states of different modal features in the same dimension, masking is used to classify dimensions with high energy for a single modality as independent activation regions, dimensions with high energy shared by multiple modalities as common activation regions, and dimensions with energy below a preset threshold as inactive regions.

[0065] Specifically, during the encoding process, this embodiment calculates the energy values ​​(such as absolute feature values ​​or norms) of each modal feature across different dimensions. If a dimension exhibits high energy only in one modality, it is masked to 0 by an independent activation mask in other modalities, thus forming an independent activation region. If a dimension exhibits high energy in multiple modalities, it is retained by a common activation mask, forming a common activation region. For dimensions with energy below a preset threshold, they are considered redundant noise and are forcibly set to zero by a non-activation mask, forming a non-activation region. This embodiment defines the structural source of the multimodal hidden space based on an energy distribution-based masking mechanism, mapping feature responses to energy distributions. This makes the boundaries of independent activation regions, common activation regions, and non-activation regions no longer dependent on manual settings, but dynamically determined by the energy level of the data itself, which is beneficial for improving the robustness and fault tolerance of cross-modal alignment.

[0066] Step A2: Generate a set of target loss functions based on the sparse coding of each sample feature.

[0067] The target loss function set is used to constrain the independent activation regions of the sparse codes of each sample to form a mutually exclusive relationship, and also to constrain the feature distribution of the common activation regions of the sparse codes of each sample to remain similar. The target loss function set includes sparse loss functions, mutually exclusive loss functions, and similarity loss functions.

[0068] A sparse loss function is a loss function that minimizes the number of non-zero elements in the encoding of a common hidden space. It constrains the sparsity of the sparse encoding of a sample, thereby increasing its sparsity. For example, the sparse loss function forces the common hidden model to ignore sky noise behind a sign and retain only a very small number of non-zero digits in the encoding.

[0069] In some embodiments, generating a set of target loss functions based on the sparse coding of each sample feature includes the following steps: generating a sparse loss function based on the sparse coding of each sample.

[0070] The process of generating a sparse loss function based on the sparse coding of each sample includes the following steps: obtaining the L1 norm of the sparse coding of the sample, and generating a sparse loss function based on the feature channel size of the common hidden space and the L1 norm of the sparse coding of the sample.

[0071] The expression for the sparse loss function is as follows:

[0072]

[0073] Sparse encoding for the i-th sample, This is a linear rectified function used to set non-positive numbers to 0. Uppercase C represents the feature channel size, and lowercase c represents the c-th feature channel in the sparse coding of the i-th sample. The loss value of the sparse loss function is used to constrain the sparsity of the sparse codes of samples in the common hidden space. For encoder.

[0074] The mutual exclusion loss function penalizes overlap between different modalities in independent activation regions. It constrains any two sample features to remain mutually exclusive within their independent activation regions, forcing the common hidden space model to strictly separate modality-specific information and prevent mutual interference. For example, the mutual exclusion loss function ensures that the encoding dimension representing "red" is activated only in the image, while the point cloud remains 0; similarly, the dimension representing "precise distance" is activated only in the point cloud, while the image remains 0.

[0075] In some embodiments, generating a set of target loss functions based on the sparse coding of each sample feature includes the following steps: generating a mutually exclusive loss function based on the sparse coding of each sample.

[0076] The steps for generating a mutual exclusion loss function based on the sparse coding of each sample are as follows: calculate the element-wise product of the sparse coding of any two samples, accumulate the element-wise products to obtain the total product, determine the number of features in the sample feature set, and generate a mutual exclusion loss function based on the number of features and the total product.

[0077] The expression for the mutual exclusion loss function is as follows:

[0078] This represents the loss value of the mutual exclusion loss function.

[0079] The similarity loss function is a function that constrains different modalities to have similar encoding distributions in the common activation region. It is used to constrain the feature distributions of any two sample features to remain similar in the common activation region. For example, it ensures that the "octagon" in the image and the "octagon" in the point cloud are mapped to almost the same position and value in the common hidden space.

[0080] In some embodiments, generating a set of target loss functions based on the sparse coding of each sample feature includes: generating a similarity loss function based on the sparse coding of each sample.

[0081] The steps for generating a similarity loss function based on the sparse coding of each sample are as follows: calculate the similarity of sparse coding of any two samples to obtain coding similarity; sum all coding similarities to obtain total coding similarity; determine the number of features in the sample feature set; and generate a similarity loss function based on the number of features and the total coding similarity.

[0082] The expression for the similarity loss function is as follows:

[0083] The loss value is the value of the similarity loss function. Let be the similarity function, where the similarity function can be cosine similarity, Euclidean distance similarity, Manhattan distance, or other similarity functions.

[0084] This application's embodiments introduce L1 norm, element-wise product, and similarity function, and supplement them with channel size and feature number for normalization, ensuring the stability and convergence of the training process, thus providing a precise mathematical means for implementing the region division rules of the common hidden space.

[0085] This application embodiment divides the overall region of sparse coding of samples into three specific, functionally independent loss functions, thereby achieving refined management of the structure of the common hidden space. The three loss functions work together to construct a feature space that is structurally rigorous, information-separated, and aligned.

[0086] Step A3: Use the shared weight decoder to be trained to decode the sparse coding of the samples to obtain the reconstructed features.

[0087] The reconstructed features are those restored by the shared-weight decoder to be trained based on the sparse encoding of the samples. The system passes the sparse encoding of the samples output by the encoder to be trained to the input layer of the shared-weight decoder to be trained. The shared-weight decoder to be trained contains multiple fully connected layers, and the specific mathematical operations performed are as follows: 1) Matrix multiplication and bias addition: The sparse encoding of the samples is multiplied by the current weight matrix to be trained in the decoder, and a bias term is added to linearly expand the low-dimensional compact vector. 2) Multi-layer cascaded dimensionality increase: Through continuous calculations of multiple network layers, the low-dimensional vector is gradually expanded, and the spatial resolution and channel dimension are continuously increased, eventually restoring it to a high-dimensional feature tensor with the same feature size as the original input sample. 3) Nonlinear activation: Nonlinear activation functions are interspersed between the layers of dimensionality increase calculation to help the network reconstruct complex nonlinear feature distributions and avoid the restored features being too smooth or distorted. 4) Output reconstructed features: After multi-layer inverse mapping dimensionality increase and nonlinear activation calculations, the decoder finally outputs a high-dimensional reconstructed feature. Subsequently, the reconstructed feature will be extracted and compared element-wise with the original sample features input to the encoder to calculate the reconstruction loss function.

[0088] During the training phase, the shared weight decoder uses the same set of weight matrices to perform the aforementioned inverse mapping calculation, regardless of whether the input image modality or the input 3D point cloud modality is sparsely encoded. This forces the model to align the features of different modalities to a unified standard during the encoding phase, facilitating the accurate reconstruction of both the image and point cloud using the same set of weights.

[0089] For example, consider training on a "red stop sign." Both image features and point cloud features are encoded into a common hidden space. For independent activation regions, bits 1-10 of the encoding are image-specific regions; only image encodings have numbers in these regions, while point cloud encodings are all 0s (point clouds have no color information). For common activation regions, bits 11-20 of the encoding are common regions; both image and point cloud encodings have numbers in these regions, and because they both describe "octagons," the distribution of these numbers is very similar. Finally, a shared-weight decoder reconstructs these encodings into the original image and original point cloud (reconstructed features). If the reconstructed image is blurry, the reconstruction loss function increases, forcing the network to readjust. Through continuous adjustments, the common hidden model eventually learns to divide these three regions.

[0090] Step A4: Generate a reconstruction loss function based on the reconstruction features and sample features.

[0091] This application's embodiments calculate the feature difference between the reconstructed features and the sample features, determine the feature channel size of the common hidden space and the number of features in the modal feature set, and generate a reconstruction loss function based on the feature channel size of the common hidden space, the number of features in the modal feature set, and the feature difference.

[0092] The expression for the reconstruction loss function is as follows:

[0093]

[0094] To reconstruct the loss value of the loss function, For the i-th reconstructed feature, For decoder, Let be the feature of the i-th sample.

[0095] For example, the decoder to be trained attempts to sparsely encode the samples of red stop signs to reconstruct the image. If the reconstructed image becomes a green stop sign or a blurry mosaic, it produces a large feature difference compared to the original input image of a clear red stop sign. This embodiment of the application uses a weighted average of this feature difference, channel size, and number of features to derive the loss value of the reconstruction loss function. This embodiment of the application adjusts the decoder parameters based on the loss value of the reconstruction loss function until the reconstructed image is almost identical to the original image.

[0096] The reconstruction loss function is the bottom line to ensure that no information is lost. By adding the reconstruction loss, the model is forced to retain enough information to reconstruct the features of the input samples while satisfying the spatial structure constraints.

[0097] Step A5: Perform model training based on the target loss function set and reconstruction loss function to obtain a pre-trained common hidden space model.

[0098] In this embodiment, the sparse loss, mutual exclusion loss, similarity loss, and reconstruction loss obtained from forward computation are weighted and summed to form a total loss. Based on this total loss, the backpropagation algorithm is used, according to the chain rule in calculus, to calculate the gradient of the total loss for all parameters in the public hidden model to be trained (including the weight matrix of the encoder and the weight matrix of the shared-weight decoder). This embodiment uses a preset optimizer to update the weight parameters of the public hidden model to be trained based on the calculated gradients. This embodiment repeats the above-described cycle of forward computation, multiple losses, backpropagation, and parameter updates. As training progresses, the total loss value continuously decreases and gradually converges. When the public hidden model to be trained can stably and accurately map different sample features to the structured public hidden space and can clearly reconstruct the data, training stops, and the weight parameters of the encoder and decoder at this point are saved, thus obtaining the pre-trained public hidden space model.

[0099] S123, The encoder based on the pre-trained public hidden space model encodes the target single-modal features to obtain single-modal sparse coding.

[0100] The pre-trained encoder uses its fixed weight matrix to perform matrix multiplication on the target unimodal features and adds a bias term, compressing and projecting the target unimodal features from the high-dimensional feature space into a fixed-length low-dimensional vector. This reduced-dimensional vector then enters a non-linear activation layer. Since the encoder has already accepted the long-term constraint of sparse loss in the "target loss function set" during pre-training, it has learned to map irrelevant or redundant information as negative numbers. Therefore, when the low-dimensional vector passes through the ReLU activation function, all values ​​less than or equal to 0 are forcibly truncated to 0, retaining only the key feature activation values ​​greater than 0, thus achieving extreme sparsity of the features. After the above dimensionality reduction and truncation processing, the encoder finally outputs a unimodal sparse code with most elements being 0, retaining only the core information of the modality. This unimodal sparse code implicitly follows the spatial structure learned during pre-training in terms of dimensionality distribution, preparing for cross-modal completion in the subsequent target prediction model.

[0101] For example, during a rainstorm, a camera captures a red stop sign ahead. A CNN model extracts image features (i.e., target unimodal features). The system then calls a pre-trained encoder. The encoder performs multiplication on the image features based on a fixed weight matrix, compressing it into a 256-dimensional number sequence, which is then truncated using the ReLU activation function. Since the encoder is aware of key features such as the red color and the octagonal sign, it calculates the dimensions corresponding to these key features as positive numbers (e.g., the 5th dimension is 2.3), while calculating irrelevant background noise as negative numbers. After truncation, the system outputs a unimodal sparse code with only a few non-zero numbers.

[0102] In this embodiment of the application, through steps S131 to S133, the unimodal sparse code is input into the pre-trained target prediction model, so that the target prediction model performs multimodal prediction processing on the unimodal sparse code to obtain a multimodal sparse code, as detailed below: Step S131: Obtain the modal feature set, which includes sample features of multiple single modalities.

[0103] This application's embodiments map sample features from multiple single modalities to a unified coordinate system and perform feature size normalization and alignment processing to obtain sample features of a unified size. Since the original data structures of different modalities differ significantly, direct fusion or encoding would introduce severe noise due to scale mismatch. Therefore, this application's embodiments introduce a unified coordinate system mechanism before inputting the sample features into the encoder to be trained. This application's embodiments use projection transformation methods, global pooling methods, or interpolation methods to forcibly align the high-dimensional feature tensors of different modalities to a unified feature size and coordinate system, eliminating spatial and scale differences between modalities.

[0104] Step S132: Train the target prediction model to be trained based on the modal feature set to obtain a pre-trained target prediction model.

[0105] The target prediction model to be trained is a model responsible for forcing the model to learn the direct mapping from single-modal features to fused features. The target prediction model can be a multilayer perceptron (MLP), containing fully connected layers and nonlinear activation functions. In the pre-training stage, this embodiment uses a set of modal features containing multiple sample features to train the target prediction model to be trained, enabling it to learn the mapping rule from the single-modal latent space to the multimodal fusion latent space. After training, the parameters are fixed. In the inference stage, this embodiment directly inputs the single-modal sparse code into the pre-trained target prediction model, which then performs multimodal prediction processing to quickly generate the predicted multimodal fusion sparse code.

[0106] For example, in this embodiment of the application, the single-modal sparse code of the image of the red stop sign is input into the target prediction model. The target prediction model learns to derive the multimodal fusion code from the single-modal code based on a fully connected layer and a nonlinear activation function. In rainy weather, when the lidar fails, the target prediction model can predict the single-modal sparse code of the camera device as a complete multimodal sparse code.

[0107] The process of training the target prediction model based on the modal feature set to obtain the pre-trained target prediction model includes the following steps: B1 inputs the sample features into the encoder of the pre-trained public hidden space model to obtain the target sparse encoding.

[0108] The target sparse encoding is obtained by inputting sample features into a pre-trained encoder with fixed parameters in the first stage, and directly outputting the encoding after calculating the forward matrix. The common hidden space model has been pre-trained, and the encoder's weight parameters have been frozen and will not be updated. In this embodiment, sample features are input into the pre-trained encoder to obtain the target sparse encoding.

[0109] B2 performs feature fusion processing on the target sparse coding of all sample features to obtain fused sparse coding.

[0110] Fusion sparse coding involves mathematically fusing the target sparse codes of all modalities to generate a complete code containing all modal information. The expression for fusion sparse coding is as follows:

[0111] in, To integrate sparse coding, For fusion function, fusion function It can be any suitable fusion method, such as additive fusion, weighted additive fusion, etc.

[0112] For example, the pre-trained encoder receives a pair of sample features, namely, image features and point cloud features for a "red stop sign". The pre-trained encoder uses a fixed weight matrix to calculate the image features and point cloud features. The image features are transformed into image sparse codes recording "red" and "octagon" (i.e., the first target sparse code), and the point cloud features are transformed into point cloud sparse codes recording "distance 15 meters" and "octagon" (i.e., the second target sparse code). In this embodiment, independent activation regions are fused. Specifically, the independent activation regions of the image sparse code (such as color and texture) and the independent activation regions of the point cloud sparse code (such as depth and normal vector) are mutually exclusive in dimension. During fusion, only the non-zero activation values ​​of each are retained.

[0113] Next, this embodiment of the application fuses the common activation area. Specifically, in the common activation area where shared information (such as object outline and spatial location) is stored, if both image sparse coding and point cloud sparse coding activate the same dimension, this embodiment of the application takes the maximum value of the two; if only one is activated, the non-zero value is retained; if both are 0, the fused value is still 0.

[0114] After the above fusion calculation, the embodiments of this application generate a new fused sparse code. The fused sparse code integrates features unique to the image modality, features unique to the point cloud modality, and common features shared by both.

[0115] B3. Input the sparse target code into the target prediction model to be trained for multimodal prediction processing to obtain the multimodal prediction code.

[0116] Multimodal prediction processing is the process by which a target prediction model receives the target sparse code as input and predicts the multimodal sparse code. Multimodal prediction encoding involves inputting the single-modal target sparse code into the target prediction model to be trained, and obtaining the prediction result after weight calculation and nonlinear activation of the target prediction model. The generative expression for multimodal prediction encoding is as follows:

[0117] For multimodal predictive coding, For target prediction models.

[0118] In this embodiment, a single-modal target sparse code is input into the target prediction model to be trained. The target prediction model performs a dot product operation between the target sparse code and the weight matrix of the first hidden layer of the target prediction model, and adds a bias term to obtain a weighted sum. The weighted sum is then input into a nonlinear activation function to help the target prediction model capture the complex nonlinear mapping relationships between modalities. After continuous calculations through multiple fully connected layers and activation functions, the target prediction model activates the missing modal dimensions (such as distance and normal vectors) that were originally 0 at the output end based on cues from known modalities. After calculations at the final output layer of the target prediction model, a new multimodal prediction code is generated. The multimodal prediction code not only attempts to retain the feature information of the original input modality, but also attempts to fill in the missing modal dimensions "out of thin air".

[0119] For example, in this embodiment, the sparse encoding of the image of the red stop sign is input into the target prediction model to be trained. In this sparse encoding, only the dimensions of the red sign and the octagon are non-zero; the values ​​of the other dimensions are all 0. The weight matrix inside the target prediction model performs continuous multiplication and activation calculations on the sparse encoding, attempting to deduce the values ​​of the other dimensions based on the clues of "red" and "octagon." Since the target prediction model is not yet properly trained, it may guess an incorrect number in the dimension representing "distance" (e.g., predicting a distance of 2 meters). Finally, the target prediction model outputs a multimodal prediction encoding filled with numbers. This embodiment compares the multimodal prediction encoding with the fused sparse encoding to obtain the error. This embodiment uses this error for backpropagation to correct the weights of the target prediction model.

[0120] B4, based on the fusion of sparse coding and multimodal predictive coding, trains the target prediction model to be trained to obtain a pre-trained target prediction model.

[0121] The embodiments of this application generate a loss optimization function based on the fusion of sparse coding and multimodal predictive coding. Based on the loss optimization function, the target prediction model to be trained is trained to obtain a pre-trained target prediction model.

[0122] Specifically, in this embodiment, the dissimilarity metric between the fused sparse coding and the multimodal predictive coding is calculated to obtain the loss optimization function. By minimizing the loss optimization function, the multimodal predictive coding is forced to approximate the true fused sparse coding, thereby completing the training of the target prediction model.

[0123] There are various methods for generating loss optimization functions. In some embodiments, this application calculates the absolute value error or mean square error between the fused sparse code and the multimodal predictive code to obtain the loss optimization function. In some embodiments, similarity indices such as cosine similarity between the fused sparse code and the multimodal predictive code are calculated to obtain the loss optimization function. In some embodiments, this application normalizes the fused sparse code and the multimodal predictive code to convert them into probability distributions, thereby obtaining a loss optimization function related to KL divergence.

[0124] KL divergence (Kullback-Leibler divergence) is a directly used divergence metric that constrains the similarity between the distribution of multimodal predictive coding and the distribution of fused sparse coding, forcing the generated results to approximate the true representation and avoiding overfitting.

[0125] The expression for the loss optimization function regarding KL divergence is as follows:

[0126] in, This represents a normalization function (such as the Softmax function) that transforms fused sparse coding or multimodal predictive coding into a probability distribution. This represents the KL divergence.

[0127] In optimizing the target prediction model, this embodiment of the application normalizes the fused sparse code (serving as the standard answer) and the multimodal prediction code output by the target prediction model, converting them into probability distributions. Then, the KL divergence between the two probability distributions is calculated, and a loss optimization function based on the KL divergence is constructed. By minimizing the loss optimization function based on the KL divergence, the distribution of the multimodal prediction code is forced to approximate the distribution of the true fused sparse code, thus completing the training of the target prediction model. This embodiment of the application, by introducing KL divergence to constrain distribution consistency, allows the target prediction model to maintain the correct overall distribution shape compared to traditional numerical approximation, enhancing the robustness and generalization ability of generated features and effectively avoiding overfitting.

[0128] In general, when training the target prediction model, given the input sample features, pre-trained encoder, shared weight decoder, and fused sparse code for reference, the sample features are first input into the pre-trained common hidden space model to obtain the target sparse code, and all target sparse codes are fused to obtain the fused sparse code as the standard answer. At the same time, the target sparse code is input into the target prediction model to be trained to obtain the multimodal prediction code. Finally, the network parameters of the target prediction model are adjusted by comparing the multimodal prediction code and the fused sparse code, so that the target prediction model learns the mapping rules of deriving multimodal features from single-modal features.

[0129] Step S133: Perform multimodal prediction processing on the single-modal sparse code based on the pre-trained target prediction model to obtain the multimodal sparse code.

[0130] The expression for multimodal sparse coding is: , It is a single-modal sparse coding. It is a multimodal sparse coding.

[0131] After training, the target prediction model has mastered the cross-modal mapping rules, and its weight parameters are fixed. When a sensor fails due to severe weather, this embodiment utilizes the pre-trained target prediction model to complete the missing modal information. Specifically, this embodiment inputs a single-modal sparse code into the pre-trained target prediction model. The pre-trained target prediction model performs multi-layer continuous dot multiplication and bias addition operations with the input single-modal sparse code and the internal fixed weight matrix. The pre-trained target prediction model utilizes the physical and semantic relationships between modalities learned during training (such as "red octagon" usually corresponding to specific "spatial distance and 3D contour"), uses the activation cues of known modalities as trigger conditions, and calculates the feature responses that the missing modality should possess in the internal hidden layer. In the output layer of the target prediction model, in order to ensure that the output results still conform to the structured sparsity characteristics of the common hidden space (i.e., containing independent activation regions, common activation regions, and most dimensions being 0), the embodiments of this application add a nonlinear activation function at the output end. If the missing modality feature value calculated by the target prediction model is greater than 0, it is retained as a high-confidence prediction result and activated. If the target prediction model believes that some dimensions do not need to be completed or have extremely low confidence, they are calculated as negative numbers and become 0 after being truncated by the nonlinear activation function.

[0132] After the aforementioned forward inference and activation truncation, the target prediction model ultimately outputs a new multimodal sparse code. This multimodal sparse code not only fully preserves the feature information of the known single modality at the input, but also successfully predicts and fills in the feature information of the missing modality.

[0133] In this embodiment, multimodal sparse coding is input into a shared-weight decoder, so that the shared-weight decoder decodes the multimodal sparse coding to obtain multimodal features. The expression for the multimodal features is: , It is a multimodal feature.

[0134] For example, during a rainstorm, the LiDAR signal of an autonomous vehicle is severely attenuated, meaning the LiDAR fails, but the camera works normally. The camera captures a red stop sign ahead, and the pre-trained encoder compresses the image information corresponding to the red stop sign into a single-modal sparse code (containing only non-zero values ​​representing color and shape, with all point cloud dimensions set to 0). This single-modal sparse code is then input into a pre-trained target prediction model. The target prediction model uses a fixed weight matrix for forward computation. The neurons inside the target prediction model recognize the "red octagon" cue and, based on the patterns learned during training, automatically calculate the corresponding point cloud distance and 3D geometric feature values. The output layer of the target prediction model is then truncated to obtain a multimodal sparse code. This multimodal sparse code not only retains the non-zero values ​​representing "red, octagon," but also generates non-zero predicted values ​​representing "distance 15 meters, column structure" on the originally 0 point cloud dimension. In this embodiment, multimodal sparse coding is input into a shared weight decoder, enabling the shared weight decoder to decode the multimodal sparse coding and obtain multimodal features. Thus, even with physically missing radar data, the system successfully completes multimodal information at the feature space level, allowing it to continue to be stably used by downstream tasks (such as 3D target detection).

[0135] In summary, the embodiments of this application have at least the following advantages: 1) Modeling of explicit multimodal common hidden space improves feature fusion quality and interpretability: Traditional multimodal mapping methods based on multilayer perceptron (MLP) have two major limitations: First, implicit feature coupling, the black box nature of MLP leads to a lack of interpretability in the feature fusion process, making it difficult to accurately locate the contribution area of ​​key modalities; Second, it is sensitive to modal differences, the significant differences in scale (such as 3D point cloud coordinates and 2D pixel values) and distribution (such as the sparsity of LiDAR and the density of RGB) of different modal features can easily introduce noise in the fusion process.

[0136] To address the aforementioned issues, this application's embodiments explicitly define activation rules for the multimodal common hidden space, enabling quantifiable analysis of the feature contributions of each modality. This significantly improves the interpretability of the common hidden space and reduces model complexity. Simultaneously, a dynamic masking mechanism is introduced to allow multimodal features to autonomously partition within the common hidden space, effectively eliminating interference from scale and distribution differences between modalities and greatly enhancing fusion robustness. This method fundamentally avoids the disruption of the fusion process caused by heterogeneous feature differences, ultimately effectively improving the accuracy of environmental perception tasks (such as object detection, semantic segmentation, and depth prediction).

[0137] 2) Cross-modal feature generation technology based on common hidden space constraints breaks through the bottleneck of single-modal perception: Traditional methods are difficult to directly generate complete multimodal features from single-modal input, resulting in limited accuracy of single-modal perception tasks (such as pure visual depth estimation) in sensor-limited scenarios.

[0138] This application proposes a method for generating multimodal features based on the modeling of a common hidden space. During the training phase, the encoder, decoder, and target prediction model for the common hidden space are jointly trained. During the inference phase, by freezing the network parameters, a complete multimodal feature output can be generated relying solely on the pre-trained network and any independent single-modal feature input. This application implements depth completion of single-modal information (e.g., completing texture features corresponding to 3D distance information, or completing corresponding depth features for texture images), which is of great significance for improving single-modal environment perception tasks.

[0139] It should be noted that in the above embodiments, there is no necessarily a certain order between the steps. Those skilled in the art can understand from the description of the embodiments of this application that the above steps may have different execution orders in different embodiments, that is, they may be executed in parallel or in turn, etc.

[0140] See Figure 2 , Figure 2 This is a schematic diagram of a computer device provided in an embodiment of this application. The computer device 200 includes one or more processors 21 and a memory 22. The memory 22 is connected to one or more processors 21, for example, via a bus.

[0141] Processor 21 is configured to support the computer device in performing the corresponding functions in the methods described in the above method embodiments. The processor may be a central processing unit (CPU), a network processor (NP), a hardware chip, or any combination thereof. The aforementioned hardware chip may be an application-specific integrated circuit (ASIC), a programmable logic device (PLD), or a combination thereof. The aforementioned PLD may be a complex programmable logic device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL), or any combination thereof.

[0142] Memory 22 is used to store program code, etc. Memory may include volatile memory (VM), such as random access memory (RAM); memory may also include non-volatile memory (NVM), such as read-only memory (ROM), flash memory, hard disk drive (HDD), or solid-state drive (SSD); memory may also include combinations of the above types of memory.

[0143] The memory 22 can be used to store non-volatile software programs, non-volatile computer-executable programs, and modules, such as the program instructions / modules corresponding to the multimodal feature generation method in the embodiments of this application. The processor executes various functional applications and data processing of the multimodal feature generation method by running the non-volatile software programs, instructions, and modules stored in the memory, thereby realizing the function of the multimodal feature generation method provided in the above method embodiments.

[0144] The memory may include a program storage area and a data storage area. The program storage area may store the operating system and application programs required for at least one function. The data storage area may store data created using the multimodal feature generation method, etc.

[0145] The one or more modules are stored in the memory. When executed by the one or more processors, they perform the multimodal feature generation method in any of the above method embodiments. For example, they perform the method steps described in the above method embodiments to realize the functions of the modules described in the above device embodiments.

[0146] This application also provides a computer-readable storage medium storing a computer program, the computer program including program instructions, which, when executed by a computer device, cause the computer device to perform the method described in the foregoing embodiments.

[0147] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. The storage medium can be a magnetic disk, optical disk, read-only memory (ROM), or random access memory (RAM), etc.

[0148] The above-disclosed embodiments are merely preferred embodiments of this application and should not be construed as limiting the scope of this application. Therefore, any equivalent variations made in accordance with the claims of this application shall still fall within the scope of this application.

Claims

1. A method for generating multimodal features, characterized in that, include: Obtain the target single-modal features; The target unimodal features are input into a pre-trained common hidden space model to obtain unimodal sparse coding, wherein the common hidden space model includes a pre-trained encoder and a shared weight decoder. The single-modal sparse code is input into a pre-trained target prediction model so that the target prediction model performs multimodal prediction processing on the single-modal sparse code to obtain a multimodal sparse code. The multimodal sparse code is input into the shared weight decoder so that the shared weight decoder can decode the multimodal sparse code to obtain multimodal features.

2. The generation method according to claim 1, characterized in that, The step of inputting the target unimodal features into a pre-trained public hidden space model to obtain unimodal sparse coding includes: Obtain a modal feature set, which includes sample features of multiple single modalities; The common hidden space model to be trained is trained based on the modal feature set to obtain a pre-trained common hidden space model. The encoder based on the pre-trained public hidden space model encodes the target unimodal features to obtain unimodal sparse coding.

3. The generation method according to claim 2, characterized in that, The process of training the public hidden space model to be trained based on the modal feature set to obtain a pre-trained public hidden space model includes: The sample features are encoded using the encoder to be trained to obtain sparse coding of the samples in the common hidden space. The sparse coding of the samples includes independent activation regions, common activation regions and non-activation regions. A set of target loss functions is generated based on the sample sparse coding of each sample feature. The set of target loss functions is used to constrain the independent activation regions of each sample sparse coding to form a mutually exclusive relationship, and also to constrain the feature distribution of the common activation regions of each sample sparse coding to remain similar. The sparse code of the sample is decoded using a shared weight decoder to be trained, and the reconstructed features are obtained. A reconstruction loss function is generated based on the reconstructed features and the sample features; Based on the set of target loss functions and the reconstruction loss function, a model training operation is performed to obtain a pre-trained common hidden space model.

4. The generation method according to claim 3, characterized in that, The target loss function set includes sparse loss functions, mutual exclusion loss functions, and similarity loss functions. The generation of the target loss function set based on the sample sparse encoding of each sample feature includes: A sparse loss function is generated based on the sparse codes of each sample. The sparse loss function is used to constrain the sparsity of the sparse codes of the samples so as to increase the sparsity of the sparse codes of the samples. A mutual exclusion loss function is generated based on the sparse coding of each of the samples. The mutual exclusion loss function is used to constrain any two of the sample features to remain mutually exclusive in the independent activation region. A similarity loss function is generated based on the sparse encoding of each sample. The similarity loss function is used to constrain the feature distribution of any two samples to remain similar in the common activation region.

5. The generation method according to claim 4, characterized in that, The step of generating a sparse loss function based on the sparse codes of each sample includes: calculating the L1 norm of the sparse codes of the samples, and generating a sparse loss function based on the feature channel size of the common hidden space and the L1 norm of the sparse codes of the samples. The step of generating a mutual exclusion loss function based on the sparse codes of each sample includes: calculating the element-wise product of any two sparse codes of the samples, summing all element-wise products to obtain a total product, determining the number of features in the sample feature set, and generating a mutual exclusion loss function based on the number of features and the total product. The step of generating a similarity loss function based on the sparse codes of each sample includes: performing similarity calculation on any two sparse codes of the samples to obtain a coding similarity; summing all coding similarities to obtain a total coding similarity; determining the number of features in the sample feature set; and generating a similarity loss function based on the number of features and the total coding similarity.

6. The generation method according to claim 3, characterized in that, The step of generating a reconstruction loss function based on the reconstructed features and the sample features includes: Calculate the feature difference between the reconstructed features and the sample features; Determine the feature channel size of the public hidden space and the number of features in the modal feature set; A reconstruction loss function is generated based on the feature channel size of the public hidden space, the number of features in the modal feature set, and the feature difference.

7. The generation method according to any one of claims 1 to 6, characterized in that, The step of inputting the single-modal sparse code into a pre-trained target prediction model, so that the target prediction model performs multimodal prediction processing on the single-modal sparse code to obtain a multimodal sparse code, includes: Obtain a modal feature set, which includes sample features of multiple single modalities; Based on the modal feature set, the target prediction model to be trained is trained to obtain a pre-trained target prediction model. The single-modal sparse code is processed by multimodal prediction based on the pre-trained target prediction model to obtain a multimodal sparse code.

8. The generation method according to claim 7, characterized in that, The process of training the target prediction model to be trained based on the modal feature set to obtain a pre-trained target prediction model includes: The sample features are input into the encoder of the pre-trained public hidden space model to obtain the target sparse encoding; The target sparse coding of all sample features is subjected to feature fusion processing to obtain fused sparse coding. The target sparse code is input into the target prediction model to be trained for multimodal prediction processing to obtain the multimodal prediction code; Based on the fused sparse coding and the multimodal predictive coding, the target prediction model to be trained is trained to obtain a pre-trained target prediction model.

9. The generation method according to claim 8, characterized in that, The step of training the target prediction model to be trained based on the fused sparse coding and the multimodal prediction coding to obtain a pre-trained target prediction model includes: A loss optimization function is generated based on the fused sparse coding and the multimodal predictive coding; Based on the loss optimization function, the target prediction model to be trained is trained to obtain a pre-trained target prediction model.

10. A computer device, characterized in that, The device includes a memory and a processor, the memory being connected to the processor, the processor being configured to execute one or more computer programs stored in the memory, the processor causing the computer device to implement the method for generating multimodal features as described in any one of claims 1-9 when executing the one or more computer programs.

Images