End-to-end autonomous driving long-tail recognition method based on contrastive learning pre-training

Abstract

The present application relates to the technical field of automatic driving end-to-end perception, in particular to an end-to-end automatic driving long-tail recognition method based on contrast learning pre-training, first, a synthetic image data with long-tail distribution characteristics is generated through a conditional diffusion model, then a fine-grained scene classifier is used to systematically organize and semantically label the generated samples, and a structured multi-modal graph-text alignment dataset is constructed; finally, the enhanced dataset and the original training set are fused, the visual-linguistic joint embedding space is optimized through a multi-task contrast loss function, and the parameter update of the pre-training model is realized. The method innovatively establishes a closed-loop optimization mechanism of generative data enhancement and contrast learning framework, effectively alleviates the data scarcity problem under the long-tail distribution scene, and significantly improves the cross-modal representation ability and downstream task generalization performance of the model on low-resource classes.

Description

Technical Field

[0001] This invention relates to the field of end-to-end perception technology for autonomous driving, and more specifically, to an end-to-end long-tail recognition method for autonomous driving based on contrastive learning pre-training. Background Technology

[0002] End-to-end perception technology is a core area of ​​autonomous driving systems. It involves the autonomous vehicle's perception of its external environment and the transmission of this perceived information to subsequent modules or for direct control output. This perception stage is crucial for autonomous driving systems because it is the starting point of the entire chain, directly impacting all subsequent vehicle behaviors. Accurate perception helps vehicles make more precise decisions, identify potential hazards and conflicts, and promptly detect obstacles and other vehicles that could affect driving safety. In complex environments, accurate perception of environmental information is affected not only by conventional vehicles but also by long-tail scenarios described by the Pareto principle. Therefore, the perception system of autonomous vehicles must not only be able to identify conventional road and traffic information but also to identify and classify long-tail scenarios. Against this backdrop, accurately identifying and classifying long-tail scenarios has become a major challenge for autonomous driving technology.

[0003] Currently, various methods have been proposed in the field of autonomous driving for long-tail scene recognition. Most of these techniques, based on scene classification and detection, combine convolutional neural networks with long short-term memory networks and use generative adversarial networks for data augmentation and synthesis. In addition, some methods utilize small sample data to incrementally learn from the original network, thereby improving the ability to recognize rare scenes.

[0004] The method combining convolutional neural networks (CNNs) and long short-term memory (LSTM) networks performs well in long-tail scene recognition, accurately identifying and classifying such scenes. However, this method is highly dependent on high-quality labeled data. In conventional training tasks, long-tail scenes are usually not trained separately and constitute a small proportion of the dataset, making the model prone to overfitting and resulting in decreased recognition accuracy when faced with new long-tail data. Furthermore, the computational complexity of CNNs and LSM networks is high, limiting their adaptability when dealing with complex and variable weather conditions and poorly labeled datasets.

[0005] Incremental learning based on small sample data has significant advantages in adapting to long-tail and newly added scenarios, but it also presents certain challenges. One of the main problems is catastrophic forgetting, meaning that when learning a new model, previously learned old scenarios and knowledge may be forgotten. Furthermore, if the newly added incremental dataset is too sparse, the model may not be able to fully learn the features of these scenarios, thus affecting its recognition accuracy and adaptability.

[0006] Therefore, while current long-tail scene recognition problems have high accuracy and precision in recognizing relatively common long-tail events, the scarcity of training datasets limits their recognition in complex and uncertain scenarios, and the computational overhead is too high, failing to meet the real-time requirements of end-to-end autonomous driving. Summary of the Invention

[0007] In view of this, the present invention proposes an end-to-end autonomous driving long-tail recognition method based on contrastive learning pre-training to solve the problems existing in the prior art.

[0008] To achieve the above objectives, this invention proposes an end-to-end autonomous driving long-tail recognition method based on contrastive learning pre-training, comprising: S1: Collect existing long-tail scene images based on public datasets and perform data annotation processing. Input image-text pairs as input sequence X into the diffusion model to generate synthetic data image-text pairs Y. S2: Input the synthetic data image pair Y into the contrastive learning pre-trained model for training and initially update the weights; S3: Images from the vehicle's forward-looking perspective are collected by the vehicle's onboard camera and fed into the incrementally trained contrastive pre-trained model for zero-shot recognition of long-tail scenes, and incremental learning is performed. S4: Use a low-rank adaptive model to update the weights of the pre-trained model in real-world scenarios, and connect it to a bridging module and a lightweight LLM to perform long-tail scene recognition and classification tasks. Preferred option: S1 includes the following steps: S101: Label data in public datasets; S102: Input the labeled dataset into the diffusion model and output the synthesized image; S103: Pack the synthesized image and its prompts into a text-image pair Y; Preferred option: S2 specifically refers to: Retrieve the image-text pair Y packaged in S1: Specifically, the images and descriptions of long-tail scenes are input into the TextEncoder and ImageEncoder of the comparison pre-trained model, respectively, for processing. In the Text Encoder, text is input into the Transformer network. The specific steps are as follows: First, the input is segmented into words, embedded, and then positional encoding is added. Here, positional encoding uses sine and cosine functions of different frequencies, specifically as follows:

[0009]

[0010]

[0011] Where pos represents the position in the input sequence Y, i.e., the position index; 2k, 2k+1 represent the dimension index ( D represents the dimension of the word embedding layer, which is the same as the total dimension of the position encoding vector. This is the word vector passed in.

[0012] Then, layer normalization is passed in, specifically represented as:

[0013] in The word vectors are after position encoding. The word vectors are after layer normalization. LayerNorm is the layer normalization process, which is as follows: Calculate the mean:

[0014] in It is the mean of the sample across all features. It is the i-th feature value of the sample.

[0015] Calculate the variance:

[0016] in It is the variance of all features of the sample.

[0017] Normalization:

[0018] in These are the normalized eigenvalues, resulting in a mean of 0 and a variance of 1. It is a very small number used to prevent the denominator from being 0.

[0019] Then, the text is normalized and passed to multiple attention layers to capture the relationships between the characters. Each head calculates the attention separately. The specific process is as follows: The formulas for calculating Q, K, and V of the input features are as follows: , ,

[0020] in, The input features are the word vectors that have undergone layer normalization. These are the learnable parameter matrices, These are three vectors representing the query, key, and value of the input feature Y.

[0021] Then, self-attention feature atten ( The formula is as follows:

[0022] Here, a multi-head attention mechanism is used, and the formula is as follows:

[0023] in This indicates the attention of the bulls. This represents the features in the splicing of multiple attention heads. Represents the number of attention heads, The weight matrix represents the multi-head attention mechanism used to capture temporal dependencies, spatial relationships, contextual information, and correlations between data points in the features. After residual processing, layer normalization is performed again, as shown in the following formula:

[0024]

[0025] in These are word vectors after positional encoding, without multiple attention layers. The word vectors are those that have passed through a multi-head attention layer. These are the word vectors after residual processing and layer normalization.

[0026] The data is then fed into a feedforward network. The feedforward network uses a linear transformation to first map the data to a high-dimensional space and then to a low-dimensional space, extracting deeper features. The feedforward network consists of two connection layers plus the GELU activation function, as shown in the following formula:

[0027]

[0028] Where FFN represents a feedforward network, and GELU represents a nonlinear activation function, short for Gaussian error linear unit. These are the learnable parameters in the network.

[0029] The word vectors after activation are further processed by residual processing and layer normalization to improve model performance and training stability.

[0030] Then the processed word vectors The attention feature values ​​are passed into the Transformer Decoder.

[0031] In the Transformer Decoder, the long-tail scene word vectors after word segmentation are processed separately. The output is right-shifted, positionally encoded, and then passed through a masked multi-head attention layer. Residual processing and layer normalization are then performed. The attention feature values ​​are input into the next multi-head attention network, where residual processing and layer normalization are repeated. A feedforward layer is then added, followed by residual processing and layer normalization. Finally, the output is mapped to the required D-dimensional word vectors in a linear layer and normalized using softmax to generate the final word distribution for subsequent image-text similarity calculations. The specific process is as follows: First, when processing input sequences in long-tail scenarios, a right shift operation is used, which involves shifting the entire word vector one position to the right and inserting a special marker vector [SOS] at the beginning of the sequence. The specific formula is as follows:

[0032]

[0033] in is the word vector in the original sequence, n is the number of input word vectors, PAD is the padding symbol used to maintain length, and SOS is the start marker.

[0034] Positional encoding is performed on the padded word vectors. Here, positional encoding is performed using sine and cosine functions of different frequencies, and the encoded vectors are then input into a masked multi-head attention layer. The specific formula is as follows:

[0035]

[0036]

[0037] , ,

[0038]

[0039]

[0040] Where pos represents the position in the input sequence Y, i.e., the position index; 2k, 2k+1 represent the dimension index ( D represents the dimension of the word embedding layer, which is the same as the total dimension of the position encoding vector. The word vectors are after position encoding. Input features A query consists of three vectors: key, value, and key. This represents a masked attention layer, where M is a mask matrix, typically a very small negative number, used to mask future positions. MSA is for multi-head attention computation. This represents the features in the splicing of multiple attention heads.

[0041] The output attention features, after residual processing and layer normalization, are input into the next multi-head attention layer, where computation is performed. Values, and simultaneously, the processed word vectors The attention feature values ​​are calculated in this multi-head attention layer. , The calculated attention features are weighted and summed, then passed through residual connections and layer normalization before being input into a feedforward neural network for nonlinear transformation and feature enhancement. The specific formula is as follows:

[0042]

[0043]

[0044]

[0045]

[0046] =

[0047]

[0048]

[0049] in This represents the word vectors after residual processing and layer normalization. LayerNorm is the layer normalization process. Represents the cross-attention layer. Represent The Q value and K value, The V value, MSA is used for multi-head attention calculation. This represents the features in the concatenation of multiple attention heads; FFN represents a feedforward network; and GELU represents a non-linear activation function. These are the learnable parameters in the network.

[0050] Finally, for Feature extraction is performed, and the extracted features are input into a linear layer for reshaping and mapping to dimension D to accommodate subsequent image-text similarity calculations. The specific formula is as follows:

[0051]

[0052] in To extract The last special marker vector, namely [EOS], The feature vector is mapped to D dimensions and used to calculate the cosine similarity between the image and the text. These are the parameters that the network can learn, and the main purpose is to project special label vectors into the D dimension.

[0053] In the Image Encoder, the image is input into a complex network, and the specific steps are as follows: Image input preprocessing begins by dividing the image into blocks with a side length of 14. The initial image size is 224x224, and the number of blocks is 196. The specific formula is as follows:

[0054]

[0055] Where p is the block size, N is the number of blocks, and C is the number of input image channels. Let be the set of all real numbers.

[0056] Then the segmented vector The input is fed into a linear layer and projected onto the hidden dimension D, as shown in the following formula:

[0057]

[0058]

[0059] Where D represents the hidden dimension, which is also the dimension of word embedding and positional encoding. The vector is the graph after projection. The parameters that can be learned by the linear network are of size . .

[0060] The image is then positionally encoded to introduce positional information into the Transformer structure, compensating for the self-attention mechanism's inability to perceive the order of the input sequence. Here, we choose learnable positional encoding, and the specific formula is as follows:

[0061]

[0062] , . in For learnable location encoding, initial random values ​​are assigned using a uniform distribution, with a size of [value missing]. The Adam optimizer is selected to update parameters based on gradient values. Let be the probability density function of a uniform distribution. , It consists of two parameters.

[0063] The location-encoded graph vector is input into a network consisting of N identical stacked Transformer layers, with each layer having the following structure: First, the input graph vector is subjected to layer normalization, as shown in the following formula:

[0064] in For graph vectors, For graph vectors that have undergone layer normalization, This is a layer normalization layer.

[0065] Subsequently, the graph vectors are input into a multi-head attention layer for processing. Each head calculates its attention independently, and the multi-head outputs are then combined. The specific formula is as follows: , ,

[0066]

[0067]

[0068] in Input features The query consists of three vectors: key, value, and key. For multi-head attention layer calculation, This represents the features in the splicing of multiple attention heads. These are the parameters that the network can learn.

[0069] Subsequently, the attention features obtained from multi-head attention calculation are fed into a multilayer perceptron network after residual connections to generate the final output, as shown in the following formula:

[0070]

[0071]

[0072] in This is the graph vector after residual processing. It is a feedforward layer. It is a non-linear activation function. For layer normalization, This is the graph vector output after passing through the feedforward layer, residual processing, and layer normalization. These are parameters that can be learned in the feedforward layer.

[0073] The following extracts the learnable classification labels from the image vectors of long-tailed scenes, and maps the features to a contrastive learning space of dimension D using a projection head. The specific formula is as follows:

[0074]

[0075] in To extract The first special marker vector, namely [CLS], The feature vector is mapped to D dimensions and used to calculate the cosine similarity between the image and the text. These are the parameters that the network can learn, and the main purpose is to project special label vectors into the D dimension.

[0076] Compare the outputs of the pre-trained model's Text Encoder and Image Encoder. and Feature normalization is performed using the following formula:

[0077]

[0078]

[0079] in For long-tail scene text features, For long-tail scene image features, For L2 normalization.

[0080] Normalized long-tail scene text features Tail scene image features after length normalization Given M pairs of inputs, calculate the similarity matrix using the following formula:

[0081] in For similarity, For the first Long-tail scene image features For the first Long-tail scene text features.

[0082] Finally, the contrast loss is calculated by taking the cross-entropy loss between the image and text and the text and image respectively, and then calculating the arithmetic mean to obtain the total loss value. The specific formula is as follows:

[0083]

[0084]

[0085] in, For long-tail scene image-to-text cross-entropy loss, For long-tail scenarios, the text-to-image cross-entropy loss is used. For similarity, For temperature coefficient, This represents the total cross-entropy loss.

[0086] The input is the S1 output image after the image has been labeled with data. The above realizes the fine-tuning of the contrast pre-trained model, making the contrast pre-trained model more sensitive to long-tail scenes and with a higher recognition rate.

[0087] In a preferred embodiment, S102 specifically involves: inputting the collected long-tail scene images into the diffusion model to generate synthetic data, which consists of similar long-tail scene images, thereby increasing the amount of data and enabling the pre-trained model to learn sufficiently complex patterns and features. The specific steps are as follows: The first step is to read the image, pass it to the VAE encoder, and then pass it to the DDIM scheduler after VAE encoding.

[0088] The original resolution of the image is 224x224x3. After convolution and downsampling by the VAE encoder, the probability distribution of latent features is calculated through two connected layers. Finally, reparameterization sampling is performed, and the final output latent variable size is 64x64x4. The specific formula is as follows:

[0089] =

[0090]

[0091]

[0092] in This is a one-dimensional vector of the exhibit after convolution and downsampling. These are the parameters that the network can learn. The mean value calculated for the VAE encoder. The variance is calculated for the VAE encoder, where x is the potential size of the final output.

[0093] The second step involves the DDIM diffusion model feeding the VAE-encoded latent variables into the Unet network for forward diffusion. The formula for forward diffusion (with added noise) is as follows:

[0094]

[0095] in Let be the latent variable into which noise is added for the t-th time. To control the noise level, a cosine scheduling table is used here. The noise is Gaussian, and T is the total number of noise additions. The third step involves using a convolutional neural network to extract image features, adjusting the number of channels to 64x64x320 to match the second layer dimension of the Unet network.

[0096] The fourth step involves backdiffusion under the guidance of DDIM scheduling, and the predicted noise is then fed back into DDIM. The specific formula is as follows:

[0097]

[0098] in This represents the latent representation (including the noisy image) at step t. This represents the latent representation (denoised image) at step t-1. Let be the noise scheduling coefficient for time step t. For the noise predicted by Unet, the input parameters at step t are... Potential representation, The parameters used to control randomness determine whether to add new noise. This is random noise sampled from a standard normal distribution.

[0099] The fifth step is to remove the noise from the backdiffusion prediction and input the denoised latent features into the VAE decoder to generate an image.

[0100] Preferably, step S4 includes the following steps: S401: Upload the fine-tuned model to the on-board computing unit and crop the image size of the on-board camera to 224x224.

[0101] S402: Utilize LoRA (Low-Rank Adaptive Model) to incrementally learn a contrastive learning pre-trained model while an end-to-end autonomous vehicle is driving on the road.

[0102] S403: The contrastive learning pre-trained model identifies long-tailed scenes in images input from the vehicle camera and outputs the corresponding long-tailed scene classification.

[0103] The preferred option, S402, specifically includes: Optimize the Transformer in the text encoder and image encoder of the contrastive learning pre-trained models respectively. , , The parameters are updated using the principle of a low-rank adaptive model. The specific process is as follows: to update... For example: The first step is to determine the size of the query matrix and the size of the hidden layer. Size, the specific formula is as follows:

[0104] in The size of the hidden layer of the Transformer. To query the size of the matrix.

[0105] Using LoRA low-rank decomposition, the data used for updating... Decomposing the model into two low-rank matrices A and B significantly reduces computational complexity, lowers existing requirements, and minimizes computational resource consumption. This allows the model to adapt to long-tail scene recognition tasks with fewer parameters compared to pre-trained models. The specific formula is as follows: =AB r=4

[0106]

[0107] in Here are the parameters used for updating, and A and B are the two low-rank matrices after decomposition.

[0108] The second step involves LoRA bypassing the computation of the original model while keeping the original model weights unchanged. The LoRA modifications take effect during forward propagation, as detailed in the following formula:

[0109] in, This is the matrix used in the model output. For learning rate, To compare the original weight matrix of the pre-trained model, A and B are two low-rank matrices after decomposition.

[0110] A bridge is built between visual encoding and language models by incorporating a bridging model, forming a vision-to-language framework. Within this framework, the visual encoding network is responsible for extracting features from images, while the language model completes its tasks by understanding and generating textual information. The Clip-former module of this invention is described below, with specific steps as follows: The first part of the Clip-former module is for self-attention. This module consists of 8 Clip-former layers, which are lightweight and effective, and can quickly pass effective visual model information to the subsequent language model module. The first part contains only a sequence of 8 queries, whose sequence length is aligned with the image features and is D-dimensional. Taking the l-th layer as an example, the specific steps are as follows:

[0111]

[0112]

[0113]

[0114]

[0115]

[0116]

[0117] in There are 8 query sequences. To query the size of the sequence, These are the Q, K, and V matrices of the query sequence, respectively. To merge the values ​​of the long positions, For the return matrix, This is the final output of the l-th layer after normalization.

[0118] The second part of the Clip-former module is the cross-attention part. This part incorporates the query sequence into the image features and uses multi-head Cross-Attention to gather useful visual information into its own embedding. Taking the l-th layer as an example, the specific steps are as follows:

[0119]

[0120]

[0121]

[0122]

[0123]

[0124] in There are 8 query sequences. Size of the query sequence The image features output by CLIP consist of 196 patches. These represent the Q-value of the query sequence, and the K and V-values ​​of the image features output by the CLIP model, respectively. To merge the values ​​of the long positions, For the return matrix, This is the final output of the l-th layer after normalization.

[0125] The third and fourth parts of the Clip-former module are both feedforward networks, responsible for expanding and then shrinking the hidden dimensions back to their original size, adding residual connections during this process. The specific steps are as follows: ,

[0126]

[0127] ,

[0128] Where H represents the feature parameters after expanding the dimensions. For query vector, All of these are learnable parameters, and GELU represents a non-linear activation function.

[0129] An LLM model is added after the Clip-former model. The CLIP model is used to extract image features. The image is input into the CLIP model, and CLIP transforms the image into an embedding vector (visual feature vector) through its visual encoder. This vector represents the semantic information in the image.

[0130] The text undergoes tokenization and embedding. The original text description requires preprocessing (such as word segmentation and word vectorization), and the Clip-former model is responsible for generating image-related text queries within this embedding space. By optimizing the Clip-former, it is ensured that it can extract key semantic information from image features, thereby better providing context for the generated text.

[0131] TinyGPT, a lightweight generative large language model, is capable of generating coherent text based on input text prompts. In this stage, TinyGPT's task is to combine text queries from the Clip-former with image features to generate detailed text descriptions.

[0132] The joint training steps for CLIP, Clip-former, and TinyGPT are as follows: CLIP's image encoder outputs image features CLIP's text encoder outputs text features. Clip-former module output Query features.

[0133] TinyGPT is based on query features and text features To generate text sequences .

[0134]

[0135] The cross-entropy loss function is a standard loss function used for text generation tasks, calculating the loss of the generated text. The difference between the actual text Z and the real text Z. The formula for calculating the cross-entropy loss is as follows:

[0136]

[0137]

[0138] in, Given a query vector The CLIP model outputs real text features. Under the conditions of the accessories, the first generation is generated. The word is The probability, It was before One generated word, It is the length of the generated text. The length of the target real text, The TinyGPT model is based on the previously generated word sequence. and the input query vector Real text features The calculated logits, The goal is to minimize the difference between the probability distribution of the generated text and the real text.

[0139] The CLIP-Clip-former-TinyGPT trained in this way not only has excellent image recognition capabilities in long-tail scenarios, but can also automatically generate high-quality text descriptions based on visual input. Its lightweight design results in a model parameter count of only a million, without affecting inference speed and accuracy at all.

[0140] Furthermore, this invention proposes a system for end-to-end autonomous driving long-tail scene recognition based on image-text comparison learning pre-training, comprising a data acquisition module, a data synthesis module, a model fine-tuning module, a text generation module, and a visualization output module connected in sequence. The acquisition input module collects image data from the vehicle's front camera, with a resolution of 224x224. The synthetic data module includes a VAE encoder module, a feature extraction module, a DDIM scheduling module, a noise addition module, two latent space modules, a UNet module, and the VAE encoder module, all connected in sequence. The feature extraction module extracts the input long-tailed learning image and downsamples it to the intermediate dimension of the UNet module using a convolutional neural network to learn the multi-dimensional features of the image. The UNet module is used to progressively add and reduce noise to learn noise for generating synthetic data. The model fine-tuning module includes an image encoder module, a text encoder module, an InfoNCE contrastive loss module, and a LoRA parameter fine-tuning module connected in sequence. The image encoder module includes a linear layer module connected in one order, a learnable position encoding module, a Transformer encoder module, and a multilayer perceptron module. The text encoder module includes an input / output embedding module, a position encoding module, a Transformer encoder module, a Transformer decoder module, a linear layer module, and a softmax module. The LoRA parameter fine-tuning module includes a low-rank decomposition module, a weight bypass module, and an optimization module, which fine-tunes the parameters of the contrastive learning pre-trained model to better adapt to long-tail scene recognition problems.

[0141] The text generation module first acquires scene images in real time using an in-vehicle camera, then extracts deep semantic features from the images using CLIP, and inputs these features into Clip-former to generate an interactive query vector for the current scene. Finally, the system passes the query vector to TinyGPT, thereby achieving accurate recognition of long-tail scenes and automatic generation of descriptive text.

[0142] The visualization output module will visualize the images and classification types of long-tail scenes.

[0143] This invention constructs a data synthesis module, trains the synthesized data, and performs long-tail scene recognition and classification. It takes the front camera of an end-to-end autonomous vehicle and long-tail scene image-text pairs as input. Secondly, the data synthesis module learns the features of long-tail scene images from multiple dimensions to achieve more accurate long-tail scene image generation. Then, the image-text pairs of the long-tail scene, along with the synthesized long-tail scene images and text, are fed into a model fine-tuning module. This module performs image-text matching by learning the similarity features of the images and text, and uses low-rank decomposition for rapid model updates, adapting the pre-trained model to the long-tail scene recognition task. Using a bridging model and a lightweight language model, it generates text for the long-tail scene. Finally, the final prediction result is output through a visualization output module.

[0144] Compared with the prior art, the beneficial effects of the present invention are as follows: This invention addresses simulation training in end-to-end autonomous driving environments, fully considering the data scarcity and recognition challenges of long-tail scenarios. It generates incremental data of similar long-tail scenarios through a diffusion model to enhance the learning ability of the contrastive learning pre-trained model for scarce long-tail scenarios. This synthetic data not only strengthens the model's representation of long-tail events but also improves its generalization performance in image-text matching tasks, enabling it to more accurately capture potential long-tail events in specific time sequences within end-to-end autonomous driving environments and provide warnings to the autonomous driving system and driver. Furthermore, the efficient parameter updating using a low-rank adaptive model significantly improves the applicability of long-tail scene recognition tasks, thereby enhancing the robustness and safety of the autonomous driving system in complex environments.

[0145] By constructing a synthetic data module, this invention introduces an image generator based on a diffusion model to synthesize rare long-tail scene data in road environments on a large scale. This method effectively alleviates the scarcity problem of long-tail scene data, enabling the model to be fully trained in diverse long-tail scenes, thereby improving the ability to recognize rare events and generalization performance. This model not only generates high-quality long-tail scene images but also incorporates a contrastive learning mechanism, allowing the pre-trained model to learn long-tail scene features more accurately and optimizing visual-language alignment capabilities. Furthermore, this method provides richer long-tail data support for end-to-end autonomous driving systems, enhancing the system's robustness and safety in complex environments, and providing a solid guarantee for the application of autonomous driving technology in real traffic environments.

[0146] By constructing a low-rank adaptive model, this invention achieves a highly efficient enhancement of long-tail scene recognition capabilities while maintaining the original model structure and parameter scale. The low-rank adaptive model introduces low-rank matrix factorization, fine-tuning only key parameters, thereby reducing computational resource consumption while improving model adaptability and training efficiency. Compared to traditional full-parameter fine-tuning methods, this method can achieve rapid adaptation and optimization for long-tail scenes under limited data and computing power. Furthermore, by combining synthetic data based on a diffusion model, this invention further enhances the model's generalization ability in long-tail scenes, enabling it to accurately capture low-frequency events in complex road environments. This provides more robust perception and decision support for end-to-end autonomous driving systems, thereby improving the safety and reliability of autonomous driving systems. Attached Figure Description

[0147] Various other advantages and benefits will become apparent to those skilled in the art upon reading the following detailed description of preferred embodiments. The accompanying drawings are for illustrative purposes only and are not intended to limit the invention. In the drawings: Figure 1 This is a flowchart of the method of the present invention; Figure 2 System flowchart; Figure 3 This is a schematic diagram of a long-tail scene generator. Figure 4 A graphic comparison diagram of pre-trained models for comparative learning; Figure 5 Flowchart for comparing and learning pre-trained models; Figure 6 To compare the structure diagram of the pre-trained model text encoder; Figure 7 To compare the image encoder structure diagram of the pre-trained model; Figure 8 Generate a text pipeline graph for long-tail scenarios; Figure 9 A visualization of the recognition results. Detailed Implementation

[0148] Exemplary embodiments of the present disclosure will now be described in more detail with reference to the accompanying drawings. While exemplary embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure may be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided to enable a more thorough understanding of the present disclosure and to fully convey the scope of the disclosure to those skilled in the art. It should be noted that, unless otherwise specified, the embodiments and features described herein can be combined with each other. The present invention will now be described in detail with reference to the accompanying drawings and embodiments.

[0149] This embodiment proposes an end-to-end autonomous driving long-tail recognition method based on contrastive learning pre-training, such as... Figure 5 As shown, one-shot testing and model training of the pre-trained model are performed using existing long-tail scene images and synthetic data for comparative learning. Figure 2 As shown, key parameters were finally fine-tuned in a real driving environment using a low-rank adaptive model, improving the model's generalization ability in long-tail scenarios. This enabled the model to accurately capture low-frequency events in complex road environments, achieving accurate and rapid identification of long-tail scenarios. Specifically, the following steps were included: (1) The long-tail scene synthetic data module, such as Figure 3 As shown: 1. Collection and processing of pre-training data: Search for 100 images of different long-tail scenes in the network graph dataset, scale them to 224x224 resolution through a convolutional neural network, and save them as .png format. These data contain RGB images with 3 channels.

[0150] 2. Data Encoding and Feature Extraction: The collected data items are processed using a specially designed neural network structure. Four convolutional layers are used for feature extraction, extracting features from different dimensions. These four convolutional layers consist of two downsampling layers, one upsampling layer, and one channel-adjusting convolutional layer. The first convolutional layer uses a 3x3 convolution with a stride of 2 and padding of 1, outputting a 64-channel (112x112) image from an input of 3 channels (214x214). The second layer continues with a 3x3 convolution, stride of 2, and padding of 1, outputting a 128-channel (56x56) image. The third layer uses a transposed convolution with a 3x3 kernel, stride of 2, and padding of 1, outputting a 256-channel (64x64) image. The fourth layer adjusts the number of channels using a 1x1 convolution to match the UNet layer, outputting a 320-channel (64x64) image. This is the output of the feature extraction and is fed into the UNet layer.

[0151] 3. Noise is added to the Unet network. The DDIM diffusion model inputs the VAE-encoded latent variables into the Unet network for forward diffusion. The formula for forward diffusion (noise addition) is as follows:

[0152]

[0153] in Let be the latent variable into which noise is added for the t-th time. To control the noise level, a cosine scheduling table is used here. The noise is Gaussian, and T is the total number of noise additions. 4. The UNet network performs inverse denoising. The DDIM model determines the denoising step size and repeatedly feeds random Gaussian noise into the UNet network. The specific formula is as follows:

[0154]

[0155] in This represents the latent representation (including the noisy image) at step t. This represents the latent representation (denoised image) at step t-1. Let be the noise scheduling coefficient for time step t. For the noise predicted by Unet, the input parameters at step t are... Potential representation, The parameters used to control randomness determine whether to add new noise. This is random noise sampled from a standard normal distribution.

[0156] 5. Remove the noise from the backdiffusion prediction, and input the denoised latent features into the VAE decoder to generate an image.

[0157] 6. The generated images are normalized and accurate long-tail scene category labels are added to them to build a high-quality image-text matching dataset, thereby further optimizing the training effect of the contrastive learning pre-trained model.

[0158] (2) Training of the contrastive learning pre-trained model for identifying long-tail scenes, such as Figure 6 , Figure 7 As shown: 1. Train the text encoder, such as Figure 6 As shown, the long-tail scene category label text is fed into the text encoder. The text is then input into the Transformer network. First, the input is tokenized, and the long-tail scene type label position is segmented into multiple tokens, which are then padded to 77 tokens. If any tokens are insufficient, zeros are added to the pads. The token word embedding is 512-dimensional, and positional encoding is added. Here, positional encoding uses sine and cosine functions of different frequencies, specifically as follows:

[0159]

[0160]

[0161] Where pos represents the position in the input sequence Y, i.e., the position index; 2k, 2k+1 represent the dimension index ( D represents the dimension of the word embedding layer, which is the same as the total dimension of the positional encoding vector, here it is 512 dimensions. This is the word vector passed in.

[0162] The word distribution is then passed through a Transformer encoder layer and a Transformer decoder layer, followed by a linear layer and a softmax layer. This output is used for subsequent image-text similarity calculations. The specific process is as follows:

[0163] , ,

[0164]

[0165]

[0166]

[0167]

[0168]

[0169] in, The input features are the word vectors that have undergone layer normalization. These are the learnable parameter matrices, These are three vectors representing the query, key, and value of the input feature Y. These are word vectors after positional encoding, without multiple attention layers. The word vectors are those that have passed through a multi-head attention layer. These are the word vectors after residual processing and layer normalization, where FFN represents a feedforward network and GELU represents a non-linear activation function, short for Gaussian error linear unit. These are the learnable parameters in the network.

[0170] The feature values ​​output by the Transformer encoder are fed into the multi-head attention layer of the Transformer decoder. Next, the word vector shifted one position to the right, with the first character being [SOS], is fed into the Transformer decoder. First, the input is segmented into tokens, with the long-tail scene type label positions segmented into multiple tokens and padded to 77 tokens. Any insufficient tokens are padded with zeros. The token embeddings are 512-dimensional, and positional encoding is added. These are then sequentially connected to a masked multi-layer attention layer, a residual connection and layer normalization layer, a multi-head attention layer, a residual connection and layer normalization layer, and a feedforward layer. The specific process of the residual connection and layer normalization layer is as follows:

[0171]

[0172]

[0173] , ,

[0174]

[0175]

[0176] ,

[0177] ,

[0178]

[0179] =

[0180]

[0181]

[0182] in Let be the word vectors in the original sequence, n be the number of input word vectors, PAD be the padding symbol used to maintain length, and SOS be the start marker. Here, pos represents the position in the input sequence Y, i.e., the position index; 2k and 2k+1 represent the dimension indices (…). D represents the dimension of the word embedding layer, which is the same as the total dimension of the position encoding vector. The word vectors are after position encoding. Input features A query consists of three vectors: key, value, and key. This represents a masked attention layer, where M is a mask matrix, typically a very small negative number, used to mask future positions. MSA is for multi-head attention computation. Represents the features in the splicing multi-attention head, among which This represents the word vectors after residual processing and layer normalization. LayerNorm is the layer normalization process. Represents the cross-attention layer. Represent The Q value and K value, The V value represents the value of the feedforward network, FFN represents the feedforward network, and GELU represents a nonlinear activation function. These are the learnable parameters in the network.

[0183] Finally, for Feature extraction is performed, and the extracted features are input into a linear layer for reshaping and mapping to 512 dimensions. This reshaping is then used as the feature vector for text labels in long-tail scenarios, which is then used for subsequent image-text similarity calculations. The specific formula is as follows:

[0184]

[0185] in To extract The last special marker vector, namely [EOS], The feature vector is mapped to D dimensions and used to calculate the cosine similarity between the image and the text. These are the parameters that the network can learn, and the main purpose is to project special label vectors onto D dimensions. 2. Train the image encoder, such as Figure 7 As shown, the image synthesized from S2 is fed into the image encoder. First, the image is divided into blocks. The original image resolution is 224x224. Dividing the original image into 196 blocks of 14x14 units, the block-wise image is mapped to D=512 dimensions through a linear layer. Then, it undergoes learnable one-dimensional position encoding with a size of 196x512. After that, it is fed into the Transformer encoder, where features are extracted and then input into a multilayer perceptron, projected onto the contrast space dimension, which is 512 dimensions. The specific process is as follows: →

[0186]

[0187]

[0188] , ,

[0189]

[0190]

[0191] ,

[0192]

[0193] in For learnable location encoding, initial random values ​​are assigned using a uniform distribution, with a size of [value missing]. The Adam optimizer is selected to update parameters based on gradient values, where For graph vectors, For graph vectors that have undergone layer normalization, For layer normalization, where Input features The query consists of three vectors: key, value, and key. For multi-head attention layer calculation, This represents the features in the splicing of multiple attention heads. These are the parameters that the network can learn, where This is the graph vector after residual processing. It is a feedforward layer. It is a non-linear activation function. For layer normalization, This is the graph vector output after passing through the feedforward layer, residual processing, and layer normalization. These are parameters that can be learned in the feedforward layer.

[0194] Finally, learnable classification labels are extracted from the image vectors of long-tail scene images, and the features are mapped to a 512-dimensional contrastive learning space using a projection head. These features serve as feature vectors for the long-tail scene image labels, which are then used for subsequent image-text similarity calculations. The specific formula is as follows:

[0195]

[0196] in To extract The first special marker vector, namely [CLS], The feature vector is mapped to D dimensions and used to calculate the cosine similarity between the image and the text. These are the parameters that the network can learn, and the main purpose is to project special label vectors into the D dimension.

[0197] 3. Normalize the text feature vector and image feature vector, calculate the pre-similarity, and finally calculate the contrast loss. The specific process is as follows:

[0198]

[0199]

[0200]

[0201]

[0202] in For long-tail scene text features, For long-tail scene image features, For L2 normalization, where, For long-tail scene image-to-text cross-entropy loss, For long-tail scenarios, the text-to-image cross-entropy loss is used. For similarity, For temperature coefficient, This represents the total cross-entropy loss.

[0203] 4. Train the model, minimizing the contrastive loss as the objective function during optimization to ensure optimal alignment of image and text features in the joint embedding space, generating results as follows: Figure 4 As shown.

[0204] (3) Deploy the contrastive learning pre-trained model on a real vehicle platform and perform one-shot testing, such as... Figure 1 As shown, the LoRA low-rank decomposition technique is used to efficiently update key parameters, enabling the model to adapt to and accurately identify long-tail scenes in real time. The specific process is as follows: The end-to-end autonomous driving model receives images in real time from the contrastive learning pre-trained model every 5 seconds. The contrastive learning pre-trained model performs image-text matching using a feature mapping library. After matching long-tail scene text labels, the model automatically stores them and updates key parameters using a bypass. The specific formula is as follows: =AB r=4、

[0205]

[0206]

[0207] in These are the parameters used for updating. Let A be the set of all real numbers, and let A and B be the two low-rank matrices after decomposition. This is the matrix used in the model output. This is the learning rate.

[0208] (4) Image features output by CLIP model and text features The model is passed into the Clip-former module to create a bridge model with TinyGPT, such as... Figure 8 As shown, it is divided into four parts. The first part contains only a sequence of 8 queries, the length of which is aligned with the image features. The second part is the cross-attention part. The third and fourth parts are both feedforward networks. Taking the l-th layer as an example, the specific steps are as follows:

[0209]

[0210]

[0211]

[0212]

[0213]

[0214]

[0215]

[0216]

[0217]

[0218]

[0219]

[0220] ,

[0221]

[0222] ,

[0223] in There are 8 query sequences. Size of the query sequence The image features output by CLIP consist of 196 patches. These represent the Q-value of the query sequence, and the K and V-values ​​of the image features output by the CLIP model, respectively. To merge the values ​​of the long positions, For the return matrix, These are the Q, K, and V matrices of the query sequence, respectively. To merge the values ​​of the long positions, For the return matrix, This is the final output of the l-th layer after normalization. The output of the l-th layer after normalization is given, and H represents the feature parameters after expanding the dimensions. For query vector, All of these are learnable parameters, and GELU represents a non-linear activation function.

[0224] Clip-former will The data is fed into TinyGPT, which generates the recognized image and text information. The specific process is as follows:

[0225] The decoder takes the start sequence [SOS] as input, followed by the embedding query vector. The text is generated using an autoregressive mechanism, and the specific formula is as follows:

[0226] in It is the output of the Transformer layer (context-dependent representation). It is the weight matrix of the output layer, with a size of , It is a bias term with dimension 1. , It is the final logits vector, with dimension . That is, the size of one's vocabulary.

[0227] TinyGPT generates text based on the highest probability of logits until that point.

[0228] (5) Visualize the recognized graphic and textual information onto the vehicle-mounted display terminal, such as... Figure 9 As shown, the interface consists of four parts: the original navigation map section of the vehicle system, the current time, signal strength, WIFI, satellite signal module, terminal control module, vehicle speed information and long-tail scene recognition reminder information module, which uses pop-up reminders.

[0229] In summary, this invention proposes an iterative optimization method based on generative data augmentation and multimodal contrastive learning. The specific implementation process is as follows: First, synthetic image data with long-tail distribution characteristics is generated through a conditional diffusion model. Then, a fine-grained scene classifier is used to systematically organize and semantically annotate the generated samples, constructing a structured multimodal image-text alignment dataset. Finally, this augmented dataset is fused with the original training set, and the visual-language joint embedding space is optimized through a multi-task contrastive loss function to update the parameters of the pre-trained model. This method innovatively establishes a closed-loop optimization mechanism for the generative data augmentation and contrastive learning framework, effectively alleviating the data scarcity problem in long-tail distribution scenarios and significantly improving the model's cross-modal representation ability and downstream task generalization performance in low-resource categories.

[0230] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit it. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that modifications or equivalent substitutions can still be made to the specific implementation of the present invention. Any modifications or equivalent substitutions that do not depart from the spirit and scope of the present invention should be covered within the scope of protection of the claims of the present invention.

Claims

1. An end-to-end autonomous driving long-tail recognition method based on contrastive learning pre-training, characterized in that, Includes the following steps: S1. Collect long-tail scene images and perform data annotation processing. Input the processed image-text pairs as the input sequence into the diffusion model to generate synthetic data image-text pairs. S2. The synthetic data image pairs are fed into the contrastive learning pre-training model for training, and the weights are initially updated; S3. Images from the vehicle's forward-facing perspective are collected using the vehicle-mounted camera and fed into the incrementally trained contrastive pre-training model for zero-shot recognition of long-tail scenes, and incremental learning is performed. S4. Use a low-rank adaptive model to update the weights of the pre-trained model in real-world scenarios, and connect it to a bridging module and a lightweight LLM to perform long-tail scene recognition and classification tasks. The process in S4 includes: The finely adjusted model was uploaded to the vehicle's computing unit, and the image from the vehicle's camera was cropped to 224x224. The LoRA model is used to incrementally learn the contrastive learning pre-trained model while the end-to-end autonomous vehicle is driving on the road; Comparative learning pre-trained models identify long-tailed scenes in images input from vehicle cameras and output corresponding long-tailed scene classifications; The process of incrementally learning a contrastive learning pre-trained model using the LoRA model while an end-to-end autonomous vehicle is driving on the road includes: The parameters of the Transformer in the text encoder and image encoder of the contrastive learning pre-trained model are optimized respectively. Specifically, the parameter size is determined according to the query matrix size and the hidden layer size, and LoRA is used to decompose the parameters into low-rank matrices. The original weight matrix of the contrastive pre-trained model is updated according to the low-rank matrices. While incrementally learning a contrastive learning pre-trained model using the LoRA model during end-to-end autonomous vehicle driving on the road, the Tiny GPT and Clip-former modules are jointly trained to fuse long-tail scene image features and generate correct text. The Clip-former module includes a self-attention part, a cross-attention part, and a feedforward network. The self-attention part includes several query sequences. The cross-attention part is used to integrate the query sequences into the image features and gather useful visual information into its own embedding through multi-head cross-attention. The feedforward network is used to expand and then shrink the hidden dimensions back to their original size, and residual connections are added in the process. The Tiny GPT is used to combine text queries from the Clip-former module with image features from CLIP to generate text descriptions.

2. The method according to claim 1, characterized in that, The process S1 includes: The public dataset is labeled; the labeled dataset is input into a diffusion model to output a synthetic image; the synthetic image and the prompt words are packaged into a synthetic data image-text pair; Specifically, in the diffusion model, the synthesized image is fed into the VAE encoder to output latent variables, and then fed into the DDIM diffusion model for scheduling; the DDIM diffusion model feeds the VAE-encoded latent variables into the Unet network for forward diffusion; image features are extracted using a convolutional neural network, and the number of channels is adjusted to 64x64x320 to match the second layer dimension of the Unet network; backdiffusion is performed under the guidance of DDIM scheduling, and the predicted noise is fed back into the DDIM diffusion model; the noise predicted by backdiffusion is removed, and the denoised latent features are fed into the VAE decoder to generate the synthesized image.

3. The method according to claim 1, characterized in that, The process of S2 includes: The long-tail scene description and long-tail scene image in the synthetic data image-text pair are respectively input into the Text Encoder and Image Encoder of the comparison pre-trained model for processing, and the long-tail scene text features and long-tail scene image features are output respectively. The text features and image features of the long-tail scene are normalized and input into M pairs to calculate the similarity matrix; Based on the similarity matrix, calculate the cross-entropy loss from image to text and the cross-entropy loss from text to image respectively, and finally calculate the arithmetic mean to obtain the total cross-entropy loss.

4. The method according to claim 3, characterized in that, The processing of the long-tail scenario description in the Text Encoder includes: The long-tail scene description is segmented, word-embedded, and position-encoded. After layer normalization, multiple attention layers, self-attention features, multi-head attention mechanism, and residual processing followed by layer normalization, the word vectors are fed into a feedforward network for linear transformation. The word vectors after activation function are then subjected to residual processing and layer normalization again. The attention feature values ​​of the processed word vectors are then fed into the Transformer Decoder to generate the final word distribution.

5. The method according to claim 3, characterized in that, The processing of the long-tailed scene image in the Image Encoder includes: The long-tail scene image is segmented into blocks, and then the segmented graph vectors are input into a linear layer and projected into the hidden dimension. The image data is then positionally encoded, and the positionally encoded graph vectors are input into a network consisting of N identical Transformer layers stacked together. The attention features obtained from multi-head attention calculation are then input into a multilayer perceptron network after residual connections to generate the final output.

6. The method according to claim 3, characterized in that, The total cross-entropy loss is expressed as follows: ， ， ， in, For long-tail scene image-to-text cross-entropy loss, For long-tail scenarios, the text-to-image cross-entropy loss is used. For similarity, For temperature coefficient, This represents the total cross-entropy loss.

7. The method according to claim 1, characterized in that, The joint training steps for CLIP, Clip-former module, and TinyGPT include: CLIP's image encoder outputs image features CLIP's text encoder outputs text features. Clip-former module output Query features; TinyGPT is based on query features and text features To generate text sequences : ， The cross-entropy loss function is a standard loss function used for text generation tasks, calculating the loss of the generated text. The difference between the actual text Z and the real text Z is calculated using the following formula: ， ， ， in, Given a query vector The CLIP model outputs real text features. Under the conditions of the accessories, the first generation is generated. The word is The probability, It was before One generated word, It is the length of the generated text. The length of the target real text, The TinyGPT model is based on the previously generated word sequence. and the input query vector Real text features The calculated logits, The goal is to minimize the difference between the probability distribution of the generated text and the real text.

Images