A multi-modal automatic driving training method based on a DeepSeek training framework

Abstract

The present application relates to the technical field of automatic driving, and more particularly to a multi-modal automatic driving training method based on a DeepSeek training framework. The method comprises the following steps: reading multi-view camera images and text instructions of a DriveLM-nuScenes dataset, splicing the images according to a look-around layout to form a panoramic representation; performing scaling, normalization and standardization on the panoramic image to obtain an image tensor; performing tokenization processing on the text instructions, inserting an image placeholder and a dialogue role mark, and structuring a text input representation; realizing dimension alignment, position encoding addition and cross-modal attention fusion of visual and text mark sequences through a multi-modal alignment module to generate a multi-modal embedding representation; inputting the embedding representation into a DeepSeek language model to generate decision text in an autoregressive manner, and taking a masked cross-entropy loss as an optimization objective. The present application solves the problems of insufficient multi-view fusion and weak modal alignment in the prior art, improves the cognitive reliability and decision interpretability in complex scenarios, and is suitable for vehicle edge deployment.

Description

Technical Field

[0001] This invention relates to the field of autonomous driving technology, and in particular to a multimodal autonomous driving training method based on the DeepSeek training framework. Background Technology

[0002] As autonomous driving technology accelerates towards Level 4 / 5, the shortcomings of existing systems in environmental perception, semantic understanding, and complex decision-making are becoming increasingly apparent. Traditional autonomous driving methods based on Convolutional Neural Networks (CNNs) and Recurrent Neural Networks (RNNs) have significant limitations in multimodal information processing, long-term modeling, and the construction of strong semantic associations: First, existing technologies are insufficient in fusing multi-view information, often processing each camera input independently, lacking integrated modeling of the geometric relationships in the surrounding space, and making it difficult to guarantee semantic coherence and spatial consistency across viewpoints; Second, existing image encoding strategies generally adopt a fixed-resolution block approach, lacking flexibility and failing to adapt to image inputs of different scales and aspect ratios in autonomous driving environments, easily leading to loss of image details or waste of computational resources; Third, the alignment and interaction capabilities between visual and linguistic modalities are limited, making it difficult to achieve deep semantic associations, resulting in insufficient performance of the system in complex traffic scenarios, driver intent recognition, and contextual command understanding; Fourth, decision generation lacks interpretability, usually relying on numerical control outputs, making it difficult to provide linguistic reasoning and explanations that conform to human cognitive habits, limiting the human-machine collaboration capabilities and reliable deployment level of autonomous driving systems.

[0003] To address the aforementioned technical challenges, this invention proposes a multimodal autonomous driving training method based on the DeepSeek training framework. The system first constructs a panoramic visual input by stitching together images from multiple cameras, thereby achieving unified spatial modeling of the environment. Second, it employs a multi-view visual encoding strategy to adaptively process image inputs of different sizes and scales, balancing local detail preservation with overall computational efficiency. Third, it introduces a cross-modal attention mechanism to perform deep semantic alignment and interaction modeling between visual features and language commands. Finally, relying on the DeepSeek series of language model structures, it generates driving decision outputs that conform to human cognitive logic while achieving scene semantic understanding.

[0004] Through the above technical solutions, this invention achieves spatial consistency modeling and dynamic coding adaptation for multi-view environmental perception. While improving the accuracy of semantic reasoning, it effectively enhances the cognitive reliability of the system in traffic participant occlusion, multi-target interaction, and complex traffic environments. At the same time, through the linguistic explanation and reasoning process, it significantly improves the interpretability of decision output and human-machine collaboration capabilities, thereby providing more robust and reliable technical support for the actual deployment of autonomous driving systems at L4 / L5 levels. Summary of the Invention

[0005] This invention addresses the technical bottlenecks of existing technologies, such as insufficient multi-view fusion, rigid image encoding, weak modal alignment, and poor decision interpretability. It provides a multimodal autonomous driving training method based on the DeepSeek training framework, which achieves multi-view spatial consistency modeling, adaptive image encoding, deep cross-modal alignment, and linguistic decision generation, and is suitable for vehicle edge deployment scenarios.

[0006] The technical solution of this invention: a multimodal autonomous driving training method based on the DeepSeek training framework, comprising the following steps:

[0007] S1: Read sample data from the DriveLM-nuScenes dataset. The sample data includes multi-view camera images and text command information. The multi-view camera images are collected by cameras deployed in a surround view layout on the experimental vehicle. The text command information is natural language information expressing driving tasks or scene understanding needs. The images from each view are stitched together in a fixed order according to the surround view layout to form a panoramic stitched image.

[0008] S2: Scale the panoramic stitched image to 384×384 pixels, convert it into a tensor, normalize the pixel values ​​to the range of [0,1], and then standardize it using the preset mean and standard deviation of each channel to obtain the preprocessed image tensor;

[0009] S3: Convert text instruction information into a preliminary marked sequence and assign position information, insert image placeholders and dialogue character tags into the sequence, and construct a structured text input representation using a preset format;

[0010] S4: Through a multimodal alignment module containing visual coding units, text coding units, and cross-modal attention units, multimodal unified representation construction, tag sequence alignment and pre-adaptation, and cross-modal attention alignment processing are performed sequentially to obtain a multimodal embedding representation that integrates visual and textual semantics;

[0011] S5: Input the multimodal embedding representation into the language model based on the DeepSeek training framework, and gradually generate semantic understanding and decision text for autonomous driving scenarios through autoregressive generation to form a prediction sequence. The prediction sequence refers to the sequence of discrete text tags generated by the language model based on the multimodal embedding representation through autoregression. The prediction sequence is compared with the real labels, and the masked cross-entropy loss is calculated as the optimization objective. The real labels refer to the manually labeled data provided by the DriveLM-nuScenes training dataset.

[0012] S6: The teacher-forced strategy is adopted to use the real label as the decoding input, with masked cross-entropy loss as the target, and gradient backpropagation and parameter update are performed through AdamW optimizer, combined with weight decay and exponential decay learning rate scheduling strategies.

[0013] Preferably, in step S1, the experimental vehicle, according to the surround view layout, includes at least the forward view, left forward view, right forward view, and rear view directions.

[0014] Preferably, step S2 specifically includes the following steps:

[0015] S21: Scale the panoramic stitched image and adjust it to a uniform input size of 384×384 pixels to obtain the scaled image;

[0016] S22: Normalize the scaled image by mapping pixel values ​​to the range [0,1] and converting them into tensor form to obtain a normalized tensor.

[0017] S23: Standardize the normalized tensor by subtracting the mean of the corresponding channel from the pixel value of each channel in the tensor, and then dividing by the standard deviation of the corresponding channel to obtain the preprocessed image tensor. The mean and standard deviation of the channels are constants obtained in advance from the dataset.

[0018] Preferably, step S3 specifically involves converting the text instruction information into a preliminary marking sequence and preparing position information. The preliminary marking sequence is a sequence composed of text symbols, which include characters and special symbols.

[0019] Image placeholders and dialogue character tags are inserted into the initial tagging sequence. Image placeholders are special symbols used to represent the corresponding input image positions, and dialogue character tags are special symbols used to distinguish between the command end and the response end in the input data. The initial tagging sequence is processed using a preset "prompt-response" input format to obtain a structured text input representation. The structured text input representation includes text symbols, image placeholders, dialogue character tags, and position information from the initial tagging sequence.

[0020] Preferably, the "prompt-response" input format includes a command field and a response field. The text in the command field contains image placeholders, and the placeholders correspond one-to-one with the image paths. The response field is the output text area that matches the command field.

[0021] Preferably, step S4 specifically includes the following steps:

[0022] S41: Multimodal unified representation construction: The visual coding unit converts the image tensor into a visual tag sequence, and the text coding unit converts the structured text input representation into a text tag sequence. A fixed length upper limit is set, and the two types of sequences are padded or truncated. Sequence mask information that identifies valid / invalid positions is generated simultaneously, and position coding information is generated.

[0023] S42: Marker Sequence Alignment and Prefitting: The feature vectors of the two types of sequences are mapped to the same dimensional space through linear projection. Sequence position codes based on sequential index are added to the text marker sequences, and spatial position codes based on two-dimensional row and column coordinates are added to the visual marker sequences. The encoding vectors are added to the corresponding feature vectors in the same dimension.

[0024] S43: Cross-modal attention alignment: Merge two types of sequence input cross-modal attention units, calculate the semantic association weights between text and visual sequences based on scaled dot products, perform attention calculation only at valid positions based on sequence mask information, and output a multimodal embedding representation with dimensions [batch_size, seq_len, hidden_dim].

[0025] Wherein, batch_size represents the batch size, seq_len represents the uniform sequence length, and hidden_dim represents the dimension of the semantic representation vector space.

[0026] Preferably, the training optimization method for S6 includes:

[0027] Loss function design: A masked cross-entropy loss function is used, which only calculates the loss of valid predicted labels and ignores invalid labels such as image placeholders and separators; the decoding stage incorporates a teacher-forced input strategy to input real labels;

[0028] Gradient optimization: The AdamW optimizer is used, along with exponentially decaying learning rate scheduling and gradient pruning techniques.

[0029] Efficiency optimization: Enable bfloat16 mixed precision training and use a parameter-by-parameter update method for optimizer parameters.

[0030] Preferably, the visual coding unit supports switching between two coding modes:

[0031] The encoding mode based on DeepSeek-VL2-Tiny: The panoramic image is divided into m×n local sub-blocks by adaptively selecting the filling scheme and generating a global thumbnail. After the features are extracted by the shared weight visual Transformer, 196-dimensional visual labels are generated by 2×2 pixel shuffling and compression. Separators (to distinguish between global and local features) and line breaks (to preserve the spatial arrangement of sub-blocks) are inserted to construct a structured sequence.

[0032] The Janus-Pro-1B-based encoding mode extracts global semantic features through a semantic understanding path and extracts local texture features through a detail generation path. The two feature paths are then fused into a unified Transformer backbone network.

[0033] Compared with existing technologies, the beneficial effects of this invention are:

[0034] This invention achieves efficient processing of high-resolution, multi-view inputs through a multi-view visual encoding strategy, overcoming the shortcomings of traditional fixed-resolution encoding. By introducing multimodal alignment and lightweight language modeling, it effectively enhances the autonomous driving system's understanding and reasoning capabilities in complex traffic scenarios. While ensuring model performance, it adapts to the in-vehicle edge computing environment, significantly improving the cognitive robustness and decision interpretability of the autonomous driving system in complex scenarios such as occlusion and multi-target interaction.

[0035] This invention realizes spatial consistency modeling and dynamic coding adaptation for multi-view environmental perception. While improving the accuracy of semantic reasoning, it effectively enhances the cognitive reliability of the system in traffic participant occlusion, multi-target interaction and complex traffic environments. At the same time, through the linguistic explanation and reasoning process, it significantly improves the interpretability of decision output and human-machine collaboration capability, thus providing more robust and reliable technical support for the actual deployment of autonomous driving systems at L4 / L5 levels. Attached Figure Description

[0036] Figure 1 This is a flowchart of the present invention;

[0037] Figure 2 This is a flowchart of the multimodal alignment module of the present invention. Detailed Implementation

[0038] Unless otherwise defined, the technical or scientific terms used in this application shall have the general meaning understood by one of ordinary skill in the art to which this application pertains. Words such as “a,” “an,” “an,” “the,” “the,” and “these” used in this application do not indicate quantitative limitation and may be singular or plural. The terms “comprising,” “including,” “having,” and any variations thereof used in this application are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or device that comprises a series of steps or modules (units) is not limited to the listed steps or modules (units) but may include steps or modules (units) not listed, or may include other steps or modules (units) inherent to these processes, methods, products, or devices. Words such as “connected,” “linked,” and “coupled” used in this application are not limited to physical or mechanical connections but may include electrical connections, whether direct or indirect. “Multiple” used in this application refers to two or more. “And / or” describes the relationship between related objects, indicating that three relationships may exist; for example, “A and / or B” can represent: A alone, A and B simultaneously, and B alone. Normally, the character " / " indicates that the objects before and after it are in an "or" relationship. The terms "first," "second," "third," etc., used in this application are merely to distinguish similar objects and do not represent a specific order of objects.

[0039] This invention provides a multimodal autonomous driving training method based on the DeepSeek training framework. Figure 1 This is a flowchart of the multimodal autonomous driving training method based on the DeepSeek training framework of the present invention, as shown below. Figure 1 As shown, the process includes the following steps:

[0040] S1: Data Loading and Preprocessing: Read sample data from the DriveLM-nuScenes dataset. The sample data includes multi-view camera images and text command information. The multi-view camera images are collected by cameras deployed in a surround view layout on the experimental vehicle. The text command information is natural language information expressing driving tasks or scene understanding needs. The images from each view are stitched together in a fixed order according to the surround view layout to form a panoramic stitched image.

[0041] S2: Image standardization processing: The panoramic stitched image is scaled to 384×384 pixels, converted into a tensor, and the pixel values ​​are normalized to the range of [0,1]. Then, it is standardized by the preset mean and standard deviation of each channel to obtain the preprocessed image tensor.

[0042] S3: Text tokenization processing: Convert text instruction information into a preliminary token sequence and assign position information, insert image placeholders and dialogue character tags into the sequence, and construct a structured text input representation using a preset format;

[0043] S4: Multimodal Alignment: Through the multimodal alignment module containing visual encoding units, text encoding units, and cross-modal attention units, multimodal unified representation construction, label sequence alignment and pre-adaptation, and cross-modal attention alignment processing are performed sequentially to obtain a multimodal embedding representation that integrates visual and textual semantics.

[0044] S5: Language Model Processing: The multimodal embedding representation is input into the language model trained on the DeepSeek framework. Semantic understanding and decision text for autonomous driving scenarios are gradually generated through autoregressive generation to form a prediction sequence. The prediction sequence refers to the sequence of discrete text tags generated by the language model based on the multimodal embedding representation through autoregression. The prediction sequence is compared with the real labels, and the masked cross-entropy loss is calculated as the optimization objective. The real labels refer to the manually labeled data provided by the DriveLM-nuScenes training dataset.

[0045] S6: Training Optimization: A teacher-forced strategy is adopted to use the real labels as decoding input, with masked cross-entropy loss as the target, and gradient backpropagation and parameter updates are performed through the AdamW optimizer, combined with weight decay and exponential decay learning rate scheduling strategies.

[0046] Specifically, in step S1, the system reads the surround view images according to a preset camera sequence and loads the multi-camera input based on the image storage path in the DriveLM-nuScenes dataset. In the DriveLM-nuScenes dataset used in this embodiment, the installation positions of the multi-view cameras on each experimental vehicle follow a unified surround view layout specification. The forward, left forward, right forward, and rearward directions remain consistent across different vehicles. Any minor deviations are corrected through extrinsic parameter calibration. To maintain the topological relationship of the scene space, the system stitches the images from each viewpoint in a fixed order to generate a panoramic stitched image. The system reads the text instruction information corresponding to the image. The text content is in natural language form and is used to express driving tasks or scene understanding requirements.

[0047] Specifically, step S2 includes the following steps:

[0048] S21: Scale the panoramic stitched image and adjust it to a uniform input size of 384×384 pixels to obtain the scaled image;

[0049] S22: Normalize the scaled image by mapping pixel values ​​to the range [0,1] and converting them into tensor form to obtain a normalized tensor.

[0050] S23: Standardize the normalized tensor by subtracting the mean of the corresponding channel from the pixel value of each channel in the tensor, and then dividing by the standard deviation of the corresponding channel to obtain the preprocessed image tensor. The mean and standard deviation of the channels are constants obtained in advance from the dataset.

[0051] Specifically, in step S2, firstly, the system standardizes the generated panoramic stitched image. First, the panoramic stitched image is uniformly scaled to a preset fixed resolution of 384×384 pixels, and a fixed-resolution thumbnail is generated simultaneously to preserve the spatial semantics of the overall scene. Then, the scaled image is converted into a tensor form, and the pixel values ​​are normalized to the [0,1] interval. Finally, the resulting tensor is normalized based on the statistical mean and standard deviation. The pixel value of each channel is subtracted from the mean of that channel, and then divided by the standard deviation of that channel, resulting in the preprocessed image tensor. The preprocessed image tensor Fo∈R3×H×W is obtained. Here, H and W represent the height and width of the preset input size, and the mean and standard deviation of each channel are constants pre-calculated on the training dataset to ensure the consistency of the input feature distribution.

[0052] In step S3, the text instruction information is converted into a preliminary marking sequence and the position information is prepared. The preliminary marking sequence is a sequence of text symbols, which include characters and special symbols.

[0053] Image placeholders and dialogue role markers are inserted into the initial marking sequence. Image placeholders are special symbols used to represent the positions of the corresponding input images, and dialogue role markers are special symbols used to distinguish between the command end and the response end in the input data. The initial marking sequence is processed using a preset "prompt-response" input format to obtain a structured text input representation. The structured text input representation includes text symbols, image placeholders, dialogue role markers, and position information in the initial marking sequence.

[0054] In addition, the "Prompt-Response" input format includes a command field and a response field. The text in the command field contains image placeholders, and the placeholders correspond one-to-one with the image paths. The response field is the output text area that matches the command field.

[0055] Specifically, the system first cleans and standardizes the input text instructions, removing invalid symbols and segmenting them appropriately based on semantics. On this basis, the text encoding module transforms the instructions into a preliminary labeled sequence composed of discrete symbols and assigns positional information to it. Image placeholders and dialogue character tags can be inserted into this sequence to ensure a strict correspondence between image and text input. Finally, the input data is organized in a "prompt-response" format to obtain a structured text input representation, which includes image placeholders, dialogue character tags, and positional information, providing the input foundation for subsequent multimodal alignment. The "prompt-response" format for organizing the input data is shown below: [

[0057] {

[0058] "role":"user",

[0059] "content":" Are there any pedestrians at the intersection ahead?

[0060] "images":["cam1.png"]

[0061] },

[0062] {

[0063] "role":"assistant",

[0064] "content":""

[0065] } ]

[0067] This input format can simultaneously express the correspondence between images and text, and provides clear dialogue role labels to identify the command end and the response end, thereby establishing a standardized input structure for subsequent multimodal alignment.

[0068] The specific steps in step S4 include the following:

[0069] S41: Multimodal unified representation construction: The visual coding unit transforms the image tensor into a visual label sequence, specifically a discrete visual feature vector sequence. The text coding unit transforms the structured text input representation into a text label sequence, specifically a text label corresponding vector sequence. A fixed length upper limit is set, and the two types of sequences are padded or truncated. Padded means the tail is filled with labels, and truncated means the excess part is discarded. Sequence mask information that identifies valid / invalid positions is generated simultaneously, and position encoding information is generated.

[0070] S42: Marker Sequence Alignment and Prefitting: The feature vectors of the two types of sequences are mapped to the same dimensional space through linear projection. Sequence position codes based on sequential index are added to the text marker sequences, and spatial position codes based on two-dimensional row and column coordinates are added to the visual marker sequences. The encoding vectors are added to the corresponding feature vectors in the same dimension.

[0071] S43: Cross-modal attention alignment: Merge two types of sequence input cross-modal attention units, calculate the semantic association weights between text and visual sequences based on scaled dot products, perform attention calculation only at valid positions based on sequence mask information, and output a multimodal embedding representation with dimensions [batch_size, seq_len, hidden_dim].

[0072] Wherein, batch_size represents the batch size, seq_len represents the uniform sequence length, and hidden_dim represents the dimension of the semantic representation vector space.

[0073] Specifically, after image and text preprocessing, the system inputs the preprocessed image tensor into the visual encoding unit, mapping it into a visual tag sequence composed of multiple discrete visual feature vectors. Simultaneously, the structured text input representation is input into the text encoding unit to obtain a numerical text tag sequence. The system uses linear projection operations to map the feature vectors of the visual tag sequence and the text tag sequence to a unified target dimension space, ensuring that cross-modal attention occurs in the same vector space. Before concatenation, the system first sets a unified sequence length as a fixed-length upper limit and aligns the lengths of the visual tag sequence and the text tag sequence with this upper limit. When the sequence is shorter than the upper limit, tags are added to the end; when it exceeds the upper limit, the excess is discarded. Simultaneously, a sequence mask with the same length as the aligned sequence is generated to identify valid and invalid positions. After length alignment, the system further adds a sequence position code based on sequential index to the text tag sequence and a spatial position code generated based on two-dimensional row and column coordinates to the visual tag sequence, adding these codes to the corresponding feature vectors in the same dimension to generate positional encoding information, which is used to preserve the positional information of the sequence and space. The system concatenates visual and text tag sequences into a unified input, which is then fed into a cross-modal attention unit for alignment. During cross-modal attention calculation, the system, based on sequence masking information, only participates in attention weight calculation at valid tag positions, omitting calculations at invalid positions. The cross-modal attention mechanism uses the text tag sequence as the query vector and the visual tag sequence as the key and value. It calculates semantic association weights based on scaled dot product attention and performs information interaction in parallel under a multi-head structure, thereby establishing correspondences between different modalities and achieving deep semantic fusion. Specifically, the semantic association weights are derived from the relevance score obtained by scaling the dot product of the query vector of the text tag sequence and the key vector of the visual tag sequence. After normalization, this score is converted into attention weights, which are then applied to the value vector of the visual tag sequence and weighted. Semantic correspondences are established between the text and visual modalities through computation, resulting in the fused multimodal embedding representation Z∈R. batch_size×seq_len×hidden_dim This representation serves as the input for subsequent language model generation. Here, batch_size represents the batch size, seq_len represents the uniform sequence length, and hidden_dim represents the dimension of the semantic representation vector space, providing a unified feature representation for semantic parsing and decision reasoning in autonomous driving scenarios.

[0074] It should be noted that the visual coding unit supports switching between two coding modes:

[0075] 1. Encoding mode based on DeepSeek-VL2-Tiny: Adaptive selection of filling scheme divides the panoramic image into m×n local sub-blocks and generates a global thumbnail. After feature extraction by shared weight visual Transformer, 196-dimensional visual labels are generated by 2×2 pixel shuffling and compression. Separators (to distinguish between global and local features) and line breaks (to preserve the spatial arrangement of sub-blocks) are inserted to construct a structured sequence.

[0076] 2. Janus-Pro-1B based encoding mode: global semantic features are extracted through the semantic understanding path, local texture features are extracted through the detail generation path, and the two feature paths are fused into a unified Transformer backbone network.

[0077] Specifically, the visual encoding, text encoding, and cross-modal attention mechanisms involved in this step are detailed in the appendix. Figure 2 The core process of the multimodal alignment module is shown.

[0078] Due to the different characteristics of the DeepSeek series models, this embodiment supports two visual feature encoding modes in a unified process. During the initialization phase, the system selects either Janus-Pro-1B or DeepSeek-VL2-Tiny through a unified adaptation interface, and can switch between the two paths depending on the task scenario. In the DeepSeek-VL2-Tiny visual encoding path, the minimum filling scheme is adaptively selected from the candidate set CR={(m·384,n·384)} based on the aspect ratio of the input image Fo, dividing the panoramic image into m×n local sub-blocks, and additionally generating a global thumbnail to preserve global scene information.

[0079] Secondly, each local sub-block and thumbnail are input into a shared-weight visual Transformer to extract a high-dimensional feature vector sequence. Then, a 2×2 pixel shuffling compression is performed on the feature vector sequence to generate 196-dimensional visual labels. Here, m, n, and 384 are preferred implementations; equivalent scales and label compression methods can also be used without affecting the essence of the invention. Next, two types of special labels are inserted into the visual label sequence: a separator label to distinguish between the global thumbnail and local sub-block features, and a newline label to preserve the arrangement relationship of the sub-blocks in two-dimensional space. Finally, a structured visual label sequence is constructed to provide input for subsequent multimodal alignment. In the Janus-Pro-1B visual encoding path, global semantic features are first extracted through a semantic understanding path. The system inputs the input image Fo into the SigLIP-Large visual encoder to obtain high-level abstract visual features, which are then projected into the embedding space of the language model via a multilayer perceptron adapter. Secondly, in the detail generation path, a vector quantization tokenizer divides the original image into several local regions. A predefined finite set of vectors is used to find the representative vector closest to each region, assigning corresponding discrete indices. The resulting vector quantization tag sequence consists of multiple discrete indices, used to preserve the fine-grained texture information of the image. Subsequently, the generative adapter maps this tag sequence to the input space of the Transformer. The encoding results from both paths are finally connected to a unified Transformer backbone network, achieving collaborative modeling of visual semantic abstraction and detail generation.

[0080] After the above processing, the system obtains a multimodal embedding representation that integrates visual and textual semantic information, which serves as the input for subsequent language model generation.

[0081] In step S5, the multimodal embedding representation is input into a language model based on the DeepSeek architecture. Semantic parsing results and decision text for the driving scene are generated progressively through an autoregressive approach, such as: "A vehicle is changing lanes in the right lane ahead; please maintain a safe distance." During the inference phase, the system uses a greedy decoding strategy to progressively generate output, selecting the highest-probability tag at each step to ensure the determinism and consistency of the results. To adapt to different task requirements, the system also supports enabling bundle search through configuration, allowing parallel exploration among multiple candidate sequences. It can also combine length penalty parameters to control the generation length preference and repetitive n-gram constraints to avoid redundant output content. This inference process is executed in a mixed-precision environment, utilizing automatic precision conversion to improve memory utilization and computational efficiency. Finally, the generated text tag sequence is parsed into driving semantics and decision results, which can be used for downstream driving control and scene understanding tasks. During the training phase, the system compares the predicted sequences generated by the language model with the real labels provided by the DriveLM-nuScenes dataset, calculating the masked cross-entropy loss as the optimization objective to ensure effective alignment between the generated results and human annotations during model training.

[0082] Specifically, the training optimization method in step S6 includes:

[0083] S61: Training Loss Function Design: During the training phase, a masked cross-entropy loss function is used as the optimization objective. The loss function is calculated only for valid predicted labels, ignoring invalid labels such as image placeholders and separators. During the training process, a teacher-mandated strategy is combined, and the real labels are used as input during the decoding phase.

[0084] S62: Gradient Optimization and Dynamic Scheduling: The AdamW optimizer is used for parameter updates; an exponentially decaying learning rate scheduling strategy is used during training; and gradient pruning is used during optimization.

[0085] S63: Accuracy Optimization and Training Acceleration: The bfloat16 mixed precision training method is enabled during training; the parameters of the optimizer are updated parameter by parameter.

[0086] Specifically, the system employs a teacher-mandated strategy, using real labels as input during the decoding process to accelerate convergence and improve generation stability. During training, a masked cross-entropy loss function is used, calculating loss only at valid predicted label locations and ignoring invalid labels such as image placeholders and separators. This ensures that gradient backpropagation is concentrated in valid regions, improving the reliability of the training process.

[0087] For parameter optimization, the AdamW optimizer is employed, combined with a weight decay mechanism and an exponential decay learning rate scheduling strategy. Gradient pruning is also used to avoid gradient anomalies, thereby improving overall convergence stability. During training, a bfloat16 mixed-precision calculation method is used to reduce memory usage and improve computational efficiency. Furthermore, optimizer configuration avoids convergence problems caused by inconsistent data types, thus ensuring numerical stability during training.

[0088] The above are merely preferred embodiments of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.

Claims

1. A multimodal autonomous driving training method based on the DeepSeek training framework, characterized in that, Includes the following steps: S1: Read sample data from the DriveLM-nuScenes dataset. The sample data includes multi-view camera images and text command information. The multi-view camera images are collected by cameras deployed in a surround view layout on the experimental vehicle. The text command information is natural language information expressing driving tasks or scene understanding needs. The images from each view are stitched together in a fixed order according to the surround view layout to form a panoramic stitched image. S2: Scale the panoramic stitched image to 384×384 pixels, convert it into a tensor and normalize it to [0,1]. Then, standardize it with the channel preset mean and standard deviation to obtain the preprocessed image tensor. S3: Convert text instruction information into a preliminary marked sequence and assign position information, insert image placeholders and dialogue character tags into the sequence, and construct a structured text input representation using a preset format; S4: Through the multimodal alignment module, multimodal unified representation construction, label sequence alignment and pre-adaptation, and cross-modal attention alignment processing are performed sequentially to obtain a multimodal embedding representation that integrates visual and textual semantics; The multimodal alignment module includes a visual encoding unit, a text encoding unit, and a cross-modal attention unit. Step S4 specifically includes the following steps: S41: Multimodal unified representation construction: The visual coding unit converts the image tensor into a visual tag sequence, and the text coding unit converts the structured text input representation into a text tag sequence. A fixed length upper limit is set, and the two types of sequences are padded or truncated. Sequence mask information that identifies valid / invalid positions is generated simultaneously, and position coding information is generated. S42: Marker Sequence Alignment and Prefitting: The feature vectors of the two types of sequences are mapped to the same dimensional space through linear projection. Sequence position codes based on sequential index are added to the text marker sequences, and spatial position codes based on two-dimensional row and column coordinates are added to the visual marker sequences. The encoding vectors are added to the corresponding feature vectors in the same dimension. S43: Cross-modal attention alignment: Merge two types of sequence input cross-modal attention units, calculate the semantic association weights between text and visual sequences based on scaled dot products, perform attention calculation only at valid positions based on sequence mask information, and output a multimodal embedding representation with dimensions [batch_size, seq_len, hidden_dim]. Wherein, batch_size represents the batch size, seq_len represents the uniform sequence length, and hidden_dim represents the vector space dimension of the semantic representation; S5: Input the multimodal embedding representation into the language model based on the DeepSeek training framework, and gradually generate semantic understanding and decision text for autonomous driving scenarios through autoregressive generation to form a prediction sequence; S6: The teacher-forced strategy is adopted to use the real label as the decoding input, with masked cross-entropy loss as the target, and gradient backpropagation and parameter update are performed through AdamW optimizer, combined with weight decay and exponential decay learning rate scheduling strategies.

2. A multimodal autonomous driving training method based on the DeepSeek training framework according to claim 1, characterized in that, In step S1, the experimental vehicle is arranged in a surround view layout that includes at least the front view, left front view, right front view, and rear view directions.

3. A multimodal autonomous driving training method based on the DeepSeek training framework according to claim 1, characterized in that, Step S2 specifically includes the following steps: S21: Scale the panoramic stitched image and adjust it to a uniform input size of 384×384 pixels to obtain the scaled image; S22: Normalize the scaled image by mapping pixel values ​​to the range [0,1] and converting them into tensor form to obtain a normalized tensor. S23: Standardize the normalized tensor by subtracting the mean of the corresponding channel from the pixel value of each channel in the tensor, and then dividing by the standard deviation of the corresponding channel to obtain the preprocessed image tensor. The mean and standard deviation of the channels are constants obtained in advance from the dataset.

4. A multimodal autonomous driving training method based on the DeepSeek training framework according to claim 1, characterized in that, Step S3 specifically involves converting the text instruction information into a preliminary marking sequence and preparing the position information. The preliminary marking sequence is a sequence composed of text symbols, which include characters and special symbols. Image placeholders and dialogue character tags are inserted into the initial tagging sequence. Image placeholders are special symbols used to represent the positions of the corresponding input images, and dialogue character tags are special symbols used to distinguish between the command end and the response end in the input data. The initial tagging sequence is processed using a preset "prompt-response" input format to obtain a structured text input representation. The structured text input representation includes text symbols, image placeholders, dialogue character tags, and position information from the initial tagging sequence.

5. A multimodal autonomous driving training method based on the DeepSeek training framework according to claim 4, characterized in that, The "Prompt-Response" input format includes a command field and a response field. The text in the command field contains image placeholders, and each placeholder corresponds to an image path. The response field is the output text area that matches the command field.

6. A multimodal autonomous driving training method based on the DeepSeek training framework according to claim 1, characterized in that, In step S5, the predicted sequence refers to the sequence of discrete text tags generated by the language model based on autoregressive multimodal embedding representation; the predicted sequence is compared with the real labels, and the masked cross-entropy loss is calculated as the optimization objective, where the real labels refer to the manually labeled data provided by the DriveLM-nuScenes training dataset.

7. A multimodal autonomous driving training method based on the DeepSeek training framework according to claim 1, characterized in that, The training optimization methods for S6 include: Loss function design: A masked cross-entropy loss function is used, which only calculates the loss of valid predicted labels and ignores invalid labels such as image placeholders and separators; the decoding stage incorporates a teacher-forced input strategy to input real labels; Gradient optimization: The AdamW optimizer is used, along with exponentially decaying learning rate scheduling and gradient pruning techniques. Efficiency optimization: Enable bfloat16 mixed precision training and use a parameter-by-parameter update method for optimizer parameters.

8. A multimodal autonomous driving training method based on the DeepSeek training framework according to claim 1, characterized in that, The visual coding unit supports switching between two coding modes: The encoding mode based on DeepSeek-VL2-Tiny: the panoramic image is divided into m×n local sub-blocks by adaptively selecting the filling scheme and generating a global thumbnail. After feature extraction by the shared weight visual Transformer, 196-dimensional visual labels are generated by 2×2 pixel shuffling and compression. Separators and line breaks are inserted to construct a structured sequence. The Janus-Pro-1B-based encoding mode extracts global semantic features through a semantic understanding path and extracts local texture features through a detail generation path. The two feature paths are then fused into a unified Transformer backbone network.

Images