Multimodal automatic driving training method based on DeepSeek training framework

Abstract

The invention relates to the technical field of automatic driving, in particular to a multi-mode automatic driving training method based on a DeepSeek training framework. Comprising the following steps: reading multi-view camera images and text instructions of a DriveLM-nuScenes data set, and splicing the images according to a look-around layout to form panoramic representation; performing zooming, normalization and standardization processing on the panoramic image to obtain an image tensor; performing marking processing on the text instruction, inserting an image placeholder and a dialogue role mark, and structuring text input representation; dimensionality alignment, position code addition and cross-modal attention fusion of vision and text marking sequences are realized through a multi-modal alignment module, and multi-modal embedding representation is generated; and inputting the embedded representation into a DeepSeek language model to generate a decision text through autoregression, and taking the cross entropy loss with a mask as an optimization target. According to the method, the problems of insufficient multi-view fusion, weak modal alignment and the like in the prior art are solved, the cognitive reliability and the decision interpretability in a complex scene are improved, and vehicle-mounted edge deployment is adapted.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of automatic driving, and in particular to a multi-modal automatic driving training method based on a DeepSeek training framework. BACKGROUND

[0002] With the acceleration of automatic driving technology to L4 / L5 level, the shortcomings of existing systems in environmental perception, semantic understanding and complex decision-making are increasingly prominent. Traditional automatic driving methods based on convolutional neural network (CNN) and recurrent neural network (RNN) have significant limitations in multi-modal information processing, long-time sequence modeling and strong semantic association construction: first, existing technologies are insufficient in multi-view information fusion, often processing each camera input independently, lacking integrated modeling of the geometry of the all-around space, and difficult to ensure semantic coherence and spatial consistency across different angles; second, existing image encoding strategies generally use fixed resolution block methods, which lack flexibility and are difficult to adapt to different scales and aspect ratios of image inputs in the automatic driving environment, leading to image detail loss or waste of computing resources; third, the alignment and interaction between visual and language modalities are limited, making it difficult to achieve deep semantic association, resulting in poor performance of the system in complex traffic scenarios, driver intent recognition and context instruction understanding; fourth, the decision generation lacks explainability, usually relying on numerical control output, making it difficult to provide language reasoning and explanation that conforms to human cognitive habits, limiting the human-machine collaboration ability and trusted deployment level of the automatic driving system.

[0003] To solve the above technical problems, the present application proposes a multi-modal automatic driving training method based on a DeepSeek training framework. The system first constructs panoramic visual input by stitching multi-angle camera images, thereby realizing unified spatial modeling of the environment; second, it uses a multi-view visual encoding strategy to adaptively process different sizes and proportions of image inputs, balancing local detail preservation and overall computational efficiency; third, it introduces a cross-modal attention mechanism to model deep semantic alignment and interaction between visual features and language instructions; finally, relying on the language model structure of the DeepSeek series, it generates driving decision output that conforms to human cognitive logic while completing scene semantic understanding.

[0004] Through the above technical solutions, the present application realizes spatial consistency modeling and dynamic encoding adaptation of multi-angle environmental perception, improves semantic reasoning accuracy, and effectively enhances the cognitive reliability of the system in traffic participant occlusion, multi-target interaction and complex traffic environment; at the same time, through the language-based explanation and reasoning process, the explainability and human-machine collaboration ability of the decision output are significantly improved, thereby providing more robust and reliable technical support for the practical deployment of the automatic driving system at L4 / L5 level. SUMMARY

[0005] The present application aims at the technical bottlenecks of insufficient multi-view fusion, rigid image coding, weak modal alignment and poor decision explainability in the prior art, and provides a multi-modal automatic driving training method based on a DeepSeek training framework, which realizes multi-view spatial consistency modeling, adaptive image coding, deep cross-modal alignment and language decision generation, and adapts to the vehicle-mounted edge deployment scene.

[0006] The technical scheme of the present application: a multi-modal automatic driving training method based on a DeepSeek training framework, comprising the following steps: S1: reading DriveLM-nuScenes dataset sample data, the sample data containing multi-view camera images and text instruction information, the multi-view camera images being obtained by deploying cameras according to a surround view layout on an experimental vehicle, and the text instruction information being natural language information expressing driving tasks or scene understanding requirements, splicing the images of each view in a fixed order according to the surround view layout to form a panoramic spliced image; S2: scaling the panoramic spliced image to 384x384 pixels, converting it into a tensor and normalizing the pixel values to the range [0, 1], and then standardizing it through the preset mean and standard deviation of each channel to obtain a preprocessed image tensor; S3: converting the text instruction information into a preliminary labeled sequence and assigning position information, inserting image placeholders and dialogue role labels in the sequence, and constructing a structured text input representation in a preset format; S4: through a multi-modal alignment module containing a visual encoding unit, a text encoding unit and a cross-modal attention unit, sequentially performing multi-modal unified representation construction, label sequence alignment and pre-adaptation, and cross-modal attention alignment processing to obtain a multi-modal embedding representation that fuses visual and text semantics; S5: inputting the multi-modal embedding representation into a language model based on the DeepSeek training framework, and gradually generating semantic understanding and decision text of the automatic driving scene through a self-recurrent generation method to form a prediction sequence, wherein the prediction sequence is a sequence composed of discrete text labels generated by the language model based on the multi-modal embedding representation through self-recurrent generation; comparing the prediction sequence with the true label to calculate a masked cross-entropy loss as an optimization target, wherein the true label is an artificial label provided by the DriveLM-nuScenes training dataset; S6: using a teacher forcing strategy to take the true label as the decoding input, taking the masked cross-entropy loss as the target, performing gradient backpropagation and parameter update through the AdamW optimizer, and combining the weight decay and exponential decay learning rate scheduling strategies.

[0007] Preferably, in step S1, the experimental vehicle according to the surround view layout includes at least front view, left front view, right front view and rear view directions.

[0008] Preferably, step S2 specifically comprises the following steps: S21: scaling processing is performed on the panoramic stitched image, which is uniformly adjusted to an input size of 384x384 pixels, to obtain a scaled image; S22: normalization operation is performed on the scaled image, pixel values are mapped to the range [0, 1] and converted into tensor form, to obtain a normalized tensor; S23: standardization processing is performed on the normalized tensor, specifically, the pixel value of each channel in the tensor is subtracted by the mean value of the corresponding channel, and then divided by the standard deviation of the corresponding channel, to finally obtain a pre-processed image tensor, and the mean value and the standard deviation of the channel are constants obtained by pre-statistics on a data set.

[0009] Preferably, step S3 specifically comprises converting the text instruction information into a preliminary mark sequence and preparing position information, the preliminary mark sequence being a sequence composed of text symbols, and the text symbols including characters and special symbols. An image placeholder and a dialogue role mark are inserted into the preliminary mark sequence, the image placeholder is used to represent a special symbol corresponding to the position of the input image, the dialogue role mark is used to distinguish the special symbol of the instruction end and the response end in the input data, the preliminary mark sequence is processed in a preset "prompt-response" input format, to obtain a structured text input representation, and the structured text input representation includes the text symbols, the image placeholder, the dialogue role mark and the position information in the preliminary mark sequence.

[0010] Preferably, the "prompt-response" input format contains an instruction end field and a response end field, the text in the instruction end field contains the image placeholder, and the placeholder corresponds to the image path one by one, and the response end field is an output text area matched with the instruction end.

[0011] Preferably, step S4 specifically comprises the following steps: S41: multi-modal unified representation construction: a visual encoding unit converts the image tensor into a visual mark sequence, a text encoding unit converts the structured text input representation into a text mark sequence, a fixed length upper limit is set, the two types of sequences are padded or truncated, sequence mask information identifying valid / invalid positions is generated synchronously, and position encoding information is generated; S42: mark sequence alignment and pre-adaptation: feature vectors of the two types of sequences are mapped to the same dimensional space through linear projection, sequence position encoding based on sequential index is added to the text mark sequence, spatial position encoding based on two-dimensional row-column coordinates is added to the visual mark sequence, and the encoding vectors are added to the corresponding feature vectors in the same dimension; S43: Cross-modal attention alignment: merge two types of sequence inputs into a cross-modal attention unit, calculate the semantic correlation weight between text and visual sequences based on scaled dot product, and only perform attention calculation at valid positions according to sequence mask information, output multi-modal embedding representation with dimensions [batch_size, seq_len, hidden_dim]; wherein, wherein the batch_size dimension represents the batch size, the seq_len dimension represents the unified sequence length, and the hidden_dim dimension represents the vector space dimension of the semantic representation.

[0012] Preferably, the training optimization method of S6 comprises: Loss function design: adopt a masked cross-entropy loss function, only calculate the loss of valid predicted labels, and ignore invalid labels such as image placeholders and delimiters; in the decoding stage, input the real label combined with the teacher forcing strategy; Gradient optimization: use AdamW optimizer, combined with exponential decay learning rate scheduling and gradient clipping technology; Efficiency optimization: enable bfloat16 mixed precision training, and use parameter-wise update for optimizer parameters.

[0013] Preferably, the visual encoding unit supports two encoding mode switching: DeepSeek-VL2-Tiny-based encoding mode: adaptively select a padding scheme to divide the panoramic image into m*n local sub-blocks and generate a global thumbnail, extract features through a shared weight visual Transformer, then generate a 196-dimensional visual label through 2*2 pixel shuffle compression, insert a delimiter (to distinguish global and local features) and a line feed (to preserve sub-block spatial arrangement) to construct a structured sequence; Janus-Pro-1B-based encoding mode: extract global semantic features through a semantic understanding path, extract local texture features through a detail generation path, and fuse the two features through a unified Transformer backbone network.

[0014] Compared with the prior art, the present application has the following advantages: The present application realizes efficient processing of high-resolution, multi-view input through a multi-view visual encoding strategy, overcomes the shortcomings of traditional fixed resolution encoding, effectively enhances the understanding and reasoning ability of the automatic driving system for complex traffic scenes by introducing multi-modal alignment and lightweight language modeling, and adapts to the vehicle-mounted edge computing environment while ensuring the performance of the model, significantly improving the cognitive robustness and decision-making interpretability of the automatic driving system in complex scenes such as occlusion and multi-target interaction.

[0015] The application realizes spatial consistency modeling and dynamic coding adaptation of multi-view environment perception, effectively enhances the cognitive reliability of the system in the occlusion of traffic participants, multi-target interaction and complex traffic environment, while improving the semantic reasoning accuracy, and significantly improves the explainability and human-machine cooperation ability of the decision output through the language interpretation and reasoning process, thereby providing more stable and reliable technical support for the actual deployment of the automatic driving system at the L4 / L5 level. BRIEF DESCRIPTION OF DRAWINGS

[0016] Figure 1 is a flowchart of the application; Figure 2 is a flowchart of the multi-modal alignment module of the application. DETAILED DESCRIPTION

[0017] Unless otherwise defined, technical terms or scientific terms used in the present application shall have the general meaning understood by a person with ordinary skill in the art to which the present application belongs. In the present application, "one", "a", "an", "the", "these" and similar words do not represent a quantitative limitation, and they can be singular or plural. In the present application, the terms "include", "contain", "have" and any variants thereof are intended to cover non-exclusive inclusion; for example, a process, method and system, product or device containing a series of steps or modules (units) are not limited to the listed steps or modules (units), but can include steps or modules (units) not listed, or can include other steps or modules (units) inherent to the process, method, product or device. In the present application, the terms "connected", "connected", "coupled" and similar words are not limited to physical or mechanical connections, but can include electrical connections, whether direct or indirect. In the present application, "multiple" refers to two or more. "And / or" describes the association between the associated objects, which means that there can be three relationships, for example, "A and / or B" can mean that A exists alone, A and B exist together, and B exists alone. In general, the character " / " represents an "or" relationship between the associated objects. In the present application, the terms "first", "second", "third" and the like are only used to distinguish similar objects, and do not represent a specific order of the objects.

[0018] In an embodiment of the present application, a multi-modal automatic driving training method based on a DeepSeek training framework is provided, Figure 1 is a flowchart of the multi-modal automatic driving training method based on the DeepSeek training framework of the present application, as Figure 1 shown, the flow includes the following steps: S1: Data loading and preprocessing: reading DriveLM-nuScenes dataset sample data, the sample data contains multi-view camera images and text instruction information, the multi-view camera images are obtained by deploying cameras according to the surround view layout on the experimental vehicle, the text instruction information is natural language information expressing the driving task or scene understanding requirement, splicing the images of each view according to the fixed order of the surround view layout to form a panoramic stitched image; S2: Image standardization processing: scaling the panoramic stitched image to 384x384 pixels, converting it to a tensor and normalizing the pixel values to the range [0, 1], and then standardizing it by the preset mean and standard deviation of each channel to obtain the preprocessed image tensor; S3: Text tokenization processing: converting the text instruction information into a preliminary token sequence and assigning position information, inserting image placeholders and dialogue role markers in the sequence, and constructing a structured text input representation in a preset format; S4: Multi-modal alignment: through the multi-modal alignment module containing a visual encoding unit, a text encoding unit and a cross-modal attention unit, sequentially performing multi-modal unified representation construction, token sequence alignment and pre-adaptation, and cross-modal attention alignment processing to obtain a multi-modal embedding representation that fuses visual and textual semantics.

[0019] S5: Language model processing: inputting the multi-modal embedding representation into the language model based on the DeepSeek training framework, and generating the semantic understanding and decision text of the autonomous driving scene step by step through the autoregressive generation method to form a prediction sequence, wherein the prediction sequence refers to a sequence composed of discrete text tokens generated by the language model based on the multi-modal embedding representation; comparing the prediction sequence with the true label to calculate the masked cross-entropy loss as the optimization objective, wherein the true label refers to the artificial annotation provided by the DriveLM-nuScenes training dataset; S6: Training optimization: using the teacher forcing strategy to use the true label as the decoding input, taking the masked cross-entropy loss as the target, performing gradient backpropagation and parameter update through the AdamW optimizer, and combining the weight decay and exponential decay learning rate scheduling strategies.

[0020] Specifically, in step S1, the system reads the surround view images according to the preset camera sequence and loads the multi-camera input based on the image storage path in the DriveLM-nuScenes dataset. In the DriveLM-nuScenes dataset used in this embodiment, the installation positions of the multi-view cameras of each experimental vehicle follow a unified surround view layout specification, and the front view, left front view, right front view, rear view and other directions remain consistent among different vehicles. If there is a slight deviation, it is corrected through external parameter calibration. In order to maintain the topological relationship of the scene space, the system splices the images of each view in a fixed order to generate a panoramic spliced image. The system reads the text instruction information corresponding to the image, and the text content is in natural language form, which is used to express the driving task or scene understanding requirement.

[0021] Specifically, step S2 specifically includes the following steps: S21: scaling the panoramic spliced image to adjust it to a uniform input size of 384x384 pixels to obtain a scaled image; S22: performing a normalization operation on the scaled image to map the pixel values to the range [0, 1] and convert it to a tensor form to obtain a normalized tensor; S23: performing a standardization process on the normalized tensor, specifically, subtracting the mean value of the corresponding channel from the pixel value of each channel in the tensor and dividing by the standard deviation of the corresponding channel, to finally obtain a preprocessed image tensor. The mean and standard deviation of the channel are constants obtained by pre-statistics on the dataset.

[0022] Specifically, in step S2, first, based on the generated panoramic spliced image, the system performs a standardization process. First, the panoramic spliced image is uniformly scaled to a preset fixed resolution input of 384x384 pixels, and a fixed resolution thumbnail can also be generated to preserve the spatial semantics of the overall scene. Then, the scaled image is converted to a tensor form, and the pixel values are normalized to the interval [0, 1]. Finally, based on the statistical mean and standard deviation, the obtained tensor is normalized, and the pixel value of each channel is subtracted from the mean value of the channel and divided by the standard deviation of the channel, to finally obtain a preprocessed image tensor. The preprocessed image tensor Fo∈R3xHxW is obtained. Wherein, H and W represent the height and width of the preset input size, and the mean and standard deviation of each channel are constants obtained by pre-statistics on the training dataset, which are used to ensure the consistency of the input feature distribution.

[0023] In step S3, the text instruction information is converted into a preliminary token sequence and the position information is prepared. The preliminary token sequence is a sequence composed of text symbols, including characters and special symbols. Image placeholders and dialogue role markers are inserted into the initial marking sequence. Image placeholders are special symbols used to represent the positions of the corresponding input images, and dialogue role markers are special symbols used to distinguish between the command end and the response end in the input data. The initial marking sequence is processed using a preset "prompt-response" input format to obtain a structured text input representation. The structured text input representation includes text symbols, image placeholders, dialogue role markers, and position information in the initial marking sequence. In addition, the "Prompt-Response" input format includes a command field and a response field. The text in the command field contains image placeholders, and the placeholders correspond one-to-one with the image paths. The response field is the output text area that matches the command field.

[0024] Specifically, the system first cleans and standardizes the input text instructions, removing invalid symbols and segmenting them appropriately based on semantics. On this basis, the text encoding module transforms the instructions into a preliminary labeled sequence composed of discrete symbols and assigns positional information to it. Image placeholders and dialogue character tags can be inserted into this sequence to ensure a strict correspondence between image and text input. Finally, the input data is organized in a "prompt-response" format to obtain a structured text input representation, which includes image placeholders, dialogue character tags, and positional information, providing the input foundation for subsequent multimodal alignment. The "prompt-response" format for organizing the input data is shown below: [ { "role":"user", "content":" Are there any pedestrians at the intersection ahead? "images":["cam1.png"] }, { "role":"assistant", "content":"" } ] This input format can simultaneously express the correspondence between images and text, and provides clear dialogue role labels to identify the command end and the response end, thereby establishing a standardized input structure for subsequent multimodal alignment.

[0025] The specific steps in step S4 include the following: S41: Multi-modal unified representation construction: the visual encoding unit converts the image tensor into a visual token sequence, specifically a discrete visual feature vector sequence, and the text encoding unit converts the structured text input representation into a text token sequence, specifically a text token corresponding vector sequence. A fixed upper limit is set, and the two types of sequences are padded or truncated. Padding is tail filling tokens, and truncation is discarding the excess part. Simultaneously, sequence mask information identifying valid / invalid positions is generated, as well as position encoding information; S42: Token sequence alignment and pre-adaptation: map the feature vectors of the two types of sequences to the same dimensional space through linear projection. Add sequence position encoding based on sequential indexing to the text token sequence, and add spatial position encoding based on two-dimensional row-column coordinates to the visual token sequence. The encoding vectors are added to the corresponding feature vectors in the same dimension; S43: Cross-modal attention alignment: merge the two types of sequence inputs into a cross-modal attention unit. Calculate the semantic association weight between the text and visual sequences based on scaled dot product. According to the sequence mask information, only perform attention calculation at valid positions. The multi-modal embedding representation with a dimension of [batch_size, seq_len, hidden_dim] is obtained. where batch_size dimension represents batch size, seq_len dimension represents unified sequence length, and hidden_dim dimension represents the vector space dimension of semantic representation.

[0026] Specifically, after the preprocessing of images and texts is completed, the system inputs the preprocessed image tensor into the visual encoding unit to map it into a visual token sequence composed of multiple discrete visual feature vectors; at the same time, the structured text input representation is input into the text encoding unit to obtain a numerical text token sequence. The system maps the feature vectors of the visual token sequence and the text token sequence to a unified target dimension space through linear projection operation to ensure that the cross-modal attention is performed in the same vector space. Before splicing, the system first sets a unified sequence length as an upper limit of fixed length, and aligns the lengths of the visual token sequence and the text token sequence with the upper limit of fixed length; when the sequence is insufficient, the tail is padded, and when the sequence exceeds the upper limit, the redundant part is discarded, and a sequence mask information consistent with the length of the aligned sequence is generated to identify the valid position and the invalid position. After the length alignment is completed, the system further appends the sequence position encoding based on the order index to the text token sequence, and appends the spatial position encoding generated based on the two-dimensional row-column coordinates to the visual token sequence, and adds the corresponding feature vectors in the same dimension, thereby generating position encoding information for retaining the position information of the sequence and the space. The system splices the visual token sequence and the text token sequence into a unified input and sends it into the cross-modal attention unit for alignment processing. During the cross-modal attention calculation process, the system only participates in the attention weight calculation at the valid token position according to the sequence mask information, and does not perform calculation at the invalid position. The cross-modal attention mechanism takes the text token sequence as the query vector, takes the visual token sequence as the key and the value, calculates the semantic correlation weight based on the scaled dot product attention, and performs information interaction in parallel under the multi-head structure, thereby establishing the corresponding relationship between different modalities and realizing deep semantic fusion. Among them, the semantic correlation weight is the relevance score obtained by performing scaled dot product calculation on the query vector of the text token sequence and the key vector of the visual token sequence, which is converted into attention weight after normalization processing, and the attention weight acts on the value vector of the visual token sequence to perform weighted combination. The semantic corresponding relationship between the text and the visual modalities is established through operation, and the fused multi-modal embedding representation Z∈R batch_size×seq_len×hidden_dim is obtained, which is used as the input of the subsequent language model generation, wherein batch_size represents the batch size, seq_len represents the unified sequence length, and hidden_dim represents the vector space dimension of semantic representation. The system provides a unified feature representation for semantic analysis and decision reasoning of the autonomous driving scene.

[0027] It should be noted that the visual encoding unit supports two encoding mode switching: 1. DeepSeek-VL2-Tiny-based encoding mode: The panoramic image is divided into m x n local sub-blocks by adaptively selecting a padding scheme to generate a global thumbnail. After feature extraction by a shared weight visual Transformer, a 196-dimensional visual tag is generated by 2 x 2 pixel shuffling compression. A separator (to distinguish global and local features) and a line feed (to preserve the spatial arrangement of sub-blocks) are inserted to construct a structured sequence. 2. Janus-Pro-1B-based encoding mode: Global semantic features are extracted through a semantic understanding path, and local texture features are extracted through a detail generation path. The two paths are fused through a unified Transformer backbone network.

[0028] Specifically, the visual encoding, text encoding, and cross-modal attention mechanism involved in this step correspond to the Figure 2 The core process of the multi-modal alignment module is shown.

[0029] Due to the different characteristics of the DeepSeek series models, this embodiment supports two visual feature encoding modes in the unified process. The system selects Janus-Pro-1B or DeepSeek-VL2-Tiny through a unified adaptation interface during the initialization stage, and can be configured to switch between the two paths according to the task scenario. In the DeepSeek-VL2-Tiny visual encoding path, first, according to the aspect ratio of the input image Fo, the smallest padding scheme is adaptively selected from the candidate set CR={(m·384,n·384)} to divide the panoramic image into m x n local sub-blocks, and a global thumbnail is additionally generated to preserve the global information of the scene.

[0030] Secondly, each local sub-block and thumbnail input is shared weight visual Transformer to extract high-dimensional feature vector sequence. After that, the feature vector sequence is executed 2x2 pixel shuffle compression to generate 196-dimensional visual tokens. Where m, n and 384 are preferred implementations, equivalent scales and token compression methods can also be used without affecting the essence of the application. Next, two types of special tokens are inserted into the visual token sequence, where the separator token is used to distinguish the global thumbnail and the local sub-block features, and the line feed token is used to preserve the arrangement relationship of the sub-blocks in the two-dimensional space, finally constructing a structured visual token sequence, providing input for subsequent multi-modal alignment. In the Janus-Pro-1B visual encoding path, first, the global semantic features are extracted through the semantic understanding path, the system inputs the input image Fo into the SigLIP-Large visual encoder to obtain high-level abstract visual features, and projects the features to the embedding space of the language model through the multi-layer perceptron adapter; Secondly, in the detail generation path, the original image is divided into several local regions using the vector quantization tokenizer, and the closest representative vector in the pre-defined limited vector set is found for each region to assign the corresponding discrete index. The vector quantization token sequence obtained in this way is composed of multiple discrete indexes, which is used to preserve the fine-grained texture information of the image. Then, the token sequence is mapped to the input space of the Transformer through the generation adapter. The encoding results of the two paths are finally input into the unified Transformer backbone network to realize the dual-path collaborative modeling of visual semantic abstraction and detail generation.

[0031] After the above processing, the system obtains a multi-modal embedding representation that fuses visual and text semantic information, which is used as input for subsequent language model generation.

[0032] In step S5, the multi-modal embedding representation is input into the language model based on the DeepSeek architecture to generate the semantic parsing result and decision text of the driving scene in a step-by-step manner through an autoregressive method, such as: "There is a vehicle changing lanes in the right lane in front, please keep a safe distance". In the inference stage, the system adopts a greedy decoding strategy to generate output step by step, that is, the highest probability mark is selected at each step to ensure the determinacy and consistency of the result. To adapt to different task requirements, the system also supports enabling beam search by configuration to perform parallel exploration among multiple candidate sequences, and can combine the length penalty parameter to control the length preference of generation, and the repeated n-gram restriction to avoid redundant output content. The inference process is executed in a mixed precision environment to improve memory utilization and computational efficiency using automatic precision conversion. Finally, the generated text mark sequence is parsed into driving semantics and decision results, which can be used for downstream driving control and scene understanding tasks. In the training stage, the system compares the predicted sequence generated by the language model with the true label provided by the DriveLM-nuScenes dataset, calculates the masked cross-entropy loss as the optimization target, so as to ensure that the generated results can be effectively aligned with the artificial annotation during the model training process.

[0033] Specifically, the training optimization method in step S6 includes: S61: Training loss function design: a masked cross-entropy loss function is used as the optimization target in the training stage, and the loss function is only calculated for valid prediction marks, ignoring invalid marks such as image placeholders and separator marks; In the training process, the teacher forcing strategy is combined, and the true label is used as input in the decoding stage; S62: Gradient optimization and dynamic scheduling: AdamW optimizer is used for parameter update; Exponential decay learning rate scheduling strategy is used in the training process; Gradient clipping technology is used in the optimization process; S63: Precision optimization and training acceleration: bfloat16 mixed precision training method is enabled in the training process; The parameters of the optimizer are updated in a parameter-by-parameter manner.

[0034] Specifically, the system uses the teacher forcing strategy to input the true label in the decoding process to accelerate convergence and improve generation stability. A masked cross-entropy loss function is used in the training process, and the loss is only calculated at the position of the valid prediction mark, ignoring invalid marks such as image placeholders and separator marks, so as to concentrate the gradient backpropagation on the effective area and improve the reliability of the training process.

[0035] In terms of parameter optimization, the AdamW optimizer is adopted, combined with weight decay mechanism and exponential decay learning rate scheduling strategy, and can be combined with gradient clipping to avoid gradient anomaly and improve overall convergence stability. In the training, the bfloat16 mixed precision calculation method is used to reduce the memory occupation and improve the calculation efficiency, and the optimizer configuration is used to avoid the convergence problem caused by the inconsistency of data types, so as to ensure the numerical stability of the training process.

[0036] The above is only a preferred specific embodiment of the present application, but the protection scope of the present application is not limited thereto, any person skilled in the art can make equivalent replacement or change according to the technical solution and the inventive concept of the present application within the technical range disclosed by the present application, which should be covered within the protection scope of the present application.

Claims

1. A multi-modal autonomous driving training method based on a DeepSeek training framework, characterized in that, The method comprises the following steps: S1: reading DriveLM-nuScenes dataset sample data, the sample data containing multi-view camera images and text instruction information, the multi-view camera images being obtained by arranging cameras according to a surround view layout and collecting images of an experimental vehicle, and the text instruction information being natural language information expressing a driving task or a scene understanding requirement; splicing the images of each view in a fixed order according to the surround view layout to form a panoramic spliced image; S2: scaling the panoramic spliced image to 384×384 pixels, normalizing the image to [0, 1] after being converted into a tensor, and performing channel preset mean and standard deviation standardization to obtain a preprocessed image tensor; S3: converting the text instruction information into a preliminary token sequence and assigning position information, inserting an image placeholder and a dialogue role token in the sequence, and constructing a structured text input representation in a preset format; S4: sequentially performing multi-modal unified representation construction, token sequence alignment and pre-adaptation, and cross-modal attention alignment processing through a multi-modal alignment module to obtain a multi-modal embedding representation that fuses visual and text semantics; S5: inputting the multi-modal embedding representation into a language model based on a DeepSeek training framework to gradually generate a semantic understanding and decision text of an autonomous driving scene in a self-recurrent manner to form a predicted sequence; S6: using a teacher forcing strategy to take the real label as the decoding input, taking the masked cross-entropy loss as the target, performing gradient backpropagation and parameter updating through an AdamW optimizer, and combining a weight decay and an exponential decay learning rate scheduling strategy. 2.The multi-modal autonomous driving training method based on the DeepSeek training framework according to claim 1, wherein, In step S1, the experimental vehicle according to the surround view layout comprises at least front view, left front view, right front view and rear view directions. 3.The multi-modal autonomous driving training method based on the DeepSeek training framework of claim 1, wherein, Step S2 specifically comprises the following steps: S21: performing scaling processing on the panoramic spliced image to uniformly adjust the image to an input size of 384×384 pixels to obtain a scaled image; S22: performing normalization on the scaled image to map the pixel value to the range of [0, 1] and convert the image into a tensor form to obtain a normalized tensor; S23: performing standardization processing on the normalized tensor, specifically, subtracting the mean value of each channel in the tensor from the pixel value of the channel, and then dividing the value by the standard deviation of the channel, to finally obtain a preprocessed image tensor, the mean value and the standard deviation of the channel being constants obtained by pre-statistics on the dataset.

4. The multi-modal autonomous driving training method based on the DeepSeek training framework according to claim 1, wherein, Step S3 specifically converts the text instruction information into a preliminary token sequence and prepares position information, the preliminary token sequence being a sequence composed of text symbols, the text symbols including characters and special symbols; inserting an image placeholder and a dialogue role token in the preliminary token sequence, the image placeholder being used to represent a special symbol corresponding to the position of the input image, the dialogue role token being used to distinguish the special symbol of the instruction end from the response end in the input data, processing the preliminary token sequence in a preset "prompt-response” input format to obtain a structured text input representation, the structured text input representation including the text symbols, the image placeholder, the dialogue role token and the position information in the preliminary token sequence.

5. The multi-modal autonomous driving training method based on the DeepSeek training framework according to claim 4, wherein, The "prompt-response" input format comprises an instruction end field and a response end field, the text of the instruction end field contains image placeholders, and the placeholders correspond to image paths one by one, and the response end field is an output text area matched with the instruction end.

6. The multi-modal autonomous driving training method based on the DeepSeek training framework according to claim 1, wherein, The multi-modal alignment module comprises a visual encoding unit, a text encoding unit and a cross-modal attention unit, and step S4 specifically comprises the following steps: S41: Multi-modal unified representation construction: the visual encoding unit converts the image tensor into a visual token sequence, the text encoding unit converts the structured text input representation into a text token sequence, sets a fixed upper limit, and performs padding or truncation processing on the two types of sequences, synchronously generates sequence mask information identifying valid / invalid positions, and generates position encoding information; S42: Token sequence alignment and pre-adaptation: map the feature vectors of the two types of sequences to the same dimensional space through linear projection, add sequence position encoding based on sequential index to the text token sequence, and add spatial position encoding based on two-dimensional row-column coordinates to the visual token sequence; the encoding vectors are added in the same dimension as the corresponding feature vectors; S43: Cross-modal attention alignment: merge the two types of sequence inputs into the cross-modal attention unit, calculate the semantic association weight between the text and the visual sequence based on scaled dot product, and only perform attention calculation at valid positions according to the sequence mask information, to obtain a multi-modal embedding representation with a dimension of [batch_size, seq_len, hidden_dim]; wherein, the batch_size dimension represents the batch size, the seq_len dimension represents the unified sequence length, and the hidden_dim dimension represents the vector space dimension of semantic representation.

7. The multi-modal autonomous driving training method based on the DeepSeek training framework according to claim 1, wherein, The predicted sequence in step S5 refers to a sequence composed of discrete text tokens generated by a language model based on the multi-modal embedding representation through autoregression; the predicted sequence is compared with the real label to calculate the masked cross-entropy loss as the optimization target, wherein the real label refers to the artificial annotation provided by the DriveLM-nuScenes training data set. 8.The multi-modal autonomous driving training method based on the DeepSeek training framework of claim 1, wherein, The training optimization method of S6 comprises: Loss function design: adopt a masked cross-entropy loss function, only calculate the loss of valid predicted tokens, and ignore invalid tokens such as image placeholders and delimiters; in the decoding stage, input the real label combined with the teacher forcing strategy; Gradient optimization: adopt AdamW optimizer, combined with exponential decay learning rate scheduling and gradient clipping technology; Efficiency optimization: enable bfloat16 mixed precision training, and adopt parameter-by-parameter update for the optimizer parameters. 9.The multi-modal autonomous driving training method based on the DeepSeek training framework of claim 1, wherein, The visual encoding unit supports two encoding mode switching: DeepSeek-VL2-Tiny-based encoding mode: adaptively select a padding scheme to divide the panoramic image into m*n local sub-blocks and generate a global thumbnail, extract features through a shared weight visual Transformer, generate 196-dimensional visual tokens through 2*2 pixel shuffle compression, and insert delimiters and line feeds to construct a structured sequence; Based on Janus-Pro-1B coding mode: extract global semantic features through semantic understanding path, extract local texture features through detail generation path, and two paths of features access unified Transformer backbone network for fusion.

Images