Human preference-fused automatic driving visual language model enhancement method and system

Abstract

This invention proposes a method and system for enhancing the visual language model of autonomous driving by incorporating human preferences. Belonging to the fields of intelligent decision-making and multimodal artificial intelligence in autonomous driving, the method includes: acquiring multi-source multimodal data; processing the multi-source multimodal data to generate a question-and-answer sample set containing questions, preferred answers, and secondary answers; processing the question-and-answer sample set to obtain a high-quality human preference thought chain dataset; supervising and fine-tuning the visual language model based on the obtained high-quality human preference thought chain dataset; aligning the supervised fine-tuned visual language model with human driving preferences to make the visual language model's decisions more consistent with human expectations; employing a grouped relative strategy optimization algorithm to optimize the model's open-ended reasoning and trajectory planning capabilities; combining a large language model-based evaluation mechanism and human preference rewards for multi-objective collaborative optimization; and using the optimized trained visual language model to generate a formatted response containing explicit reasoning processes and the final answer / trajectory.

Description

Technical Field

[0001] This invention belongs to the field of autonomous driving intelligent decision-making and multimodal artificial intelligence technology, and particularly relates to an autonomous driving visual language model enhancement method and system that integrates human preferences. Background Technology

[0002] The statements in this section are merely background information related to the present invention and do not necessarily constitute prior art.

[0003] Current autonomous driving systems have made significant progress in handling deterministic tasks with clear truth values ​​and objective evaluation metrics, such as trajectory planning and obstacle avoidance, thanks to their reliance on large-scale labeled data and end-to-end optimization paradigms. However, autonomous driving systems struggle to handle open driving problems. These problems do not have a single, definitive answer; their rationality highly depends on the dynamic context, implicit traffic rules, and common sense and value judgments in human society. This requires the system not only to perceive the environment but also to perform deep semantic understanding, causal reasoning, and the generation of reasonable explanations. Existing solutions based on visual language models mainly employ a supervised fine-tuning paradigm, which essentially learns existing, fixed question-and-answer patterns and expressions from training data. While this method enables the model to master basic scene description and command response capabilities, when dealing with the infinitely diverse and uncertain open scenarios in the real world, the model often exhibits problems such as rigid reasoning processes, broken logical chains, or a disconnect between generated content and scene semantics.

[0004] Furthermore, purely data-driven models often prioritize optimizing objective metrics. Their decisions may be technically correct but contradict human intuition and comfort, leading to overly conservative or abrupt driving styles. This stems from the current training framework's lack of a systematic approach to incorporating human subjective preferences and ethical judgments into the model. While reinforcement learning has been introduced to optimize decision-making, its reward design is largely limited to measurable objective metrics, failing to encompass crucial subjective value dimensions.

[0005] While existing technologies, such as vision-language models, near-end policy optimization, and diffusion generation, have been used for autonomous driving decision-making or trajectory prediction, they still generally suffer from the following shortcomings: (1) When faced with open-ended semantic reasoning questions such as “why / is it safe”, there is a lack of stable and optimizable training objectives and interpretable reasoning processes, which leads to the tendency of answers and decisions to be templated and lack of generalization ability.

[0006] (2) The training objectives are mostly objective indicators, such as collision rate and deviation distance, which are difficult to explicitly inject human driving preferences, such as comfort, courtesy, conservative / aggressive style, etc., which may result in strategies that are technically correct but have poor experience.

[0007] (3) The ability to align multimodal information in time sequence, assess counterfactual security and explain causality is insufficient, and unstable or untraceable trajectory output is likely to occur in long-tail scenarios. Summary of the Invention

[0008] To overcome the shortcomings of the prior art, this invention provides an autonomous driving visual language model enhancement method and system that integrates human preferences. This method is used in autonomous driving systems that can simultaneously process open semantic reasoning and make decisions that conform to human preferences and values. By directly learning human preference data through direct preference optimization, the decisions and explanations output by the model are closer to the thinking patterns and comfort zones of human drivers, thereby improving the safety and trustworthiness of human-machine collaboration.

[0009] To achieve the above objectives, one or more embodiments of the present invention provide the following technical solutions: Firstly, a method for enhancing the visual language model for autonomous driving by incorporating human preferences is disclosed, including: Acquire multi-source, multi-modal data, process the multi-source, multi-modal data to generate a question-and-answer sample set containing questions, preferred answers, and secondary answers, and further process the question-and-answer sample set to obtain a high-quality human preference thought chain dataset; The visual language model was trained under supervised fine-tuning based on the obtained high-quality human preference thought chain dataset. Aligning the supervised fine-tuned visual language model with human driving preferences makes the visual language model's decisions more in line with human expectations. A group-based relative strategy optimization algorithm is adopted to improve the open-end reasoning and trajectory planning capabilities of the model. Combined with a judgment mechanism based on a large language model and human preference rewards, multi-objective collaborative optimization is achieved. The optimized visual language model is used to generate a formatted response containing explicit reasoning process and final answer / trajectory.

[0010] As a further technical solution, acquiring multi-source, multi-modal data specifically includes: Vehicle-mounted perception information is collected in the same scene in a real road or simulation environment. The perception information includes at least a sequence of visible light camera image frames and a sequence of radar frames. The vehicle status record corresponding to each frame of image and each frame of radar data, as well as road structure and traffic control related information, are saved simultaneously. At the same time, precise timestamps are written to the image frames, radar frames and their corresponding vehicle status records, and cross-modal alignment is completed using the timestamps as an index to obtain the aligned continuous sequence.

[0011] As a further technical solution, it also includes: based on the aligned continuous sequence, selecting the key decision moment as the starting point of trajectory labeling, extracting the vehicle's actual driving trajectory from the preset prediction time domain after that moment and labeling it as the future trajectory ground value. The future trajectory ground value is stored in the form of a discrete two-dimensional coordinate sequence. The resulting multimodal scene dataset is organized in units of samples. Each sample contains at least a scene ID, a timestamp sequence, aligned camera image data, aligned radar data, the future trajectory ground value, and information related to road structure and traffic control.

[0012] As a further technical solution, for the key decision moment and the true value of the future trajectory of each sample in the multimodal scenario dataset, based on the vehicle state and vehicle kinematic / dynamic constraints at that moment, an alternative trajectory corresponding to the real trajectory is artificially constructed. The alternative trajectory remains dynamically feasible under the physical constraints of acceleration and steering continuity, but reflects irrationality at the driving strategy level, thus constituting counterfactual trajectory analysis data.

[0013] As a further technical solution, around the key decision-making moments of each sample, the aligned camera images, radar data, and road and traffic control information are organized into a question context to generate open-ended question texts that require deep reasoning. Multiple candidate answers are generated under the same question and the same question context input. The candidate answers are then preliminarily scored and ranked using a large language model. Subsequently, the selected candidate answers are submitted to domain experts for review, forming a question-and-answer sample set with a clear preference relationship of "preferred answer - secondary answer". The multimodal scenario dataset, alternative trajectories, and the aforementioned preference question-and-answer sample set are integrated and stored in a structured manner to form a high-quality human preference thought chain dataset.

[0014] As a further technical solution, the visual language model includes a visual encoder, a projection layer, and an autoregressive language model decoder; The aforementioned high-quality human preference thought chain dataset is organized into an input sequence that can be directly processed by a visual language model: For the camera image sequence contained in each sample, a visual encoder is used to extract and convert it into visual minimal semantic units, and then a projection layer is used to map the visual features corresponding to the visual minimal semantic units to the embedding space of the language model; subsequently, the mapped visual minimal semantic unit embeddings are concatenated with the sample's open-ended question text, reasoning process text, answer text, and the text minimal semantic unit embeddings corresponding to the labels and special delimiters to form a complete input sequence for training.

[0015] As a further technical solution, during supervised fine-tuning training, the model generates a target token sequence one by one under the input sequence conditions in an autoregressive manner. The target token sequence consists of tokens corresponding to the labeled text of the reasoning process and the final answer or trajectory-related conclusions. The model uses standard cross-entropy loss to minimize the difference between the predicted distribution and the actual label.

[0016] As a further technical solution, it also includes: using low-rank decomposition to efficiently fine-tune the parameters of the pre-trained model: selecting the linear transformation layer that needs to be adapted in the pre-trained model, marking its original weight matrix and keeping it frozen, while constructing a low-rank adapter parallel to the original weight matrix, consisting of a dimensionality reduction matrix and an dimensionality increase matrix; during forward propagation... Based on low-rank update During backpropagation, only the gradients of the reduced-dimensional matrix and the increased-dimensional matrix are updated, and the frozen original weight matrix does not receive gradient updates. After the fine-tuning termination condition is met, the low-rank update is merged with the original weights to obtain an equivalent weight matrix for inference deployment.

[0017] As a further technical solution, the supervised fine-tuning model is aligned with human driving preferences, making the model's decisions more consistent with human expectations. This includes: From the constructed high-quality human preference thought chain dataset, sample triples for preference optimization are extracted and organized in the form of "scenario input - preferred answer - secondary answer". The scenario context input corresponds to the aligned camera image sequence, radar data, vehicle status information, and road and traffic control information at the same key decision moment. The preferred answer corresponds to the better answer confirmed by domain experts under the scenario context and open-ended question text. The secondary answer corresponds to the relatively worse answer confirmed by domain experts. In terms of preference modeling, the Bradley-Terry preference model is used to describe the preference probability that "the preferred answer is better than the secondary answer under the same scenario context and the same question text". The preference probability is obtained by the difference between the implicit reward of the preferred answer and the secondary answer through sigmoid mapping.

[0018] As a further technical solution, it also includes: introducing implicit KL divergence constraints, keeping the reference model fixed, updating only the current policy model, and controlling the degree of deviation of the policy model from the reference model through the hyperparameter β, so that the preference learning process is carried out in a way that "the first choice answer is more promoted than the reference model and the second choice answer is more suppressed than the reference model". The model after supervised fine-tuning is further fine-tuned using the aforementioned triplet, optimizing the direct preference optimization objective until the fine-tuning termination condition is met.

[0019] As a further technical solution, regarding the evaluation mechanism and human preference-based rewards based on large language models, the following are included: For each open-ended question sample in the dataset, the model generates candidate answers under the same question context input and calculates multi-dimensional rewards: First, a semantic similarity reward, which uses a pre-trained semantic encoder to calculate sentence embeddings for the generated answer text and the corresponding reference answer text, and uses the cosine similarity between the two embeddings as the semantic reward; Second, a trajectory planning reward, which calculates the L2 distance between the generated trajectory contained in the candidate answer and the ground truth of the future trajectory of the sample at the corresponding time points of the 1st, 2nd, and 3rd seconds, and summarizes them in an exponential decay manner, which can be expressed as the summation of terms at each time point; At the same time, a fixed negative reward is applied when the generated trajectory collides with an obstacle; Third, a preference consistency reward, which evaluates the probability that the generated answer is consistent with human preferences through a large language model and uses it as the reward component.

[0020] As a further technical solution, in multi-objective collaborative optimization, a grouped relative policy optimization method is used for training: for each sample's scene context and question text, G candidate responses are generated from the current policy model and their generation probabilities are recorded; during training and updating, the probability ratio of the new and old policies is introduced, and the probability ratio is restricted to a set interval by a truncation function to stabilize the optimization. At the same time, a KL divergence penalty term is added to constrain the deviation of the current policy model from the reference model. The KL divergence can be approximated by Monte Carlo averaging the difference in log probabilities of the group of candidate responses under the current policy and the reference model.

[0021] As a further technical solution, for each candidate response, the semantic similarity reward, trajectory planning reward and preference consistency reward are weighted and summed to obtain a comprehensive reward value. Then, the mean and standard deviation of the comprehensive reward are calculated within the group and normalized to a relative advantage value to drive policy updates. During training, the weights are dynamically adjusted according to the convergence of each reward component.

[0022] Secondly, a visual language model enhancement system for autonomous driving that incorporates human preferences was disclosed, including: The high-quality human preference thought chain dataset construction module is configured to: acquire multi-source multimodal data, process the multi-source multimodal data to generate a question-and-answer sample set containing questions, preferred answers and secondary answers, and further process the question-and-answer sample set to obtain a high-quality human preference thought chain dataset; The visual language model building module is configured to: supervise and fine-tune the training of the visual language model based on the obtained high-quality human preference thought chain dataset; The preference alignment module is configured to align the supervised fine-tuned visual language model with human driving preferences, making the visual language model's decisions more in line with human expectations. The optimization module is configured to: employ a grouped relative strategy optimization algorithm to optimize the model's open-end reasoning and trajectory planning capabilities; combine a large language model-based evaluation mechanism and human preference rewards to achieve multi-objective collaborative optimization; and optimize the trained visual language model to generate a formatted response containing explicit reasoning processes and the final answer / trajectory.

[0023] The above one or more technical solutions have the following beneficial effects: This invention constructs scene samples based on camera images, radar data, vehicle status, and road and traffic control information. It then labels the true future trajectory and dynamically feasible but unreasonable alternative trajectories, generating a high-quality dataset of human preference thought chains containing open-ended questions, preferred answers, and secondary answers. By supervising and low-rank adaptation fine-tuning the pre-trained visual language model, the basic reasoning and expressive capabilities of the model are established. Furthermore, direct preference optimization aligns the model output with human driving preferences, and combined with grouped relative strategy optimization and a multi-dimensional reward evaluation mechanism, the open-ended reasoning and trajectory planning capabilities are synergistically improved. This enhances the safety, compliance, and comfort of autonomous driving decisions, making it suitable for planning and decision-making scenarios in autonomous driving systems and demonstrating promising application prospects.

[0024] The decision-making achieved by the technical solution of this invention is more in line with human expectations, and has higher safety and acceptability: by directly learning human preference data through direct preference optimization, the decision and explanation output by the model are closer to the thinking mode and comfort zone of human drivers, reducing the "weirdness" of machine decision-making and improving the safety and trustworthiness of human-machine collaboration.

[0025] The technical solution of this invention achieves simultaneous superior performance in open reasoning and trajectory planning: the composite reward and joint training mechanism enable the model to reach a leading level in both reasoning and trajectory planning tasks, breaking through the limitation of previous models that could only focus on one aspect, and realizing a leap in comprehensive capabilities.

[0026] The technical solution of this invention achieves more stable and efficient training while reducing risks and costs: the dual policy constraint mechanism effectively prevents policy collapse and performance oscillations common in reinforcement learning; the modular training process reduces the difficulty and risk of end-to-end training and improves data utilization efficiency.

[0027] Advantages of additional aspects of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. Attached Figure Description

[0028] The accompanying drawings, which form part of this invention, are used to provide a further understanding of the invention. The illustrative embodiments of the invention and their descriptions are used to explain the invention and do not constitute an improper limitation of the invention.

[0029] Figure 1 This is a schematic diagram of the process for enhancing the visual language model of autonomous driving by incorporating human preference alignment, as described in an embodiment of the present invention. Figure 2 This is a schematic diagram of the pre-trained visual language model structure according to an embodiment of the present invention. Detailed Implementation

[0030] It should be noted that the following detailed descriptions are exemplary and intended to provide further illustration of the invention. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains.

[0031] It should be noted that the terminology used herein is for the purpose of describing particular implementations only and is not intended to limit the exemplary implementations of the present invention.

[0032] Where there is no conflict, the embodiments and features in the embodiments of the present invention can be combined with each other.

[0033] Terminology Explanation Human preference alignment: This refers to the introduction of preference information formed by humans or domain experts when selecting the superiority or inferiority of multiple candidate outputs under the same multimodal input conditions during the training or optimization of autonomous driving visual language models. The model parameters are updated accordingly, making the model more inclined to generate the selected output under the same input. The preference information is represented by relative preference relationships, usually in the form of preference pairs consisting of preferred and secondary outputs. The model output may include text used to express intermediate reasoning and formatted responses such as the final answer or final trajectory, so that the model-generated results are more in line with human selection tendencies in evaluation dimensions such as safety, rationality and compliance, logical consistency and driving comfort.

[0034] Thought chain: The serialized text content output by the model before generating the final answer or final trajectory, which is used to express the intermediate reasoning steps. The text content together with the final output constitutes a complete response. The reasoning part and the final output part are usually distinguished by a preset format or separator mark, so that the training phase can perform supervised learning or preference optimization for the reasoning process and the final output respectively, and make the answering process of open questions traceable and evaluable.

[0035] Preference alignment data refers to the dataset used to train or optimize models to achieve human preference alignment. It includes multimodal inputs corresponding to the same scenario and the same key decision moment, multiple candidate outputs generated by the base model, and the superiority or inferiority selection results given by humans or domain experts among the candidate outputs. The data is usually organized in the form of triples, where the input is multimodal information, the preferred output is the selected preferred answer or preferred trajectory, the secondary output is the judged poor answer or trajectory, and may further include annotation information such as future trajectory truth, counterfactual or alternative trajectories to form a basis for comparison and judgment, so as to support the training stages such as supervised fine-tuning, direct preference optimization and policy optimization.

[0036] Direct Preference Optimization (DPO) is an optimization method that updates model parameters based on preference alignment data. Its training objective is that "the preferred output should be given a higher generation probability by the model than the second-best output." During the optimization process, the reference model parameters are kept fixed to constrain the update direction of the policy model.

[0037] Visual language model: A multimodal model capable of jointly processing visual and textual information and generating text sequence output in an autoregressive manner. It typically includes a visual encoder for extracting features from images or image sequences, a projection layer for mapping visual features to the language model embedding space, and a language model decoder for generating the output sequence. The output sequence may contain formatted content such as inference process text and final answer or final trajectory, thereby achieving semantic understanding, causal explanation, and decision expression for autonomous driving scenarios.

[0038] Multimodal information is a collection of multi-source heterogeneous data used to describe autonomous driving scenarios and vehicle states. It includes at least images or image sequences collected by visual sensors, and may further include environmental perception data such as LiDAR and millimeter-wave radar, as well as vehicle state data such as IMU, GPS, speed and attitude. The multi-source data is aligned through timestamps or synchronization mechanisms, and can be combined with scene identifiers, key decision moment markers, future trajectory ground truth and alternative trajectory annotations to form the input and reference basis required for training and evaluation.

[0039] Example 1 See appendix Figure 1 As shown, this embodiment discloses a method for enhancing the visual language model of autonomous driving by incorporating human preferences, including: Step 1: By integrating multi-source and multimodal data, generate preference pairs containing questions, preferred answers, and secondary answers, introduce counterfactual trajectory analysis data, and inject causal reasoning and prior knowledge of safety judgment to form a high-quality human preference thought chain dataset; Step 2: Using the high-quality human preference thought chain dataset output from Step 1, train the model to generate formatted responses containing explicit reasoning processes and final answers / trajectories, thus establishing basic reasoning and expressive capabilities for the autonomous driving visual language model; Step 3: Using direct preference optimization techniques, the supervised fine-tuned model is aligned with human driving preferences, making the model's decisions more consistent with human expectations. During the optimization process, the reference model is kept fixed, and only the current policy model is optimized. Implicit KL divergence constraints prevent the policy from deviating excessively from the reference model. Step 4: While maintaining alignment with human preferences, further optimize the model's open-ended reasoning and trajectory planning capabilities. Employ a grouped relative strategy optimization algorithm, combined with a large language model-based evaluation mechanism and human preference rewards, to achieve multi-objective collaborative optimization.

[0040] In the technical solution of this embodiment, the visual language model solves the problem of fragmented multimodal information, unifying visual perception and semantic understanding into a single model. The thought chain enhances the ability of the large language model to solve complex problems through step-by-step logical reasoning, solving the black-box problem of model decision-making and making its reasoning logic traceable and evaluable. Supervised fine-tuning uses thought chain data, solving the problem of initializing the model's basic capabilities in professional domains. The evaluation mechanism based on the large language model utilizes the general knowledge of the large model as the evaluator, solving the dilemma of difficulty in quantifying and evaluating the quality of responses in open scenarios, and providing continuous reward signals for reinforcement learning. Direct preference optimization directly aligns with human preferences through contrastive learning, solving the efficiency and stability problems of traditional reinforcement learning training complex reward models from human feedback. The grouped relative policy optimization algorithm solves the stability and convergence problems of policy optimization under multi-objective driving.

[0041] In one implementation example, specifically in step one, vehicle-mounted perception information in the same scene is collected in a real road or simulation environment. The perception information includes at least a sequence of visible light camera image frames and a sequence of radar frames. The vehicle status record (including position, speed, heading, etc.) corresponding to each frame of image and each frame of radar data, as well as road structure and traffic control related information, are saved simultaneously. At the same time, precise timestamps are written to the image frames, radar frames and their corresponding vehicle status records, and cross-modal alignment is completed using the timestamps as an index.

[0042] Specifically, referring to the synchronization method of the nuScenes acquisition platform: to achieve cross-modal alignment between the LiDAR and the camera, trigger synchronization can be adopted at the hardware level. When the LiDAR's scan line sweeps across the center of the camera's field of view, the camera exposure is triggered. The image timestamp is defined as the "exposure trigger moment," and the LiDAR timestamp is defined as the "moment when the LiDAR completes one full rotation in this frame." Since the camera exposure time is approximately instantaneous, this method can achieve good cross-modal alignment results. Subsequently, vehicle positioning / motion compensation is combined to further eliminate pose errors caused by residual time differences.

[0043] Alternatively, multimodal data from the NuScenes dataset can be used, including camera images from visible light sensors, LiDAR, millimeter-wave radar, IMU, and GPS data. This simulates visible light sensor data corresponding to eight different positions and angles on the vehicle, recording the timestamp, vehicle position, and attitude for each frame.

[0044] Based on the aligned continuous sequence, key decision moments are selected as the starting points for trajectory labeling. For example, in the aligned continuous multimodal sequence, a vehicle approaches an intersection while proceeding straight. A slow-moving vehicle appears in the same lane ahead, and the left lane is empty. The system needs to choose between "maintaining lane speed and following the vehicle ahead" and "changing lanes to the left to bypass the vehicle." In this case, the "moment when the distance to the vehicle ahead decreases to a threshold and the vehicle's planned action switches from maintaining straight lane speed to intending to change lanes" can be defined as the key decision moment. This is the starting point of the trajectory labeling. A preset prediction time domain is taken from this moment onwards. For example, 3 seconds, at fixed time intervals. (For example, 0.5 s) Sample the actual driving trajectory of the vehicle to obtain The true value points of the future trajectory form a discrete two-dimensional coordinate sequence:

[0045] And key decision moments can be shown in the trajectory diagram. Using the vehicle's position as the origin, plot the curve of the true value trajectory point as it progresses over time.

[0046] The vehicle's actual driving trajectory is extracted from the preset prediction time domain after that moment and labeled as the future trajectory ground truth. The future trajectory ground truth is stored in the form of a discrete two-dimensional coordinate sequence, for example, according to... Records, including Indicates key decision-making moments For reference time, in the selected coordinate system, the vehicle is at the [time point]. The dataset consists of two-dimensional location coordinates for each prediction step, where t represents the index of the critical decision moment and n represents the number of prediction steps. The resulting multimodal scene dataset is organized in units of samples. Each sample contains at least a scene ID, a timestamp sequence, aligned camera image data, aligned radar data, ground truth future trajectory values, and information related to road structure and traffic control.

[0047] For each sample in the dataset, considering the critical decision moment and its future trajectory ground truth, based on the vehicle's state and kinematic / dynamic constraints at that moment, discrete time intervals are defined. The vehicle status is The control variable is acceleration. With front wheel steering angle Sampling period wheelbase It can then be written as:

[0048]

[0049]

[0050] .

[0051] An alternative trajectory corresponding to the real trajectory is artificially constructed. Kinematic / dynamic constraints include state recursion based on a bicycle model, with boundaries set for velocity, acceleration, steering angle, and lateral acceleration. Acceleration continuity and steering continuity are achieved through jerk constraints and steering rate of change constraints, respectively. Continuity is usually expressed as "rate of change," equivalent to limiting the jerk and steering angular velocities. Jerk constraint, acceleration continuity: Steering continuity, steering angle change rate constraint:

[0052] The yaw rate constraint is closer to the physical meaning of a "sharp turn": Alternative trajectories remain dynamically feasible under physical constraints such as acceleration and steering continuity, but exhibit irrationality at the driving strategy level, thus constituting counterfactual trajectory analysis data. Counterfactual trajectory analysis data is used to construct training samples for 'safety assessment / consequence reasoning for a given candidate trajectory', and can also serve as a negative example comparison in the preference alignment stage. Furthermore, it participates in training together with trajectory rewards and collision penalties in the joint optimization stage, thereby improving the quality and safety of trajectory planning.

[0053] For example, more aggressive trajectories can be constructed, including sharp turns at high speeds.

[0054] Around the key decision-making moments of each sample, aligned camera images, radar data, and road and traffic control information are organized into a question context to generate open-ended question texts requiring deep reasoning. Multiple candidate answers are generated under the same question and the same question context input. A large language model is then used to preliminarily score and rank the candidate answers. Subsequently, the selected candidate answers are submitted to domain experts for review. The domain experts determine the better answer and select a relatively poor control answer based on criteria such as driving safety, the degree of rationality and compliance of behavior, and the consistency and comfort of reasoning logic. This forms a question-and-answer sample set with a clear preference relationship of "preferred answer and second-choice answer".

[0055] The above question-and-answer sample set is input from the same question context. The following two candidate answers constitute the answer, including the preferred answer. and the second choice answer That is, forming preference pairs with clear preference relationships. The preferred answer is the better answer confirmed by domain experts, and the second-best answer is the relatively worse control answer.

[0056] In one embodiment, open-ended questions can be set as "Why does the vehicle take a certain action" from the perspective of causal explanation, "What will happen if a certain condition changes" from the perspective of hypothetical reasoning, "Is the current behavior safe" from the perspective of safety assessment, and "What might happen next" from the perspective of predictive analysis.

[0057] In one embodiment, the base model is Qwen3-Max-Thinking, which generates no fewer than 8 candidate answers for each question, and the judging model is DeepSeek-V3.2. Finally, the multimodal scene dataset, alternative trajectories, and the aforementioned preference question-and-answer samples are uniformly integrated and structured for storage, forming a high-quality human preference thought chain dataset.

[0058] In one implementation example, specifically in step two, a pre-trained visual language model is selected as the base model, which includes a visual encoder, a projection layer, and an autoregressive language model decoder.

[0059] In this embodiment, see Appendix Figure 2 As shown, the macroscopic structure of the pre-trained visual language model selected in step two remains unchanged, still consisting of a visual encoder, a projection layer, and an autoregressive language model decoder. In the fine-tuning stage, to reduce training costs and maintain the capabilities of the basic model, low-rank decomposition / low-rank adapter (LoRA) is used to efficiently fine-tune the parameters of some linear transformation layers of the pre-trained model: the original weight matrix is ​​frozen. Introducing a dimension reduction matrix into its parallel branch With increasing dimension matrix The constructed low-rank adapter only updates Parameters. After fine-tuning, the low-rank update matrix can be used. Compared with the original weights The weights are merged into equivalent complete weights for inference deployment. Therefore, the structural level remains "visual encoder, projection layer, autoregressive decoder," with the main changes being the parameter update method during the training phase and the optional adapter branch.

[0060] In one embodiment, the pre-trained model is Qwen3-VL-8B-Instruct.

[0061] The aforementioned high-quality human preference thought chain dataset is organized into an input sequence that can be directly processed by a visual language model. Specifically, this involves: extracting and converting the camera image sequence contained in each sample into visual minimal semantic units using a visual encoder; then mapping the visual features corresponding to the visual minimal semantic units to the embedding space of the language model through a projection layer; subsequently concatenating the mapped visual minimal semantic unit embeddings with the sample's open-ended question text, reasoning process text, answer text, and the text minimal semantic unit embeddings corresponding to the labels and special delimiters to form a complete input sequence for training.

[0062] By extracting camera image sequences through a visual encoder and projecting them into the visual minimum semantic units of the language model embedding space, and then uniformly concatenating them with the text minimum semantic units embedded for open-ended question text, reasoning process text, answer text, and their corresponding labels / special delimiters to form a complete input sequence, visual and linguistic information are fused end-to-end under the same autoregressive sequence modeling framework, and the "question-reasoning-answer" structure is explicitly distinguished. This improves the stability, controllability, and compatibility with subsequent preference alignment and policy optimization training without significantly changing the main structure of the model.

[0063] During supervised fine-tuning training, the complete input sequence is used as conditional information, and the model employs an autoregressive approach. The steps of autoregressive generation: Given a multimodal input, use it as conditional information. .

[0064] The target text sequence consists of the smallest semantic unit of the token, denoted as ,in The sequence length is given.

[0065] Generated step-by-step using an autoregressive approach: [The first...] Step under known conditions With the preceding token In the case of predicting the conditional distribution of the next token .

[0066] During training, the conditional probability of each step is calculated using the pre-order tokens of the real labels (teacher forcing), and the sum (or mean) of the negative log-likelihood / cross-entropy of the entire sequence is used as the loss, which is then minimized.

[0067] Conditional probability modeling formula

[0068] in This represents the smallest semantic unit of the preceding sequence that has been generated, the token.

[0069] The smallest semantic unit of the target text sequence is generated sequentially. The target text sequence consists of a sequence of tokens corresponding to the reasoning process and the final answer, and the sequence length is determined by the labeled text of that sample. During training, the conditional probability of "predicting the next token given the generated preceding tokens and the above input sequence" is modeled, and standard cross-entropy loss is used for the ground truth labeled sequence. The cross-entropy loss can be written as: Standard cross-entropy loss minimizes the difference between the model's predicted distribution and the true labeled token sequence for the sample.

[0070] Considering that the human preference thought chain dataset in step one contains a large number of labels and special delimiters, in order to strengthen the model's distinction between the reasoning process and the answer and reduce the impact of labels and special delimiters on training, the cross-entropy loss can be weighted according to the token type. The weight of the tokens corresponding to the reasoning process and the answer is significantly higher than the weight of the tokens corresponding to the labels and special delimiters, thus forming a weighted supervised fine-tuning training objective.

[0071] Specifically, the model generates target token sequences one by one under the above input sequence conditions using an autoregressive approach. The target token sequence consists of tokens corresponding to the labeled text of the reasoning process and the final answer (or trajectory-related conclusion). The model uses standard cross-entropy loss to minimize the difference between the model's predicted distribution and the true label. Considering that the dataset contains a large number of labels and special delimiters, in order to strengthen the model's distinction between the reasoning process and the answer and reduce the impact of labels and special delimiters on training, the cross-entropy loss can be weighted according to the token type, so that the weight of the tokens corresponding to the reasoning process and the answer is significantly higher than the weight of the tokens corresponding to the labels and special delimiters. In one embodiment, the weight of the reasoning process and the answer is set to 1, the weight of the format label is set to 0.5, and the weight of other special delimiters is set to 0.3.

[0072] Low-rank decomposition is used for efficient parameter fine-tuning of the pre-trained model: The linear transformation layer to be adapted is selected in the pre-trained model, and its original weight matrix is ​​denoted as... And keep it frozen so that it does not participate in gradient updates during fine-tuning; at the same time construct and Parallel low-rank adapters, derived from dimensionality reduction matrices With increasing dimension matrix Composition, in which The adapter output is zero at the beginning of training by using A random Gaussian initialization and B zero initialization.

[0073] The original weight matrix of the selected linear transformation layer is a parameter already present in the pre-trained model. It is frozen during fine-tuning and the gradient is not updated. The set of real numbers Represent a real matrix with d rows and k columns; : Output dimension of the linear layer, output vector length / number of output channels; : The input dimension of this linear layer, input vector length / number of input channels; if the input vector The original output .

[0074] Dimension reduction matrix: transforms the input from... Dimensional compression to dimension: ,in Dimensionality reduction is reflected in: the number of rows is ,Compare Much smaller Upgraded matrix: This refers to the compressed matrix... Dimension Remapping Back Dimensional output space: ,in The dimensional shift is reflected in: the output dimension returning to... , : It is a low-rank / bottleneck dimension (rank), much smaller than and Therefore, the number of parameters that need to be trained is from Down to Training is more memory-efficient and more effective. During forward propagation, for the input vector of this layer... The layer output is obtained by adding the original path output and the adapter path output, that is, in Based on low-rank update (in Depend on (This indicates that) only updates during backpropagation. and gradient, frozen Gradient updates are not accepted; after the fine-tuning termination condition is met, the low-rank update is merged with the original weights to obtain an equivalent weight matrix. Used for inference deployment.

[0075] The input vector of this linear layer has a dimension of , The original weight matrix of this linear layer in the pre-trained model is frozen and not updated. :enter The original path output, obtained through linear transformation of the original weights, is the baseline output. It is the output of this layer of the pre-trained model. The weight increment matrix generated by the low-rank adapter represents the change in weights from the original weights. "Patch / fix" ,in Dimensionality reduction matrix (reducing the dimensionality of the matrix) Dimensional input pressure dimension) : Upgraded dimension matrix, Dimension Remapping Back dimension. The increment / correction of the output by the low-rank adapter path, and... The sums form the final output.

[0076] In one embodiment, the fine-tuning termination condition is that the number of fine-tuning rounds exceeds a threshold N; in another embodiment, the fine-tuning termination condition is that the rate of change between training loss and validation loss is lower than a threshold; in yet another embodiment, the fine-tuning termination condition is that the validation set loss does not change significantly for M consecutive rounds.

[0077] In one embodiment, in step three, the supervised fine-tuning model is aligned with human driving preferences using direct preference optimization techniques, making the model's decisions more consistent with human expectations. During the optimization process, the reference model is kept fixed, and only the current policy model is optimized. Implicit KL divergence constraints prevent the policy from deviating excessively from the reference model.

[0078] From the high-quality human preference thought chain dataset constructed in Step 1, sample triples for preference optimization are extracted and organized in the form of "scenario input - preferred answer - secondary answer". The scenario context input corresponds to the aligned camera image sequence, radar data, vehicle status information, and road and traffic control information at the same critical decision point. The preferred answer corresponds to the better answer confirmed by domain experts within the scenario context and open-ended question text (including reasoning process and conclusion expression). The secondary answer corresponds to the relatively worse answer confirmed by domain experts (also including reasoning process and conclusion expression). These triples constitute the training data for preference learning, on a sample basis.

[0079] A triplet refers to a training sample used for preference learning / preference optimization (such as DPO) consisting of three elements:

[0080] (Scene Input / Context): Input conditions at the same critical decision moment, including aligned camera image sequences, radar data, vehicle status information, road and traffic control information, and corresponding open-ended question text, are generally also fed into the model as part of the context.

[0081] (Preferred Answer): The better answer confirmed by domain experts in this scenario and problem, which usually includes the reasoning process (CoT) + final conclusion / trajectory.

[0082] (Secondary Response): A relatively poor control response confirmed by domain experts, which also includes the reasoning process (CoT) + final conclusion / trajectory.

[0083] In other words, a triple is "the input under the same scenario and the same problem". +A better output + A poor output This is used to teach the model to prefer generating under the same conditions. Instead .

[0084] In terms of preference modeling, the Bradley–Terry preference model is used to describe the preference probability that “the preferred answer is better than the second-choice answer in the same scenario context and the same question text”. The preference probability is obtained by mapping the difference in implicit reward between the preferred answer and the second-choice answer through a sigmoid function, where the implicit reward function is used to characterize the degree of preference of a certain answer in the scenario context.

[0085] To prevent the optimized policy model from deviating excessively from the reference model, an implicit KL divergence constraint is introduced. This establishes a correspondence between the implicit reward and the logarithmic ratio of the probability of the model to generate the answer in this scenario context to the probability of the reference model generating the answer in the same scenario context. The degree of deviation is controlled by the hyperparameter β. Based on this, the above relationship is substituted into the Bradley-Terry preference probability expression. The preference probability can be written as a sigmoid mapping of the difference between the logarithmic probability ratio of the preferred answer and the logarithmic probability ratio of the second-best answer. Based on the maximum likelihood principle, a direct preference optimization loss is constructed with the loglikelihood of this preference probability as the objective. This makes the model more inclined to generate the preferred answer labeled by domain experts rather than the second-best answer in the training data distribution.

[0086] Analytical relationship between optimal policy and implicit reward under implicit KL divergence constraint: in: Hyperparameters for controlling the degree of deviation, This is the partition function (normalization term).

[0087] Correspondence: within the same scene context The model to be optimized and the reference model give the same answer The log-probability ratio satisfies:

[0088] Using the aforementioned "scenario context input - preferred answer - second-choice answer" triplet, the obtained model is further fine-tuned and trained to optimize the direct preference optimization loss, and the process is iterated until the fine-tuning termination condition is reached.

[0089]

[0090] in Indicates multimodal input, This indicates the preferred answer chosen by human experts. This indicates the second-choice answer as marked by human experts.

[0091] In this embodiment, the reference model is kept fixed, and only the current policy model is updated. The deviation of the policy model from the reference model is controlled by the hyperparameter β, so that the preference learning process is carried out in a manner where "the preferred answer is more promoted relative to the reference model, and the second-best answer is more suppressed relative to the reference model." The supervised fine-tuning model is further fine-tuned using the above triples to optimize the direct preference optimization objective until the fine-tuning termination condition is met. In one embodiment, the fine-tuning termination condition is that the number of fine-tuning rounds exceeds a threshold N; in another embodiment, the fine-tuning termination condition is that the preference alignment loss value tends to stabilize. In yet another embodiment, the fine-tuning termination condition is that the KL divergence between the policy model and the reference model reaches the target range.

[0092] In one implementation example, regarding step four, while maintaining the human preference alignment capability already possessed by the obtained visual language model, further optimization is performed to enhance open-ended reasoning and trajectory planning capabilities. Each open-ended question sample in the dataset consists of aligned camera image sequences at the same critical decision moment, radar data, vehicle status information, road and traffic control information, and other scene contexts, along with the corresponding open-ended question text. Therefore, for each open-ended question sample in the dataset, the model generates candidate answers under the same question context input and calculates multi-dimensional rewards. Calculate multi-dimensional reward information: Step 1: Calculate the semantic similarity reward between the generated answer and the reference answer using a pre-trained semantic encoder, as shown in the following formula:

[0093] in This is a sentence embedding function.

[0094] Step 2: Calculate the trajectory planning task reward based on the exponential decay function of the L2 distance at each time point, using the following formula:

[0095] in Let L2 be the distance at second t. This represents the time decay coefficient. If the generated trajectory collides with an obstacle, a negative reward is applied, as shown in the following formula:

[0096] Step 3: Evaluate the consistency between the generated response and human preferences using a large language model:

[0097] The first is a semantic similarity reward. A pre-trained semantic encoder is used to calculate sentence embeddings for both the generated answer text and the corresponding reference answer text, and the cosine similarity between the two embeddings is used as the semantic reward. The second is a trajectory planning reward. For the generated trajectory contained in the candidate answer, the L2 distance between the discrete coordinate sequence output at preset time points and the ground truth of the sample's future trajectory at the corresponding times (1st, 2nd, and 3rd seconds) is calculated and summed exponentially to form the trajectory reward. The time decay coefficient reflects the weight decay at later time points, which can be formally expressed as the weight of each time point item... The summation of, where For the L2 distance at the corresponding time, This is the time decay coefficient; simultaneously, a fixed negative reward is applied when the generated trajectory collides with an obstacle, and the penalty is applied if a collision is detected. Otherwise, it is 0; the third is the preference consistency reward, which uses a large language model to evaluate the probability that the generated answer is consistent with human preferences and uses it as the reward component.

[0098] For multi-objective collaborative optimization, a grouped relative policy optimization method is used for training. For each sample's scene context and question text, G candidate responses are generated from the current policy model, and their generation probabilities are recorded. During training updates, the probability ratio between the old and new policies is introduced, and a truncation function is used to limit the probability ratio to a certain value. The interval is optimized for stability, while a KL divergence penalty term is added to constrain the deviation of the current policy model from the reference model. The KL divergence can be approximated by Monte Carlo averaging the log probability difference between the current policy and the reference model for the group of candidate responses, thereby achieving a joint optimization form of group relative advantage, truncation update and KL constraint.

[0099] Configure a strategy optimization objective function based on the relative advantage of grouping.

[0100] in: The probability ratio between the old and new strategies. The function restricts the probability ratio to an interval. Inside, Let KL divergence penalty coefficient be denoted as . The KL divergence between the policy model and the reference model is calculated using Monte Carlo estimation:

[0101] For each candidate response, the semantic similarity reward, trajectory planning reward (including collision penalty), and preference consistency reward are weighted and summed to obtain a comprehensive reward value. Then, the mean and standard deviation of the comprehensive reward are calculated within the group, and the comprehensive reward of each candidate response is normalized to a relative advantage value to drive policy updates. During training, the weights are dynamically adjusted based on the convergence of each reward component. Weight updates are performed exponentially, for example, they can be written as: ,in To adjust the rate, The target reward value is used; fine-tuning stops when a termination condition is met, resulting in the final visual language model. In one embodiment, the fine-tuning termination condition is that the number of fine-tuning rounds exceeds a threshold N; in another embodiment, the fine-tuning termination condition is that the changes in the overall reward and the rewards of each component tend to stabilize; in yet another embodiment, the fine-tuning termination condition is that the weights of each reward component are stable and no single reward dominates the optimization.

[0102] Regarding the calculation of the comprehensive reward value for each candidate response ,in This is the weighted sum of the above rewards. Calculate the relative dominance value of the responses within the group:

[0103] in and These are the mean and standard deviation of the group rewards, respectively. To prevent division by zero for small constants.

[0104] It should be noted that during the training process, the process of "generating G candidate responses - calculating the comprehensive reward and relative advantage - optimizing the target update parameters according to the grouped relative strategy" is repeated, and the corresponding weights are dynamically adjusted according to the convergence of each reward component. The weight update adopts an exponential form and is jointly controlled by the adjustment rate and the target reward value. Training stops when the fine-tuning termination condition is met, and the final visual language model is obtained.

[0105] The merged weights are used for inference deployment, serving as the output for open-end inference and trajectory planning during the inference phase. Training stops once the fine-tuning meets the termination condition, yielding the final visual language model. During inference deployment, the weight updates generated by low-rank adaptation are merged with the original weights to form an equivalent complete weight matrix for deployment. In the deployment phase, the model takes aligned camera image sequences, radar data, vehicle status information, and road and traffic control information—including contextual information from key decision moments and open-ended question text—as input. This visual language model generates a formatted response containing the inference process and the final conclusion / trajectory, thus supporting the generation and invocation of outputs related to open-end inference, decision interpretation, and trajectory planning in real-world autonomous driving scenarios.

[0106] In one embodiment, the fine-tuning termination condition is that the number of fine-tuning rounds exceeds a threshold N.

[0107] In one embodiment, the fine-tuning termination condition is that the changes in the overall reward and each component reward tend to stabilize.

[0108] In one embodiment, the fine-tuning termination condition is that the weights of each reward component are stable and no single reward dominates the optimization.

[0109] Example 2 The purpose of this embodiment is to provide a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps of the above-described method.

[0110] Example 3 The purpose of this embodiment is to provide a computer-readable storage medium.

[0111] A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, performs the steps of the above method.

[0112] Example 4 The purpose of this embodiment is to provide an autonomous driving visual language model enhancement system that incorporates human preferences, including: The high-quality human preference thought chain dataset construction module is configured to: acquire multi-source multimodal data, process the multi-source multimodal data to generate a question-and-answer sample set containing questions, preferred answers and secondary answers, and further process the question-and-answer sample set to obtain a high-quality human preference thought chain dataset; The visual language model building module is configured to: supervise and fine-tune the training of the visual language model based on the obtained high-quality human preference thought chain dataset; The preference alignment module is configured to align the supervised fine-tuned visual language model with human driving preferences, making the visual language model's decisions more in line with human expectations. The optimization module is configured to: employ a grouped relative strategy optimization algorithm to optimize the model's open-end reasoning and trajectory planning capabilities; combine a large language model-based evaluation mechanism and human preference rewards to achieve multi-objective collaborative optimization; and optimize the trained visual language model to generate a formatted response containing explicit reasoning processes and the final answer / trajectory.

[0113] Example 5 The purpose of this embodiment is to provide a computer program product containing instructions that, when run on a computer, cause the computer to perform the methods and functions involved in any of the above embodiments.

[0114] The steps and methods involved in the apparatus of the above embodiments correspond to those in Embodiment 1. For specific implementation details, please refer to the relevant description section of Embodiment 1. The term "computer-readable storage medium" should be understood as a single medium or multiple media including one or more instruction sets; it should also be understood as including any medium capable of storing, encoding, or carrying an instruction set for execution by a processor and enabling the processor to perform any of the methods in this invention.

[0115] Those skilled in the art will understand that the modules or steps of the present invention described above can be implemented using general-purpose computer devices. Optionally, they can be implemented using computer-executable program code, thereby allowing them to be stored in a storage device for execution by a computer device, or they can be fabricated as separate integrated circuit modules, or multiple modules or steps can be fabricated as a single integrated circuit module. The present invention is not limited to any particular combination of hardware and software.

[0116] While the specific embodiments of the present invention have been described above in conjunction with the accompanying drawings, this is not intended to limit the scope of protection of the present invention. Those skilled in the art should understand that various modifications or variations that can be made by those skilled in the art without creative effort based on the technical solutions of the present invention are still within the scope of protection of the present invention.

Claims

1. An autonomous driving visual language model enhancement method incorporating human preferences, characterized by: include: Acquire multi-source, multi-modal data, process the multi-source, multi-modal data to generate a question-and-answer sample set containing questions, preferred answers, and secondary answers, and further process the question-and-answer sample set to obtain a high-quality human preference thought chain dataset; The visual language model was trained under supervised fine-tuning based on the obtained high-quality human preference thought chain dataset. Aligning the supervised fine-tuned visual language model with human driving preferences makes the visual language model's decisions more in line with human expectations. A group-based relative strategy optimization algorithm is adopted to improve the open-end reasoning and trajectory planning capabilities of the model. Combined with a judgment mechanism based on a large language model and human preference rewards, multi-objective collaborative optimization is achieved. The optimized visual language model is used to generate a formatted response containing explicit reasoning process and final answer / trajectory.

2. The method for enhancing the visual language model of autonomous driving by incorporating human preferences as described in claim 1, characterized in that, Acquiring multi-source, multi-modal data specifically includes: Vehicle-mounted perception information is collected in the same scene in a real road or simulation environment. The perception information includes at least a sequence of visible light camera image frames and a sequence of radar frames. The vehicle status record corresponding to each frame of image and each frame of radar data, as well as road structure and traffic control related information, are saved simultaneously. At the same time, precise timestamps are written to the image frames, radar frames and their corresponding vehicle status records, and cross-modal alignment is completed using the timestamps as an index to obtain the aligned continuous sequence.

3. The method for enhancing the visual language model of autonomous driving by incorporating human preferences as described in claim 2, characterized in that, Also includes: Based on the aligned continuous sequence, key decision moments are selected as the starting point for trajectory labeling. The actual driving trajectory of the vehicle is extracted from the preset prediction time domain after that moment and labeled as the future trajectory ground value. The future trajectory ground value is stored in the form of a discrete two-dimensional coordinate sequence. The resulting multimodal scene dataset is organized in units of samples. Each sample contains at least a scene ID, a timestamp sequence, aligned camera image data, aligned radar data, the future trajectory ground value, and information related to road structure and traffic control.

4. The method for enhancing the visual language model of autonomous driving by incorporating human preferences as described in claim 3, characterized in that, For each sample in the multimodal scenario dataset, the key decision moment and its future trajectory true value are used. Based on the vehicle state and vehicle kinematics / dynamic constraints at that moment, an alternative trajectory corresponding to the true trajectory is artificially constructed. The alternative trajectory remains dynamically feasible under the physical constraints of acceleration and steering continuity, but shows irrationality at the driving strategy level, thus constituting counterfactual trajectory analysis data. Around the key decision-making moments of each sample, aligned camera images, radar data, and road and traffic control information are organized into a question context to generate open-ended question texts that require deep reasoning. Multiple candidate answers are generated under the same question and the same question context input. A large language model is then used to preliminarily score and rank the candidate answers. Subsequently, the selected candidate answers are submitted to domain experts for review, forming a question-and-answer sample set with clear preference relationships of "preferred answer and second-choice answer". The multimodal scenario dataset, alternative trajectories, and the aforementioned preference question-and-answer sample set are integrated and stored in a structured manner to form a high-quality human preference thought chain dataset.

5. The method for enhancing the visual language model of autonomous driving by incorporating human preferences as described in claim 1, characterized in that, The visual language model includes a visual encoder, a projection layer, and an autoregressive language model decoder. The aforementioned high-quality human preference thought chain dataset is organized into an input sequence that can be directly processed by a visual language model: For the camera image sequence contained in each sample, a visual encoder is used to extract and convert it into visual minimal semantic units, and then a projection layer is used to map the visual features corresponding to the visual minimal semantic units to the embedding space of the language model; subsequently, the mapped visual minimal semantic unit embeddings are concatenated with the sample's open-ended question text, reasoning process text, answer text, and the text minimal semantic unit embeddings corresponding to the labels and special delimiters to form a complete input sequence for training.

6. The method for enhancing the visual language model of autonomous driving by incorporating human preferences as described in claim 1, characterized in that, During supervised fine-tuning training, the model generates a target token sequence one by one in an autoregressive manner under the input sequence conditions. The target token sequence consists of tokens corresponding to the labeled text of the reasoning process and the final answer or trajectory-related conclusions. The model uses the standard cross-entropy loss to minimize the difference between the predicted distribution and the actual label. Preferably, it also includes: using low-rank decomposition to efficiently fine-tune the parameters of the pre-trained model: selecting the linear transformation layer to be adapted in the pre-trained model, marking its original weight matrix and keeping it frozen, while constructing a low-rank adapter parallel to the original weight matrix, consisting of a dimension reduction matrix and an dimension increase matrix; during forward propagation... Based on low-rank update During backpropagation, only the gradients of the reduced-dimensional matrix and the increased-dimensional matrix are updated, and the frozen original weight matrix does not receive gradient updates. After the fine-tuning termination condition is met, the low-rank update is merged with the original weights to obtain an equivalent weight matrix for inference deployment. Preferably, the supervised fine-tuning of the model is aligned with human driving preferences, making the model's decisions more consistent with human expectations. This includes: From the constructed high-quality human preference thought chain dataset, sample triples for preference optimization are extracted and organized in the form of "scenario input, preferred answer, and secondary answer". The scenario context input corresponds to the aligned camera image sequence, radar data, vehicle status information, and road and traffic control information at the same key decision moment. The preferred answer corresponds to the better answer confirmed by the domain expert under the scenario context and open-ended question text. The secondary answer corresponds to the relatively worse answer confirmed by the domain expert. In terms of preference modeling, the Bradley-Terry preference model is used to describe the preference probability that "the preferred answer is better than the secondary answer under the same scenario context and the same question text". The preference probability is obtained by the difference between the implicit reward of the preferred answer and the secondary answer through sigmoid mapping. Preferably, it also includes: introducing implicit KL divergence constraints, keeping the reference model fixed, updating only the current policy model, and controlling the degree of deviation of the policy model from the reference model through the hyperparameter β, so that the preference learning process is carried out in a way that "the preferred answer is more promoted relative to the reference model and the second-choice answer is more suppressed relative to the reference model"; The model after supervised fine-tuning is further fine-tuned using the aforementioned triplet, optimizing the direct preference optimization objective until the fine-tuning termination condition is met.

7. The method for enhancing the visual language model of autonomous driving by incorporating human preferences as described in claim 1, characterized in that, Regarding the evaluation mechanism and human preference-based rewards based on large language models, including: For each open-ended question sample in the dataset, the model generates candidate answers under the same question context input and calculates multi-dimensional rewards: First, a semantic similarity reward, which uses a pre-trained semantic encoder to calculate sentence embeddings for the generated answer text and the corresponding reference answer text, and uses the cosine similarity between the two embeddings as the semantic reward; Second, a trajectory planning reward, which calculates the L2 distance between the generated trajectory contained in the candidate answer and the ground truth of the sample's future trajectory at the corresponding time points of the 1st, 2nd, and 3rd seconds, and summarizes them in an exponential decay manner, which can be expressed as the summation of terms at each time point; at the same time, a fixed negative reward is applied when the generated trajectory collides with an obstacle; Third, a preference consistency reward, which evaluates the probability that the generated answer is consistent with human preferences through a large language model and uses it as the reward component; Preferably, in multi-objective collaborative optimization, a grouped relative policy optimization method is used for training: for each sample's scene context and question text, G candidate responses are generated from the current policy model and their generation probabilities are recorded; during training and updating, the probability ratio of the new and old policies is introduced, and the probability ratio is restricted to a set interval by a truncation function to stabilize the optimization. At the same time, a KL divergence penalty term is added to constrain the deviation of the current policy model from the reference model. The KL divergence can be approximated by Monte Carlo averaging the difference in log probabilities of the group of candidate responses under the current policy and the reference model. Preferably, for each candidate response, the semantic similarity reward, trajectory planning reward, and preference consistency reward are weighted and summed to obtain a comprehensive reward value. Then, the mean and standard deviation of the comprehensive reward are calculated within the group and normalized to a relative advantage value to drive policy updates. During training, the weights are dynamically adjusted based on the convergence of each reward component.

8. An autonomous driving visual language model enhancement system incorporating human preferences, characterized in that, include: The high-quality human preference thought chain dataset construction module is configured to: acquire multi-source multimodal data, process the multi-source multimodal data to generate a question-and-answer sample set containing questions, preferred answers and secondary answers, and further process the question-and-answer sample set to obtain a high-quality human preference thought chain dataset; The visual language model building module is configured to: supervise and fine-tune the training of the visual language model based on the obtained high-quality human preference thought chain dataset; The preference alignment module is configured to align the supervised fine-tuned visual language model with human driving preferences, making the visual language model's decisions more in line with human expectations. The optimization module is configured to: employ a grouped relative strategy optimization algorithm to optimize the model's open-end reasoning and trajectory planning capabilities; combine a large language model-based evaluation mechanism and human preference rewards to achieve multi-objective collaborative optimization; and optimize the trained visual language model to generate a formatted response containing explicit reasoning processes and the final answer / trajectory.

9. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the steps of the method described in any one of claims 1-7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it performs the steps of the method described in any one of claims 1-7.

Images