Vehicle control method, device, vehicle and computer readable storage medium

Abstract

The application relates to a vehicle control method, device, vehicle and computer readable storage medium. The method comprises: determining a target input sequence based on image information and vehicle state information of a target vehicle during driving; the target input sequence comprises a visual token sequence; performing planning and prediction processing on the target input sequence by using a planning decision module to obtain initial driving information of the target vehicle; performing adversarial reasoning on the initial driving information and the visual token sequence by using a safety review module to obtain risk warning information of the target vehicle; and performing arbitration processing on the initial driving information and the risk warning information by using an arbitration module to obtain a control instruction of the target vehicle, so that the target vehicle drives according to the control instruction. The application can improve the path decision safety of a vehicle in a complex traffic scene during automatic driving.

Description

Technical Field

[0001] This application relates to the field of autonomous driving technology, specifically to a vehicle control method, apparatus, vehicle, and computer-readable storage medium. Background Technology

[0002] With the continuous development of artificial intelligence and vehicle networking technologies, people's demand for intelligent driving of vehicles is growing. Autonomous driving systems need to continuously perceive changes in the external environment and make inferences and decisions based on context, which places higher demands on the cognitive ability, logical reasoning ability and behavioral planning ability of the trajectory planning model in the autonomous driving system.

[0003] However, existing trajectory planning models suffer from path planning errors in complex traffic scenarios. Therefore, improving the safety of path decision-making in complex traffic scenarios during autonomous driving is an urgent problem to be solved. Summary of the Invention

[0004] This application provides a vehicle control method, apparatus, vehicle, and computer-readable storage medium, which can improve the safety of path decision-making in complex traffic scenarios during autonomous driving.

[0005] This application provides a vehicle control method, the method including:

[0006] Based on the image information and vehicle status information of the target vehicle during its driving process, the target input sequence is determined; the target input sequence includes a visual token sequence.

[0007] The planning and decision module is used to perform planning and prediction processing on the target input sequence to obtain the initial driving information of the target vehicle. The initial driving information includes the scene information of the target vehicle, the behavioral intention of the target vehicle, and the trajectory planning information of the target vehicle. The planning and decision module is a model trained based on historical image information and historical vehicle state information.

[0008] Using a safety review module, adversarial reasoning is performed on initial driving information and visual token sequences to obtain risk warning information for the target vehicle. The risk warning information includes risk identification information and risk assessment information for the target vehicle. The safety review module is a model trained based on historical driving information and historical visual token sequences.

[0009] An arbitration module is used to arbitrate the initial driving information and risk warning information to obtain control commands for the target vehicle, so that the target vehicle can drive according to the control commands. The arbitration module is a model trained based on historical driving information and historical risk warning information.

[0010] Based on the aforementioned technical means, a target input sequence is determined using multi-dimensional information such as image information and vehicle state information. Using the obtained target input sequence and the planning and decision-making module, the initial driving information of the target vehicle can be determined more accurately. Furthermore, after determining the initial driving information, it is not directly used as the vehicle's control command. Instead, potential risks in the current scenario are considered, and adversarial reasoning is performed based on the initial driving information and the safety review module to obtain risk warning information present in the target vehicle's current scenario. Furthermore, the initial driving information and risk warning information can be fused using an arbitration module to obtain both efficient and safe driving actions, thereby improving the safety of path decision-making in complex traffic scenarios during autonomous driving.

[0011] In some embodiments, a target input sequence is determined based on image information and vehicle state information of the target vehicle during its driving process, including:

[0012] Acquire image information and vehicle status information of the target vehicle during its driving process; the vehicle status information includes the vehicle's operating status and navigation instruction information.

[0013] The image information is image encoded using a visual encoder to obtain a visual token sequence; the vehicle operating status and navigation command information are state encoded using a state encoder to obtain a state token sequence corresponding to the vehicle operating status and a command token sequence corresponding to the navigation command information.

[0014] The visual token sequence, status token sequence, and instruction token sequence are concatenated to obtain the target input sequence.

[0015] Based on the above technical means, by acquiring vehicle image information and vehicle state information, and encoding the image information and vehicle state information respectively, a visual token sequence, a state token sequence, and an instruction token sequence are obtained. The three are then concatenated to obtain the target input sequence, which can be used as input for subsequent models. This provides complete and accurate input information for subsequent models, thereby facilitating accurate prediction and reasoning by the subsequent models using the input information.

[0016] In some embodiments, a security review module is used to perform adversarial reasoning on initial driving information and visual token sequences to obtain risk warning information for the target vehicle, including:

[0017] Based on image information and visual token sequences, determine the target complexity score of the image information;

[0018] If the target complexity score is greater than the preset score, the security review module is used to perform adversarial reasoning on the initial driving information and visual token sequence to obtain risk warning information for the target vehicle.

[0019] Based on the aforementioned technical means, the target complexity score of image information can be determined based on image information and visual token sequences. The security review module is activated only when the target complexity score is greater than the preset score. In other words, the security review module is only needed to conduct risk assessment when the complexity of the current scenario is high. This can further improve the safety of path decision-making in complex traffic scenarios during autonomous driving. When the complexity of the current scenario is low, only the initial driving information output by the planning decision module needs to be used. This can reduce computing resources and lower computing energy consumption while ensuring decision-making safety.

[0020] In some embodiments, the target complexity score of the image information is determined based on the image information and the visual token sequence, including:

[0021] The visual feature entropy is determined based on the attention weight of each visual token in the visual token sequence; the visual feature entropy is used to characterize the uniformity of the distribution of visual content in image information.

[0022] The occlusion rate of the image information is determined based on the number of pixels in the occluded area and the number of pixels in the unoccluded area of ​​the image information.

[0023] The dynamic complexity score of the image information is determined based on the weight of each object in the image information, the speed of each dynamic object in the image information relative to the target vehicle, and the spatial orientation weight of each object; among which, the spatial orientation weight is determined based on the azimuth angle of the object in the target vehicle coordinate system.

[0024] The target complexity score is obtained by weighting the visual feature entropy, the occlusion rate of image information, and the dynamic complexity score.

[0025] Based on the aforementioned technical means, visual feature entropy, occlusion rate, and dynamic complexity score can be determined based on image information and visual token sequence. In other words, the current scene can be evaluated in multiple dimensions. By fusing scene density, occlusion rate, and scene complexity, the target complexity score of the current scene can be determined more accurately. This can make the activation of the security review module more precise and further improve the safety of path decision-making in complex traffic scenarios during autonomous driving.

[0026] In some embodiments, an arbitration module is used to arbitrate initial driving information and risk warning information to obtain control commands for the target vehicle, including:

[0027] The first semantic feature vector corresponding to the first preset layer when the planning and decision module outputs initial driving information is obtained, and the second semantic feature vector corresponding to the second preset layer when the safety review module outputs risk warning information is obtained;

[0028] Using the arbitration module, the control command for the target vehicle is determined based on the cosine distance between the first semantic feature vector and the second semantic feature vector.

[0029] Based on the aforementioned technical means, the first semantic feature vector corresponding to the last layer when the planning and decision-making module generates initial driving information and the second semantic feature vector corresponding to the last layer when the safety review module generates risk warning information can be determined. The difference between the two generated texts can be determined based on the cosine distance between the first and second semantic feature vectors, and different arbitration strategies can be executed according to different differences. In this way, the safety of path decision-making in complex traffic scenarios during autonomous driving can be further improved.

[0030] In some embodiments, the arbitration module is used to determine the control command for the target vehicle based on the cosine distance between the first semantic feature vector and the second semantic feature vector, including:

[0031] When the cosine distance is less than the first distance threshold, the control command for the target vehicle is determined based on the initial driving information;

[0032] When the cosine distance is less than the second distance threshold and greater than or equal to the first distance threshold, the control command for the target vehicle is determined based on the fusion network; wherein, the fusion network is trained based on a two-stage training strategy.

[0033] If the cosine distance is greater than or equal to the second distance threshold, the control command for the target vehicle is determined based on the risk warning information.

[0034] Based on the aforementioned technical means, when the cosine distance is less than the first distance threshold, it indicates that the safety review module has not detected any safety risks in the current traffic scenario, and control commands can be directly determined based on the initial driving information, thus ensuring efficient response in normal scenarios. When the cosine distance is less than the second distance threshold but greater than or equal to the first distance threshold, the fusion network optimized by the two-stage training strategy is activated to dynamically balance aggressive and conservative strategies, generating driving decisions that are closer to expert levels. When the cosine distance is greater than or equal to the second distance threshold, risk warning information is prioritized to ensure that safety is the primary objective in extreme or high-risk scenarios. This not only improves the adaptability of the autonomous driving system to complex scenarios but also effectively balances driving efficiency and safety.

[0035] In some embodiments, determining control commands for the target vehicle based on the fusion network includes:

[0036] The first token sequence output by the planning decision module, the second token sequence output by the security review module, and the special separator token are determined; the special separator token is used to identify the boundary between different types of sequences.

[0037] The target input sequence, the first token sequence, the second token sequence, and the special delimiter token are subjected to a first processing to obtain the second target input sequence; wherein, the first processing includes at least concatenation processing and position encoding processing;

[0038] By using a fusion network, the second target input sequence is fused and predicted to obtain the control command for the target vehicle.

[0039] Based on the aforementioned technical means, by determining the first token sequence of the planning and decision-making module and the second token sequence of the security review module, and introducing a special separator token to clearly identify the boundary between the two types of sequences, information confusion is effectively avoided. Subsequently, through splicing and position encoding, a second target input sequence containing sequence order and position information is constructed, enabling the fusion network to fully perceive the temporal structure and contextual relationships of different strategies. Finally, the fusion network optimized through two-stage training is used for deep fusion prediction to generate accurate vehicle control commands. This not only enhances the model's semantic understanding of complex driving scenarios but also ensures the organic integration of aggressive and conservative strategies at the decision-making level, thereby improving the decision-making quality of the autonomous driving system in various scenarios.

[0040] In some embodiments, the fusion network includes a dynamic weight prediction module, an adversarial awareness attention module, and an inference generation module; using the fusion network, the second target input sequence is subjected to fusion prediction processing to obtain control commands for the target vehicle, including:

[0041] Using the dynamic weight prediction module, the first token sequence and the second token sequence are weighted to predict the weights and determine the first weight coefficient corresponding to the planning decision module and the second weight coefficient corresponding to the security review module.

[0042] Using an adversarial awareness attention module, attention weights are adjusted on the attention bias matrix, query matrix, key matrix, and value matrix based on the first and second weight coefficients to obtain the fused feature vector;

[0043] Using the reasoning generation module, reasoning is performed on the second target input sequence and the fused feature vector to obtain the action token sequence;

[0044] The action token sequence is decoded using an action decoder to obtain the control commands for the target vehicle.

[0045] Based on the aforementioned technical means, the dynamic weight prediction module adaptively generates the weight coefficients of aggressive and conservative strategies according to the real-time scene; the adversarial perception attention module dynamically adjusts the bias matrix based on these weights, guiding the model to focus on more important policy regions, thereby achieving deep fusion at the feature level; the inference generation module adopts an autoregressive approach to generate coherent inference text and action token sequences based on the fused features; finally, the action decoder converts the token sequences into executable control commands, thereby improving the decision-making quality, safety, and reliability of the autonomous driving system in complex scenarios.

[0046] In some embodiments, a dynamic weight prediction module is used to perform weight prediction processing on the first token sequence and the second token sequence to determine the first weight coefficient corresponding to the planning decision module and the second weight coefficient corresponding to the security review module, including:

[0047] Determine the first feature vector corresponding to the last token in the first token sequence, and the second feature vector corresponding to the last token in the second token sequence;

[0048] The risk feature vector is obtained by average pooling the hidden states corresponding to all tokens in the second token sequence.

[0049] The first feature vector, the second feature vector, and the risk feature vector are concatenated to obtain the target vector;

[0050] The target vector is input into the dynamic weight prediction module for weight prediction processing to obtain the first weight coefficient and the second weight coefficient.

[0051] Based on the aforementioned technical means, the feature vector of the last token in the first token sequence is extracted as the global representation of the aggressive strategy. At the same time, the feature vector of the last token in the second token sequence is combined with the risk feature vector obtained by average pooling to comprehensively capture the semantic information of the conservative strategy. Subsequently, the target vector constructed by concatenating the three is input into the dynamic weight prediction module to generate the first weight coefficient and the second weight coefficient that match the current scenario. This allows the model to dynamically adjust the contribution ratio of the aggressive and conservative strategies according to the real-time environment. In low-risk scenarios, the aggressive strategy is given higher weight to pursue efficiency, while in high-risk scenarios, the conservative strategy is strengthened to ensure safety. This achieves the optimal balance between efficiency and safety, thereby enhancing the intelligence and robustness of the autonomous driving decision-making system.

[0052] This application provides a vehicle control device, the device comprising:

[0053] The determining unit is used to determine the target input sequence based on the image information and vehicle state information of the target vehicle during its driving process; the target input sequence includes a visual token sequence.

[0054] The planning and prediction unit is used to perform planning and prediction processing on the target input sequence using the planning and decision module to obtain the initial driving information of the target vehicle. The initial driving information includes the scene information of the target vehicle, the behavioral intention of the target vehicle, and the trajectory planning information of the target vehicle. The planning and decision module is a model trained based on historical image information and historical vehicle state information.

[0055] The adversarial reasoning unit is used to perform adversarial reasoning on the initial driving information and visual token sequence using the safety review module to obtain risk warning information of the target vehicle. The risk warning information includes risk identification information and risk assessment information of the target vehicle. The safety review module is a model trained based on historical driving information and historical visual token sequences.

[0056] The arbitration unit is used to arbitrate the initial driving information and risk warning information using the arbitration module to obtain the control command of the target vehicle, so that the target vehicle can drive according to the control command. The arbitration module is a model trained based on historical driving information and historical risk warning information.

[0057] This application provides a vehicle including a processor and a memory. The memory stores a computer program that can run on the processor. When the processor executes the computer program, it implements the steps in any of the above methods.

[0058] This application provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the steps in any of the above methods.

[0059] This application provides a computer program product, including a computer program or instructions, which, when executed by a processor, implement the steps of any of the above methods.

[0060] The beneficial effects of this application are:

[0061] (1) The target input sequence is determined based on multi-dimensional information such as image information and vehicle status information. Based on the obtained target input sequence and planning decision module, the initial driving information of the target vehicle can be determined more accurately. Furthermore, after determining the initial driving information, the initial driving information is not directly used as the control command of the vehicle. Instead, the potential risks in the current scenario of the vehicle are further considered. Adversarial reasoning is performed based on the initial driving information and the safety review module to obtain the risk warning information in the current scenario of the target vehicle. Furthermore, the initial driving information and risk warning information can be fused according to the arbitration module to obtain a driving action that is both efficient and safe, thereby improving the safety of path decision-making in complex traffic scenarios during the autonomous driving process of the vehicle.

[0062] (2) By acquiring the vehicle's image information and vehicle status information, and encoding the image information and vehicle status information respectively, a visual token sequence, a status token sequence and an instruction token sequence are obtained. The three are then concatenated to obtain the target input sequence, which can be used as the input for subsequent models. This provides complete and accurate input information for subsequent models, which is beneficial for subsequent models to make accurate predictions and inferences using the input information.

[0063] (3) The target complexity score of the image information can be determined based on the image information and the visual token sequence. The security review module is activated only when the target complexity score is greater than the preset score. In other words, the security review module is only needed to conduct risk assessment when the complexity of the current scenario is high. This can further improve the safety of path decision-making in complex traffic scenarios during the autonomous driving process of the vehicle. When the complexity of the current scenario is low, only the initial driving information output by the planning decision module needs to be planned. This can reduce computing resources and reduce computing energy consumption while ensuring the safety of decision-making.

[0064] (4) Visual feature entropy, occlusion rate and dynamic complexity score can be determined based on image information and visual token sequence. In other words, the current scene is evaluated in multiple dimensions. Based on scene density, occlusion rate and scene complexity fusion, the target complexity score of the current scene can be determined more accurately, which can make the activation of the security review module more precise and further improve the safety of path decision-making in complex traffic scenarios during the autonomous driving process of vehicles.

[0065] (5) The first semantic feature vector corresponding to the last layer of the planning decision module when generating initial driving information and the second semantic feature vector corresponding to the last layer of the safety review module when generating risk warning information can be determined. The difference between the two generated texts can be determined based on the cosine distance between the first semantic feature vector and the second semantic feature vector. Different arbitration strategies can be executed according to different differences. In this way, the safety of path decision-making in complex traffic scenarios during the autonomous driving process of the vehicle can be further improved.

[0066] (6) When the cosine distance is less than the first distance threshold, it indicates that the safety review module has not found any safety risks in the current traffic scenario, and the control command can be determined directly based on the initial driving information, which can ensure efficient response in normal scenarios; when the cosine distance is less than the second distance threshold and greater than or equal to the first distance threshold, the fusion network optimized by the two-stage training strategy is activated to dynamically balance the aggressive and conservative strategies and generate driving decisions that are closer to the level of experts; when the cosine distance is greater than or equal to the second distance threshold, risk warning information is adopted first to ensure that safety is the primary goal in extreme or high-risk scenarios; this not only improves the adaptability of the autonomous driving system to complex scenarios, but also effectively balances driving efficiency and safety.

[0067] (7) By determining the first token sequence of the planning decision module and the second token sequence of the security review module, and introducing special separator tokens to clearly identify the boundaries of the two types of sequences, information confusion is effectively avoided. Then, through splicing and position encoding, a second target input sequence containing sequence order and position information is constructed, enabling the fusion network to fully perceive the temporal structure and contextual relationship of different strategies. Finally, the fusion network optimized by two-stage training is used to perform deep fusion prediction and generate accurate vehicle control commands. This not only enhances the model's semantic understanding of complex driving scenarios, but also ensures the organic integration of aggressive and conservative strategies at the decision-making level, thereby improving the decision-making quality of the autonomous driving system in various scenarios.

[0068] (8) The dynamic weight prediction module adaptively generates the weight coefficients of aggressive and conservative strategies based on the real-time scenario; the adversarial perception attention module dynamically adjusts the bias matrix based on the weights to guide the model to focus on more important policy regions and achieve deep fusion at the feature level; the reasoning generation module adopts an autoregressive approach to generate coherent reasoning text and action token sequences based on the fused features; finally, the action decoder converts the token sequences into executable control commands, thereby improving the decision-making quality, safety and reliability of the autonomous driving system in complex scenarios.

[0069] (9) By extracting the feature vector of the last token of the first token sequence as the global representation of the aggressive strategy, and combining the feature vector of the last token of the second token sequence with the risk feature vector obtained by average pooling, the semantic information of the conservative strategy is fully captured. Then, the target vector constructed by splicing the three is input into the dynamic weight prediction module to generate the first weight coefficient and the second weight coefficient that match the current scenario. This allows the model to dynamically adjust the contribution ratio of the aggressive and conservative strategies according to the real-time environment. In low-risk scenarios, the aggressive strategy is given higher weight to pursue efficiency, while in high-risk scenarios, the conservative strategy is strengthened to ensure safety. This achieves the optimal balance between efficiency and safety, thereby enhancing the intelligence and robustness of the autonomous driving decision-making system. Attached Figure Description

[0070] Figure 1 A flowchart illustrating a vehicle control method provided in this application embodiment. Figure 1 ;

[0071] Figure 2 A flowchart illustrating a vehicle control method provided in this application embodiment. Figure 2 ;

[0072] Figure 3 A flowchart illustrating a vehicle control method provided in this application embodiment. Figure 3 ;

[0073] Figure 4 This is a schematic diagram of the composition structure of a vehicle control device provided in an embodiment of this application;

[0074] Figure 5 This is a schematic diagram of the hardware entity of a vehicle provided in an embodiment of this application.

[0075] It should be noted that the terms "first" and "second" mentioned above are only used to distinguish between different options and do not represent the degree of superiority or inferiority of the options or their priority in the implementation process. Detailed Implementation

[0076] The embodiments of this application will be described below with reference to the accompanying drawings and preferred embodiments. Those skilled in the art can easily understand other advantages and effects of this application from the content disclosed in this specification. This application can also be implemented or applied through other different specific embodiments, and various details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of this application. It should be understood that the preferred embodiments are only for illustrating this application and are not intended to limit the scope of protection of this application.

[0077] It should be noted that the illustrations provided in the following embodiments are only schematic representations of the basic concept of this application. Therefore, the drawings only show the components related to this application and are not drawn according to the actual number, shape and size of the components in the actual implementation. In the actual implementation, the form, quantity and proportion of each component can be arbitrarily changed, and the layout of the components may also be more complex.

[0078] In the following description, references are made to “some embodiments,” which describe a subset of all possible embodiments. However, it is understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0079] In the following description, the terms "first, second, third" are used merely to distinguish similar objects and do not represent a specific ordering of objects. It is understood that "first, second, third" may be interchanged in a specific order or sequence where permitted, so that the embodiments of this application described herein can be implemented in an order other than that illustrated or described herein.

[0080] In this embodiment, the term "and / or" is merely a description of the relationship between related objects, indicating that there can be three relationships. For example, object A and / or object B can represent three situations: object A exists alone, object A and object B exist simultaneously, and object B exists alone.

[0081] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.

[0082] With the continuous development of artificial intelligence and vehicle networking technologies, people's demand for intelligent driving of vehicles is growing. Autonomous driving systems need to continuously perceive changes in the external environment and make inferences and decisions based on context, which places higher demands on the model's cognitive ability, logical reasoning ability and behavior planning ability.

[0083] Furthermore, with the rapid development of deep learning technology, end-to-end autonomous driving technology has gradually become a research hotspot in both academia and industry. Traditional modular autonomous driving systems separate perception, prediction, planning, and control into independent modules. While this facilitates debugging and maintenance, it also leads to problems such as information transmission loss and error accumulation. End-to-end autonomous driving systems attempt to directly map sensor inputs to control outputs through a single neural network, hoping to achieve better overall performance.

[0084] In recent years, Vision-Language-Action (VLA) framework models have shown great potential in the field of autonomous driving. These models combine the scene understanding capabilities of Vision Language Models (VLMs) with the reasoning capabilities of language models, enabling them to generate interpretable driving decisions. This end-to-end planning approach based on large models has become a research hotspot. Typical examples include DriveVLM and AutoVLA. These models employ a Transformer architecture and utilize a Chain-of-Thought (CoT) mechanism. Through visual feature extraction and textual reasoning, the model generates a scene description and reasoning process before outputting actions, thereby improving the interpretability of decisions. However, existing VLA models suffer from the following key technical problems:

[0085] (1) The limitation of unidirectional reasoning: Existing models generally adopt a unidirectional linear CoT reasoning process, that is, generating in the order of "vision → scene understanding → intent → action". This method can achieve good results in conventional driving scenarios, but the unidirectional reasoning generation mode lacks the ability to actively explore potential risks. In actual driving, human drivers often engage in counterfactual thinking (e.g., "What if a pedestrian suddenly runs out of an obscured area ahead?"). At this time, human drivers may take defensive measures in advance, such as reducing the speed to pass through the area, in order to deal with possible dangerous events in the future. However, existing models lack this "self-questioning" ability.

[0086] (2) Insufficient ability to handle long-tail scenarios: The decision-making of existing VLA models mainly relies on the distribution of training data and implicitly learns the risks. For long-tail scenarios with low probability but high risk (such as "ghost peek", sudden braking of the car in front, sudden lane change of the obstructing vehicle, etc.), the model tends to output a high-probability "average" decision rather than providing corresponding solutions for potential risks. Therefore, it lacks clear counterfactual reasoning ability.

[0087] (3) Lack of “self-dialectical thinking” in model decision-making: When making decisions, current models usually output the best action based on a single inference chain, without any further thinking mechanisms to “self-examine” their decisions. Although mechanisms like CoT can provide some interpretability, the model only explains “why to do it this way”, but not “why not to do it that way”. It lacks a dialectical thinking process like the human brain. A complete decision should include an analysis of rejected options, which is crucial for the safety of autonomous driving systems in complex scenarios.

[0088] Therefore, improving the safety of path decision-making in complex traffic scenarios during autonomous driving is an urgent problem to be solved.

[0089] Based on this, embodiments of this application provide a vehicle control method, the method comprising: determining a target input sequence based on image information and vehicle state information of a target vehicle during its driving process; the target input sequence including a visual token sequence; using a planning and decision module to perform planning and prediction processing on the target input sequence to obtain initial driving information of the target vehicle; wherein, the initial driving information includes scene information of the target vehicle, the behavioral intention of the target vehicle, and trajectory planning information of the target vehicle; using a security review module to perform adversarial reasoning on the initial driving information and the visual token sequence to obtain risk warning information of the target vehicle; wherein, the risk warning information includes risk identification information and risk assessment information of the target vehicle; and using an arbitration module to arbitrate the initial driving information and the risk warning information to obtain control commands for the target vehicle, so that the target vehicle drives according to the control commands. In this way, the target input sequence is determined based on multi-dimensional information such as image information and vehicle state information. Based on the obtained target input sequence and the planning and decision-making module, the initial driving information of the target vehicle can be determined more accurately. Furthermore, after determining the initial driving information, it is not directly used as the vehicle control command. Instead, the potential risks in the current scenario of the vehicle are further considered. Adversarial reasoning is performed based on the initial driving information and the safety review module to obtain risk warning information of the target vehicle in the current scenario. Furthermore, the initial driving information and risk warning information can be fused by the arbitration module to obtain driving actions that are both efficient and safe, thereby improving the safety of path decision-making in complex traffic scenarios during autonomous driving.

[0090] The technical solutions in the embodiments of this application will now be clearly and completely described with reference to the accompanying drawings.

[0091] It should be noted that the vehicle control method provided in the embodiments of this application can be executed by an electronic device, wherein the electronic device can be an electronic control unit in the vehicle.

[0092] Figure 1 A flowchart illustrating a vehicle control method provided in this application embodiment. Figure 1 ,like Figure 1 As shown, it may include S101 to S104, wherein:

[0093] S101, Based on the image information and vehicle status information of the target vehicle during its driving process, determine the target input sequence; wherein, the target input sequence includes a visual token sequence.

[0094] Here, image information refers to multiple images captured by one or more cameras installed around the vehicle during its operation; the images may include, but are not limited to, motor vehicles, pedestrians, animals, and non-motorized vehicles.

[0095] Here, vehicle status information refers to the vehicle's operating status during driving; for example, vehicle status information may include, but is not limited to, vehicle operating status and navigation instruction information.

[0096] Among these, vehicle operating status refers to the physical motion data of the vehicle during driving, which may include the vehicle's speed, heading angle, acceleration, etc.; navigation instruction information refers to the specific instructions or suggestions issued by the navigation system to the driver or vehicle control system to guide the vehicle to travel along a predetermined route. For example, navigation instruction information may include, but is not limited to, turn instructions (e.g., "Turn right in 200 meters", "Please turn left", "Keep going straight"), lane guidance (e.g., "Please enter the rightmost lane", "Please use the two leftmost lanes", "Please follow the main road ahead"), etc.

[0097] Here, the target input sequence refers to a series of ordered feature vectors obtained after preprocessing the acquired image information and vehicle state information. The visual token sequence refers to a set of visual feature vectors after transforming the original image data.

[0098] Next, embodiments of this application will describe the specific implementation of determining the target input sequence.

[0099] In one possible implementation, the acquired image information and vehicle state information of the target vehicle can be encoded separately, and the encoded token sequences can be concatenated to obtain the target input sequence.

[0100] In another possible implementation, the acquired image information and state information can be encoded as independent token sequences, and after input, the two can interact with each other through the model's cross-attention layer to form a fused target input sequence in the deep network.

[0101] S102, using the planning and decision-making module, performs planning and prediction processing on the target input sequence to obtain the initial driving information of the target vehicle.

[0102] The planning and decision-making module is a model trained based on historical image information and historical vehicle status information.

[0103] Here, the planning and decision-making module refers to the model that generates the vehicle's initial driving intention based on the target input sequence; it can be understood that the planning and decision-making model can be built within the VLA model.

[0104] Understandably, the planning and decision-making module can employ a deep learning-based neural network model, such as a trajectory prediction-decision joint model combining the Transformer architecture and multi-head attention mechanisms. The driving decision-making problem can be modeled as a sequence-to-sequence generation task: the input is the target vehicle's historical state sequence (including position, speed, and heading angle) and its surrounding environmental context information (such as adjacent lane structures, traffic signs, and surrounding vehicle trajectories). An encoder extracts high-dimensional features, and a decoder outputs the driving behavior at future moments, including discrete decision intentions (such as following, changing lanes, and avoiding obstacles) and continuous planned trajectories.

[0105] It should be noted that the planning and decision-making module can also be called the scheme proposal module or the proposer module, and this application embodiment does not limit it.

[0106] Here, initial driving information refers to an efficient driving plan that complies with traffic rules, generated by the planning and decision-making module based on the current scenario (target input sequence). This plan may include, but is not limited to, scenario information of the target vehicle, the target vehicle's behavioral intentions, and the target vehicle's trajectory planning information.

[0107] Among them, scene information refers to the description of the traffic environment in which the vehicle is currently located, such as road type, surrounding vehicles, traffic lights, etc.; behavioral intention refers to the driving intention of the vehicle in the future, such as keeping the current lane, changing lanes to the left, turning right, etc.; trajectory planning refers to the sequence of trajectory points of the target vehicle in the future, such as the sequence of trajectory points of the vehicle in the next 4 seconds, with one node every 0.5 seconds, including position, speed, and heading angle.

[0108] In some embodiments, the target input sequence and target cue words can be input together into the planning and decision model to output the initial driving intention of the target vehicle. The target cue words help the planning and decision model better understand the task objective, thus making the output initial driving intention more accurate; for example, the target cue words could be to generate an efficient driving plan that complies with traffic rules.

[0109] It should be noted that the decision-making principles of the planning decision-making model may include: 1. prioritizing traffic efficiency; 2. complying with traffic rules; 3. assuming that the behavior of other traffic participants conforms to regulations; and 4. prioritizing the most direct route planning.

[0110] For example, after receiving input multi-view image data, vehicle status, and navigation commands, the electronic control unit extracts features through a visual encoder and a state encoder to obtain input X (target input sequence). Then, the proposer module generates initial driving information based on the input sequence X. The output format is: "<Scene Description><Behavioral Intent><Track Planning>". Output example: "Scene Description: Green light ahead, slow-moving vehicle in the left lane, clear right lane; Behavioral Intent: Change lane to the right lane and accelerate through the intersection; Trajectory Planning: [t=0.5s: offset 0.5m to the right, v=15m / s; t=1.0s: complete lane change, v=16m / s; ...; t=4.0s: maintain right lane, v=20m / s]".

[0111] S103 utilizes the security review module to perform adversarial reasoning on the initial driving information and visual token sequence to obtain risk warning information for the target vehicle.

[0112] The safety review module is a model trained based on historical driving information and historical visual token sequences.

[0113] It is understandable that historical visual token sequences can be determined based on historical image information and historical vehicle state information; historical driving information can be determined based on historical target input sequences; and historical target input sequences can be determined based on historical image information and historical vehicle state information.

[0114] Here, the safety review module is responsible for identifying potential risk points in the current environment. The safety review module refers to a model that generates risk warning information about the target vehicle's current traffic environment based on the initial driving information output by the planning and decision-making module and the visual token sequence. It is understandable that the safety review module can also be built within the VLA model.

[0115] Understandably, the security review module can adopt a hybrid model that combines adversarial generative networks with anomaly detection. For example, an adversarial risk reasoning network can be designed. The core idea of ​​this model is to treat the driving scenario as a dynamic game process: the generator attempts to simulate normal or potential adversarial driving behaviors, while the discriminator is responsible for identifying the risks implied in these behaviors.

[0116] It should be noted that the security review module can also be called a risk assessment module, a critic module, or a Critic module, and this application embodiment does not limit it.

[0117] Here, risk warning information refers to warning information obtained through a critical evaluation of initial driving information by a safety review module, which may include, but is not limited to, risk identification information, risk assessment information, and counterfactual scenarios of the target vehicle.

[0118] Among them, risk identification information refers to all potential risk points in the current traffic environment of the target vehicle; risk assessment information refers to the safety assessment of all potential risk points and the risk level (low / medium / high) of each potential risk point; and counterfactual scenario refers to the description of the worst possible situation.

[0119] In other words, after generating initial driving information, the initial driving information is not directly applied to the vehicle's trajectory planning. Instead, the impact of blind spots, unexpected behaviors of other vehicles, and environmental uncertainties on the target vehicle is further considered to obtain risk warning information for the target vehicle.

[0120] In one possible implementation, initial driving information, a visual token sequence, and a target prompt word can be jointly input into a safety review module, thereby outputting risk warning information for the target vehicle. The target prompt word can be used to provide direction for the output of the safety review module; for example, the target prompt word could be a critical evaluation of a given driving plan.

[0121] In another possible implementation, if the complexity of the current traffic scenario exceeds a preset complexity, the safety review module is activated. Initial driving information, visual token sequences, and target prompts are input into the safety review module, which can then output risk warning information for the target vehicle. If the complexity of the current traffic scenario is less than or equal to the preset complexity, the initial driving scenario output by the planning and decision module is used as the control command for the target vehicle.

[0122] S104, using the arbitration module, arbitrates the initial driving information and risk warning information to obtain control commands for the target vehicle, so that the target vehicle can drive according to the control commands.

[0123] The arbitration module is a model trained based on historical driving information and historical risk warning information.

[0124] Understandably, historical risk warning information can be determined based on historical driving information and historical visual token sequences.

[0125] Here, the arbitration module can be used to dialectically integrate the outputs of the planning decision-making module and the safety review module. In other words, the arbitration module refers to the current control command of the target vehicle jointly generated by the planning decision-making module and the safety review module.

[0126] Understandably, the arbitration module can employ a hierarchical game arbitration model based on a combination of reinforcement learning and game theory. The core idea of ​​this model is to view the allocation of driving control as a multi-agent game process, in which the "efficiency demands" represented by the initial driving information and the "safety demands" represented by the risk warning information interact with each other, ultimately reaching an optimal equilibrium solution.

[0127] It should be noted that the arbitration module can also be built within the VLA model.

[0128] The control commands may include the target speed, target acceleration, and steering wheel angle of the target vehicle.

[0129] In some embodiments, initial driving information and risk warning information can be jointly input into a pre-trained arbitration module, which can assess the degree of disagreement between the parties to generate an action token sequence. Further, the action token sequence can be input into an action decoder to decode it into specific vehicle control commands. Further, the vehicle can drive based on the vehicle control commands.

[0130] It should be noted that the motion decoder module can use a multilayer perceptron network to map the 1024-dimensional motion token into continuous control quantities (i.e., control commands), including target speed, target acceleration, steering wheel angle, etc.

[0131] In this embodiment, the target input sequence is determined based on multi-dimensional information such as image information and vehicle state information. Based on the obtained target input sequence and the planning and decision-making module, the initial driving information of the target vehicle can be determined more accurately. Furthermore, after determining the initial driving information, it is not directly used as the vehicle's control command. Instead, potential risks in the current scenario are considered, and adversarial reasoning is performed based on the initial driving information and the safety review module to obtain risk warning information present in the current scenario of the target vehicle. Furthermore, the initial driving information and risk warning information can be fused by the arbitration module to obtain both efficient and safe driving actions, thereby improving the safety of path decision-making in complex traffic scenarios during autonomous driving.

[0132] In some embodiments, the step of "determining the target input sequence based on the image information and vehicle state information of the target vehicle during its driving process" in S101 above may include the following steps:

[0133] S1011, acquire image information and vehicle status information of the target vehicle during its driving process; wherein, the vehicle status information includes vehicle operating status and navigation instruction information.

[0134] S1012, use a visual encoder to encode the image information to obtain a visual token sequence; use a state encoder to encode the vehicle operating status and navigation command information to obtain a state token sequence corresponding to the vehicle operating status and a command token sequence corresponding to the navigation command information.

[0135] Here, the state token sequence refers to a set of discrete or continuous feature vectors obtained by extracting and vectorizing the vehicle's kinematic data (such as driving speed, heading angle, acceleration, etc.) through a state encoder; the command token sequence refers to a set of feature vector sequences obtained by encoding the guidance information issued by the navigation system (such as steering prompts, lane selection, remaining distance, etc., structured or textual commands) through a state encoder (or text embedding layer).

[0136] In some embodiments, image information from multiple cameras can be received as input, and each image information can be preprocessed. Preprocessing may include image normalization and resolution adjustment. Furthermore, a visual encoder can be used to encode the multi-view images into a unified sequence of visual tokens, resulting in... Where n is the total number of visual tokens.

[0137] The visual encoder can be a pre-trained large visual model (such as Qwen3-VL).

[0138] In some embodiments, a preset encoder can be used to encode the vehicle's status information; wherein, the preset encoder may include a status encoder and a text encoder, that is, the status encoder can be used to encode the vehicle's operating status information to obtain a status token sequence; and the text encoder can be used to encode the vehicle's navigation command information to obtain a command token sequence.

[0139] For example, the state encoder can employ a three-layer fully connected network to uniformly map the original state dimension (assuming 32 dimensions) to a 1024-dimensional embedding space. The specific network structure can be:

[0140] Layer 1: Linear(32→128) + LayerNorm (layer normalization) + SwiGLU (activation function);

[0141] Layer 2: Linear(128→512) + LayerNorm + SwiGLU;

[0142] Layer 3: Linear(512→1024);

[0143] It should be noted that, in order to capture the temporal dependency of vehicle state information, a preset encoder can be used to encode the state information of the vehicle's historical 10 frames to obtain a state token sequence.

[0144] For example, a historical state sequence can be collected. (For example, sampling at a frequency of 10Hz to maintain the historical state of 10 frames in the past second), extracting temporal features through a one-dimensional temporal convolutional network, and finally outputting the temporally aggregated state. .

[0145] As another example, navigation instructions are given in natural language (such as "turn left at the next intersection"), so the navigation instruction information can be encoded using a VLM text encoder to obtain an instruction token sequence. ,in The instruction length is the number of tokens obtained after the navigation instruction text is segmented.

[0146] S1013, the visual token sequence, status token sequence and instruction token sequence are concatenated to obtain the target input sequence.

[0147] For example, the visual token V and the state token can be... Concatenate it with the instruction Token G to form a unified input sequence (that is, the target input sequence): ,in, For embedding dimensions. In this embodiment, vehicle image information and vehicle state information are acquired, and the image information and vehicle state information are encoded respectively to obtain a visual token sequence, a state token sequence, and an instruction token sequence. The three are then concatenated to obtain the target input sequence, which can be used as input for subsequent models. This provides complete and accurate input information for subsequent models, thereby facilitating accurate prediction and inference by the subsequent models using the input information.

[0148] Understandably, the security review module is not activated for all scenarios, but rather determines whether adversarial reasoning is required by scoring the complexity of the current scenario.

[0149] In some embodiments, the step S103 above, "using the security review module to perform adversarial reasoning on the initial driving information and visual token sequence to obtain risk warning information for the target vehicle," may include the following steps:

[0150] S1031, Based on image information and visual token sequence, determine the target complexity score of image information;

[0151] S1032, if the target complexity score is greater than the preset score, the security review module is used to perform adversarial reasoning on the initial driving information and visual token sequence to obtain the risk warning information of the target vehicle.

[0152] Here, the objective complexity score can be used to characterize the traffic complexity of the current scenario in which the vehicle is located.

[0153] In one possible embodiment, image information and visual token sequences can be jointly input into a pre-trained complexity evaluation network or visual perception scoring model, thereby directly outputting a numerical score representing the degree of image clutter, information density, or scene complexity through the model's regression or classification layers.

[0154] In another possible embodiment, a function relation for complexity scoring can be pre-constructed based on historical image information and historical visual token sequences, and the current image information and the current visual token sequence can be input into the function relation to determine the target complexity score of the image information.

[0155] In some embodiments, after determining the target complexity score, it can be determined whether the target complexity score is greater than a preset score. If the target complexity score is greater than the preset score, the initial driving information and visual token sequence can be input into the safety review module to obtain risk warning information for the target vehicle. If the target complexity score is less than or equal to the preset score, the initial driving information can be directly used as the vehicle control command.

[0156] For example, the Critic module is activated only when the objective complexity score exceeds a pre-set threshold. This module reads initial driving information and a sequence of visual tokens, specifically searches for potential risks, and generates counterfactual warnings. The output format is: "<Risk Identification><Counterfactual Scenario><Risk Assessment>". Example output: "Warning: There is a blind spot in the right lane (obscured by a parked vehicle). Counterfactual Assumption: A pedestrian may be preparing to cross the road in the blind spot. Risk Assessment: The current lane change acceleration plan will result in a collision risk under the assumed scenario; Hazard Level: High."

[0157] In this embodiment, the target complexity score of the image information can be determined based on the image information and the visual token sequence. The security review module is activated only when the target complexity score is greater than the preset score. In other words, the security review module is only needed to conduct risk assessment when the complexity of the current scenario is high. This can further improve the safety of path decision-making in complex traffic scenarios during autonomous driving. When the complexity of the current scenario is low, only the initial driving information output by the planning decision module needs to be used. This can reduce computing resources and reduce computing energy consumption while ensuring the safety of decision-making.

[0158] In some embodiments, Figure 2 A flowchart illustrating a vehicle control method provided in this application embodiment. Figure 2 ,like Figure 2 As shown, the step S1031 above, "determining the target complexity score of image information based on image information and visual token sequence," may further include the following steps:

[0159] S201, determine the visual feature entropy based on the attention weight of each visual token in the visual token sequence.

[0160] Among them, visual feature entropy is used to characterize the uniformity of the distribution of visual content in image information.

[0161] Here, the attention weight of each visual token can be calculated by inputting the sequence of visual tokens into the attention layer of the Transformer model. These weights reflect the degree of attention the model pays to different visual regions when processing the current task.

[0162] For example, the formula for calculating visual feature entropy can be expressed as:

[0163] (1)

[0164] in, Represents visual feature entropy; This represents the attention weight of the i-th visual token.

[0165] It should be noted that the entropy value of visual feature entropy is directly proportional to the degree of chaos, information density, or scene complexity of the current scene. In other words, the higher the entropy value of visual feature entropy, the higher the degree of chaos, information density, and scene complexity of the current scene.

[0166] S202, determine the occlusion rate of the image information based on the number of pixels in the occluded area and the number of pixels in the unoccluded area of ​​the image information.

[0167] Here, the number of pixels in the occluded area refers to the total number of pixels in the current image captured by the camera that are obscured by obstacles (such as vehicles, buildings, rain, snow, or dirt) and therefore cannot obtain effective environmental information; the number of pixels in the unoccluded area (also known as the number of effective field-of-view pixels) refers to the total number of pixels in the current image captured by the camera that can clearly see effective environmental information such as roads, vehicles, and pedestrians, that is, all effective field-of-view pixels in the image excluding the occluded area; the occlusion rate is the ratio of the number of pixels in the occluded area to the number of pixels in the unoccluded area in the image captured by the camera, used to quantify the severity of the current visual perception obstruction.

[0168] It is understandable that the pixels in the occluded area can be obtained by performing semantic segmentation, depth estimation, or background modeling on multiple image information (such as identifying the foreground obstacle mask through a pre-trained segmentation model);

[0169] For example, the formula for calculating the occlusion rate of image information can be expressed as:

[0170] (2)

[0171] in, This is expressed as the number of pixels in the occluded area; This represents the number of pixels in the unobstructed area. This is expressed as the occlusion rate of image information.

[0172] S203. Based on the weights of each object in the image information, the speeds of each dynamic object in the image information relative to the target vehicle, and the spatial orientation weights of each object, determine the dynamic complexity score of the image information.

[0173] In some embodiments, the dynamic complexity score can be determined based on the total number of dynamic objects in the image information, the weight of each object in the image information, the speed of each dynamic object relative to the target vehicle, the spatial orientation weight of each object, and the distance value of the speed of each object relative to the target vehicle.

[0174] In one possible implementation, the first ratio can be determined based on the ratio between the speed of each dynamic object relative to the target vehicle and the highest speed among all dynamic objects in the current scene;

[0175] The target exponential function is determined based on the distance values ​​and distance decay constants of the speeds of each object relative to the target vehicle;

[0176] The dynamic complexity score of image information can be determined based on the sum of the products of the weights of each object in the image information, the first ratio, the spatial orientation weights of each object, and the target exponential function.

[0177] Here, the weight of each object in the image information refers to the weight of different objects in the image information. Different weights can be set according to the object category in the image information. For example, if the object category is a vehicle, the weight can be 0.5; if the object category is a pedestrian, the weight can be 0.7; if the object category is a non-motorized vehicle, the weight can be 0.6, etc.

[0178] Here, the spatial orientation weight is determined based on the object's azimuth angle in the target vehicle's coordinate system. For example, the spatial orientation weight of object j can be expressed as:

[0179] (3)

[0180] in, This represents the spatial orientation weight of object j. If object j is located in front of the target vehicle at an azimuth angle between -30° and 30°, the spatial orientation weight of object j is 1. If object j is located in front of the target vehicle at an azimuth angle between 30° and 60° or between -60° and -30°, the spatial orientation weight of object j is 0.9. If object j is located in front of the target vehicle at an azimuth angle between 60° and 120° or between -120° and -60°, the spatial orientation weight of object j is 0.7. If object j is located behind the target vehicle at an azimuth angle between 120° and 180° or between -180° and -120°, the spatial orientation weight of object j is 0.3.

[0181] For example, the formula for calculating the dynamic complexity score of image information can be expressed as:

[0182] (4)

[0183] in, The dynamic complexity score of the image information is represented by M, which represents the total number of dynamic objects in the current scene (including pedestrians, vehicles, non-motorized vehicles, animals, etc.). This represents the weight of object j; Let be the velocity of the j-th dynamic object relative to the target vehicle; Let j be the Euclidean distance from object j to the target vehicle; The distance decay constant can take values ​​in the range [0,1]; exponential function Distance-weighted weighting means that objects that are closer to the object have a higher weight.

[0184] S204 calculates the target complexity score by weighting the visual feature entropy, the occlusion rate of image information, and the dynamic complexity score.

[0185] In some embodiments, the visual feature entropy, the occlusion rate of image information, and the dynamic complexity score can be weighted according to preset weight values ​​to obtain the target complexity score.

[0186] For example, the objective complexity score can be expressed as:

[0187] (5)

[0188] in, This is represented as the objective complexity score; The weight values ​​can be adjusted.

[0189] In this embodiment, visual feature entropy, occlusion rate, and dynamic complexity score can be determined based on image information and visual token sequence. In other words, the current scene is evaluated in multiple dimensions. By fusing scene density, occlusion rate, and scene complexity, the target complexity score of the current scene can be determined more accurately. This makes the activation of the security review module more precise and can further improve the safety of path decision-making in complex traffic scenarios during autonomous driving.

[0190] In some embodiments, the step S104 above, "using the arbitration module to arbitrate the initial driving information and risk warning information to obtain control commands for the target vehicle," may include the following steps:

[0191] S1041, obtain the first semantic feature vector corresponding to the first preset layer when the planning and decision module outputs initial driving information, and obtain the second semantic feature vector corresponding to the second preset layer when the safety review module outputs risk warning information;

[0192] Here, the first preset layer can be used to indicate the last layer or any layer of the planning and decision module when generating initial driving information, and the first semantic feature vector refers to the semantic feature vector corresponding to the last token or any token when the planning and decision module generates initial driving information.

[0193] It should be noted that the first semantic feature vector can also be called the first hidden state, but this is not limited in the embodiments of this application.

[0194] Here, the second preset layer can be used to indicate the last layer or any layer when the security review module generates risk warning information, and the second semantic feature vector refers to the semantic feature vector corresponding to the last token or any token when the security review module generates risk warning information.

[0195] It should be noted that the second semantic feature vector can also be called the second hidden state, but this is not limited in the embodiments of this application.

[0196] In some embodiments, during the text generation process of the planning decision module and the security review module, the output vector of the corresponding specific token (usually the last generated token or any specified token) can be extracted from the specified network layer of each model (i.e., the first preset layer and the second preset layer, which can be the last layer or any intermediate layer). This vector is the semantic feature vector of the module at that moment, also known as the hidden state.

[0197] S1042, using the arbitration module, determine the control command for the target vehicle based on the cosine distance between the first semantic feature vector and the second semantic feature vector.

[0198] In some embodiments, the cosine distance between the first semantic feature vector and the second semantic feature vector can be calculated, and based on the range of values ​​for the cosine distance, the arbitration module can determine different arbitration strategies and generate control commands for the target vehicle based on the different arbitration strategies.

[0199] For example, the cosine distance can be expressed as:

[0200] (6)

[0201] in, Represented as the first semantic feature vector; Represented as the second semantic feature vector; Represents cosine distance; The norm (i.e., length) of the first semantic feature vector; The norm (i.e., length) of the second semantic feature vector.

[0202] In this embodiment, the first semantic feature vector corresponding to the last layer when the planning and decision module generates initial driving information and the second semantic feature vector corresponding to the last layer when the safety review module generates risk warning information can be determined. The difference between the two generated texts is determined based on the cosine distance between the first and second semantic feature vectors. Different arbitration strategies are executed according to different differences. In this way, the safety of path decision-making in complex traffic scenarios during autonomous driving can be further improved.

[0203] In some embodiments, the step S1042 above, "using the arbitration module to determine the control command of the target vehicle based on the cosine distance between the first semantic feature vector and the second semantic feature vector," may further include the following steps:

[0204] S10421, when the cosine distance is less than the first distance threshold, determine the control command for the target vehicle based on the initial driving information.

[0205] In this embodiment of the application, the cosine distance is less than the first distance threshold, which can be understood as the output text of the planning decision module and the output text of the security review module being basically the same, that is, the security review module did not find any security risks in the current traffic scenario.

[0206] In some embodiments, after determining the cosine distance, the cosine distance is compared with a first distance threshold. If the cosine distance is less than the first distance threshold, the initial driving information output by the planning and decision module can be directly used as the control command for the target vehicle. In other words, the initial driving information can be parsed to obtain the control command for the target vehicle.

[0207] For example, the first distance threshold can be set to 0.3. Of course, it can also be set according to different traffic scenarios. This application embodiment does not limit this.

[0208] S10422, when the cosine distance is less than the second distance threshold and greater than or equal to the first distance threshold, the control command of the target vehicle is determined according to the fusion network.

[0209] It should be noted that the second distance threshold is greater than the first distance threshold.

[0210] In this embodiment of the application, the cosine distance is less than the second distance threshold and greater than or equal to the first distance threshold. This can be understood as the fact that although there are certain discrepancies and differences between the output text of the planning decision module and the output text of the security review module, they can be adjusted. In other words, it is necessary to take into account both output texts.

[0211] In some embodiments, after determining the cosine distance, the cosine distance can be compared with a first distance threshold and a second distance threshold. If it is determined that the cosine distance is less than the second distance threshold and greater than or equal to the first distance threshold, the initial driving information output by the planning decision module and the risk warning information output by the safety review module can be input into the pre-trained fusion network for fusion reasoning and judgment, and finally the control command of the target vehicle can be obtained.

[0212] For example, the second distance threshold can be 0.8. Of course, it can also be set according to different traffic scenarios or customized by the user. This application embodiment does not limit this.

[0213] For example, when the cosine distance is less than 0.8 and greater than or equal to 0.3, the arbitration module can execute a dialectical fusion mechanism (i.e., a fusion network) to generate a synthesized final reasoning and action token. The output format is: "<Synthetic Reasoning><Final Action>". Output example: "Considering traffic efficiency and potential risks: Maintain the current lane, moderately reduce speed to 40km / h, and maintain continuous monitoring of the right blind spot. If a pedestrian appears in the blind spot, immediately perform emergency braking. Final action: [Maintain lane, reduce speed, prepare to brake]".

[0214] Understandably, the fusion network could be a lightweight Transformer Decoder layer specifically responsible for the fusion process of the initial driving information output by the planning and decision-making module and the risk warning information output by the safety review module.

[0215] The fusion network is trained based on a two-stage training strategy.

[0216] Next, the training process of the fusion network will be described in detail in the embodiments of this application.

[0217] In some embodiments, the two-stage training strategy may include a first stage, which utilizes an adversarial triplet comprising an aggressive strategy, a conservative strategy, and an expert-balanced strategy for supervised fusion pre-training, and a second stage, which introduces regret value constraints for reinforcement learning fine-tuning based on the parallel execution results of normal and adversarial environments.

[0218] In the first stage, adversarial examples are constructed and pre-trained. Adversarial triples are constructed from the training dataset for each scenario. Construct training samples ,in "Aggressive strategy" (generated by Proposer); "Conservative strategy" (generated by Critic); The "balancing strategy" (ground truth) is labeled by experts. Then, supervised fusion learning is performed, using cross-entropy loss to train the fusion network, learning how to... and generate .

[0219] Furthermore, the loss function can be defined as:

[0220] (7)

[0221] in, Let i represent the scenario of the i-th sample; Let i represent the aggressive strategy for the i-th sample; This represents the conservative strategy for the i-th sample; This represents the parameters of the fusion network; understandably, this loss function allows the fusion model to directly learn "when to listen to the Proposer and when to listen to the Critic".

[0222] Building upon the first stage, reinforcement learning can be used for fine-tuning to further optimize the fusion strategy, ensuring the fusion model remains robust even in adversarial environments. For each training scenario... Generate two parallel training environments: normal environment and Adversarial environments (introducing random perturbations, such as sudden braking of the car in front or sudden appearance of a pedestrian). Then, an aggressive policy is implemented in each of these two environments. Conservative strategy and integration strategies Each strategy earns a reward in each environment:

[0223] (8)

[0224] in: As a collision penalty, when a collision occurs It is 1 if it is true, otherwise it is 0. ,in, Indicates the amount of displacement change; Indicates the change over time. It is used to measure average driving speed; the higher the value, the higher the efficiency.

[0225] To avoid the fusion strategy falling into extremes of being overly aggressive or overly conservative, this application defines a regret value to penalize these two extreme cases. Under normal conditions, the fusion strategy should not be significantly worse than the aggressive strategy to ensure efficiency; furthermore, under adversarial conditions, the fusion strategy should not be significantly worse than the conservative strategy to ensure security. This application defines the regret value of the fusion strategy as follows:

[0226] (9)

[0227] Therefore, the overall optimization objective of the fusion strategy is:

[0228] (10)

[0229] in, This represents the regret penalty coefficient. Finally, this application can use the GRPO algorithm to train and update the fusion network, obtaining the trained fusion network.

[0230] S10423, when the cosine distance is greater than or equal to the second distance threshold, determine the control command for the target vehicle based on the risk warning information.

[0231] In this embodiment of the application, the cosine distance being greater than or equal to the second distance threshold can be understood as a serious discrepancy between the output text of the planning decision module and the output text of the security review module, that is, the security review module has found a security risk in the current traffic scenario.

[0232] In some embodiments, after determining the cosine distance, the cosine distance is compared with the second distance threshold. If the cosine distance is greater than or equal to the second distance threshold, the risk warning information output by the safety review module can be used as the control command for the target vehicle. In other words, the risk warning information can be parsed to obtain the control command for the target vehicle.

[0233] In this embodiment, when the cosine distance is less than the first distance threshold, it indicates that the safety review module has not found any safety risks in the current traffic scenario, and the control command can be directly determined based on the initial driving information, thus ensuring efficient response in normal scenarios. When the cosine distance is less than the second distance threshold but greater than or equal to the first distance threshold, the fusion network optimized by the two-stage training strategy is activated to dynamically balance aggressive and conservative strategies, generating driving decisions that are closer to expert levels. When the cosine distance is greater than or equal to the second distance threshold, risk warning information is prioritized to ensure that safety is the primary goal in extreme or high-risk scenarios. This not only improves the adaptability of the autonomous driving system to complex scenarios but also effectively balances driving efficiency and safety.

[0234] In some embodiments, Figure 3 A flowchart illustrating a vehicle control method provided in this application embodiment. Figure 3 ,like Figure 3 As shown, the step S10422 above, "determining the control command of the target vehicle according to the fusion network," may further include the following steps:

[0235] S301, determine the first token sequence output by the planning decision module, the second token sequence output by the security review module, and the special separator token.

[0236] Understandably, the planning and decision-making module can generate a series of token sequences representing aggressive driving behaviors. These tokens are arranged in chronological order, and each token encodes specific driving action information, such as aggressive strategies aimed at improving driving efficiency, like rapid acceleration, quick lane changes, and overtaking. For example, the first token sequence can be represented as... .

[0237] Understandably, the safety review module can generate a series of token sequences representing conservative driving behavior based on safety constraints and risk assessment mechanisms. These tokens are arranged in chronological order, and each token encodes a safety-priority driving action, such as slowing down, maintaining a safe distance, or avoiding obstacles—conservative strategies. For example, the second token sequence can be represented as follows: .

[0238] The special delimiter token is used to identify the boundary between different types of sequences; that is, the special delimiter token can be used as the boundary between the first token sequence and the second token sequence. For example, the special delimiter token is a predefined marker used in the model to clearly distinguish token sequences from different sources. When the first token sequence output by the Proposer module and the second token sequence output by the Critic module are concatenated and input into the fusion network, the special delimiter token is inserted between the two sequences, acting as a boundary marker. This helps the model clearly identify the end position of the aggressive policy sequence and the start position of the conservative policy sequence, thereby more accurately understanding the contextual relationship between the two policies and making effective fusion decisions.

[0239] S302, perform a first processing on the target input sequence, the first token sequence, the second token sequence, and the special delimiter token to obtain the second target input sequence.

[0240] The first process includes at least splicing and position encoding.

[0241] In some embodiments, the target input sequence, the first token sequence, the second token sequence, and the special delimiter token are concatenated. That is, the special delimiter token can be inserted between the target input sequence, the first token sequence, and the second token sequence to obtain the concatenated input sequence. For example, the concatenated input sequence can be represented as:

[0242] (11)

[0243] in, Represented as the target input sequence; Represented as a special separator token

[0244] Furthermore, the concatenated input sequence can be positionally encoded to obtain the second target input sequence.

[0245] S303, using a fusion network, performs fusion prediction processing on the second target input sequence to obtain the control command for the target vehicle.

[0246] In some embodiments, the second target input sequence can be input into a pre-trained fusion network for fusion prediction processing, thereby obtaining control commands for the target vehicle.

[0247] In this embodiment, by determining the first token sequence of the planning decision module and the second token sequence of the security review module, and introducing a special separator token to clearly identify the boundary between the two types of sequences, information confusion is effectively avoided. Then, through splicing and position encoding, a second target input sequence containing sequence order and position information is constructed, enabling the fusion network to fully perceive the temporal structure and contextual relationship of different strategies. Finally, the fusion network optimized through two-stage training is used for deep fusion prediction to generate accurate vehicle control commands. This not only enhances the model's semantic understanding of complex driving scenarios but also ensures the organic integration of aggressive and conservative strategies at the decision-making level, thereby improving the decision-making quality of the autonomous driving system in various scenarios.

[0248] In some embodiments, the fusion network includes a dynamic weight prediction module, an adversarial awareness attention module, and an inference generation module; the step S303 above, "using the fusion network to perform fusion prediction processing on the second target input sequence to obtain the control command of the target vehicle," may include the following steps:

[0249] S3031, using the dynamic weight prediction module, perform weight prediction processing on the first token sequence and the second token sequence to determine the first weight coefficient corresponding to the planning decision module and the second weight coefficient corresponding to the security review module.

[0250] Here, the dynamic weight prediction module can automatically predict the fusion weights based on the current scene characteristics, the initial driving information output by the planning and decision-making module, and the degree of discrepancy between the risk warning information output by the safety review module. For example, the dynamic weight prediction network can be a two-layer fully connected network.

[0251] Understandably, the first weighting factor can reflect the importance of the initial driving information output by the planning and decision-making module; the second weighting factor can reflect the importance of the risk warning information output by the safety review module.

[0252] In one possible implementation, the first token sequence and the second token sequence can be directly input into a pre-trained dynamic weight prediction module, thereby outputting the first weight coefficient corresponding to the planning decision module and the second weight coefficient corresponding to the security review module.

[0253] In another possible implementation, features can be extracted from the first token sequence and the second token sequence to obtain a first feature vector and a second feature vector. The first feature vector and the second feature vector are then input into a pre-trained dynamic weight prediction module to output the first weight coefficient and the second weight coefficient.

[0254] S3032, using the adversarial awareness attention module, adjusts the attention weights of the attention bias matrix, query matrix, key matrix and value matrix based on the first weight coefficient and the second weight coefficient to obtain the fused feature vector;

[0255] Understandably, in order to enable the fusion network to dynamically adjust its attention to both positive and negative opinions based on weights, this application's embodiments introduce a learnable attention bias matrix on top of standard self-attention. .

[0256] In some embodiments, adjusting the attention weights of the attention bias matrix, query matrix, key matrix, and value matrix based on a first weight coefficient and a second weight coefficient may include:

[0257] A learnable attention bias matrix is ​​established. Further, the corresponding elements in the attention bias matrix are adjusted according to the first and second weight coefficients, and the element values ​​corresponding to the Proposer token sequence region or the Critic token sequence region in the bias matrix are increased in a targeted manner. The bias matrix is ​​superimposed on the scaled dot product result of the query matrix and the key matrix, and after softmax normalization, it is multiplied with the value matrix V to obtain the fused feature vector.

[0258] For example, the fused feature vector can be represented as:

[0259] (12)

[0260] Where B is the attention bias matrix; Q is the query matrix; K is the key matrix; and V is the value matrix; This is the fused feature vector.

[0261] It should be noted that when the weight predictor determines that more attention should be paid to the proposer's opinion (i.e., the first weight coefficient is larger), the bias matrix causes the query token to be biased towards the proposer's opinion when calculating attention. The score of the region increases; conversely, when the second weighting coefficient is large, the score of the region decreases. Increased attention to the region.

[0262] S3033, using the inference generation module, performs inference processing on the second target input sequence and the fused feature vector to obtain the action token sequence.

[0263] Here, the action token sequence refers to a series of discrete tokens ultimately output by the inference generation module, used to directly control the target vehicle. These tokens are arranged in time step order, and each token corresponds to a specific driving action command (such as steering, acceleration, braking, etc.), which can be directly converted into executable control commands for the vehicle after decoding. In some embodiments, after determining the second target input sequence and the fused feature vector, the second target input sequence and the fused feature vector can be input into the inference generation module, which uses an autoregressive approach to generate the action token sequence and the comprehensive inference text.

[0264] Among them, the comprehensive reasoning text refers to the reasoning logic that integrates key information from the planning and decision-making module and the security review module, and presents a balanced approach.

[0265] In some embodiments, the fused feature vector and the concatenated input sequence are combined. The data is input as a condition into the inference generation module; subsequently, an autoregressive generation method is used, where at each time step, the model is based on the already generated historical token sequence and... Utilizing learnable parameters The reasoning generation module predicts the token for the current step. The conditional probability distribution P is obtained; the joint probability of the entire output sequence is obtained by multiplying the conditional probabilities at each time step. Finally, when the generation process reaches the preset length T, the model outputs a combined inference text and action token sequence.

[0266] For example, the calculation formula for inference processing can be expressed as:

[0267] (13)

[0268] in, To integrate the learnable parameters of the network, This represents all tokens generated before time t; This represents a comprehensive reasoning text; This represents a sequence of action tokens.

[0269] S3034 decodes the action token sequence based on the action decoder to obtain the control command for the target vehicle.

[0270] In some embodiments, after determining the action token sequence, the action token sequence can be decoded based on the action decoder to obtain the control commands for the target vehicle.

[0271] For example, the motion decoder module can decode the motion token sequence output by the arbitration module into specific vehicle control commands. The motion decoder module can employ a multilayer perceptron network to map the 1024-dimensional motion token sequence into continuous control quantities (i.e., control commands), which may include, but are not limited to, target speed, target acceleration, steering wheel angle, etc.

[0272] In this embodiment, the dynamic weight prediction module adaptively generates weight coefficients for aggressive and conservative strategies based on the real-time scenario; the adversarial perception attention module dynamically adjusts the bias matrix based on these weights, guiding the model to focus on more important policy regions, thereby achieving deep fusion at the feature level; the inference generation module adopts an autoregressive approach to generate coherent inference text and action token sequences based on the fused features; finally, the action decoder converts the token sequences into executable control commands, thereby improving the decision-making quality, safety, and reliability of the autonomous driving system in complex scenarios.

[0273] In some embodiments, the step S3031 above, "using the dynamic weight prediction module to perform weight prediction processing on the first token sequence and the second token sequence to determine the first weight coefficient corresponding to the planning decision module and the second weight coefficient corresponding to the security review module," may further include the following steps:

[0274] S30311, determine the first feature vector corresponding to the last token in the first token sequence, and the second feature vector corresponding to the last token in the second token sequence.

[0275] In some embodiments, the hidden state (i.e., the first feature vector) of the last token sequence can be extracted from the first token sequence; simultaneously, the hidden state (i.e., the second feature vector) of the last token sequence can be extracted from the second token sequence; wherein, the first feature vector can be represented as... The second eigenvector can be represented as: .

[0276] S30312, average pooling is performed on the hidden states corresponding to all tokens in the second token sequence to obtain the risk feature vector.

[0277] In some embodiments, the hidden state (i.e., the third feature vector corresponding to each token) can be determined from the second token sequence, and all hidden states (third feature vectors) can be averaged to obtain the risk feature vector.

[0278] For example, a risk feature vector can be represented as:

[0279] (14)

[0280] in, Represented as a risk feature vector; It is represented as the hidden state (third feature vector) corresponding to the i-th token.

[0281] S30313, concatenate the first feature vector, the second feature vector, and the risk feature vector to obtain the target vector;

[0282] S30314, The target vector is input into the dynamic weight prediction module for weight prediction processing to obtain the first weight coefficient and the second weight coefficient.

[0283] In some embodiments, after determining the first feature vector, the second feature vector, and the risk feature vector, the three feature vectors can be concatenated to obtain the target vector; the target vector can be directly input into the dynamic weight prediction module composed of two fully connected layers to obtain the first weight coefficient and the second weight coefficient.

[0284] For example, the formulas for calculating the first weighting coefficient and the second weighting coefficient can be expressed as follows:

[0285] (15)

[0286] in, This is represented as the first weighting coefficient; This is represented as the second weighting coefficient.

[0287] In this embodiment, the feature vector of the last token in the first token sequence is extracted as the global representation of the aggressive strategy. At the same time, the feature vector of the last token in the second token sequence is combined with the risk feature vector obtained by average pooling to comprehensively capture the semantic information of the conservative strategy. Then, the target vector constructed by concatenating the three is input into the dynamic weight prediction module to generate the first weight coefficient and the second weight coefficient that match the current scenario. This allows the model to dynamically adjust the contribution ratio of the aggressive and conservative strategies according to the real-time environment. In low-risk scenarios, the aggressive strategy is given higher weight to pursue efficiency, while in high-risk scenarios, the conservative strategy is strengthened to ensure safety. This achieves the optimal balance between efficiency and safety, thereby enhancing the intelligence and robustness of the autonomous driving decision-making system.

[0288] The following describes the application of the vehicle control method provided in the embodiments of this application in a real-world scenario.

[0289] This application belongs to the interdisciplinary technology field of intelligent driving, artificial intelligence, and multimodal large-scale models. Addressing the three problems of existing models mentioned above—unidirectional reasoning, insufficient ability to handle long-tail scenarios, and a lack of "self-dialectical thinking" in model decision-making—this application proposes a dialectical adversarial reasoning decision-making method for VLA models in the field of autonomous driving. Specifically, it involves a Vision-Language-Action (VLA) planning and decision-making method for end-to-end autonomous driving systems. This method introduces proactive risk exploration and adversarial thinking mechanisms into the model, enabling it to "self-question" and perform counterfactual reasoning like a human driver. For example, in uncertain scenarios such as visual occlusion and complex traffic, the model can identify low-probability, high-risk events in advance; in long-tail scenarios such as "ghosting" or sudden braking, it outputs defensive decisions rather than "average" decisions based on data distribution, significantly improving decision safety. Simultaneously, the dialectical reasoning process enhances the interpretability of the decision, enabling the model not only to explain the chosen action but also why it rejects other potential actions, forming a complete decision-making argument chain.

[0290] To implement this method, this application constructs two opposing "thinking roles" within the VLA model: a radical "proposer" and a defensive "critic." The final driving decision is generated through dialectical reasoning between the two. Specifically, the proposer is responsible for generating an initial driving intention based on the current scenario, while the criterion specifically identifies potential risks and raises counterfactual challenges. Finally, a comprehensive arbitration module integrates the viewpoints of both parties, outputting a driving action that is both efficient and safe. This mechanism simulates the defensive driving mindset of human drivers, improving the system's safety in long-tail scenarios.

[0291] Next, the embodiments of this application introduce the overall system architecture. The specific implementation of this application is based on the current mainstream Transformer architecture. The model mainly includes the following core modules: (1) visual encoder module; (2) vehicle state encoder module; (3) scheme proposal module (that is, the planning decision module in the above embodiments); (4) critic module (that is, the security review module in the above embodiments); (5) comprehensive arbitration module; (6) action decoder module.

[0292] The specific implementation method has been explained in detail in the above embodiments, and will not be repeated here for the sake of brevity.

[0293] The present application will be further described in detail below with reference to specific embodiments.

[0294] (1) Required hardware and software environment:

[0295] Training server: 8×NVIDIA A100 80GB GPUs, AMD EPYC 7742 64-core CPUs, 1TB of memory;

[0296] Inference platform: NVIDIA Orin or higher computing power automotive computing platform;

[0297] Sensor configuration: 6 surround-view cameras (1920×1080 resolution, 30fps).

[0298] Deep learning frameworks: PyTorch 2.0+, Transformers 4.30+;

[0299] Basic model: Qwen3-VL-30B (multimodal language model);

[0300] Simulation platform: nuPlan simulator.

[0301] (2) Data preparation: Training data was extracted from the nuPlan and Waymo datasets, and high-risk scenarios were selected. The Qwen3-VL-235B model was used to perform adversarial annotation on the high-risk scenarios, generating training data in the format of "<Scene>→<Initial Plan>→<Risk Criticism>→<Revised Plan>→<Action Token>". A total of about 500,000 adversarial inference data were generated.

[0302] (3) Pre-training of the base model: Supervised fine-tuning (SFT) was performed using Qwen3-VL-7B as the base model.

[0303] Training epochs: 20;

[0304] Learning rate: 2e-5, using cosine annealing scheduling;

[0305] Batch size: 32;

[0306] Optimizer: AdamW, weight decay 0.01;

[0307] Mixed precision training: Enable FP16 / BF16 mixed precision;

[0308] (4) Enhanced fine-tuning: The GRPO algorithm is used for enhanced fine-tuning.

[0309] Training epochs: 10;

[0310] Learning rate: 1e-5;

[0311] Batch size: 16;

[0312] (5) Scenario example: Unprotected right turn encountering obstruction

[0313] The target vehicle is preparing to turn right on a city road. A parked bus in front of it is obstructing its view, and a pedestrian may be crossing in front of the bus. The system processing procedure is as follows:

[0314] The Proposer module outputs: "Current green light, right turn lane is clear and there are no obvious obstacles. Intent: Complete the right turn at normal speed. Trajectory planning: [t=0.5s: Enter the right turn lane, v=20km / h; t=1.0s: Complete the turn, v=20km / h; t=2.0s: Enter the target lane, v=25km / h]".

[0315] Complexity assessment: The overall complexity score c=0.72>τ(0.6), thus activating the adversarial criticism module.

[0316] The Critic module outputs: "Risk Identification: A bus parked to the right front creates a significant blind spot, obscuring approximately 25% of the visual area. It is impossible to confirm whether there are pedestrians or non-motorized vehicles in the zebra crossing area. Counterfactual Scenario: There may be a pedestrian preparing to cross the zebra crossing in the blind spot, or an electric bike / bicycle may emerge from in front of the bus. If turning right at the current speed (20km / h), the reaction distance upon detecting a sudden target is less than 5 meters, and the braking time is insufficient. Risk Assessment: The current right-turn trajectory has a high collision risk under the hypothetical scenario. Hazard Level: High."

[0317] Comprehensive Arbitration Output: "Comprehensive Reasoning: Before blind spots are eliminated, traffic efficiency should give way to safety. Adopt a defensive driving strategy: first, decelerate to a creeping state (5km / h), while simultaneously shifting slightly to the left (approximately 0.3 meters) to expand the field of vision and continuously monitor the blind spot. After confirming that there are no pedestrians / non-motorized vehicles on the zebra crossing and in the blind spot, smoothly complete the right turn. Final actions: [t=0.5s: decelerate to 5km / h, shift 0.3m to the left; t=1.0s: continue creeping and observing; t=2.0s: accelerate after confirming safety; t=3.0s: complete the right turn, v=15km / h]."

[0318] In summary, compared to the "direct right turn" decision of the traditional end-to-end model, the defensive decision of this application can avoid collisions when there are indeed pedestrians in the blind spot, while only adding about 2 seconds of passage delay when the blind spot is safe.

[0319] Based on the above embodiments, this application also provides a vehicle control device. Figure 4 This is a schematic diagram of the composition structure of a vehicle control device provided in an embodiment of this application, as shown below. Figure 4 As shown, the vehicle control device 400 includes a determination unit 401, a planning and prediction unit 402, an adversarial reasoning unit 403, and an arbitration unit 404, wherein:

[0320] The determining unit 401 is used to determine the target input sequence based on the image information and vehicle state information of the target vehicle during its driving process; the target input sequence includes a visual token sequence.

[0321] The planning and prediction unit 402 is used to perform planning and prediction processing on the target input sequence using the planning and decision module to obtain the initial driving information of the target vehicle. The initial driving information includes the scene information of the target vehicle, the behavioral intention of the target vehicle, and the trajectory planning information of the target vehicle. The planning and decision module is a model trained based on historical image information and historical vehicle state information.

[0322] The adversarial reasoning unit 403 is used to perform adversarial reasoning on the initial driving information and visual token sequence using the safety review module to obtain risk warning information of the target vehicle; wherein, the risk warning information includes risk identification information and risk assessment information of the target vehicle, and the safety review module is a model trained based on historical driving information and historical visual token sequence.

[0323] Arbitration unit 404 is used to arbitrate the initial driving information and risk warning information using the arbitration module to obtain the control command of the target vehicle so that the target vehicle can drive according to the control command. The arbitration module is a model trained based on historical driving information and historical risk warning information.

[0324] In some embodiments of this application, the determining unit 401 is further configured to acquire image information and vehicle status information of the target vehicle during its driving process; wherein, the vehicle status information includes vehicle operating status and navigation instruction information; the image information is image encoded using a visual encoder to obtain a visual token sequence; the vehicle operating status and navigation instruction information are state encoded using a state encoder to obtain a state token sequence corresponding to the vehicle operating status and an instruction token sequence corresponding to the navigation instruction information; the visual token sequence, state token sequence and instruction token sequence are concatenated to obtain the target input sequence.

[0325] In some embodiments of this application, the adversarial reasoning unit 403 is further configured to determine the target complexity score of the image information based on the image information and the visual token sequence; if the target complexity score is greater than the preset score, the security review module is used to perform adversarial reasoning on the initial driving information and the visual token sequence to obtain the risk warning information of the target vehicle.

[0326] In some embodiments of this application, the adversarial inference unit 403 is further configured to determine the visual feature entropy based on the attention weight of each visual token in the visual token sequence; the visual feature entropy is used to characterize the uniformity of the distribution of visual content in the image information; the occlusion rate of the image information is determined based on the number of pixels in the occluded area and the number of pixels in the unoccluded area of ​​the image information; the dynamic complexity score of the image information is determined based on the weight of each object in the image information, the speed of each dynamic object in the image information relative to the target vehicle, and the spatial orientation weight of each object; wherein, the spatial orientation weight is determined based on the azimuth angle of the object in the target vehicle coordinate system; and the target complexity score is obtained by weighting the visual feature entropy, the occlusion rate of the image information, and the dynamic complexity score.

[0327] In some embodiments of this application, the arbitration unit 404 is further configured to obtain the first semantic feature vector corresponding to the first preset layer when the planning decision module outputs initial driving information, and the second semantic feature vector corresponding to the second preset layer when the safety review module outputs risk warning information; and to use the arbitration module to determine the control command of the target vehicle based on the cosine distance between the first semantic feature vector and the second semantic feature vector.

[0328] In some embodiments of this application, the arbitration unit 404 is further configured to determine the control command of the target vehicle based on initial driving information when the cosine distance is less than a first distance threshold; and to determine the control command of the target vehicle based on a fusion network when the cosine distance is less than a second distance threshold but greater than or equal to the first distance threshold; wherein the fusion network is trained based on a two-stage training strategy; and to determine the control command of the target vehicle based on risk warning information when the cosine distance is greater than or equal to the second distance threshold.

[0329] In some embodiments of this application, the arbitration unit 404 is further configured to determine the first token sequence output by the planning decision module, the second token sequence output by the security review module, and a special separator token; wherein the special separator token is used to identify the boundary between different types of sequences; the target input sequence, the first token sequence, the second token sequence, and the special separator token are subjected to a first processing to obtain a second target input sequence; wherein the first processing includes at least splicing processing and position encoding processing; and the second target input sequence is subjected to fusion prediction processing using a fusion network to obtain the control command of the target vehicle.

[0330] In some embodiments of this application, the fusion network includes a dynamic weight prediction module, an adversarial awareness attention module, and an inference generation module; the arbitration unit 404 is further configured to use the dynamic weight prediction module to perform weight prediction processing on the first token sequence and the second token sequence to determine the first weight coefficient corresponding to the planning decision module and the second weight coefficient corresponding to the security review module; use the adversarial awareness attention module to adjust the attention weights of the attention bias matrix, query matrix, key matrix, and value matrix based on the first and second weight coefficients to obtain the fused feature vector; use the inference generation module to perform inference processing on the second target input sequence and the fused feature vector to obtain the action token sequence; and decode the action token sequence based on the action decoder to obtain the control command of the target vehicle.

[0331] In some embodiments of this application, the arbitration unit 404 is further configured to determine the first feature vector corresponding to the last token in the first token sequence and the second feature vector corresponding to the last token in the second token sequence; perform average pooling on the hidden states corresponding to all tokens in the second token sequence to obtain a risk feature vector; concatenate the first feature vector, the second feature vector and the risk feature vector to obtain a target vector; and input the target vector into the dynamic weight prediction module for weight prediction processing to obtain a first weight coefficient and a second weight coefficient.

[0332] It should be noted that, in the embodiments of this application, if the above methods are implemented as software functional modules and sold or used as independent products, they can also be stored in a computer-readable storage medium. Based on this understanding, the technical solutions of the embodiments of this application, or the parts that contribute to related technologies, can be embodied in the form of software products. These software products are stored in a storage medium and include several instructions to cause an electronic device (which may be a personal computer, server, or network device, etc.) to execute all or part of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory, magnetic disks, or optical disks. Thus, the embodiments of this application are not limited to any specific hardware and software combination.

[0333] This application also provides an electronic device, including a memory and a processor, wherein the memory stores a computer program that can run on the processor, and the processor executes the computer program to implement the above-described method.

[0334] This application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the above-described method. The computer-readable storage medium can be transient or non-transient.

[0335] This application also provides a computer program product, including a computer program or instructions, which, when executed by a processor, implement some or all of the steps in the above-described method. This computer program product can be implemented specifically through hardware, software, or a combination thereof.

[0336] In one alternative embodiment, the computer program product is specifically embodied in a computer storage medium; in another alternative embodiment, the computer program product is specifically embodied in a software product, such as a software development kit (SDK), etc.

[0337] It should be noted that, Figure 5 This is a schematic diagram of the hardware entity of a vehicle provided in an embodiment of this application, such as... Figure 5 As shown, the hardware entity of the vehicle 500 includes: a processor 501, a communication interface 502, and a memory 503, wherein:

[0338] The processor 501 typically controls the overall operation of the vehicle 500.

[0339] Communication interface 502 enables vehicle 500 to communicate with other terminals or servers via a network.

[0340] The memory 503 is configured to store instructions and applications executable by the processor 501, and can also cache data to be processed or already processed by the processor 501 and various modules in the vehicle 500 (e.g., image data, audio data, voice communication data, and video communication data), and can be implemented using flash memory or RAM. Data can be transferred between the processor 501, the communication interface 502, and the memory 503 via the bus 504.

[0341] It should be noted that the descriptions of the storage medium and device embodiments above are similar to the descriptions of the method embodiments above, and have similar beneficial effects. For technical details not disclosed in the storage medium and device embodiments of this application, please refer to the descriptions of the method embodiments of this application for understanding.

[0342] It should be understood that the phrase "one embodiment" or "an embodiment" throughout the specification means that a specific feature, structure, or characteristic related to the embodiment is included in at least one embodiment of this application. Therefore, "in one embodiment" or "in an embodiment" appearing throughout the specification does not necessarily refer to the same embodiment. Furthermore, these specific features, structures, or characteristics can be combined in any suitable manner in one or more embodiments. It should be understood that in the various embodiments of this application, the sequence numbers of the above steps / processes do not imply a sequential order of execution; the execution order of each step / process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application. The sequence numbers of the above embodiments of this application are merely descriptive and do not represent the superiority or inferiority of the embodiments.

[0343] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0344] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are merely illustrative. For example, the division of the units described above is only a logical functional division, and in actual implementation, there may be other division methods, such as: multiple units or components may be combined, or integrated into another system, or some features may be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the various components shown or discussed may be through some interfaces, and the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0345] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units. They may be located in one place or distributed across multiple network units. Some or all of the units may be selected to achieve the purpose of this embodiment according to actual needs.

[0346] In addition, each functional unit in the various embodiments of this application can be integrated into one processing unit, or each unit can be a separate unit, or two or more units can be integrated into one unit; the integrated unit can be implemented in hardware or in the form of hardware plus software functional units.

[0347] Those skilled in the art will understand that all or part of the steps of the above method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it performs the steps of the above method embodiments. The aforementioned storage medium includes various media that can store program code, such as mobile storage devices, read-only memory, magnetic disks, or optical disks.

[0348] Alternatively, if the integrated units described above are implemented as software functional modules and sold or used as independent products, they can also be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence or the part that contributes to related technologies, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as mobile storage devices, ROMs, magnetic disks, or optical disks.

[0349] The above description is merely an embodiment of this application, but the scope of protection of this application is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application.

Claims

1. A vehicle control method, characterized in that, The method includes: Based on the image information and vehicle status information of the target vehicle during its driving process, a target input sequence is determined; the target input sequence includes a visual token sequence. The planning and decision-making module is used to perform planning and prediction processing on the target input sequence to obtain the initial driving information of the target vehicle; wherein, the initial driving information includes the scene information of the target vehicle, the behavioral intention of the target vehicle, and the trajectory planning information of the target vehicle, and the planning and decision-making module is a model trained based on historical image information and historical vehicle state information; Using a security review module, adversarial reasoning is performed on the initial driving information and the visual token sequence to obtain risk warning information for the target vehicle; wherein, the risk warning information includes risk identification information and risk assessment information of the target vehicle, and the security review module is a model trained based on historical driving information and historical visual token sequences; The initial driving information and the risk warning information are arbitrated using an arbitration module to obtain control commands for the target vehicle, so that the target vehicle can drive according to the control commands. The arbitration module is a model trained based on the historical driving information and historical risk warning information. The step of using a security review module to perform adversarial reasoning on the initial driving information and the visual token sequence to obtain risk warning information for the target vehicle includes: Based on the image information and the visual token sequence, determine the target complexity score of the image information; If the target complexity score is greater than the preset score, the security review module is used to perform adversarial reasoning on the initial driving information and the visual token sequence to obtain the risk warning information of the target vehicle.

2. The method according to claim 1, characterized in that, The determination of the target input sequence based on image information and vehicle state information of the target vehicle during its driving process includes: The image information and vehicle status information of the target vehicle during its driving process are acquired; wherein, the vehicle status information includes vehicle operating status and navigation command information; The image information is image encoded using a visual encoder to obtain a visual token sequence; the vehicle operating status and navigation command information are state encoded using a state encoder to obtain a state token sequence corresponding to the vehicle operating status and a command token sequence corresponding to the navigation command information. The visual token sequence, the state token sequence, and the instruction token sequence are concatenated to obtain the target input sequence.

3. The method according to claim 1, characterized in that, The step of determining the target complexity score of the image information based on the image information and the visual token sequence includes: Based on the attention weight of each visual token in the visual token sequence, the visual feature entropy is determined; the visual feature entropy is used to characterize the uniformity of the distribution of visual content in the image information. The occlusion rate of the image information is determined based on the number of pixels in the occluded area and the number of pixels in the unoccluded area of ​​the image information. The dynamic complexity score of the image information is determined based on the weights of each object in the image information, the speeds of each dynamic object in the image information relative to the target vehicle, and the spatial orientation weights of each object; wherein, the spatial orientation weights are determined based on the azimuth angles of the objects in the target vehicle coordinate system. The target complexity score is obtained by weighting the visual feature entropy, the occlusion rate of the image information, and the dynamic complexity score.

4. The method according to any one of claims 1 to 3, characterized in that, The arbitration module is used to arbitrate the initial driving information and the risk warning information to obtain control commands for the target vehicle, including: The first semantic feature vector corresponding to the first preset layer when the planning and decision module outputs initial driving information is obtained, and the second semantic feature vector corresponding to the second preset layer when the safety review module outputs risk warning information is obtained. Using the arbitration module, the control command for the target vehicle is determined based on the cosine distance between the first semantic feature vector and the second semantic feature vector.

5. The method according to claim 4, characterized in that, The step of using the arbitration module to determine the control command for the target vehicle based on the cosine distance between the first semantic feature vector and the second semantic feature vector includes: If the cosine distance is less than a first distance threshold, the control command for the target vehicle is determined based on the initial driving information; If the cosine distance is less than a second distance threshold and greater than or equal to the first distance threshold, the control command for the target vehicle is determined according to the fusion network; wherein the fusion network is trained based on a two-stage training strategy. If the cosine distance is greater than or equal to the second distance threshold, the control command for the target vehicle is determined based on the risk warning information.

6. The method according to claim 5, characterized in that, The step of determining the control command for the target vehicle based on the fusion network includes: The first token sequence output by the planning decision module, the second token sequence output by the security review module, and a special separator token are determined; wherein, the special separator token is used to identify the boundary between different types of sequences; The target input sequence, the first token sequence, the second token sequence, and the special delimiter token are subjected to a first process to obtain a second target input sequence; wherein, the first process includes at least concatenation processing and position encoding processing; The second target input sequence is fused and predicted using the fusion network to obtain the control command for the target vehicle.

7. The method according to claim 6, characterized in that, The fusion network includes a dynamic weight prediction module, an adversarial awareness attention module, and an inference generation module; the step of using the fusion network to perform fusion prediction processing on the second target input sequence to obtain the control command for the target vehicle includes: Using the dynamic weight prediction module, the first token sequence and the second token sequence are processed for weight prediction to determine the first weight coefficient corresponding to the planning decision module and the second weight coefficient corresponding to the security review module. Using the adversarial awareness attention module, attention weights are adjusted on the attention bias matrix, query matrix, key matrix and value matrix based on the first weight coefficient and the second weight coefficient to obtain the fused feature vector; Using the inference generation module, the second target input sequence and the fused feature vector are subjected to inference processing to obtain an action token sequence; The action token sequence is decoded using an action decoder to obtain the control commands for the target vehicle.

8. The method according to claim 7, characterized in that, The step of using the dynamic weight prediction module to perform weight prediction processing on the first token sequence and the second token sequence to determine the first weight coefficient corresponding to the planning decision module and the second weight coefficient corresponding to the security review module includes: Determine the first feature vector corresponding to the last token in the first token sequence, and the second feature vector corresponding to the last token in the second token sequence; The hidden states corresponding to all tokens in the second token sequence are averaged and pooled to obtain the risk feature vector. The first feature vector, the second feature vector, and the risk feature vector are concatenated to obtain the target vector; The target vector is input into the dynamic weight prediction module for weight prediction processing to obtain the first weight coefficient and the second weight coefficient.

9. A vehicle control device, characterized in that, The device includes: The determining unit is used to determine a target input sequence based on image information and vehicle status information of the target vehicle during its driving process; the target input sequence includes a visual token sequence. The planning and prediction unit is used to perform planning and prediction processing on the target input sequence using the planning and decision module to obtain the initial driving information of the target vehicle; wherein, the initial driving information includes the scene information where the target vehicle is located, the behavioral intention of the target vehicle, and the trajectory planning information of the target vehicle, and the planning and decision module is a model trained based on historical image information and historical vehicle state information; The adversarial reasoning unit is used to perform adversarial reasoning on the initial driving information and the visual token sequence using the security review module to obtain risk warning information of the target vehicle; wherein, the risk warning information includes risk identification information and risk assessment information of the target vehicle, and the security review module is a model trained based on historical driving information and historical visual token sequences; An arbitration unit is used to arbitrate the initial driving information and the risk warning information using an arbitration module to obtain control commands for the target vehicle, so that the target vehicle can drive according to the control commands. The arbitration module is a model trained based on the historical driving information and historical risk warning information. The adversarial reasoning unit is specifically used to determine the target complexity score of the image information based on the image information and the visual token sequence; if the target complexity score is greater than a preset score, the security review module is used to perform adversarial reasoning on the initial driving information and the visual token sequence to obtain the risk warning information of the target vehicle.

10. A vehicle, characterized in that, It includes a processor and a memory, the memory storing a computer program that can run on the processor, and the processor executing the computer program to implement the steps of the method described in any one of claims 1 to 8.

11. A computer-readable storage medium, characterized in that, It stores a computer program that, when executed by a processor, implements the steps of the method described in any one of claims 1 to 8.

Images