Decision-making methods, devices, equipment, storage media and products for vehicle assisted driving

Abstract

The present application provides a decision-making method, device, equipment, storage medium and product for vehicle assisted driving, which relates to the field of vehicle technology. The method includes: obtaining feedback data and first environmental perception data of the vehicle executing a first assisted driving decision; performing abnormality identification on the first assisted driving decision based on the feedback data and the first environmental perception data to obtain an abnormality identification result; when the abnormality identification result indicates that the first assisted driving decision has an abnormality, triggering the decision-making big model to execute the decision tracing task to generate a model update guidance signal, and updating and optimizing the environmental perception big model based on the model update guidance signal to obtain an updated environmental perception big model; outputting second environmental perception data to the decision-making big model based on the updated environmental perception big model, and the decision-making big model outputs a second assisted driving decision of the vehicle in combination with the second environmental perception data. The use of this application can improve the accuracy of assisted driving decisions in complex scenarios.

Description

Technical Field

[0001] The present application relates to the field of vehicle technology, and in particular to a decision-making method, device, equipment, storage medium and product for vehicle assisted driving. Background Art

[0002] Vehicle-assisted driving decisions need to be made quickly in complex and changing traffic environments (such as multi-objective games, mixed traffic of people and vehicles, emergencies, etc.). This places higher demands on the vehicle-assisted driving system for environmental perception, real-time reasoning, and overall robustness.

[0003] Among related technologies, Large Language Models (LLMs) and Visual-Language Models (VLMs) have been gradually introduced into the field of assisted driving due to their excellent generalization and reasoning capabilities. Leveraging these models, assisted driving systems can effectively improve perception interpretability and decision-making flexibility. However, assisted driving systems still rely primarily on static perception and decision-making based on these large models. This makes the system susceptible to decision-making errors in dynamic and complex scenarios due to biased perception information. Summary of the Invention

[0004] The main purpose of the embodiments of this application is to propose a decision-making method, device, equipment, storage medium and product for vehicle assisted driving, aiming to compensate for the perception information deviation of the vehicle assisted driving system in dynamic and complex scenes, thereby improving the accuracy of the system's decision-making in complex scenes.

[0005] To achieve the above objectives, a first aspect of an embodiment of the present application provides a decision-making method for vehicle assisted driving, which is applied to a vehicle and includes:

[0006] Obtaining feedback data and first environmental perception data from the vehicle executing a first assisted driving decision; the first assisted driving decision is output by a decision-making model of a cloud device, and the first environmental perception data is output by the vehicle's environmental perception model, and the cloud device communicates with the vehicle;

[0007] performing abnormality identification on the first assisted driving decision based on the feedback data and the first environmental perception data to obtain an abnormality identification result;

[0008] When the abnormality identification result indicates that the first assisted driving decision has an abnormality, a decision tracing instruction is sent to the cloud device to trigger the decision large model to perform a decision tracing task, and the environment perception large model is updated and optimized based on the model update guidance signal fed back by the cloud device to obtain an updated environment perception large model; the model update guidance signal is generated by the decision large model performing the decision tracing task and output to the environment perception large model;

[0009] Based on the updated environmental perception large model, the environmental perception task is executed to output second environmental perception data to the decision large model, and the second assisted driving decision of the vehicle output by the decision large model in combination with the second environmental perception data is received.

[0010] In some embodiments, the method further comprises:

[0011] When the abnormality recognition result indicates that the first assisted driving decision is abnormal, constructing an adversarial simulation scenario based on the environmental perception large model;

[0012] Based on the adversarial simulation scenario, a stress testing instruction is sent to the cloud device to trigger the decision-making model to execute a decision stress testing task; the decision-making model is updated and optimized based on the execution of the decision stress testing task to obtain an updated decision-making model;

[0013] Based on the environmental perception big model, the third environmental perception data is output to the updated decision big model, and the third assisted driving decision of the vehicle output by the updated decision big model in combination with the third environmental perception data is received.

[0014] In some embodiments, constructing an adversarial simulation scenario based on the large environmental perception model includes:

[0015] performing an abnormality analysis on the assisted driving scenario corresponding to the first assisted driving decision to obtain key interference factors in the assisted driving scenario;

[0016] An adversarial simulation scenario is constructed based on the environmental perception large model and the key interference factors.

[0017] In some embodiments, sending a stress testing instruction to a cloud device based on the adversarial simulation scenario to trigger the decision model to perform a decision stress testing task includes:

[0018] Converting the adversarial simulation scenario into multimodal adversarial input data;

[0019] The multimodal adversarial input data is sent as a stress testing instruction to the cloud device using a preset heterogeneous data collaborative transmission method to trigger the decision model to perform a decision stress testing task based on the multimodal adversarial input data.

[0020] In some embodiments, the identifying anomalies in the first assisted driving decision based on the feedback data and the first environmental perception data includes:

[0021] performing multi-dimensional bad case detection based on the feedback data, the model confidence corresponding to the first environmental perception data, and the first assisted driving decision, to obtain multiple bad case detection results;

[0022] Anomalies are identified for the first assisted driving decision based on the multiple bad case detection results.

[0023] In some embodiments, the multi-dimensional bad case detection is performed based on the feedback data, the model confidence corresponding to the first environmental perception data, and the first assisted driving decision to obtain multiple bad case detection results, including:

[0024] performing bad case detection of a decision dimension based on the feedback data and the first assisted driving decision to obtain decision deviation data; wherein the multiple bad case detection results include the decision deviation data;

[0025] performing bad case detection in a perception dimension based on the model confidence corresponding to the first environmental perception data to obtain perception confidence data; the multiple bad case detection results also include the perception confidence data;

[0026] The performing abnormality identification on the first assisted driving decision based on the multiple bad case detection results includes:

[0027] Comparing the decision deviation data with a preset deviation threshold, and comparing the perception confidence data with a preset confidence threshold, to identify abnormalities in the first assisted driving decision;

[0028] In which, when the decision deviation data is greater than the deviation threshold, and / or when the perception confidence data is less than the confidence threshold, an abnormality recognition result indicating that there is an abnormality in the first assisted driving decision is obtained.

[0029] To achieve the above objectives, a second aspect of an embodiment of the present application provides another vehicle assisted driving decision-making method, which is applied to a cloud device communicating with a vehicle, and includes:

[0030] Obtaining a decision tracing trigger instruction sent by the vehicle; wherein the vehicle sends the decision tracing trigger instruction to the cloud device when it identifies an abnormality in the first assisted driving decision, the first assisted driving decision being output by the decision model of the cloud device;

[0031] In response to the decision tracing trigger instruction, the decision big model is controlled to execute the decision tracing task and generate a model update guidance signal;

[0032] Sending the model update guidance signal to the vehicle, so that the vehicle updates and optimizes the environment perception large model based on the model update guidance signal to obtain an updated environment perception large model;

[0033] The second environmental perception data output by the updated environmental perception large model to the decision-making large model is obtained, and a second assisted driving decision of the vehicle is output to the environmental perception large model based on the decision-making large model in combination with the second environmental perception data.

[0034] In some embodiments, the method further comprises:

[0035] Obtaining a stress test instruction sent by the vehicle; wherein, when the vehicle identifies that there is an abnormality in the first assisted driving decision, constructing an adversarial simulation scenario based on the environmental perception large model, and sending the stress test instruction to the cloud device based on the adversarial simulation scenario;

[0036] In response to the stress testing instruction, triggering the decision big model to execute the decision stress testing task to perform updating and optimization to obtain an updated decision big model;

[0037] Obtain the third environmental perception data output by the environmental perception big model to the updated decision big model, and output the third assisted driving decision of the vehicle to the environmental perception big model based on the updated decision big model and the third environmental perception data.

[0038] To achieve the above objectives, a third aspect of an embodiment of the present application provides a vehicle assisted driving decision-making device, the device comprising:

[0039] an acquisition module, configured to acquire feedback data of the vehicle executing a first assisted driving decision and first environmental perception data; the first assisted driving decision being output by a decision-making model of a cloud device, and the first environmental perception data being output by the vehicle's environmental perception model, the cloud device communicating with the vehicle;

[0040] an abnormality identification module, configured to identify abnormalities in the first assisted driving decision based on the feedback data and the first environmental perception data, and obtain an abnormality identification result;

[0041] a reflection module, configured to, when the abnormality identification result indicates that the first assisted driving decision is abnormal, send a decision tracing instruction to the cloud device to trigger the decision large model to perform a decision tracing task, and update and optimize the environment perception large model based on the model update guidance signal fed back by the cloud device to obtain an updated environment perception large model; the model update guidance signal is generated by the decision large model performing the decision tracing task and output to the environment perception large model;

[0042] A decision module is used to perform an environmental perception task based on the updated environmental perception model, output second environmental perception data to the decision model, and receive a second assisted driving decision of the vehicle output by the decision model in combination with the second environmental perception data.

[0043] To achieve the above-mentioned purpose, the fourth aspect of an embodiment of the present application proposes a vehicle assisted driving decision-making device, which includes a memory and a processor, the memory stores a computer program, and the processor implements the vehicle assisted driving decision-making method described in the first aspect and / or the second aspect when executing the computer program.

[0044] To achieve the above-mentioned purpose, the fifth aspect of an embodiment of the present application proposes a vehicle, which is equipped with a vehicle-assisted driving decision-making device, and the vehicle-assisted driving decision-making device includes a memory and a processor, and the memory stores a computer program. When the processor executes the computer program, it implements the vehicle-assisted driving decision-making method described in the first aspect above.

[0045] To achieve the above-mentioned purpose, the sixth aspect of an embodiment of the present application proposes a cloud device, on which a vehicle assisted driving decision-making device is configured, and the vehicle assisted driving decision-making device includes a memory and a processor, and the memory stores a computer program, and when the processor executes the computer program, it implements the vehicle assisted driving decision-making method described in the second aspect above.

[0046] To achieve the above-mentioned purpose, the seventh aspect of an embodiment of the present application proposes a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, it implements the vehicle assisted driving decision-making method described in the first aspect and / or the second aspect above.

[0047] To achieve the above-mentioned objectives, the eighth aspect of an embodiment of the present application proposes a computer program product, which includes a computer program. When the computer program is executed by a processor, it implements the vehicle assisted driving decision-making method provided in the first aspect and / or the second aspect above.

[0048] The vehicle assisted driving decision-making method, apparatus, device, vehicle, cloud device, computer-readable storage medium and computer program product proposed in the present application obtain feedback data and first environmental perception data of the vehicle executing a first assisted driving decision; the first assisted driving decision is output by the decision big model of the cloud device, and the first environmental perception data is output by the environmental perception big model of the vehicle, and the cloud device communicates with the vehicle; based on the feedback data and the first environmental perception data, the first assisted driving decision is identified as abnormal to obtain an abnormal identification result; when the abnormal identification result indicates that there is an abnormality in the first assisted driving decision, a decision tracing instruction is sent to the cloud device to trigger the decision big model to execute the decision tracing task, and the environmental perception big model is updated and optimized based on the model update guidance signal feedback from the cloud device to obtain an updated environmental perception big model; the model update guidance signal is generated by the decision big model executing the decision tracing task and output to the environmental perception big model; based on the updated environmental perception big model, the environmental perception task is executed to output second environmental perception data to the decision big model, and the second assisted driving decision of the vehicle output by the decision big model in combination with the second environmental perception data is received.

[0049] Compared to the vehicle assisted driving decision-making method in the related art that adopts a similar edge-cloud architecture but is mainly based on static perception and static decision-making, the embodiment of the present application is based on automatically identifying whether there is an abnormality in the first assisted driving decision output by the decision-making large model deployed on the cloud device, and when it is identified that the decision is abnormal, the decision-making large model is triggered to perform a decision tracing task to generate a model update guidance signal, and based on the signal, the environment perception large model deployed locally on the vehicle is updated and optimized to obtain an updated environment perception large model, and then based on the updated environment perception large model, the second environment perception data is output to the decision-making large model, so as to generate the second assisted driving decision of the vehicle based on the decision-making large model combined with the second environment perception data. In this way, the embodiment of the present application is based on triggering the decision-making large model to perform a decision tracing task to analyze its own errors, and generate a model update guidance signal to reversely guide the environment perception large model to update and optimize, which can enable the environment perception large model to significantly improve the target detection and scene understanding capabilities in complex scenes with occlusion or multiple vehicles, thereby effectively compensating for the perception information deviation of the vehicle assisted driving system in dynamic and complex scenes, and improving the accuracy of the system's decision-making in complex scenes. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] Figure 1 A schematic flow chart of the steps of the vehicle assisted driving decision-making method provided in some embodiments of the present application;

[0051] Figure 2A schematic flow chart of steps in other embodiments of the vehicle assisted driving decision-making method provided in the embodiment of the present application;

[0052] Figure 3 for Figure 2 Schematic diagram of the detailed steps of step S201;

[0053] Figure 4 for Figure 2 Schematic diagram of the detailed steps of step S202;

[0054] Figure 5 for Figure 1 Schematic diagram of the detailed steps of step S102;

[0055] Figure 6 for Figure 5 Schematic diagram of the detailed steps of step S501;

[0056] Figure 7 A schematic flow chart of steps in further embodiments of the vehicle assisted driving decision-making method provided in the embodiments of the present application;

[0057] Figure 8 A schematic flow chart of steps in further embodiments of the vehicle assisted driving decision-making method provided in the embodiments of the present application;

[0058] Figure 9 This is a diagram of the VLM+LLM cross-layer bidirectional reflection technology architecture involved in a complete embodiment of the vehicle assisted driving decision-making method provided by the embodiment of the present application;

[0059] Figure 10 A schematic diagram of the structure of a decision-making device for assisted driving of a vehicle provided in an embodiment of the present application;

[0060] Figure 11 This is a schematic diagram of the hardware structure of the vehicle assisted driving decision-making device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0061] In order to make the purpose, technical solutions and advantages of this application more clear, the following further describes this application in detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application.

[0062] It should be noted that although the device schematics illustrate functional module divisions and the flowcharts illustrate logical sequences, in certain circumstances, the steps shown or described may be performed in a sequence that differs from the module divisions in the device or the sequence in the flowcharts. The terms "first," "second," and so on, in the specification, claims, and drawings, are used to distinguish similar items and are not necessarily used to describe a specific sequence or precedence.

[0063] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the art to which this application pertains. The terms used herein are for the purpose of describing the embodiments of this application only and are not intended to limit this application.

[0064] First, the overall concept of the vehicle assisted driving decision-making method provided in the embodiment of the present application is explained.

[0065] With the rapid development of vehicle assisted driving technology, the decision-making system in vehicle assisted driving has gradually become the most challenging part of the core functional module.

[0066] Vehicle-assisted driving decisions need to be made quickly in complex and changing traffic environments (such as multi-objective games, mixed traffic of people and vehicles, emergencies, etc.). This places higher demands on the vehicle-assisted driving system for environmental perception, real-time reasoning, and overall robustness.

[0067] Among related technologies, Large Language Models (LLMs) and Visual Language Models (VLMs) have been gradually introduced into the field of assisted driving due to their excellent generalization and reasoning capabilities. LLMs effectively enhance perception interpretability and decision-making flexibility. The Visual Language Model (VLM) is a multimodal model capable of image understanding and natural language generation, while the Large Language Model (LLM) is a natural language processing model based on the Transformer deep learning architecture.

[0068] However, the computing power consumption and latency issues caused by large-scale model deployment restrict its comprehensive application in the automotive field.

[0069] Edge-Cloud Collaboration (ECC) has become a key approach to addressing this bottleneck. ECC refers to an intelligent architecture that achieves data processing and decision optimization through the coordinated operation of the edge (on-board VLM) and the cloud (high-performance LLM), balancing real-time performance and computing efficiency. By deploying a lightweight VLM at the vehicle's edge for real-time environmental perception and a high-performance LLM on cloud devices or high-computing in-vehicle computing platforms for complex reasoning, the overall system performance and response efficiency can be effectively improved. Furthermore, to overcome the problems of knowledge aging and decreased generalization ability of models during long-term operation, the introduction of self-learning mechanisms with "reflective capabilities" has become a new trend in current intelligent driving research.

[0070] However, most existing edge-cloud architectures rely on static perception and decision-making, making it difficult for the system to continuously evolve and optimize in dynamic and complex scenarios. This makes it very easy for decision-makers to make decisions in these dynamic and complex scenarios due to biased perception information. In complex scenarios, vehicle sensors are limited by weather, occlusion, or blind spots, and the VLM's perception results are prone to bias, which in turn affects the LLM's decision-making and reasoning. For example, camera blur in rainy or snowy weather can cause the VLM to generate an incorrect scene caption, which in turn can lead to misjudgment by the LLM, potentially risking a vehicle collision.

[0071] Based on this, the embodiments of the present application provide a decision-making method, device, equipment, vehicle, cloud device, computer-readable storage medium and computer program product for vehicle assisted driving, aiming to overcome the shortcomings of the above-mentioned related technologies by establishing a mechanism to identify and correct perception errors, that is, based on triggering LLM to perform decision tracing tasks to analyze its own errors, and generate a model update guidance signal to reversely guide VLM to update and optimize, the VLM can significantly improve target detection and scene understanding capabilities in complex scenes with occlusion or multiple vehicles. In this way, it effectively compensates for the perception information deviation of the vehicle assisted driving system in dynamic and complex scenes, and improves the accuracy of the system in making vehicle assisted driving decisions in complex scenes.

[0072] It should be noted that in this embodiment, the VLM is responsible for perceiving the vehicle's internal and external environments. It can use Chain of Thought (CoT) reasoning to output a caption (or scene text description), attention distribution, and reasoning path. Furthermore, the LLM is responsible for integrating the VLM output, vehicle status information, and user profile to make assisted driving decisions, and it also possesses self-reflexive capabilities.

[0073] In addition, considering that the "perception" and "decision-making" of the system in the related art are in a serial relationship, that is, the VLM outputs the caption and transmits it to the LLM. In this way, there is a lack of reverse correction capability, and it is impossible to realize the dynamic adjustment of the VLM perception weight by the LLM decision feedback. For example, in the case of a failed deflection turn, the LLM cannot point out its perception blind spot to the VLM, and long-term operation can easily lead to "blindness" of the system. Therefore, the embodiment of the present application also introduces a perception-decision closed-loop feedback mechanism, which realizes the perception-decision closed-loop feedback and collaborative evolution within the model based on the cross-layer reflection mechanism, thereby improving the adaptability and long-term robustness of the system.

[0074] Furthermore, considering that the current large model has high updating costs after deployment and lacks self-assessment and self-learning capabilities, performance degradation is likely to occur in the face of new scenarios during long-term operation. For example, low-speed congestion scenarios frequently occur in narrow urban roads, and the model needs to constantly adjust its strategy, while traditional static models are difficult to adaptively evolve. To this end, the embodiment of the present application also designs a collaborative system with reflective and continuous learning capabilities. That is, through the LLM initiating guidance and reverse tuning to the VLM, dynamic perception focus migration is achieved, and the VLM automatically generates a virtual adversarial environment to perform stress testing on the LLM, thereby promoting LLM strategy optimization. This mechanism can effectively compensate for perception errors, optimize model decision logic, and realize system self-learning, solving the closed-loop rupture problem of "misperception-misdecision-no feedback" in traditional systems. In addition, the embodiments of the present application combine lightweight VLM to achieve efficient environmental perception at the edge and high-performance LLM to perform complex decision reasoning in the cloud, constructing a two-way reflection mechanism for perception-decision linkage. By identifying abnormal perception or decision in real time, and performing reverse optimization and virtual confrontation scenario construction, it can also achieve adaptive evolution and robustness enhancement of the system, breaking the limitations of traditional perception and decision modules of "one-way transmission and difficult to correct", thereby realizing continuous learning and evolution of vehicle assisted driving systems.

[0075] Next, the decision-making method, device, equipment, vehicle, cloud device, computer-readable storage medium and computer program product for vehicle assisted driving provided in the embodiments of the present application are specifically described through the following embodiments, and the decision-making method for vehicle assisted driving provided in the embodiments of the present application is first described in detail.

[0076] It should be noted that in each specific embodiment of the present application, when it comes to the need to perform relevant processing based on data related to the user's identity or characteristics, such as user information, user behavior data, user historical data, and user location information, the user's permission or consent will be obtained first, and the collection, use, and processing of such data will comply with relevant laws, regulations, and standards. In addition, when the embodiment of the present application needs to obtain the user's sensitive personal information, the user's separate permission or consent will be obtained through a pop-up window or by jumping to a confirmation page. After clearly obtaining the user's separate permission or consent, the necessary user-related data for the normal operation of the embodiment of the present application will be obtained.

[0077] It should be noted that the vehicle-assisted driving decision-making method provided in the embodiments of the present application can be applied to a terminal or a server, and can also be software running on a terminal or server. In some embodiments, the terminal can be an onboard terminal on the vehicle, a high-computing vehicle-mounted computing platform, or a computer device such as a smartphone, tablet computer, laptop computer, or desktop computer associated with the vehicle. The terminal's association with the vehicle means that the terminal can communicate and exchange data with the vehicle over a network. The server can be a backend server terminal device in the vehicle, such as a cloud device that communicates with the vehicle. It can be configured as a standalone physical server, a server cluster or distributed system consisting of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms. The software can include an application that implements the vehicle-assisted driving decision-making method, a computer program, and a storage medium that carries the computer program. It should be understood that based on different design requirements of actual applications, in different feasible embodiments, the terminal, server side, and software that apply the vehicle assisted driving decision-making method provided in the embodiments of the present application may of course also be in other forms not listed here, and the vehicle assisted driving decision-making method provided in the embodiments of the present application does not specifically limit this.

[0078] Furthermore, the present application may be used in a wide variety of general-purpose or specialized computer system environments or configurations. For example, vehicles, personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer computer devices, personal computers (PCs), minicomputers, mainframe computers, distributed computing environments including any of the above, and the like. The present application may be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, and the like that perform specific tasks or implement specific abstract data types. The present application may also be practiced in distributed computing environments in which tasks are performed by remote processing devices connected via a communications network. In distributed computing environments, program modules may be located in local and remote computer storage media, including storage devices.

[0079] For ease of understanding and explanation, the following text uses the vehicle-mounted terminal (directly configured on the vehicle or associated with the vehicle) and the cloud device to apply the vehicle-assisted driving decision-making method provided in the embodiments of the present application as an example to describe in detail various specific embodiments of the present application. The implementation of the vehicle-assisted driving decision-making method provided in the embodiments of the present application in any of the above-mentioned forms of subject matter can refer to the process of applying the vehicle-assisted driving decision-making method by the vehicle-mounted terminal and / or cloud device described below.

[0080] Please refer to Figure 1 , Figure 1 The following is a flowchart of the steps in some embodiments of the vehicle assisted driving decision-making method provided in the embodiment of the present application. It should be understood that although Figure 1 The following flowcharts show the execution order of some method steps, but based on the different design requirements of actual applications, the vehicle assisted driving decision-making method provided in the embodiment of the present application can certainly adopt an execution order different from the method steps shown in the figure. Figure 1 The order of the steps in the method shown does not constitute a limitation on the execution logic order of the decision-making method for vehicle assisted driving provided in the embodiment of the present application. Figure 1 Reasonable changes in the order of the steps of the method shown should be included in the scope of protection of the vehicle assisted driving decision-making method provided in the embodiments of the present application.

[0081] like Figure 1 As shown, in some embodiments, when the vehicle assisted driving decision-making method provided in the embodiments of the present application is applied to a vehicle, it may include steps S101 to S104 as shown below.

[0082] Step S101: Obtain feedback data and first environmental perception data of the vehicle executing a first assisted driving decision; the first assisted driving decision is output by the decision model of the cloud device, and the first environmental perception data is output by the vehicle's environmental perception model, and the cloud device communicates with the vehicle.

[0083] It should be noted that the decision-making big model can be the above-mentioned LLM, and the environmental perception big model can be the above-mentioned VLM. For the convenience of explanation and understanding, the following text will use LLM instead of the decision-making big model and VLM instead of the environmental perception big model to explain the embodiments in detail. In addition, the cloud device can be a cloud server or a high-computing-power vehicle-mounted computing platform in the edge-cloud architecture when the decision-making LLM and VLM are deployed using the edge-cloud architecture. The vehicle can be the vehicle itself or an on-board terminal configured on the vehicle. The following text will uniformly explain the on-board terminal.

[0084] When the vehicle terminal combines the local VLM and the LLM in the cloud device to make intelligent decisions for assisted driving, the local lightweight VLM first processes data such as camera images inside and outside the vehicle, radar point clouds, and inertial measurement unit (IMU) data to obtain the scene text caption, attention area attention map, reasoning path trace, and confidence information output by the VLM. The vehicle terminal then obtains the scene text caption, attention area attention map, reasoning path trace, and confidence information output by the VLM as the first environmental perception data and transmits this first environmental perception data to the LLM in the cloud device.

[0085] After receiving the first environmental perception data, LLM integrates the data with the vehicle status data and user portrait information to generate vehicle control instructions (such as steering angle, acceleration and deceleration decisions, etc.). The vehicle control instruction is the first assisted driving decision for the vehicle, and the cloud device transmits the first assisted driving decision to the vehicle for execution.

[0086] During the process of the vehicle executing the first assisted driving decision, the on-board terminal continuously obtains feedback data of the vehicle executing the first assisted driving decision (for example, the steering angle is too large / too small, the deceleration is too rapid / untimely, etc.).

[0087] Step S102: performing abnormality identification on the first assisted driving decision based on the feedback data and the first environmental perception data to obtain an abnormality identification result.

[0088] After obtaining the first environmental perception data output by the VLM and the feedback data of the vehicle executing the first assisted driving decision, the vehicle-mounted terminal combines the first environmental perception data and the feedback data to perform abnormality identification on the first assisted driving decision, thereby obtaining an abnormality identification result for the first assisted driving decision.

[0089] In some embodiments, the vehicle terminal can use a Bad Case Detector to identify anomalies in the first assisted driving decision. As a module for detecting model output anomalies, the Bad Case Detector can comprehensively determine whether the first assisted driving decision is in error compared to an ideal (reference) decision based on three multi-dimensional indicators: perception confidence (derived from the first environmental perception data), decision deviation (derived from the first assisted driving decision), and execution feedback (derived from feedback data). The ideal (reference) decision can be derived from preset rules or expert experience.

[0090] Step S103: When the abnormal recognition result indicates that there is an abnormality in the first assisted driving decision, a decision tracing instruction is sent to the cloud device to trigger the decision big model to perform the decision tracing task, and the environment perception big model is updated and optimized based on the model update guidance signal fed back by the cloud device to obtain an updated environment perception big model; the model update guidance signal is generated by the decision big model when it performs the decision tracing task and output to the environment perception big model.

[0091] After the on-board terminal performs abnormality identification on the first assisted driving decision and obtains an abnormality identification result, if the abnormality identification result indicates that there is an abnormality in the first assisted driving decision, for example, there is an error in the first assisted driving decision compared to the ideal (reference) decision, the on-board terminal sends a decision tracing instruction to the cloud device. After receiving the decision tracing instruction, the cloud device immediately responds to the instruction to control the LLM to perform the decision tracing task. In the process of performing the decision tracing task, the LLM performs its own error analysis and generates and outputs a model update guidance signal that reversely guides the VLM to perform update optimization. After the LLM outputs the model update guidance signal, the cloud device sends the model update guidance signal to the on-board terminal. By receiving the model update guidance signal, the on-board terminal controls the VLM to perform update optimization based on the model update guidance signal to obtain an updated VLM.

[0092] In some embodiments, the model update guidance signal can be an attention guidance vector that guides the VLM to adjust its attention area, allowing the LLM to perform its own error analysis during decision tracing. When the vehicle terminal updates and optimizes the VLM based on the model update guidance signal, the VLM can adjust the original attention mechanism based on the attention guidance vector to obtain an updated VLM.

[0093] For example, when the vehicle-mounted terminal identifies an abnormality in the first assisted driving decision through the Bad Case detector, it triggers the LLM to enter the "decision tracing" mode by sending a decision tracing instruction to the cloud device.

[0094] In this "decision traceability" mode, the LLM performs a decision traceability task, analyzing its own decision-making process in reverse order, determining which inputs (such as the primary environmental perception data output by the VLM, vehicle status data, and user profile information) play a key role in the decision. LLM can leverage attention-based attribution to analyze this information. Specifically, LLM can quantify the influence of input features (such as the primary environmental perception data output by the VLM, vehicle status data, and user profile information) using techniques such as integrated gradients (IG), SHapley additive exPlanations (SHAP), or Transformer Attention Weights (TW), a deep learning model.

[0095] LLM generates attention guidance vectors in the process of reverse analysis of its own decision-making process. That is, LLM maps the focus of attention into a set of attention guidance vectors vec{a}, which are as follows: , where (x i ,y i ) represents the coordinates of the area in the image that should be focused on, w i Indicates the attention intensity corresponding to the region coordinates.

[0096] After generating the attention guidance vector, the LLM outputs the attention guidance vector as a model update guidance signal to the VLM. That is, the cloud device sends the attention vector vec{a} to the VLM module through a communication protocol (for example, a communication protocol based on the JSON+protobuf format) along with instruction information (such as descriptions of the target object, color, motion characteristics, etc.).

[0097] After receiving the model update guidance signal sent by the LLM side, the VLM merges the guidance vector into the original attention area attention map and follows: Adjust the original attention mechanism, where Attention' is the adjusted attention mechanism and Attention is the original attention mechanism. It is a dynamic adjustment coefficient used to balance original attention and guided attention.

[0098] In some embodiments, after the vehicle-mounted terminal performs abnormality identification on the first assisted driving decision and obtains an abnormality identification result, if the abnormality identification result indicates that there is no abnormality in the first assisted driving decision, for example, there is no error in the first assisted driving decision compared to the ideal (reference) decision, the vehicle-mounted terminal continues to jointly use the VLM and LLM to perform a new round of intelligent decision-making for vehicle assisted driving, and continues to execute the above-mentioned process of obtaining data and performing abnormality identification.

[0099] Step S104: Execute the environmental perception task based on the updated environmental perception large model to output second environmental perception data to the decision large model, and receive the second assisted driving decision of the vehicle output by the decision large model in combination with the second environmental perception data.

[0100] After obtaining the updated VLM, the vehicle terminal continues to re-execute the environmental perception task based on the updated VLM, and performs subsequent decision iterations or training replays based on the results output by the updated VLM. That is, based on the updated VLM, the vehicle terminal continues to process the camera images inside and outside the vehicle, the radar point cloud, and the IMU data to obtain the scene text description caption, attention area attention map, reasoning path reasoning trace, and confidence information output by the updated VLM. Then, the vehicle terminal obtains the scene text description caption, attention area attention map, reasoning path reasoning trace, and confidence information output by the updated VLM as the second environmental perception data, and transmits the second environmental perception data to the LLM of the cloud device. After receiving the second environmental perception data, the LLM fuses the data with the vehicle status data and the user profile information to generate a vehicle control instruction. The vehicle control instruction serves as the second assisted driving decision for the vehicle and is transmitted from the cloud device to the vehicle for execution.

[0101] In an embodiment of the present application, the scene text description caption, attention area attention map, reasoning path reasoning trace and confidence information output by the VLM are obtained as the first environmental perception data through the vehicle terminal, and the feedback data of the vehicle executing the first assisted driving decision is continuously obtained through the vehicle terminal during the process of the vehicle executing the first assisted driving decision. Afterwards, the first assisted driving decision is identified as abnormal by the vehicle terminal in combination with the first environmental perception data and the feedback data, thereby obtaining an abnormal identification result for the first assisted driving decision. If the abnormal identification result indicates that the first assisted driving decision is abnormal, a decision tracing instruction is sent to the cloud device through the vehicle terminal. After receiving the decision tracing instruction, the cloud device immediately responds to the instruction to control the LLM to execute the decision tracing task. In the process of executing the decision tracing task, the LLM performs its own error analysis and generates and outputs a model update guidance signal that reversely guides the VLM to update and optimize. After the LLM outputs the model update guidance signal, the cloud device sends the model update guidance signal to the vehicle terminal. The on-board terminal receives the model update guidance signal and controls the VLM to update and optimize based on the model update guidance signal to obtain an updated VLM. Finally, the on-board terminal continues to process the camera images inside and outside the vehicle, the radar point cloud, and the IMU data based on the updated VLM to obtain the scene text description caption, attention area attention map, reasoning path reasoning trace, and confidence information output by the updated VLM. Then, the on-board terminal obtains the scene text description caption, attention area attention map, reasoning path reasoning trace, and confidence information output by the updated VLM as the second environmental perception data, and transmits the second environmental perception data to the LLM of the cloud device. After receiving the second environmental perception data, the LLM integrates the data with the vehicle status data and the user profile information to generate a vehicle control instruction. The vehicle control instruction is transmitted from the cloud device to the vehicle for execution as the second assisted driving decision for the vehicle.

[0102] Compared to the vehicle assisted driving decision-making method in the related art that adopts a similar edge-cloud architecture but is mainly based on static perception and static decision-making, the embodiment of the present application is based on automatically identifying whether there is an abnormality in the first assisted driving decision output by the LLM deployed on the cloud device, and when it is identified that the decision is abnormal, it triggers the LLM to perform a decision tracing task to generate a model update guidance signal, and based on the signal, updates and optimizes the VLM deployed locally in the vehicle to obtain an updated VLM, and then outputs the second environmental perception data to the LLM based on the updated VLM, so as to generate the second assisted driving decision of the vehicle based on the LLM combined with the second environmental perception data. In this way, the embodiment of the present application is based on triggering the LLM to perform a decision tracing task to analyze its own errors, and generating a model update guidance signal to reversely guide the VLM to update and optimize, which can enable the VLM to significantly improve the target detection and scene understanding capabilities in complex scenes with occlusion or multiple vehicles, thereby effectively compensating for the perception information deviation of the vehicle assisted driving system in dynamic and complex scenes, and improving the accuracy of the system's decision-making in complex scenes.

[0103] In some embodiments, when the vehicle assisted driving decision-making method provided in the embodiments of the present application is applied to a vehicle, the vehicle-mounted terminal can also automatically generate a virtual adversarial environment based on the VLM to perform stress testing on the LLM when it identifies that there is an abnormality in the first vehicle assisted driving decision, thereby promoting LLM strategy optimization.

[0104] Please refer to Figure 2 , Figure 2 A schematic flowchart of the steps of the vehicle assisted driving decision-making method provided in the embodiment of the present application in other embodiments.

[0105] like Figure 2 As shown, in some embodiments, when the vehicle assisted driving decision-making method provided in the embodiments of the present application is applied to a vehicle, it can also include steps S201 to S203 as shown below.

[0106] Step S201: When the abnormality recognition result indicates that the first assisted driving decision is abnormal, construct an adversarial simulation scenario based on the environmental perception large model.

[0107] After the vehicle-mounted terminal performs anomaly identification on the first assisted driving decision and obtains an anomaly identification result, if the anomaly identification result indicates that there is an anomaly in the first assisted driving decision, it can also generate an adversarial simulation scenario based on the vehicle's local VLM and combined with the cross-modal correlation data fed back when the anomaly identification was performed on the first assisted driving decision (such as anomaly images, scene text description captions, attention area attention maps, and environmental context sensor meta information).

[0108] In some embodiments, when the vehicle-mounted terminal uses a Bad Case detector to perform abnormality identification on the first assisted driving decision, the Bad Case detector can output the above-mentioned cross-modal correlation data to the vehicle's local VLM when it comprehensively judges that there is an error in the first assisted driving decision compared with the ideal (reference) decision.

[0109] Step S202: Based on the adversarial simulation scenario, a stress testing instruction is sent to the cloud device to trigger the decision-making model to execute a decision stress testing task; the decision-making model is updated and optimized based on the execution of the decision stress testing task to obtain an updated decision-making model.

[0110] After constructing an adversarial simulation scenario based on the VLM, the vehicle terminal further generates a stress test instruction for the adversarial simulation scenario and sends the stress test instruction to the cloud device. The stress test instruction must at least include relevant data for the adversarial simulation scenario. After receiving the stress test instruction, the cloud device responds to the stress test instruction and controls the LLM to perform a decision stress test task using the adversarial simulation scenario as input, thereby implementing a policy stress test on the LLM and updating and optimizing the LLM to obtain an updated LLM.

[0111] Step S203: Outputting third environmental perception data to the updated decision-making model based on the environmental perception big model, and receiving a third assisted driving decision of the vehicle output by the updated decision-making big model in combination with the third environmental perception data.

[0112] After the vehicle terminal sends a stress test command to the cloud device to trigger the LLM to execute the decision stress test task, when the LLM obtains an updated LLM based on the execution of the task, the vehicle terminal further processes the camera images inside and outside the vehicle, the radar point cloud, and the IMU data based on the VLM to obtain the scene text description caption, attention area attention map, reasoning path reasoning trace, and confidence information output by the VLM. The vehicle terminal then obtains the scene text description caption, attention area attention map, reasoning path reasoning trace, and confidence information output by the VLM as third environmental perception data, and transmits the third environmental perception data to the updated LLM on the cloud device. After receiving the third environmental perception data, the updated LLM integrates the data with the vehicle status data and user profile information to generate a vehicle control instruction. The vehicle control instruction is transmitted from the cloud device to the vehicle for execution as the third assisted driving decision for the vehicle.

[0113] In some embodiments, when constructing an adversarial simulation scenario based on the VLM, the vehicle-mounted terminal can perform abnormal case analysis on cross-modal correlation data based on the VLM, thereby constructing an adversarial simulation scenario based on the analysis results.

[0114] Please refer to Figure 3 , Figure 3 for Figure 2 Schematic diagram of the detailed steps of step S201.

[0115] like Figure 3 As shown, in some embodiments, the above-mentioned step S201: constructing an adversarial simulation scenario based on the environmental perception large model may include steps S301 and S302 as shown below.

[0116] Step S301: performing an abnormality analysis on the assisted driving scenario corresponding to the first assisted driving decision to obtain key interference factors in the assisted driving scenario.

[0117] When the vehicle terminal uses the VLM to perform anomaly case analysis on cross-modal correlation data to construct an adversarial simulation scenario, it first uses the VLM to perform anomaly case analysis based on the cross-modal correlation data (including anomaly images, captions, attention maps, and sensor metadata). Specifically, the VLM analyzes the environmental structure of the vehicle's assisted driving scenario, such as traffic layout, key objects, weather conditions, and lighting conditions, corresponding to the first assisted driving decision currently being executed. The vehicle terminal then uses the VLM to extract interference factors from these analyzed environmental structures that may have caused an anomaly in the first assisted driving decision. These interference factors are then used as key interference factors in the assisted driving scenario corresponding to the first assisted driving decision. These key interference factors include, but are not limited to, object occlusion, object ambiguity, complex lighting / rain / snow / glare, and behavioral patterns such as irregular motion paths and sudden crossings.

[0118] Step S302: Construct an adversarial simulation scenario based on the environmental perception large model and the key interference factors.

[0119] After obtaining the key interference factors in the assisted driving scenario, the VLM further uses these key interference factors to generate a set of disturbance factors, thereby constructing the function: , where E is the original scene and D is the set of perturbation factors. The onboard terminal then generates an adversarial simulation scenario based on the function that the VLM should construct, using image-level perturbations, entity-level perturbations, and / or semantic-level perturbations. Image-level perturbations can include adding rain, snow, blur, or brightness and contrast perturbations; entity-level perturbations can include changing target positions, suddenly appearing targets, or adding false targets; and semantic-level perturbations can include adjusting traffic signal states and interfering with sign recognition.

[0120] In some embodiments, when the vehicle-mounted terminal performs abnormal case analysis on cross-modal correlation data based on VLM to construct an adversarial simulation scenario, it can also choose to introduce a graph neural network to enhance the context modeling capability of VLM and use the generative adversarial network GAN to generate a more realistic adversarial simulation scenario.

[0121] In some embodiments, when the vehicle-mounted terminal sends a stress testing instruction to the cloud device based on an adversarial simulation scenario to trigger the decision model to perform a decision stress testing task, the stress testing instruction can be transmitted using a preset heterogeneous data collaborative transmission method.

[0122] It should be noted that the heterogeneous data collaborative transmission method can be the data transmission method based on the communication protocol in the JSON+protobuf format mentioned above. Data transmission between the vehicle terminal and the cloud device or between the VLM and LLM can also be carried out using this data transmission method.

[0123] Please refer to Figure 4 , Figure 4 for Figure 2 Schematic diagram of the detailed process flow of step S202.

[0124] like Figure 4 As shown, in some embodiments, the above-mentioned step S202: sending a stress testing instruction to the cloud device based on the adversarial simulation scenario to trigger the decision model to perform a decision stress testing task, may include the following steps.

[0125] Step S401: converting the adversarial simulation scenario into multimodal adversarial input data.

[0126] After constructing an adversarial simulation scenario based on the VLM, the vehicle terminal converts the generated adversarial simulation scenario into multimodal adversarial input data. This multimodal adversarial input data includes but is not limited to: synthetic images, semantic captions, attention maps of the desired areas of interest in the simulation scenario, and simulated radar / IMU / positioning sensor-fused data.

[0127] Step S402: The multimodal adversarial input data is sent as a stress testing instruction to a cloud device using a preset heterogeneous data collaborative transmission method to trigger the decision model to perform a decision stress testing task based on the multimodal adversarial input data.

[0128] After the onboard terminal converts the adversarial simulation scenario into multimodal adversarial input data, it can directly use this multimodal adversarial input data as a stress testing instruction. It then processes the stress testing instruction using a pre-configured communication protocol based on the JSON+protobuf format, and transmits the stress testing instruction to the cloud device based on the JSON+protobuf format. After receiving the JSON+protobuf formatted data, the cloud device recovers the data to obtain the multimodal adversarial input data, which it then uses as input to the LLM to control the LLM's execution of decision stress testing tasks for policy stress testing.

[0129] In this embodiment, the vehicle terminal performs an anomaly analysis of the assisted driving scenario corresponding to the first assisted driving decision to extract key interference factors in the assisted driving scenario. An adversarial simulation scenario is then constructed based on these key interference factors. Using a pre-defined heterogeneous data collaborative transmission method, the multimodal adversarial input data corresponding to this scenario is transmitted to the cloud device in JSON+protobuf format. The LLM then performs a policy retest based on this multimodal adversarial input data. In this way, this embodiment not only reversely guides the VLM through its own error analysis to update and optimize, but also enables the VLM to automatically generate complex test environments based on bad cases to challenge the LLM, thereby updating and optimizing the LLM. In other words, this embodiment implements a reverse reasoning mechanism in which the LLM guides the VLM, and a two-way reflection mechanism in which the VLM generates virtual adversarial scenarios to stress-test the LLM. This overcomes the limitations of traditional perception and decision modules, which are limited by one-way transmission and difficulty in correction, and enables continuous learning and evolution of the vehicle assisted driving system.

[0130] In addition, this embodiment uses virtual confrontation scenarios to update and optimize the LLM, which can also improve the generalization ability of the LLM, thereby effectively improving the stability and security of the LLM in extreme edge cases.

[0131] In some embodiments, when the vehicle-mounted terminal identifies anomalies for the first assisted driving decision based on the first environmental perception data and the feedback data when the vehicle executes the first assisted driving decision, it can accurately identify the anomalies by setting a multi-threshold rule function to trigger the LLM to update the VLM through a reverse reasoning mechanism, and / or trigger the VLM to generate a virtual confrontation scenario and perform a two-way reflection mechanism for stress testing the LLM.

[0132] Please refer to Figure 5 , Figure 5 for Figure 1 Schematic diagram of the detailed process flow of step S102.

[0133] like Figure 5 As shown, in some embodiments, in the above-mentioned step S102, the step of "identifying abnormalities in the first assisted driving decision based on the feedback data and the first environmental perception data" may include steps S501 and S502 as shown below.

[0134] Step S501: Perform multi-dimensional bad case detection based on the feedback data, the model confidence corresponding to the first environmental perception data, and the first assisted driving decision to obtain multiple bad case detection results.

[0135] When the vehicle-mounted terminal uses the Bad Case detector to identify anomalies in the first assisted driving decision, the Bad Case detector may perform multi-dimensional bad case detection based on the feedback data, the model confidence corresponding to the first environmental perception data, and the first assisted driving decision, thereby obtaining multiple bad case detection results. The multiple bad case detection results may be at least two of three multi-dimensional indicators: perception confidence based on the first environmental perception data, decision deviation based on the first assisted driving decision, and execution feedback based on the feedback data.

[0136] Step S502: performing abnormality identification on the first assisted driving decision based on the multiple bad case detection results.

[0137] After obtaining multiple bad case detection results based on the Bad Case detector, the vehicle-mounted terminal further utilizes the multiple bad case detection results based on the Bad Case detector to perform a comprehensive multi-dimensional evaluation of the first assisted driving decision, thereby identifying whether the first assisted driving decision has errors compared to the ideal (reference) decision. If the Bad Case detector identifies that the first assisted driving decision has errors compared to the ideal (reference) decision, the first assisted driving decision is considered abnormal. Conversely, if the Bad Case detector identifies that the first assisted driving decision has no errors compared to the ideal (reference) decision, the first assisted driving decision is considered normal.

[0138] In some embodiments, the vehicle terminal may perform multi-dimensional bad case detection based on the Bad Case Detector, and obtain multiple bad case detection results, which may include at least decision deviation data and perception confidence data. Based on this, the vehicle terminal may combine the decision deviation data and perception confidence data to determine whether the first assisted driving decision contains an error compared to the ideal (reference) decision, thereby obtaining a corresponding anomaly identification result.

[0139] Please refer to Figure 6 , Figure 6 for Figure 5 Schematic diagram of the detailed steps of step S501.

[0140] like Figure 5 As shown, in some embodiments, the above-mentioned step S501: performing multi-dimensional bad case detection based on the feedback data, the model confidence corresponding to the first environmental perception data, and the first assisted driving decision to obtain multiple bad case detection results may include steps S601 and S602 as shown below.

[0141] Step S601: performing bad case detection of a decision dimension based on the feedback data and the first assisted driving decision to obtain decision deviation data; the multiple bad case detection results include the decision deviation data.

[0142] It should be noted that the decision deviation data can reflect the degree of difference between the first assisted driving decision output by the LLM and the ideal decision. The larger the value, the greater the error.

[0143] When the vehicle terminal performs multi-dimensional bad case detection based on the Bad Case detector, Formula 1 shown below can be used to perform bad case detection in the decision dimension based on the feedback data and the first assisted driving decision, thereby obtaining decision deviation data.

[0144] , formula 1

[0145] Among them, is the decision deviation data, Refers to the decision instruction actually output by the LLM model, that is, the first auxiliary driving decision, Refers to the ideal (reference) decision, which can be determined by referring to preset rules or expert experience based on the first-aided driving decision, vehicle status data and feedback data.

[0146] Step S602: performing bad case detection in the perception dimension based on the model confidence corresponding to the first environmental perception data to obtain perception confidence data; the multiple bad case detection results also include the perception confidence data.

[0147] It should be noted that the perception confidence data can reflect the overall confidence level of the first environmental perception data output by the VLM. The higher the value, the higher the credibility of the perception result, and vice versa.

[0148] When the vehicle-mounted terminal performs multi-dimensional bad case detection based on the Bad Case detector, it can also use Formula 2 shown below to perform bad case detection in the perception dimension based on the model confidence corresponding to the first environmental perception data, thereby obtaining perception confidence data.

[0149] , formula 2

[0150] Among them, Conf i Indicates the confidence of the i-th perception target or feature in the first environmental perception data output by VLM. i It can be directly output by VLM while outputting the first environmental perception data. γ is the perception confidence data, where each Conf i The average perceptual confidence of .

[0151] In this case, the above-mentioned step S502: identifying abnormalities in the first assisted driving decision based on the multiple bad case detection results may include the following steps:

[0152] The decision deviation data is compared with a preset deviation threshold, and the perception confidence data is compared with a preset confidence threshold to identify abnormalities in the first assisted driving decision.

[0153] It should be noted that when the decision deviation data is greater than the deviation threshold, and / or when the perception confidence data is less than the confidence threshold, an abnormality recognition result is obtained indicating that there is an abnormality in the first assisted driving decision.

[0154] After the vehicle-mounted terminal performs multi-dimensional bad case detection based on the Bad Case detector to obtain decision deviation data and perception confidence data, it compares the decision deviation data with a preset deviation threshold (which can be set based on actual needs), and also compares the perception confidence data with a preset confidence threshold (which can be set based on actual needs). Thus, if the comparison finds that the decision deviation data is greater than the deviation threshold, and if the perception confidence data is less than the confidence threshold, the vehicle-mounted terminal determines that an abnormality identification result has been obtained, indicating that the first assisted driving decision is abnormal. Alternatively, the vehicle-mounted terminal may determine that an abnormality identification result has been obtained, indicating that the first assisted driving decision is abnormal, only if the comparison finds that the decision deviation data is greater than the deviation threshold. Alternatively, the vehicle-mounted terminal may determine that an abnormality identification result has been obtained, indicating that the first assisted driving decision is abnormal, if the comparison finds that the perception confidence data is less than the confidence threshold.

[0155] In some embodiments, the vehicle terminal may also trigger the above-mentioned VLM to generate a virtual confrontation scene and perform a stress test on the LLM when the comparison finds that the decision deviation data is greater than the deviation threshold, thereby obtaining an abnormal recognition result indicating that the first assisted driving decision is abnormal, thereby triggering the reflection mechanism of the LLM to update and optimize the LLM to obtain an updated LLM. Alternatively, the vehicle terminal may also trigger the above-mentioned LLM to perform a reverse reasoning mechanism to guide the VLM's attention when the comparison finds that the perception confidence data is less than the confidence threshold, thereby obtaining an abnormal recognition result indicating that the first assisted driving decision is abnormal, thereby triggering the LLM to perform a reverse reasoning mechanism to update and optimize the VLM to obtain an updated VLM. Alternatively, the vehicle terminal may also trigger the LLM to perform a two-way reflection mechanism to guide the VLM's attention to update the VLM and to generate a virtual confrontation scene and perform a stress test on the LLM to update and optimize the LLM when the comparison finds that the decision deviation data is greater than the deviation threshold and the perception confidence data is less than the confidence threshold, thereby obtaining an abnormal recognition result indicating that the first assisted driving decision is abnormal.

[0156] In this embodiment, the vehicle terminal detects bad cases based on perception confidence, decision deviation, and execution feedback, triggering the LLM to direct the VLM's attention and update the VLM. Alternatively, the VLM can be triggered to generate virtual adversarial scenarios and stress-test the LLM to update and optimize the LLM. This enables closed-loop feedback and coordinated evolution of perception and decision-making within the model, thereby improving the system's adaptability and long-term robustness.

[0157] Please refer to Figure 7 , Figure 7 A schematic flowchart of the steps of the vehicle assisted driving decision-making method provided in the embodiments of the present application in some further embodiments.

[0158] like Figure 7 As shown, in some embodiments, when the vehicle assisted driving decision-making method provided in the embodiments of the present application is applied to a cloud device, it may include steps S701 to S704 as shown below.

[0159] Step S701: Obtain a decision tracing trigger instruction sent by the vehicle; wherein, when the vehicle identifies that there is an abnormality in the first assisted driving decision, the vehicle sends the decision tracing trigger instruction to the cloud device, and the first assisted driving decision is output by the decision model of the cloud device.

[0160] After obtaining the first environmental perception data output by the VLM and the feedback data from the vehicle's execution of the first assisted driving decision, the onboard terminal combines the first environmental perception data and feedback data to identify anomalies in the first assisted driving decision, thereby obtaining an anomaly identification result for the first assisted driving decision. If the anomaly identification result indicates an anomaly in the first assisted driving decision, for example, an error in the first assisted driving decision compared to the ideal (reference) decision, the onboard terminal sends a decision tracing instruction to the cloud device, which then receives the decision tracing instruction.

[0161] Step S702: In response to the decision tracing trigger instruction, the decision large model is controlled to execute the decision tracing task to generate a model update guidance signal.

[0162] After receiving the decision tracing instruction, the cloud device immediately responds to the instruction to control the LLM to perform the decision tracing task. During the execution of the decision tracing task, the LLM performs its own error analysis and generates and outputs a model update guidance signal that in turn guides the VLM to perform update optimization.

[0163] Step S703: Send the model update guidance signal to the vehicle, so that the vehicle updates and optimizes the environment perception large model based on the model update guidance signal to obtain an updated environment perception large model.

[0164] After the LLM outputs the model update guidance signal, the cloud device sends it to the vehicle terminal. The vehicle terminal receives the model update guidance signal and controls the VLM to update and optimize based on the model update guidance signal, resulting in an updated VLM.

[0165] Step S704: Obtain the second environmental perception data output by the updated environmental perception big model to the decision big model, and output the second assisted driving decision of the vehicle to the environmental perception big model based on the decision big model and the second environmental perception data.

[0166] After obtaining the updated VLM, the vehicle terminal continues to re-execute the environmental perception task based on the updated VLM, and performs subsequent decision iterations or training replays based on the results output by the updated VLM. That is, based on the updated VLM, the vehicle terminal continues to process the camera images inside and outside the vehicle, the radar point cloud, and the IMU data to obtain the scene text description caption, attention area attention map, reasoning path reasoning trace, and confidence information output by the updated VLM. Then, the vehicle terminal obtains the scene text description caption, attention area attention map, reasoning path reasoning trace, and confidence information output by the updated VLM as the second environmental perception data, and transmits the second environmental perception data to the LLM of the cloud device. After receiving the second environmental perception data, the LLM fuses the data with the vehicle status data and the user profile information to generate a vehicle control instruction. The vehicle control instruction serves as the second assisted driving decision for the vehicle and is transmitted from the cloud device to the vehicle for execution.

[0167] In an embodiment of the present application, based on automatic identification of whether there is an abnormality in the first assisted driving decision output by the LLM deployed on the cloud device, and when it is identified that there is an abnormality in the decision, the LLM is triggered to perform a decision tracing task to generate a model update guidance signal, and based on the signal, the VLM deployed locally on the vehicle is updated and optimized to obtain an updated VLM, and then the second environmental perception data is output to the LLM based on the updated VLM, so as to generate the second assisted driving decision of the vehicle based on the LLM combined with the second environmental perception data. In this way, the embodiment of the present application is based on triggering the LLM to perform a decision tracing task to analyze its own errors, and generating a model update guidance signal to reversely guide the VLM to perform update optimization, which can enable the VLM to significantly improve the target detection and scene understanding capabilities in complex scenes with occlusion or multiple vehicles, thereby effectively compensating for the perception information deviation of the vehicle assisted driving system in dynamic and complex scenes, and improving the accuracy of the system's decision-making in complex scenes.

[0168] Please refer to Figure 8 , Figure 8 A schematic flowchart of the steps of the vehicle assisted driving decision-making method provided in the embodiments of the present application in some further embodiments.

[0169] like Figure 8 As shown, in some embodiments, when the vehicle assisted driving decision-making method provided in the embodiments of the present application is applied to a cloud device, it can also include steps S801 to S803 as shown below.

[0170] Step S801: Obtain a stress test instruction sent by the vehicle; wherein, when the vehicle identifies that there is an abnormality in the first assisted driving decision, it constructs an adversarial simulation scenario based on the environmental perception large model, and sends the stress test instruction to the cloud device based on the adversarial simulation scenario.

[0171] After the vehicle-mounted terminal performs abnormal identification on the first assisted driving decision and obtains an abnormal identification result, if the abnormal identification result indicates that the first assisted driving decision has an abnormality, it can also generate an adversarial simulation scenario based on the vehicle's local VLM and combined with the cross-modal correlation data fed back when the abnormal identification is performed on the first assisted driving decision. In addition, the vehicle-mounted terminal further generates a stress test instruction for the adversarial simulation scenario and sends the stress test instruction to the cloud device, so that the cloud device can receive the stress test instruction. Among them, the stress test instruction needs to include at least the relevant data of the adversarial simulation scenario.

[0172] Step S802: In response to the stress testing instruction, the decision big model is triggered to execute the decision stress testing task to perform update optimization to obtain an updated decision big model.

[0173] After receiving the stress testing instruction, the cloud device controls the LLM to perform a decision stress testing task with the adversarial simulation scenario as input in response to the stress testing instruction, thereby implementing a policy stress test on the LLM to update and optimize the LLM and obtain an updated LLM.

[0174] Step S803: Obtain the third environmental perception data output by the environmental perception big model to the updated decision big model, and output the third assisted driving decision of the vehicle to the environmental perception big model based on the updated decision big model and the third environmental perception data.

[0175] After the vehicle terminal sends a stress test command to the cloud device to trigger the LLM to execute the decision stress test task, when the LLM obtains an updated LLM based on the execution of the task, the vehicle terminal further processes the camera images inside and outside the vehicle, the radar point cloud, and the IMU data based on the VLM to obtain the scene text description caption, attention area attention map, reasoning path reasoning trace, and confidence information output by the VLM. The vehicle terminal then obtains the scene text description caption, attention area attention map, reasoning path reasoning trace, and confidence information output by the VLM as third environmental perception data, and transmits the third environmental perception data to the updated LLM on the cloud device. After receiving the third environmental perception data, the updated LLM integrates the data with the vehicle status data and user profile information to generate a vehicle control instruction. The vehicle control instruction is transmitted from the cloud device to the vehicle for execution as the third assisted driving decision for the vehicle.

[0176] In this embodiment, the LLM is optimized by reversely guiding the VLM through its own error analysis. Furthermore, the VLM automatically generates complex test environments based on bad cases to challenge the LLM, thereby optimizing the LLM. This approach implements a reverse reasoning mechanism, with the LLM guiding the VLM, and a bidirectional reflection mechanism, with the VLM generating virtual adversarial scenarios to stress-test the LLM. This overcomes the limitations of traditional perception and decision-making modules, which are limited by one-way transmission and difficulty in correction, and enables continuous learning and evolution of the vehicle assisted driving system.

[0177] Next, a complete embodiment of the vehicle assisted driving decision-making method provided in the embodiment of the present application is proposed.

[0178] In a complete embodiment of the vehicle assisted driving decision-making method, the vehicle assisted driving decision-making method provided in the embodiment of the present application can perform continuous learning and evolution of the intelligent driving system through an architecture composed of a VLM perception module, an LLM decision module, a Bad Case detector, a reflection mechanism module (LLM→VLM, VLM→LLM), and a communication protocol and data fusion module (for example, an intelligent vehicle edge-cloud collaborative large model architecture with bidirectional cross-layer reflection capabilities), thereby achieving accurate, efficient and stable vehicle assisted driving decision-making.

[0179] It should be noted that the VLM perception module, based on the VLM, takes in-vehicle and exterior camera images, radar point clouds, and IMU data as inputs. It performs CoT inference on this data to generate semantic information and perception confidence metrics, thereby outputting scene captions, attention maps, and reasoning traces as environmental perception data. This environmental perception data can be any of the first, second, or third environmental perception data described above. Furthermore, the LLM decision module, based on the LLM, takes in VLM outputs, vehicle status data, and user profile information as inputs. Through multimodal fusion, it makes complex traffic decisions and obtains vehicle control commands (such as steering angles, acceleration and deceleration decisions). These vehicle control commands can be any of the first, second, or third assisted driving decisions described above. Furthermore, the LLM decision module possesses reverse reasoning capabilities. The bad case detector, taking in LLM outputs, VLM confidence, and vehicle execution feedback (feedback data), employs an algorithm with multiple threshold rule functions (such as decision deviation and perception confidence) to comprehensively determine whether the assisted driving decisions output by the LLM are abnormal. The reflection mechanism module is designed to trigger the LLM→VLM reflection mechanism, which reversely generates attention guidance through the LLM and modifies the VLM attention vector A' = f_{LLM}(err_{trace}). It also triggers the VLM→LLM reflection mechanism to construct a simulation scenario S^ = Sim(VLM, BadCase) based on the VLM analysis of abnormal cases, and then re-verify the strategy by the LLM. The communication protocol and data fusion module is designed to transmit VLM output using the JSON+protobuf format, fusing multi-source information and standardizing and efficiently transmitting it.

[0180] Please refer to Figure 9 , Figure 9 This is a diagram of the VLM+LLM cross-layer bidirectional reflection technology architecture involved in a complete embodiment of the vehicle assisted driving decision-making method provided in an embodiment of the present application.

[0181] like Figure 9 As shown, the vehicle assisted driving decision-making method provided in the embodiment of the present application first performs multi-dimensional judgment through the above-mentioned BadCase detector. When the judgment on the assisted driving decision output by the LLM is that the decision deviation exceeds the threshold, the judgment on the environmental perception data output by the VLM is that the perception confidence is low, and / or the judgment on the execution effect of the assisted driving decision is that the execution effect is poor, the reflection mechanism is triggered to enter the subsequent cross-layer reflection workflow.

[0182] During the cross-layer reflection workflow, LLM-VLM reverse reasoning and VLM→LLM environment reconstruction can be performed simultaneously or separately. During the LLM-VLM reverse reasoning workflow, the LLM first performs decision tracing analysis to generate perception and attention guidance. The VLM then adjusts the VLM attention weights according to these perception and attention guidance, resulting in an updated VLM. The updated VLM then performs a new round of environmental perception. Furthermore, during the VLM→LLM environment reconstruction workflow, a virtual adversarial scenario is first constructed based on the VLM. The LLM then performs a decision stress test based on this virtual adversarial scenario, optimizing the LLM decision strategy and obtaining an improved LLM decision. The improved LLM decision then undergoes a new round of decision reasoning. The intelligent assisted driving system then makes a new assisted driving decision based on the results of this new round of environmental perception and decision reasoning. Furthermore, the bad case detector continues to monitor the system's assisted driving decisions to determine whether to trigger the reflection mechanism.

[0183] In some embodiments, during the reflective workflow of VLM to LLM environment reconstruction, while inputting virtual confrontation scenarios into the LLM for stress testing, the LLM's decision output (including path planning, game judgment, action control, etc.) under the new extreme scenario can also be observed. Furthermore, strategic weakness analysis and recording are performed on the LLM. Specifically, by continuously recording the LLM's performance (whether errors occur again, whether decision confidence decreases, and whether decision latency increases, etc.), a decision behavior log and anomaly diagnosis report are generated. This decision behavior log and anomaly diagnosis report can then be used for subsequent LLM strategy corrections or for future sample annotation training.

[0184] See also Figure 10 , an embodiment of the present application also provides a vehicle assisted driving decision-making device, which can implement the above-mentioned vehicle assisted driving decision-making method.

[0185] like Figure 10 As shown, the vehicle assisted driving decision-making device provided in the embodiment of the present application includes an acquisition module 1001, an abnormality recognition module 1002, a reflection module 1003 and a decision module 1004.

[0186] Acquisition module 1001, configured to acquire feedback data regarding a vehicle's execution of a first assisted driving decision and first environmental perception data; the first assisted driving decision is output by a decision-making model of a cloud device, and the first environmental perception data is output by the vehicle's environmental perception model, with the cloud device communicating with the vehicle;

[0187] an abnormality identification module 1002, configured to identify abnormalities in the first assisted driving decision based on the feedback data and the first environmental perception data, and obtain an abnormality identification result;

[0188] A reflection module 1003 is configured to, when the abnormality identification result indicates that the first assisted driving decision is abnormal, send a decision tracing instruction to the cloud device to trigger the decision large model to perform a decision tracing task, and update and optimize the environment perception large model based on the model update guidance signal fed back by the cloud device to obtain an updated environment perception large model; the model update guidance signal is generated by the decision large model performing the decision tracing task and output to the environment perception large model;

[0189] The decision module 1004 is used to perform the environmental perception task based on the updated environmental perception model, output the second environmental perception data to the decision model, and receive the second assisted driving decision of the vehicle output by the decision model in combination with the second environmental perception data.

[0190] In some embodiments, the reflection module 1003 is further configured to, when the abnormality recognition result indicates that the first assisted driving decision has an abnormality, construct an adversarial simulation scenario based on the environmental perception big model; and, based on the adversarial simulation scenario, send a stress testing instruction to the cloud device to trigger the decision big model to perform a decision stress testing task; the decision big model is updated and optimized based on the execution of the decision stress testing task to obtain an updated decision big model;

[0191] The decision module 1004 is also used to output third environmental perception data to the updated decision model based on the environmental perception model, and receive the third assisted driving decision of the vehicle output by the updated decision model in combination with the third environmental perception data.

[0192] In some embodiments, the reflection module 1003 is also used to perform an abnormal analysis on the assisted driving scene corresponding to the first assisted driving decision to obtain key interference factors in the assisted driving scene; and to construct an adversarial simulation scene based on the environmental perception large model combined with the key interference factors.

[0193] In some embodiments, the reflection module 1003 is also used to convert the adversarial simulation scenario into multimodal adversarial input data; and, using a preset heterogeneous data collaborative transmission method, send the multimodal adversarial input data as a stress testing instruction to the cloud device to trigger the decision model to perform a decision stress testing task based on the multimodal adversarial input data.

[0194] In some embodiments, the anomaly identification module 1002 is also used to perform multi-dimensional bad case detection based on the feedback data, the model confidence corresponding to the first environmental perception data, and the first assisted driving decision to obtain multiple bad case detection results; and, based on the multiple bad case detection results, perform anomaly identification on the first assisted driving decision.

[0195] In some embodiments, the anomaly identification module 1002 is further configured to perform bad case detection in a decision dimension based on the feedback data and the first assisted driving decision to obtain decision deviation data; the multiple bad case detection results include the decision deviation data; perform bad case detection in a perception dimension based on a model confidence corresponding to the first environmental perception data to obtain perception confidence data; the multiple bad case detection results also include the perception confidence data; compare the decision deviation data with a preset deviation threshold, and compare the perception confidence data with a preset confidence threshold to identify anomalies in the first assisted driving decision;

[0196] In which, when the decision deviation data is greater than the deviation threshold, and / or when the perception confidence data is less than the confidence threshold, an abnormality recognition result indicating that there is an abnormality in the first assisted driving decision is obtained.

[0197] In some embodiments, the reflection module 1003 is further configured to obtain a decision tracing trigger instruction sent by the vehicle; wherein, when the vehicle identifies an abnormality in the first assisted driving decision, the vehicle sends the decision tracing trigger instruction to the cloud device, and the first assisted driving decision is output by the decision big model of the cloud device; in response to the decision tracing trigger instruction, the decision big model is controlled to execute the decision tracing task to generate a model update guidance signal; and the model update guidance signal is sent to the vehicle, so that the vehicle updates and optimizes the environment perception big model based on the model update guidance signal to obtain an updated environment perception big model.

[0198] The decision module 1004 is also used to obtain the second environmental perception data output by the updated environmental perception model to the decision model, and output the second assisted driving decision of the vehicle to the environmental perception model based on the decision model and the second environmental perception data.

[0199] In some embodiments, the reflection module 1003 is further configured to obtain a stress test instruction sent by the vehicle; wherein, when the vehicle identifies an abnormality in the first assisted driving decision, it constructs an adversarial simulation scenario based on the environment perception big model, and sends the stress test instruction to the cloud device based on the adversarial simulation scenario; and, in response to the stress test instruction, it triggers the decision big model to execute a decision stress test task to update and optimize the decision big model to obtain an updated decision big model;

[0200] The decision module 1004 is also used to obtain the third environmental perception data output by the environmental perception big model to the updated decision big model, and output the third assisted driving decision of the vehicle to the environmental perception big model based on the updated decision big model and the third environmental perception data.

[0201] It should be noted that the specific implementation of the vehicle assisted driving decision-making device provided in the embodiment of the present application is basically the same as the specific implementation of the above-mentioned visual language model evaluation method, and will not be repeated here.

[0202] An embodiment of the present application also provides a decision-making device for vehicle assisted driving, which includes a memory and a processor. The memory stores a computer program, and the processor implements the above-mentioned visual language model evaluation method when executing the computer program.

[0203] In some embodiments, the decision-making device for vehicle assisted driving can be any smart terminal such as an on-board computer, a tablet computer, a smart phone, a wearable device, etc.

[0204] See also Figure 11 , Figure 11 The hardware structure of a vehicle-assisted driving decision-making device according to an embodiment is shown. The vehicle-assisted driving decision-making device includes:

[0205] The processor 1101 can be implemented as a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present application;

[0206] The memory 1102 can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 1102 can store an operating system and other application programs. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 1102 and is called by the processor 1101 to execute the decision-making method for vehicle assisted driving in the embodiments of this application;

[0207] Input / output interface 1103, used to implement information input and output;

[0208] Communication interface 1104, used to implement communication interaction between this device and other devices, which can be achieved through wired means (such as USB, network cable, etc.) or wireless means (such as mobile network, WiFi, Bluetooth, etc.);

[0209] Bus 1105 , which transmits information between various components of the device (e.g., processor 1101 , memory 1102 , input / output interface 1103 , and communication interface 1104 );

[0210] The processor 1101 , the memory 1102 , the input / output interface 1103 and the communication interface 1104 are connected to each other in communication within the device via a bus 1105 .

[0211] An embodiment of the present application also provides a vehicle, which is equipped with a vehicle-assisted driving decision-making device. The vehicle-assisted driving decision-making device includes a memory and a processor. The memory stores a computer program, and the processor implements the above-mentioned vehicle-assisted driving decision-making method when executing the computer program.

[0212] An embodiment of the present application also provides a cloud device, on which a decision-making device for vehicle assisted driving is configured. The decision-making device for vehicle assisted driving includes a memory and a processor. The memory stores a computer program, and the processor implements the above-mentioned decision-making method for vehicle assisted driving when executing the computer program.

[0213] An embodiment of the present application also provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, it implements the above-mentioned vehicle assisted driving decision-making method.

[0214] The memory, as a non-transient computer-readable storage medium, can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory may include a high-speed random access memory and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory may optionally include a memory remotely arranged relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0215] An embodiment of the present application also provides a computer program product, including a computer program. The steps implemented when the computer program is executed by a processor are basically the same as the specific embodiments of the above-mentioned vehicle assisted driving decision-making method, and will not be repeated here.

[0216] The embodiments described in the embodiments of this application are intended to more clearly illustrate the technical solutions of the embodiments of this application and do not constitute a limitation on the technical solutions provided by the embodiments of this application. Those skilled in the art will appreciate that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided in the embodiments of this application are also applicable to similar technical problems.

[0217] Those skilled in the art will understand that the technical solutions shown in the figures do not constitute a limitation on the embodiments of the present application, and may include more or fewer steps than shown in the figures, or a combination of certain steps, or different steps.

[0218] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, i.e., they may be located in one place or distributed across multiple network units. Some or all of the modules may be selected based on actual needs to achieve the objectives of this embodiment.

[0219] Those skilled in the art will appreciate that all or some of the steps in the methods, systems, and functional modules / units in the devices disclosed above may be implemented as software, firmware, hardware, or appropriate combinations thereof.

[0220] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequential order. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0221] It should be understood that in this application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the previous and next associated objects are in an "or" relationship. "At least one of the following items" or similar expressions refers to any combination of these items, including any combination of single items or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0222] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are merely schematic. For example, the division of the above-mentioned units is only a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0223] The units described above as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0224] In addition, the functional units in the various embodiments of the present application may be integrated into a single processing unit, or each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0225] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium and includes multiple instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of various embodiments of the present application. The aforementioned storage medium includes: various media that can store programs, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0226] The preferred embodiments of the present invention are described above with reference to the accompanying drawings, but are not intended to limit the scope of the present invention. Any modifications, equivalent substitutions, and improvements made by those skilled in the art without departing from the scope and essence of the present invention should be within the scope of the present invention.

Claims

1. A decision-making method for vehicle assisted driving, characterized in that: The method is applied to a vehicle, and comprises: Obtaining feedback data and first environmental perception data from the vehicle executing a first assisted driving decision; the first assisted driving decision is output by a decision-making model of a cloud device, and the first environmental perception data is output by the vehicle's environmental perception model, and the cloud device communicates with the vehicle; performing abnormality identification on the first assisted driving decision based on the feedback data and the first environmental perception data to obtain an abnormality identification result; When the abnormality identification result indicates that the first assisted driving decision has an abnormality, a decision tracing instruction is sent to the cloud device to trigger the decision large model to perform a decision tracing task, and the environment perception large model is updated and optimized based on the model update guidance signal fed back by the cloud device to obtain an updated environment perception large model; the model update guidance signal is generated by the decision large model performing the decision tracing task and output to the environment perception large model; Based on the updated environmental perception large model, the environmental perception task is executed to output second environmental perception data to the decision large model, and the second assisted driving decision of the vehicle output by the decision large model in combination with the second environmental perception data is received.

2. The method according to claim 1, characterized in that The method further comprises: When the abnormality recognition result indicates that the first assisted driving decision is abnormal, constructing an adversarial simulation scenario based on the environmental perception large model; Based on the adversarial simulation scenario, a stress testing instruction is sent to the cloud device to trigger the decision-making model to execute a decision stress testing task; the decision-making model is updated and optimized based on the execution of the decision stress testing task to obtain an updated decision-making model; Based on the environmental perception big model, the third environmental perception data is output to the updated decision big model, and the third assisted driving decision of the vehicle output by the updated decision big model in combination with the third environmental perception data is received.

3. The method according to claim 2, characterized in that The constructing of an adversarial simulation scenario based on the large environmental perception model includes: performing an abnormality analysis on the assisted driving scenario corresponding to the first assisted driving decision to obtain key interference factors in the assisted driving scenario; An adversarial simulation scenario is constructed based on the environmental perception large model and the key interference factors.

4. The method according to claim 2, characterized in that The sending of a stress testing instruction to the cloud device based on the adversarial simulation scenario to trigger the decision model to perform a decision stress testing task includes: Converting the adversarial simulation scenario into multimodal adversarial input data; The multimodal adversarial input data is sent as a stress testing instruction to the cloud device using a preset heterogeneous data collaborative transmission method to trigger the decision model to perform a decision stress testing task based on the multimodal adversarial input data.

5. The method according to claim 1, characterized in that The performing abnormality identification on the first assisted driving decision based on the feedback data and the first environmental perception data includes: performing multi-dimensional bad case detection based on the feedback data, the model confidence corresponding to the first environmental perception data, and the first assisted driving decision, to obtain multiple bad case detection results; Anomalies are identified for the first assisted driving decision based on the multiple bad case detection results.

6. The method according to claim 5, characterized in that The multi-dimensional bad case detection is performed based on the feedback data, the model confidence corresponding to the first environmental perception data, and the first assisted driving decision, to obtain multiple bad case detection results, including: performing bad case detection of a decision dimension based on the feedback data and the first assisted driving decision to obtain decision deviation data; wherein the multiple bad case detection results include the decision deviation data; performing bad case detection in a perception dimension based on the model confidence corresponding to the first environmental perception data to obtain perception confidence data; the multiple bad case detection results also include the perception confidence data; The performing abnormality identification on the first assisted driving decision based on the multiple bad case detection results includes: Comparing the decision deviation data with a preset deviation threshold, and comparing the perception confidence data with a preset confidence threshold, to identify abnormalities in the first assisted driving decision; In which, when the decision deviation data is greater than the deviation threshold, and / or when the perception confidence data is less than the confidence threshold, an abnormality recognition result indicating that there is an abnormality in the first assisted driving decision is obtained.

7. A decision-making method for vehicle assisted driving, characterized in that: The method is applied to a cloud device that communicates with a vehicle, and the method includes: Obtaining a decision tracing trigger instruction sent by the vehicle; wherein the vehicle sends the decision tracing trigger instruction to the cloud device when it identifies an abnormality in the first assisted driving decision, the first assisted driving decision being output by the decision model of the cloud device; In response to the decision tracing trigger instruction, the decision big model is controlled to execute the decision tracing task and generate a model update guidance signal; Sending the model update guidance signal to the vehicle, so that the vehicle updates and optimizes the environment perception large model based on the model update guidance signal to obtain an updated environment perception large model; The second environmental perception data output by the updated environmental perception large model to the decision-making large model is obtained, and a second assisted driving decision of the vehicle is output to the environmental perception large model based on the decision-making large model in combination with the second environmental perception data.

8. The method according to claim 7, characterized in that The method further comprises: Obtaining a stress test instruction sent by the vehicle; wherein, when the vehicle identifies that there is an abnormality in the first assisted driving decision, constructing an adversarial simulation scenario based on the environmental perception large model, and sending the stress test instruction to the cloud device based on the adversarial simulation scenario; In response to the stress testing instruction, triggering the decision big model to execute the decision stress testing task to perform updating and optimization to obtain an updated decision big model; Obtain the third environmental perception data output by the environmental perception big model to the updated decision big model, and output the third assisted driving decision of the vehicle to the environmental perception big model based on the updated decision big model and the third environmental perception data.

9. A vehicle assisted driving decision-making device, characterized in that: The device comprises: an acquisition module, configured to acquire feedback data of the vehicle executing a first assisted driving decision and first environmental perception data; the first assisted driving decision being output by a decision-making model of a cloud device, and the first environmental perception data being output by the vehicle's environmental perception model, the cloud device communicating with the vehicle; an abnormality identification module, configured to identify abnormalities in the first assisted driving decision based on the feedback data and the first environmental perception data, and obtain an abnormality identification result; a reflection module, configured to, when the abnormality identification result indicates that the first assisted driving decision is abnormal, send a decision tracing instruction to the cloud device to trigger the decision large model to perform a decision tracing task, and update and optimize the environment perception large model based on the model update guidance signal fed back by the cloud device to obtain an updated environment perception large model; the model update guidance signal is generated by the decision large model performing the decision tracing task and output to the environment perception large model; A decision module is used to perform an environmental perception task based on the updated environmental perception model, output second environmental perception data to the decision model, and receive a second assisted driving decision of the vehicle output by the decision model in combination with the second environmental perception data.

10. A vehicle assisted driving decision-making device, characterized in that: The vehicle assisted driving decision-making device includes a memory and a processor, the memory stores a computer program, and the processor implements the vehicle assisted driving decision-making method according to any one of claims 1 to 8 when executing the computer program.

11. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, which, when executed by a processor, implements the vehicle assisted driving decision-making method according to any one of claims 1 to 8.

12. A computer program product, characterized in that The computer program product includes a computer program, and when the computer program is executed by a processor, the decision-making method for vehicle assisted driving according to any one of claims 1 to 8 is implemented.

Images