Decision-making method, device and equipment for vehicle auxiliary driving, storage medium and product

Abstract

The invention provides a decision-making method, device and equipment for vehicle aided driving, a storage medium and a product, and relates to the technical field of vehicles, and the method comprises the steps: obtaining feedback data and first environment perception data of a vehicle executing a first aided driving decision; performing anomaly recognition on the first auxiliary driving decision based on the feedback data and the first environment perception data to obtain an anomaly recognition result; when the abnormity identification result indicates that the first auxiliary driving decision is abnormal, triggering a decision large model to execute a decision traceability task to generate a model updating guide signal, and updating and optimizing an environment perception large model based on the model updating guide signal to obtain an updated environment perception large model; and outputting second environment perception data to the decision-making large model based on the updated environment perception large model, and outputting a second auxiliary driving decision of the vehicle by the decision-making large model in combination with the second environment perception data. According to the invention, the accuracy of aided driving decision making in a complex scene can be improved.

Description

Technical Field

[0001] This application relates to the technical field of vehicles, and in particular, to a decision-making method, device, equipment, storage medium and product for vehicle assisted driving. Background Art

[0002] The decision-making of vehicle assisted driving needs to make decisions quickly in the face of complex and changeable traffic environments (such as multi-objective games, scenarios of mixed traffic of people and vehicles, emergencies, etc.), which puts higher requirements on the environmental perception, real-time reasoning and overall robustness of the vehicle assisted driving system.

[0003] In related technologies, large language models (LLMs) and visual-language models (VLMs) have been gradually introduced into the field of vehicle assisted driving due to their excellent generalization ability and reasoning ability. The vehicle assisted driving system can effectively improve the perception interpretability and decision-making flexibility with the help of large models. However, the decision-making of the vehicle assisted driving system based on large models still mainly focuses on static perception and static decision-making, which causes the system to be very prone to decision-making errors due to perception information deviation in dynamic complex scenarios. Summary of the Invention

[0004] The main purpose of the embodiments of this application is to propose a decision-making method, device, equipment, storage medium and product for vehicle assisted driving, aiming to make up for the perception information deviation of the vehicle assisted driving system in dynamic complex scenarios, so as to improve the decision-making accuracy of the system in complex scenarios.

[0005] To achieve the above object, the first aspect of the embodiments of this application proposes a decision-making method for vehicle assisted driving, which is applied to a vehicle. The method includes: Obtain the feedback data of the vehicle executing the first assisted driving decision and the first environmental perception data; the first assisted driving decision is output by the decision-making large model of the cloud device, and the first environmental perception data is output by the environmental perception large model of the vehicle. The cloud device communicates with the vehicle; Perform anomaly identification on the first assisted driving decision based on the feedback data and the first environmental perception data to obtain an anomaly identification result; In the case where the anomaly identification result indicates that the first assisted driving decision is abnormal, send a decision traceability instruction to the cloud device to trigger the decision-making large model to execute a decision traceability task, and update and optimize the environmental perception large model based on the model update guidance signal fed back by the cloud device to obtain an updated environmental perception large model; the model update guidance signal is generated by the decision-making large model executing the decision traceability task and output to the environmental perception large model; Execute an environment perception task based on the updated environment perception large model, output second environment perception data to the decision-making large model, and receive a second assisted driving decision of the vehicle output by the decision-making large model in combination with the second environment perception data.

[0006] In some embodiments, the method further includes: When the anomaly recognition result indicates that the first assisted driving decision is abnormal, construct an adversarial simulation scenario based on the environment perception large model; Send a stress test instruction to the cloud device based on the adversarial simulation scenario to trigger the decision-making large model to execute a decision stress test task; the decision-making large model is updated and optimized based on executing the decision stress test task to obtain an updated decision-making large model; Output third environment perception data to the updated decision-making large model based on the environment perception large model, and receive a third assisted driving decision of the vehicle output by the updated decision-making large model in combination with the third environment perception data.

[0007] In some embodiments, the constructing an adversarial simulation scenario based on the environment perception large model includes: Perform anomaly analysis on the assisted driving scenario corresponding to the first assisted driving decision to obtain key interference factors in the assisted driving scenario; Construct an adversarial simulation scenario based on the environment perception large model in combination with the key interference factors.

[0008] In some embodiments, the sending a stress test instruction to the cloud device based on the adversarial simulation scenario to trigger the decision-making large model to execute a decision stress test task includes: Convert the adversarial simulation scenario into multi-modal adversarial input data; Use a preset heterogeneous data collaborative transmission method to send the multi-modal adversarial input data to the cloud device as a stress test instruction to trigger the decision-making large model to execute a decision stress test task based on the multi-modal adversarial input data.

[0009] In some embodiments, the anomaly recognition of the first assisted driving decision based on the feedback data and the first environment perception data includes: Perform multi-dimensional bad case detection based on the feedback data, the model confidence corresponding to the first environment perception data, and the first assisted driving decision to obtain multiple bad case detection results; Perform anomaly recognition on the first assisted driving decision based on the multiple bad case detection results.

[0010] In some embodiments, multi-dimensional bad case detection is performed based on the feedback data, the model confidence corresponding to the first environmental perception data, and the first assisted driving decision, and multiple bad case detection results are obtained, including: Performing bad case detection in the decision dimension based on the feedback data and the first assisted driving decision to obtain decision deviation data; the multiple bad case detection results include the decision deviation data; Performing bad case detection in the perception dimension based on the model confidence corresponding to the first environmental perception data to obtain perception confidence data; the multiple bad case detection results further include the perception confidence data; The abnormal identification of the first assisted driving decision based on the multiple bad case detection results includes: Comparing the decision deviation data with a preset deviation threshold, and comparing the perception confidence data with a preset confidence threshold to perform abnormal identification of the first assisted driving decision; Wherein, in the case that the decision deviation data is greater than the deviation threshold, and / or in the case that the perception confidence data is less than the confidence threshold, an abnormal identification result indicating that the first assisted driving decision is abnormal is obtained.

[0011] To achieve the above object, a second aspect of the embodiments of the present application proposes another decision method for vehicle assisted driving. The method is applied to a cloud device in communication with a vehicle, and the method includes: Obtaining a decision traceability trigger instruction sent by the vehicle; wherein, the vehicle sends the decision traceability trigger instruction to the cloud device when it identifies that the first assisted driving decision is abnormal, and the first assisted driving decision is output by the decision large model of the cloud device; Controlling the decision large model to execute a decision traceability task to generate a model update guidance signal in response to the decision traceability trigger instruction; Sending the model update guidance signal to the vehicle for the vehicle to update and optimize the environmental perception large model based on the model update guidance signal to obtain an updated environmental perception large model; Obtaining second environmental perception data output by the updated environmental perception large model to the decision large model, and outputting a second assisted driving decision of the vehicle to the environmental perception large model based on the decision large model in combination with the second environmental perception data.

[0012] In some embodiments, the method further includes: Obtain the stress test instruction sent by the vehicle; wherein, when the vehicle recognizes that the first assisted driving decision is abnormal, it constructs an adversarial simulation scenario based on the environmental perception large model, and sends the stress test instruction to the cloud device based on the adversarial simulation scenario; In response to the stress test instruction, trigger the decision large model to execute a decision stress test task for update and optimization to obtain an updated decision large model; Obtain the third environmental perception data output by the environmental perception large model to the updated decision large model, and based on the updated decision large model and in combination with the third environmental perception data, output the third assisted driving decision of the vehicle to the environmental perception large model.

[0013] To achieve the above object, a third aspect of the embodiments of the present application proposes a decision-making device for vehicle assisted driving, and the device includes: An acquisition module, configured to acquire feedback data and first environmental perception data of the vehicle executing a first assisted driving decision; the first assisted driving decision is output by a decision large model of a cloud device, the first environmental perception data is output by an environmental perception large model of the vehicle, and the cloud device communicates with the vehicle; An anomaly recognition module, configured to perform anomaly recognition on the first assisted driving decision based on the feedback data and the first environmental perception data to obtain an anomaly recognition result; A reflection module, configured to, when the anomaly recognition result indicates that the first assisted driving decision is abnormal, send a decision traceability instruction to the cloud device to trigger the decision large model to execute a decision traceability task, and update and optimize the environmental perception large model based on the model update guidance signal fed back by the cloud device to obtain an updated environmental perception large model; the model update guidance signal is generated by the decision large model executing the decision traceability task and output to the environmental perception large model; A decision module, configured to execute an environmental perception task based on the updated environmental perception large model to output second environmental perception data to the decision large model, and receive the second assisted driving decision of the vehicle output by the decision large model in combination with the second environmental perception data.

[0014] To achieve the above object, a fourth aspect of the embodiments of the present application proposes a decision-making device for vehicle assisted driving, and the decision-making device for vehicle assisted driving includes a memory and a processor, the memory stores a computer program, and when the processor executes the computer program, it implements the decision-making method for vehicle assisted driving described in the first aspect and / or the second aspect above.

[0015] To achieve the above object, a fifth aspect of the embodiments of the present application provides a vehicle, on which a decision-making device for vehicle assisted driving is configured. The decision-making device for vehicle assisted driving includes a memory and a processor. The memory stores a computer program, and when the processor executes the computer program, the decision-making method for vehicle assisted driving described in the first aspect above is implemented.

[0016] To achieve the above object, a sixth aspect of the embodiments of the present application provides a cloud device, on which a decision-making device for vehicle assisted driving is configured. The decision-making device for vehicle assisted driving includes a memory and a processor. The memory stores a computer program, and when the processor executes the computer program, the decision-making method for vehicle assisted driving described in the second aspect above is implemented.

[0017] To achieve the above object, a seventh aspect of the embodiments of the present application provides a computer-readable storage medium storing a computer program, and when the computer program is executed by a processor, the decision-making method for vehicle assisted driving described in the first aspect and / or the second aspect above is implemented.

[0018] To achieve the above object, an eighth aspect of the embodiments of the present application provides a computer program product including a computer program, and when the computer program is executed by a processor, the decision-making method for vehicle assisted driving provided in the first aspect and / or the second aspect above is implemented.

[0019] The decision-making method, device, equipment, vehicle, cloud device, computer-readable storage medium, and computer program product provided by the present application obtain feedback data and first environmental perception data for the vehicle to execute the first assisted driving decision; the first assisted driving decision is output by the decision-making large model of the cloud device, and the first environmental perception data is output by the environmental perception large model of the vehicle, and the cloud device communicates with the vehicle; based on the feedback data and the first environmental perception data, perform anomaly identification on the first assisted driving decision to obtain an anomaly identification result; in the case where the anomaly identification result indicates that the first assisted driving decision is abnormal, send a decision traceability instruction to the cloud device to trigger the decision-making large model to execute a decision traceability task, and update and optimize the environmental perception large model based on the model update guidance signal fed back by the cloud device to obtain an updated environmental perception large model; the model update guidance signal is generated by the decision-making large model executing the decision traceability task and output to the environmental perception large model; perform an environmental perception task based on the updated environmental perception large model and output second environmental perception data of the vehicle to the decision-making large model, and receive the second assisted driving decision of the vehicle output by the decision-making large model in combination with the second environmental perception data.

[0020] Compared with the vehicle assisted driving decision-making method in the related art that adopts a similar edge-cloud architecture but mainly focuses on static perception and static decision-making, the embodiment of the present application determines whether there is an abnormality in the first assisted driving decision output by the decision-making large model deployed on the cloud device, and when it is recognized that there is an abnormality in the decision, triggers the decision-making large model to execute a decision traceability task to generate a model update guidance signal, and updates and optimizes the environment perception large model deployed on the vehicle local based on this signal to obtain an updated environment perception large model. Then, based on the updated environment perception large model, second environment perception data is output to the decision-making large model, so as to generate a second assisted driving decision for the vehicle based on this decision-making large model in combination with the second environment perception data. In this way, the embodiment of the present application triggers the decision-making large model to execute a decision traceability task to analyze its own errors and generate a model update guidance signal to reversely guide the update and optimization of the environment perception large model, which can significantly improve the target detection and scene understanding capabilities of the environment perception large model in occluded or multi-vehicle complex scenarios, thereby effectively compensating for the perception information deviation of the vehicle assisted driving system in dynamic complex scenarios and improving the accuracy of the system's decision-making in complex scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 It is a schematic flowchart of the steps of the decision-making method for vehicle assisted driving provided by the embodiment of the present application in some embodiments; Figure 2 It is a schematic flowchart of the steps of the decision-making method for vehicle assisted driving provided by the embodiment of the present application in some other embodiments; Figure 3 is Figure 2 a schematic flowchart of the detailed steps of step S201 in; Figure 4 is Figure 2 a schematic flowchart of the detailed steps of step S202 in; Figure 5 is Figure 1 a schematic flowchart of the detailed steps of step S102 in; Figure 6 is Figure 5 a schematic flowchart of the detailed steps of step S501 in; Figure 7 It is a schematic flowchart of the steps of the decision-making method for vehicle assisted driving provided by the embodiment of the present application in some other embodiments; Figure 8 It is a schematic flowchart of the steps of the decision-making method for vehicle assisted driving provided by the embodiment of the present application in some other embodiments; Figure 9 It is a VLM+LLM cross-layer bidirectional reflection technical architecture diagram involved in a complete embodiment of the decision-making method for vehicle assisted driving provided by the embodiment of the present application; Figure 10 The structural schematic diagram of the decision-making device for vehicle assisted driving provided by the embodiment of the present application; Figure 11 It is the hardware structural schematic diagram of the decision-making device for vehicle assisted driving provided by the embodiment of the present application. Specific embodiments

[0022] In order to make the purpose, technical solutions and advantages of the present application more clear, the present application will be further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0023] It should be noted that although the functional modules are divided in the device schematic diagram and the logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in a different order from the module division in the device or the flowchart. Terms such as "first" and "second" in the specification, claims and the above-mentioned drawings are used to distinguish similar objects and do not necessarily need to describe a specific order or sequence.

[0024] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the technical field to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0025] First, the overall concept of the decision-making method for vehicle assisted driving provided by the embodiment of the present application will be described.

[0026] With the rapid development of vehicle assisted driving technology, the decision-making system in vehicle assisted driving has gradually become the most challenging part of the core functional modules.

[0027] The decision-making of vehicle assisted driving needs to make decisions quickly in the face of complex and changeable traffic environments (such as multi-objective games, scenarios of mixed traffic of people and vehicles, emergencies, etc.), which puts higher requirements on the environmental perception, real-time reasoning and overall robustness of the vehicle assisted driving system.

[0028] In the related art, the large language model LLM and the vision language model VLM have been gradually introduced into the field of vehicle assisted driving due to their excellent generalization ability and reasoning ability. The vehicle assisted driving system can effectively improve the perception interpretability and decision-making flexibility with the help of large models. Among them, the vision language model VLM is a multi-modal model with image understanding and natural language generation capabilities, and the large language model LLM is a natural language processing model based on the deep learning model Transformer architecture.

[0029] However, the computing power consumption and latency issues brought about by the deployment of large models have restricted their full application in the vehicle field.

[0030] Edge-Cloud Collaboration (ECC) has become an important direction to address this bottleneck. Edge-Cloud Collaboration (ECC) refers to an intelligent architecture that achieves data processing and decision optimization through the collaborative operation of the edge side (in-vehicle VLM) and the cloud side (high-performance LLM), taking into account both real-time performance and computing power efficiency. By deploying lightweight VLM on the vehicle edge side to sense the environment in real time and high-performance LLM on cloud devices or high-computing-power in-vehicle computing platforms for complex reasoning, the overall performance and response efficiency of the system can be effectively improved. At the same time, in order to overcome the problems of knowledge aging and decreased generalization ability of the model during long-term operation, introducing a self-learning mechanism with "reflective ability" has become a new trend in current intelligent driving research.

[0031] However, most of the existing edge-cloud architectures are mainly based on static perception and static decision-making, resulting in the system being difficult to continuously evolve and optimize in dynamic complex scenarios, and thus it is very easy to make decision errors due to perception information deviation in dynamic complex scenarios. In complex scenarios, in-vehicle sensors are limited by weather, occlusion, or blind spots, and the perception results of VLM are prone to deviation, which in turn affects the decision-making reasoning of LLM. For example, in rainy or snowy weather, the camera is blurred, resulting in VLM generating an incorrect scene text description caption, and LLM makes a misjudgment based on this, inducing the risk of vehicle collision.

[0032] Based on this, the embodiments of the present application provide a decision-making method, device, equipment, vehicle, cloud device, computer-readable storage medium, and computer program product for vehicle assisted driving, aiming to overcome the deficiencies of the above related technologies. By establishing a mechanism to identify and correct perception errors, that is, based on triggering LLM to execute a decision tracing task to analyze its own errors and generate a model update guidance signal to reverse-guide the update and optimization of VLM, the target detection and scene understanding capabilities of VLM can be significantly improved in occlusion or multi-vehicle complex scenarios. In this way, the perception information deviation of the vehicle assisted driving system in dynamic complex scenarios can be effectively compensated, and the accuracy of vehicle assisted driving decision-making in complex scenarios can be improved.

[0033] It should be noted that in the embodiments of the present application, VLM is responsible for perceiving the internal and external environment of the vehicle, and can output an environment description caption (or called scene text description), attention distribution, and reasoning path through Chain of Thought (CoT) reasoning. In addition, LLM is responsible for fusing the output of VLM, vehicle state information, and user portrait, making decisions on vehicle assisted driving, and has the ability of self-reflection.

[0034] In addition, considering that the "perception" and "decision-making" of the system in the related art are in a serial relationship, that is, after the VLM outputs the caption, it is transmitted to the LLM. In this way, the lack of reverse correction ability makes it impossible to realize the dynamic adjustment of the VLM perception weight by the LLM decision feedback. For example, in the case of a failed deflection turn, the LLM cannot point out the perception blind spot to the VLM, and long-term operation is likely to form system "blind vision". Therefore, the embodiment of the present application also introduces a perception-decision closed-loop feedback mechanism, which realizes the perception-decision closed-loop feedback and co-evolution inside the model based on the cross-layer reflection mechanism, and improves the adaptability and long-term robustness of the system.

[0035] Furthermore, considering that the update cost is high after the current large model is deployed, and it lacks self-evaluation and self-learning capabilities, performance degradation is likely to occur when facing new scenarios during long-term operation. For example, in the frequent low-speed congestion scenarios in urban narrow roads, the model needs to continuously adjust strategies, while traditional static models are difficult to adaptively evolve. In response to this, the embodiment of the present application also designs a collaborative system with reflection ability and continuous learning ability. That is, by the LLM initiating guidance and reverse tuning to the VLM, the dynamic perception focus migration is realized, and the VLM automatically generates a virtual adversarial environment to perform stress testing on the LLM, thereby promoting the optimization of the LLM strategy. Through this mechanism, the perception error can be effectively compensated, the model decision logic can be optimized, and the system self-learning can be realized, solving the problem of the closed-loop breakage of "misperception-misdecision-no feedback" in the traditional system. Moreover, the embodiment of the present application combines the lightweight VLM to achieve efficient environmental perception at the edge and the high-performance LLM to perform complex decision-making reasoning in the cloud, constructs a two-way reflection mechanism of perception-decision linkage, and can also realize the adaptive evolution and robustness enhancement of the system by real-time identifying abnormal perception or decision-making and performing reverse optimization and virtual adversarial scenario construction, breaking the limitation of the traditional perception and decision-making modules of "one-way transmission and difficult to correct", so as to realize the continuous learning and evolution of the vehicle assisted driving system.

[0036] Next, the decision method, device, equipment, vehicle, cloud device, computer-readable storage medium, and computer program product for vehicle assisted driving provided by the embodiments of the present application will be specifically described through the following embodiments, and first, the decision method for vehicle assisted driving provided by the embodiments of the present application will be described in detail.

[0037] It should be noted that in each specific embodiment of the present application, when it comes to relevant processing based on data related to the user's identity or characteristics, such as user information, user behavior data, user historical data, and user location information, the user's permission or consent will be obtained first. Moreover, the collection, use, and processing of these data will comply with relevant laws, regulations, and standards. In addition, when the embodiments of the present application need to obtain the user's sensitive personal information, the user's separate permission or separate consent will be obtained through methods such as pop-up windows or redirecting to a confirmation page. After clearly obtaining the user's separate permission or separate consent, the necessary user-related data for the normal operation of the embodiments of the present application will be obtained.

[0038] It should be noted that the decision-making method for vehicle assisted driving provided by the embodiments of the present application can be applied to a terminal, can also be applied to a server side, or can be software running on a terminal or a server side. In some embodiments, the terminal can be an in-vehicle terminal on a vehicle, a high-computing-power in-vehicle computing platform, or can also be a computer device such as a smart phone, a tablet computer, a notebook computer, or a desktop computer associated with the vehicle. The association between the terminal and the vehicle means that the terminal can perform communication data interaction with the vehicle based on a network. The server side can be a background server terminal device of the vehicle, such as a cloud device communicating with the vehicle, etc., and can be configured as an independent physical server, or can be configured as a server cluster or a distributed system composed of multiple physical servers, or can also be configured as a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, Content Delivery Network (CDN), and big data and artificial intelligence platforms. The software can be an application implementing the decision-making method for vehicle assisted driving, a computer program, and a storage medium carrying the computer program, etc. It should be understood that based on different design requirements of actual applications, in different feasible embodiments, the terminal, server side, and software applying the decision-making method for vehicle assisted driving provided by the embodiments of the present application, of course, can also be other forms not listed here, and the decision-making method for vehicle assisted driving provided by the embodiments of the present application does not specifically limit this.

[0039] In addition, the present application can also be used in numerous general-purpose or special-purpose computer system environments or configurations. For example: vehicles, personal computers, server computers, handheld devices or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer computer devices, personal computers (PCs), minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, and so on. The present application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present application can also be practiced in a distributed computing environment where tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media including storage devices.

[0040] For ease of understanding and elaboration, in the following text, taking the in-vehicle terminal (directly configured on a vehicle or associated with a vehicle) and the cloud device applying the decision-making method for vehicle assisted driving provided by the embodiments of the present application as an example, each specific embodiment of the present application will be described in detail. The implementation of the decision-making method for vehicle assisted driving provided by the embodiments of the present application for any of the above forms of the subject can refer to the process of the in-vehicle terminal and / or the cloud device applying the decision-making method for vehicle assisted driving described later.

[0041] Please refer to Figure 1 , Figure 1 which is a schematic flowchart of the steps in some embodiments of the decision-making method for vehicle assisted driving provided by the embodiments of the present application. It should be understood that although Figure 1 and the subsequent other schematic flowcharts show the execution order of some method steps, due to different design requirements in actual applications, the decision-making method for vehicle assisted driving provided by the embodiments of the present application can of course adopt an execution order different from that shown in the figures. That is, Figure 1 the order of the shown method steps does not constitute a limitation on the execution logic order of the decision-making method for vehicle assisted driving provided by the embodiments of the present application, and any reasonable changes based on Figure 1 the order of the shown method steps should be included within the protection scope of the decision-making method for vehicle assisted driving provided by the embodiments of the present application.

[0042] As Figure 1 shown, in some embodiments, when the decision-making method for vehicle assisted driving provided by the embodiments of the present application is applied to a vehicle, it can include steps S101 to S104 as shown below.

[0043] Step S101: Obtain the feedback data and the first environmental perception data for the vehicle to execute the first assisted driving decision; the first assisted driving decision is output by the decision-making large model of the cloud device, and the first environmental perception data is output by the environmental perception large model of the vehicle. The cloud device communicates with the vehicle.

[0044] It should be noted that the decision-making large model can be the above-mentioned LLM, and the environmental perception large model can be the above-mentioned VLM. For the convenience of description and understanding, the decision-making large model will be replaced by LLM and the environmental perception large model will be replaced by VLM in the following detailed description of the embodiments. In addition, when the cloud device deploys the decision-making LLM and VLM using the edge-cloud architecture, the cloud device can be the cloud server or the high-computing power vehicle-mounted computing platform in the edge-cloud architecture. The vehicle can be the vehicle itself or the vehicle-mounted terminal configured on the vehicle. In the following, it will be uniformly described as the vehicle-mounted terminal.

[0045] When the vehicle-mounted terminal makes an intelligent decision on vehicle assisted driving by combining the local VLM and the LLM of the cloud device, it first processes data such as in-vehicle and out-of-vehicle camera images, radar point clouds, and inertial measurement unit (IMU) data through the local lightweight VLM to obtain the scene text description caption, attention area attention map, reasoning path reasoning trace, and confidence information output by the VLM. Then, the vehicle-mounted terminal obtains the scene text description caption, attention area attention map, reasoning path reasoning trace, and confidence information output by the VLM as the first environmental perception data, and transmits the first environmental perception data to the LLM of the cloud device.

[0046] After receiving the first environmental perception data, the LLM fuses the data with the vehicle state data and the user portrait information to generate a vehicle control instruction (such as steering angle, acceleration and deceleration decision, etc.). This vehicle control instruction is the first assisted driving decision for the vehicle, and the cloud device then transmits the first assisted driving decision to the vehicle for execution.

[0047] During the process of the vehicle-mounted terminal executing the first assisted driving decision, it continuously obtains the feedback data of the vehicle executing the first assisted driving decision (such as too large / small steering angle, too sharp / insufficient deceleration, etc.).

[0048] Step S102: Based on the feedback data and the first environmental perception data, perform anomaly recognition on the first assisted driving decision to obtain an anomaly recognition result.

[0049] After the vehicle-mounted terminal obtains the first environmental perception data output by the VLM and the feedback data of the vehicle executing the first assisted driving decision, it combines the first environmental perception data and the feedback data to perform anomaly identification on the first assisted driving decision, so as to obtain an anomaly identification result for the first assisted driving decision.

[0050] In some embodiments, the vehicle-mounted terminal can perform anomaly identification on the first assisted driving decision through a Bad Case detector. As a module for detecting model output anomalies, the Bad Case detector can comprehensively judge whether there is an error in the first assisted driving decision compared with the ideal (reference) decision based on three multi-dimensional indicators: perception confidence (obtained based on the first environmental perception data), decision deviation (obtained based on the first assisted driving decision), and execution feedback (obtained based on the feedback data). Among them, the ideal (reference) decision can be derived from preset rules or expert experience.

[0051] Step S103: When the anomaly identification result indicates that the first assisted driving decision is abnormal, send a decision traceability instruction to the cloud device to trigger the decision-making large model to execute the decision traceability task, and update and optimize the environmental perception large model based on the model update guidance signal fed back by the cloud device to obtain an updated environmental perception large model; the model update guidance signal is generated by the decision-making large model executing the decision traceability task and output to the environmental perception large model.

[0052] After the vehicle-mounted terminal performs anomaly identification on the first assisted driving decision and obtains an anomaly identification result, if the anomaly identification result indicates that the first assisted driving decision is abnormal, for example, there is an error in the first assisted driving decision compared with the ideal (reference) decision, the vehicle-mounted terminal immediately sends a decision traceability instruction to the cloud device. After receiving the decision traceability instruction, the cloud device immediately responds to the instruction to control the LLM to execute the decision traceability task. During the execution of the decision traceability task, the LLM performs its own error analysis and generates a model update guidance signal for guiding the update and optimization of the reverse guidance VLM and outputs it. After the cloud device receives the model update guidance signal output by the LLM, it sends the model update guidance signal to the vehicle-mounted terminal. The vehicle-mounted terminal controls the VLM to be updated and optimized based on the model update guidance signal, so as to obtain an updated VLM.

[0053] In some embodiments, the model update guidance signal can be an attention guidance vector for the LLM to perform its own error analysis during the decision traceability task and reverse guide the VLM to adjust the attention area. When the vehicle-mounted terminal updates and optimizes the VLM based on the model update guidance signal, the VLM can adjust the original attention mechanism based on the attention guidance vector to obtain an updated VLM.

[0054] Exemplarily, when the in-vehicle terminal identifies that there is an abnormality in the first assisted driving decision through the Bad Case detector, it triggers the LLM to enter the "decision traceability" mode by sending a decision traceability instruction to the cloud device.

[0055] In this "decision traceability" mode, the LLM executes the decision traceability task to perform a reverse analysis of its own decision-making process, and determines which information in the input (such as the first environmental perception data output by the VLM, vehicle status data, user profile information, etc.) plays a key role in the decision. Among them, the LLM can use Attention-based Attribution to analyze which information plays a key role in the decision. Specifically, techniques such as Integrated Gradients (IG), SHapley Additive exPlanations (SHAP), or Transformer Attention Weight of the self-attention weights of the deep learning model can be used to quantify the influence degree of the input features (i.e., the first environmental perception data output by the VLM, vehicle status data, user profile information, etc.).

[0056] During the process of the LLM performing a reverse analysis of its own decision-making process, it generates an attention guidance vector. That is, the LLM maps the focus of attention to a set of attention guidance vectors vec{a}, which are in the form of: , where (x i , y i ) represents the coordinates of the area in the image that should be focused on, and w i represents the attention intensity corresponding to the area coordinates.

[0057] After the LLM generates the attention guidance vector, it outputs this attention guidance vector as a model update guidance signal to the VLM. That is, the cloud device sends the attention vector vec{a} to the VLM module through a communication protocol (such as a communication protocol based on the JSON+protobuf format), and attaches instruction information (such as descriptions of the target object, color, motion characteristics, etc.).

[0058] After the VLM receives the model update guidance signal sent from the LLM side, it fuses the guidance vector into the original attention area attention map and adjusts the original attention mechanism according to: where Attention' is the adjusted attention mechanism, Attention is the original attention mechanism, is a dynamic adjustment coefficient used to balance the original attention and the guided attention.

[0059] In some embodiments, after the vehicle terminal performs anomaly recognition on the first assisted driving decision and obtains an anomaly recognition result, if the anomaly recognition result indicates that the first assisted driving decision is normal, for example, the first assisted driving decision has no error compared to the ideal (reference) decision, the vehicle terminal will continue to jointly use the VLM and LLM to make a new round of intelligent decisions for vehicle assisted driving, and continue to execute the above processes of obtaining data and performing anomaly recognition.

[0060] Step S104: Perform an environment perception task based on the updated environment perception large model, output second environment perception data to the decision large model, and receive the second assisted driving decision of the vehicle output by the decision large model in combination with the second environment perception data.

[0061] After the vehicle terminal obtains the updated VLM, it continues to perform the environment perception task based on the updated VLM, and performs subsequent decision iteration or training playback based on the results output by the updated VLM. That is, based on the updated VLM, continue to process the in-vehicle and out-of-vehicle camera images, radar point clouds, and IMU data to obtain the scene text description caption, attention area attention map, reasoning path reasoning trace, and confidence information output by the updated VLM. Then, the vehicle terminal obtains the scene text description caption, attention area attention map, reasoning path reasoning trace, and confidence information output by the updated VLM as the second environment perception data, and transmits the second environment perception data to the LLM of the cloud device. After receiving the second environment perception data, the LLM fuses the data with the vehicle state data and user profile information to generate a vehicle control instruction, and the vehicle control instruction, as the second assisted driving decision for the vehicle, is transmitted by the cloud device to the vehicle for execution.

[0062] In the embodiments of the present application, the in-vehicle terminal is used to obtain the scenario text description caption, attention area attention map, reasoning trace, and confidence information output by the VLM as the first environmental perception data. Moreover, during the process of the vehicle executing the first assisted driving decision, the in-vehicle terminal continuously obtains the feedback data of the vehicle executing the first assisted driving decision. Subsequently, the in-vehicle terminal combines the first environmental perception data and the feedback data to perform anomaly identification on the first assisted driving decision, thereby obtaining an anomaly identification result for the first assisted driving decision. If the anomaly identification result indicates that the first assisted driving decision is abnormal, the in-vehicle terminal sends a decision traceability instruction to the cloud device. After receiving the decision traceability instruction, the cloud device immediately responds to the instruction to control the LLM to perform a decision traceability task. During the process of performing the decision traceability task, the LLM conducts its own error analysis and generates a model update guidance signal for guiding the VLM to perform update and optimization and outputs it. After the cloud device receives the model update guidance signal output by the LLM, it sends the model update guidance signal to the in-vehicle terminal. The in-vehicle terminal controls the VLM to perform update and optimization based on the model update guidance signal, thereby obtaining an updated VLM. Finally, the in-vehicle terminal continues to process the in-vehicle and out-of-vehicle camera images, radar point clouds, and IMU data based on the updated VLM, and obtains the scenario text description caption, attention area attention map, reasoning trace, and confidence information output by the updated VLM. Then, the in-vehicle terminal obtains the scenario text description caption, attention area attention map, reasoning trace, and confidence information output by the updated VLM as the second environmental perception data, and transmits the second environmental perception data to the LLM of the cloud device. After receiving the second environmental perception data, the LLM combines the data with the vehicle state data and user portrait information to generate a vehicle control instruction, and the vehicle control instruction serves as the second assisted driving decision for the vehicle and is transmitted by the cloud device to the vehicle for execution.

[0063] Compared with the vehicle assisted driving decision-making method in the related art that adopts a similar edge-cloud architecture but mainly focuses on static perception and static decision-making, in the embodiments of the present application, it is based on automatically identifying whether there is an abnormality in the first assisted driving decision output by the LLM deployed on the cloud device. When it is recognized that there is an abnormality in this decision, the LLM is triggered to execute a decision traceability task to generate a model update guidance signal, and based on this signal, the VLM deployed on the vehicle local is updated and optimized to obtain an updated VLM. Then, based on the updated VLM, the second environmental perception data is output to the LLM, so as to generate the second assisted driving decision of the vehicle based on the LLM combined with the second environmental perception data. In this way, the embodiments of the present application trigger the LLM to execute a decision traceability task to analyze its own errors and generate a model update guidance signal to reversely guide the update and optimization of the VLM, which can significantly improve the target detection and scene understanding capabilities of the VLM in occlusion or multi-vehicle complex scenarios, thereby effectively compensating for the perception information deviation of the vehicle assisted driving system in dynamic complex scenarios and improving the accuracy of the system's decision-making in complex scenarios.

[0064] In some embodiments, when the vehicle assisted driving decision-making method provided by the embodiments of the present application is applied to a vehicle, the in-vehicle terminal can also, when it is recognized that there is an abnormality in the first vehicle assisted driving decision, automatically generate a virtual adversarial environment based on the VLM to perform a stress test on the LLM, thereby promoting the optimization of the LLM strategy.

[0065] Please refer to Figure 2 , Figure 2 which is a schematic diagram of the step flow of the vehicle assisted driving decision-making method provided by the embodiments of the present application in some other embodiments.

[0066] As Figure 2 shown, in some embodiments, when the vehicle assisted driving decision-making method provided by the embodiments of the present application is applied to a vehicle, it may further include steps S201 to S203 as follows.

[0067] Step S201: When the abnormality recognition result indicates that the first assisted driving decision is abnormal, construct an adversarial simulation scene based on the environmental perception large model.

[0068] After the in-vehicle terminal obtains an abnormality recognition result for the first assisted driving decision, if this abnormality recognition result indicates that the first assisted driving decision is abnormal, it can also generate an adversarial simulation scene based on the VLM on the vehicle local, in combination with the cross-modal correlation data (such as abnormal images, scene text descriptions caption, attention areas attention map, and environmental context sensor meta information) fed back when performing the abnormality recognition for the first assisted driving decision.

[0069] In some embodiments, when the in-vehicle terminal performs anomaly recognition for the first assisted driving decision through the Bad Case detector, if the Bad Case detector comprehensively determines that there is an error in the first assisted driving decision compared to the ideal (reference) decision, it can output the above cross-modal association data to the local VLM of the vehicle.

[0070] Step S202: Send a stress test instruction to the cloud device based on the adversarial simulation scenario to trigger the decision-making large model to execute a decision stress test task; the decision-making large model is updated and optimized based on executing the decision stress test task to obtain an updated decision-making large model.

[0071] After the in-vehicle terminal constructs an adversarial simulation scenario based on the VLM, it further generates a stress test instruction from the adversarial simulation scenario and sends the stress test instruction to the cloud device. Among them, the stress test instruction needs to include at least the relevant data of the adversarial simulation scenario. After receiving the stress test instruction, the cloud device controls the LLM to execute the decision stress test task with the adversarial simulation scenario as the input in response to the stress test instruction, so as to realize the strategy stress test of the LLM and update and optimize the LLM to obtain an updated LLM.

[0072] Step S203: Output third environmental perception data to the updated decision-making large model based on the environmental perception large model, and receive the third assisted driving decision of the vehicle output by the updated decision-making large model in combination with the third environmental perception data.

[0073] After the in-vehicle terminal sends a stress test instruction to the cloud device to trigger the LLM to execute the decision stress test task, when the LLM obtains an updated LLM based on executing this task, the in-vehicle terminal further processes the in-vehicle and out-of-vehicle camera images, radar point clouds, and IMU data based on the VLM to obtain the scene text description caption, attention area attention map, reasoning path reasoning trace, and confidence information output by the VLM. Then, the in-vehicle terminal obtains the scene text description caption, attention area attention map, reasoning path reasoning trace, and confidence information output by the VLM as the third environmental perception data, and transmits the third environmental perception data to the updated LLM on the cloud device. After receiving the third environmental perception data, the updated LLM fuses the data with the vehicle state data and user portrait information to generate a vehicle control instruction, and the vehicle control instruction is used as the third assisted driving decision for the vehicle and is transmitted by the cloud device to the vehicle for execution.

[0074] In some embodiments, when the in-vehicle terminal constructs an adversarial simulation scenario based on the VLM, it can perform an abnormal case analysis on cross-modal association data based on the VLM, and thus construct an adversarial simulation scenario based on the analysis results.

[0075] Please refer to Figure 3 , Figure 3 as Figure 2 the detailed step flow diagram of step S201 in

[0076] As Figure 3 shown, in some embodiments, the above step S201: constructing an adversarial simulation scenario based on the environmental perception large model may include step S301 and step S302 shown below.

[0077] Step S301: Perform an abnormal analysis on the assisted driving scenario corresponding to the first assisted driving decision to obtain the key interference factors in the assisted driving scenario.

[0078] When the in-vehicle terminal performs an abnormal case analysis on cross-modal association data based on the VLM to construct an adversarial simulation scenario, it first performs an abnormal case analysis based on the cross-modal association data (including abnormal images, captions, attention maps, and sensor meta information) through the VLM, that is, based on this data, analyze the environmental structure in the assisted driving scenario of the vehicle corresponding to the first assisted driving decision executed by the current vehicle, such as traffic layout, key targets, weather conditions, lighting conditions, etc. Then, the in-vehicle terminal further extracts, from these environmental structures obtained through the analysis, the interference factors that may cause abnormalities in the first assisted driving decision, and thus uses the extracted interference factors as the key interference factors in the assisted driving scenario corresponding to the first assisted driving decision. Among them, the key interference factors include, but are not limited to, target occlusion, ambiguity in object class, complex lighting / rain / snow / glare, etc., and behavior patterns such as irregular movement paths / sudden crossings.

[0079] Step S302: Construct an adversarial simulation scenario based on the environmental perception large model in combination with the key interference factors.

[0080] After the in-vehicle terminal obtains the key interference factors in the assisted driving scenario, the VLM further uses these key interference factors to generate a set of perturbation factors, thereby constructing a function: , where E is the original scenario and D is the set of perturbation factors. Then, the in-vehicle terminal further generates an adversarial simulation scenario based on the function that should be constructed by the VLM, using image-level perturbation, entity-level perturbation, and / or semantic-level perturbation. Among them, the image-level perturbation can be adding rain and snow, blurring, and brightness and contrast perturbation; the entity-level perturbation can be target position change, sudden appearance of a target, and adding false targets; and the semantic-level perturbation can be adjusting the traffic signal state and interfering with sign recognition.

[0081] In some embodiments, when the in-vehicle terminal performs abnormal case analysis on cross-modal association data based on the VLM to construct an adversarial simulation scenario, it can also choose to introduce a graph neural network to enhance the context modeling ability of the VLM and use the generative adversarial network GAN to generate a more realistic adversarial simulation scenario.

[0082] In some embodiments, when the in-vehicle terminal sends a stress test instruction to the cloud device based on the adversarial simulation scenario to trigger the decision-making large model to execute the decision-making stress test task, it can use a preset heterogeneous data collaborative transmission method to transmit the stress test instruction.

[0083] It should be noted that the heterogeneous data collaborative transmission method can be the above-mentioned data transmission method based on the communication protocol of the JSON+protobuf format. The data transmission between the in-vehicle terminal and the cloud device or between the VLM and the LLM can all adopt this data transmission method.

[0084] Please refer to Figure 4 , Figure 4 For Figure 2 the detailed step flow schematic diagram of step S202 in

[0085] As Figure 4 shown, in some embodiments, the above-mentioned step S202: Sending a stress test instruction to the cloud device based on the adversarial simulation scenario to trigger the decision-making large model to execute the decision-making stress test task may include the following steps.

[0086] Step S401: Convert the adversarial simulation scenario into multi-modal adversarial input data.

[0087] After the in-vehicle terminal constructs an adversarial simulation scenario based on the VLM, it converts the generated adversarial simulation scenario into multi-modal adversarial input data. Among them, the multi-modal adversarial input data includes, but is not limited to: synthetic image, caption, attention map of the expected area of interest in the simulation scenario, and simulated radar / IMU / localization data (sensor-fused data).

[0088] Step S402: Send the multi-modal adversarial input data to the cloud device as a stress test instruction by using a preset heterogeneous data collaborative transmission method, so as to trigger the decision-making large model to execute a decision stress test task based on the multi-modal adversarial input data.

[0089] After the vehicle terminal converts the adversarial simulation scenario into multi-modal adversarial input data, it can directly use the multi-modal adversarial input data as a stress test instruction, and perform corresponding data processing on the stress test instruction by using a pre-set communication protocol based on the JSON+protobuf format, so as to transmit the stress test instruction to the cloud device based on the JSON+protobuf format. After receiving the data in the JSON+protobuf format, the cloud device restores the data to obtain the multi-modal adversarial input data, so as to use the multi-modal adversarial input data as the input of the LLM to control the LLM to execute a decision stress test task for policy stress verification.

[0090] In this embodiment, the vehicle terminal performs anomaly analysis on the assisted driving scenario corresponding to the first assisted driving decision to extract the key interference factors in the assisted driving scenario, then constructs an adversarial simulation scenario based on these key interference factors, and uses a preset heterogeneous data collaborative transmission method to transmit the multi-modal adversarial input data corresponding to the scenario to the cloud device according to the JSON+protobuf format, so that the LLM re-verifies the policy based on the multi-modal adversarial input data. In this way, this embodiment can not only use the LLM to perform its own error analysis to reversely guide the update and optimization of the VLM, but also use the VLM to automatically generate a complex test environment based on bad cases to perform challenging training on the LLM, so as to update and optimize the LLM. That is to say, this embodiment realizes the reverse inference mechanism of the LLM guiding the VLM, and the two-way reflection mechanism of the VLM generating a virtual adversarial scenario to perform a stress test on the LLM, thus breaking the limitation of the traditional perception and decision-making modules of "one-way transmission and difficult to correct", and enabling the continuous learning and evolution of the vehicle assisted driving system.

[0091] In addition, this embodiment uses the virtual adversarial scenario to update and optimize the LLM, which can also improve the generalization ability of the LLM, thus effectively improving the stability and security of the LLM in extreme edge cases.

[0092] In some embodiments, when the vehicle terminal performs anomaly recognition on the first assisted driving decision based on the first environmental perception data and the feedback data when the vehicle executes the first assisted driving decision, it can accurately identify anomalies by setting a multi-threshold rule function to trigger the reverse inference mechanism of the LLM to guide the update of the VLM, and / or trigger the two-way reflection mechanism of the VLM to generate a virtual adversarial scenario and perform a stress test on the LLM.

[0093] Please refer to Figure 5 , Figure 5 which is Figure 1 a schematic diagram of the refined step flow for step S102 in

[0094] As Figure 5 shown, in some embodiments, in the above step S102, the step of "performing anomaly recognition on the first assisted driving decision based on the feedback data and the first environmental perception data" may include steps S501 and S502 as shown below.

[0095] Step S501: Perform multi-dimensional bad case detection based on the feedback data, the model confidence corresponding to the first environmental perception data, and the first assisted driving decision, to obtain multiple bad case detection results.

[0096] When the vehicle-mounted terminal performs anomaly recognition on the first assisted driving decision through the Bad Case detector, the Bad Case detector can perform multi-dimensional bad case detection based on the feedback data, the model confidence corresponding to the first environmental perception data, and the first assisted driving decision respectively, so as to obtain multiple bad case detection results. Among them, the multiple bad case detection results can be at least two of the three multi-dimensional indicators: the perception confidence obtained based on the first environmental perception data, the decision deviation obtained based on the first assisted driving decision, and the execution feedback obtained based on the feedback data.

[0097] Step S502: Perform anomaly recognition on the first assisted driving decision based on the multiple bad case detection results.

[0098] After the vehicle-mounted terminal obtains multiple bad case detection results based on the Bad Case detector, it further uses the Bad Case detector to perform a comprehensive judgment of multi-dimensional indicators on the first assisted driving decision based on the multiple bad case detection results, so as to identify whether there is an error in the first assisted driving decision compared with the ideal (reference) decision. Among them, if the Bad Case detector identifies that there is an error in the first assisted driving decision compared with the ideal (reference) decision, it is considered that the first assisted driving decision is abnormal, and vice versa. If the Bad Case detector identifies that there is no error in the first assisted driving decision compared with the ideal (reference) decision, it is considered that the first assisted driving decision is normal.

[0099] In some embodiments, the in-vehicle terminal performs multi-dimensional bad case detection based on a Bad Case detector, and the multiple bad case detection results can at least include decision deviation data and perception confidence data. Based on this, the in-vehicle terminal can comprehensively use the decision deviation data and the perception confidence data to determine whether there is an error in the first assisted driving decision compared to the ideal (reference) decision, so as to obtain a corresponding anomaly recognition result.

[0100] Please refer to Figure 6 , Figure 6 is Figure 5 a detailed step flow diagram of step S501 in

[0101] As Figure 5 shown, in some embodiments, the above step S501: performing multi-dimensional bad case detection based on the feedback data, the model confidence corresponding to the first environmental perception data, and the first assisted driving decision to obtain multiple bad case detection results, can include step S601 and step S602 as shown below.

[0102] Step S601: Performing bad case detection in the decision dimension based on the feedback data and the first assisted driving decision to obtain decision deviation data; the multiple bad case detection results include the decision deviation data.

[0103] It should be noted that the decision deviation data can reflect the degree of difference between the first assisted driving decision output by the LLM and the ideal decision, and the larger its value, the greater the error.

[0104] When the in-vehicle terminal performs multi-dimensional bad case detection based on the Bad Case detector, it can use the following formula 1 to perform bad case detection in the decision dimension based on the feedback data and the first assisted driving decision, so as to obtain the decision deviation data.

[0105] , formula 1 Among them, is the decision deviation data, refers to the decision instruction actually output by the LLM model, that is, the first assisted driving decision, refers to the ideal (reference) decision, which can be determined with reference to preset rules or expert experience for the first assisted driving decision, vehicle status data, and feedback data.

[0106] Step S602: Performing bad case detection in the perception dimension based on the model confidence corresponding to the first environmental perception data to obtain perception confidence data; the multiple bad case detection results further include the perception confidence data.

[0107] It should be noted that the perception confidence data can reflect the overall confidence level of the first environmental perception data output by the VLM. The higher the value, the higher the credibility of the perception result, and vice versa.

[0108] When the vehicle-mounted terminal performs multi-dimensional bad case detection based on the Bad Case detector, it can also use the following formula 2 to perform bad case detection on the perception dimension based on the model confidence corresponding to the first environmental perception data, so as to obtain the perception confidence data.

[0109] , formula 2 Among them, Conf i represents the confidence confidence of the i-th perception target or feature in the first environmental perception data output by the VLM. This Conf i can be directly output by the VLM when outputting the first environmental perception data. γ is the perception confidence data, which is the average value of the perception confidence of each Conf i .

[0110] In this case, the above step S502: performing anomaly recognition on the first assisted driving decision based on the multiple bad case detection results may include the following steps: Comparing the decision deviation data with a preset deviation threshold, and comparing the perception confidence data with a preset confidence threshold to perform anomaly recognition on the first assisted driving decision.

[0111] It should be noted that in the case where the decision deviation data is greater than the deviation threshold, and / or in the case where the perception confidence data is less than the confidence threshold, an anomaly recognition result indicating that the first assisted driving decision is abnormal is obtained.

[0112] After the vehicle terminal performs multi-dimensional bad case detection based on the Bad Case detector to obtain decision deviation data and perception confidence data, it compares the decision deviation data with a preset deviation threshold (the size can be set according to actual needs), and compares the perception confidence data with a preset confidence threshold (the size can be set according to actual needs). Thus, when the vehicle terminal compares and finds that the decision deviation data is greater than the deviation threshold, and the perception confidence data is less than the confidence threshold, it determines an abnormal recognition result indicating that the first assisted driving decision is abnormal. Or, the vehicle terminal can also determine an abnormal recognition result indicating that the first assisted driving decision is abnormal only when it compares and finds that the decision deviation data is greater than the deviation threshold. Or, the vehicle terminal can also compare and find that when the perception confidence data is less than the confidence threshold, it determines an abnormal recognition result indicating that the first assisted driving decision is abnormal.

[0113] In some embodiments, when the vehicle terminal compares and finds that the decision deviation data is greater than the deviation threshold, and thus obtains an abnormal recognition result indicating that the first assisted driving decision is abnormal, it triggers the above-mentioned reflection mechanism of the VLM to generate a virtual adversarial scenario and perform a stress test on the LLM, so as to update and optimize the LLM to obtain an updated LLM. Or, when the vehicle terminal compares and finds that the perception confidence data is less than the confidence threshold, and thus obtains an abnormal recognition result indicating that the first assisted driving decision is abnormal, it triggers the above-mentioned reverse reasoning mechanism of the LLM to guide the attention of the VLM, so as to update and optimize the VLM to obtain an updated VLM. Or, when the vehicle terminal compares and finds that the decision deviation data is greater than the deviation threshold and the perception confidence data is less than the confidence threshold, and thus obtains an abnormal recognition result indicating that the first assisted driving decision is abnormal, it simultaneously triggers the LLM to guide the attention of the VLM to update the VLM and the VLM to generate a virtual adversarial scenario and perform a stress test on the LLM to update and optimize the LLM's two-way reflection mechanism.

[0114] In this embodiment, the vehicle terminal performs Bad Case detection based on perception confidence, decision deviation and execution feedback, and triggers the LLM to guide the attention of the VLM to update the VLM, and / or triggers the VLM to generate a virtual adversarial scenario and perform a stress test on the LLM to update and optimize the LLM. In this way, it is possible to achieve the perception-decision closed-loop feedback and collaborative evolution inside the model, thereby improving the adaptability and long-term robustness of the system.

[0115] Please refer to Figure 7 , Figure 7 which is a schematic diagram of the step flow of the decision-making method for vehicle assisted driving provided by the embodiments of this application in some other embodiments.

[0116] As Figure 7 shown, in some embodiments, when the decision-making method for vehicle assisted driving provided by the embodiments of the present application is applied to a cloud device, it may include steps S701 to S704 as shown below.

[0117] Step S701: Obtain a decision traceability trigger instruction sent by the vehicle; wherein, the vehicle sends the decision traceability trigger instruction to the cloud device when it recognizes that there is an abnormality in the first assisted driving decision, and the first assisted driving decision is output by the decision-making large model of the cloud device.

[0118] After the in-vehicle terminal obtains the first environmental perception data output by the VLM and the feedback data of the vehicle executing the first assisted driving decision, it combines the first environmental perception data and the feedback data to perform abnormality recognition on the first assisted driving decision, so as to obtain an abnormality recognition result for the first assisted driving decision. Then, if the abnormality recognition result indicates that the first assisted driving decision is abnormal, for example, the first assisted driving decision has an error compared to the ideal (reference) decision, the in-vehicle terminal immediately sends a decision traceability instruction to the cloud device, and the cloud device then receives the decision traceability instruction.

[0119] Step S702: In response to the decision traceability trigger instruction, control the decision-making large model to execute a decision traceability task to generate a model update guidance signal.

[0120] After the cloud device receives the decision traceability instruction, it immediately responds to the instruction to control the LLM to execute the decision traceability task. During the execution of the decision traceability task, the LLM performs its own error analysis and generates a model update guidance signal for guiding the VLM to be updated and optimized and outputs it.

[0121] Step S703: Send the model update guidance signal to the vehicle for the vehicle to update and optimize the environmental perception large model based on the model update guidance signal to obtain an updated environmental perception large model.

[0122] After the cloud device outputs the model update guidance signal, it sends the model update guidance signal to the in-vehicle terminal. The in-vehicle terminal controls the VLM to be updated and optimized based on the model update guidance signal by receiving the model update guidance signal, so as to obtain an updated VLM.

[0123] Step S704: Obtain the second environmental perception data output by the updated environmental perception large model to the decision-making large model, and based on the decision-making large model, combine the second environmental perception data to output a second assisted driving decision of the vehicle to the environmental perception large model.

[0124] After the vehicle terminal obtains the updated VLM, it continues to perform the environment perception task based on the updated VLM, and performs subsequent decision iteration or training playback based on the results output by the updated VLM. That is, based on the updated VLM, it continues to process the in-vehicle and out-of-vehicle camera images, radar point clouds, and IMU data to obtain the scene text description caption, attention area attention map, reasoning trace, and confidence information output by the updated VLM. Then, the vehicle-mounted terminal obtains the scene text description caption, attention area attention map, reasoning trace, and confidence information output by the updated VLM as the second environment perception data, and transmits the second environment perception data to the LLM of the cloud device. After receiving the second environment perception data, the LLM fuses the data with the vehicle state data and the user portrait information to generate a vehicle control instruction, and the vehicle control instruction is used as the second assisted driving decision for the vehicle and is transmitted by the cloud device to the vehicle for execution.

[0125] In the embodiments of the present application, it is determined whether there is an abnormality in the first assisted driving decision output by the LLM deployed on the cloud device automatically, and when it is recognized that the decision is abnormal, the LLM is triggered to execute a decision traceability task to generate a model update guidance signal, and the VLM deployed locally on the vehicle is updated and optimized based on the signal to obtain an updated VLM. Then, based on the updated VLM, the second environment perception data is output to the LLM, so that the second assisted driving decision for the vehicle is generated based on the LLM in combination with the second environment perception data. In this way, the embodiments of the present application trigger the LLM to execute a decision traceability task to analyze its own errors and generate a model update guidance signal to reverse-guide the update and optimization of the VLM, which can significantly improve the target detection and scene understanding capabilities of the VLM in occluded or multi-vehicle complex scenarios, thereby effectively compensating for the perception information deviation of the vehicle assisted driving system in dynamic complex scenarios and improving the accuracy of system decision-making in complex scenarios.

[0126] Please refer to Figure 8 , Figure 8 which is a schematic diagram of the step flow of the decision-making method for vehicle assisted driving provided by the embodiments of the present application in some other embodiments.

[0127] As Figure 8 shown, in some embodiments, when the decision-making method for vehicle assisted driving provided by the embodiments of the present application is applied to a cloud device, it may further include steps S801 to S803 as follows.

[0128] Step S801: Obtain the stress test instruction sent by the vehicle; wherein, when the vehicle identifies that the first assisted driving decision is abnormal, it constructs an adversarial simulation scenario based on the environment perception large model, and sends the stress test instruction to the cloud device based on the adversarial simulation scenario.

[0129] After the in-vehicle terminal performs abnormal identification on the first assisted driving decision and obtains an abnormal identification result, if the abnormal identification result indicates that the first assisted driving decision is abnormal, it can also generate an adversarial simulation scenario based on the VLM on the vehicle local and the cross-modal association data fed back when performing abnormal identification on the first assisted driving decision. Moreover, the in-vehicle terminal further generates a stress test instruction from the adversarial simulation scenario and sends the stress test instruction to the cloud device. In this way, the cloud device can receive the stress test instruction. Among them, the stress test instruction needs to include at least the relevant data of the adversarial simulation scenario.

[0130] Step S802: In response to the stress test instruction, trigger the decision large model to execute a decision stress test task for update and optimization to obtain an updated decision large model.

[0131] After the cloud device receives the stress test instruction, in response to the stress test instruction, it controls the LLM to execute a decision stress test task with the adversarial simulation scenario as the input, so as to realize the policy stress test of the LLM and update and optimize the LLM to obtain an updated LLM.

[0132] Step S803: Obtain the third environment perception data output by the environment perception large model to the updated decision large model, and based on the updated decision large model and the third environment perception data, output the third assisted driving decision of the vehicle to the environment perception large model.

[0133] After the in-vehicle terminal sends a stress test instruction to the cloud device to trigger the LLM to execute a decision stress test task, when the LLM is updated based on the execution of this task, the in-vehicle terminal further continues to process the in-vehicle and out-of-vehicle camera images, radar point clouds, and IMU data based on the VLM, and obtains the scene text description caption, attention area attention map, reasoning trace, and confidence information output by the VLM. Then, the in-vehicle terminal obtains the scene text description caption, attention area attention map, reasoning trace, and confidence information output by the VLM as the third environmental perception data, and transmits this third environmental perception data to the updated LLM on the cloud device. After receiving this third environmental perception data, the updated LLM fuses this data with the vehicle state data and user profile information to generate a vehicle control instruction, and this vehicle control instruction is used as the third assisted driving decision for the vehicle and is transmitted by the cloud device to the vehicle for execution.

[0134] In this embodiment, on the basis of updating and optimizing the VLM by reverse guiding it through the self-error analysis of the LLM, the LLM is further challenged and trained by the VLM to automatically generate a complex test environment based on bad cases, so as to update and optimize the LLM. In this way, this embodiment realizes the reverse reasoning mechanism in which the LLM guides the VLM, and the two-way reflection mechanism in which the VLM generates a virtual adversarial scenario to perform a stress test on the LLM, thus breaking the limitation of the traditional perception and decision-making module of "one-way transmission and difficult to correct", and enabling the continuous learning and evolution of the vehicle assisted driving system.

[0135] Next, a complete embodiment of the decision-making method for vehicle assisted driving provided by the embodiments of the present application is presented.

[0136] In a complete embodiment of the decision-making method for vehicle assisted driving, the decision-making method for vehicle assisted driving provided by the embodiments of the present application can be used to continuously learn and evolve the intelligent driving system through an architecture composed of a VLM perception module, an LLM decision module, a Bad Case detector, a reflection mechanism module (LLM→VLM, VLM→LLM), and a communication protocol and data fusion module (for example, an intelligent vehicle edge-cloud collaborative large model architecture with two-way cross-layer reflection capabilities), so as to accurately, efficiently, and stably make decisions for vehicle assisted driving.

[0137] It should be noted that the VLM perception module can be based on the VLM, taking in-vehicle and out-of-vehicle camera images, radar point clouds, IMU data, etc. as inputs, and performing CoT reasoning on these data to generate semantic information and perception confidence metrics, thereby outputting scene text descriptions (captions), attention regions (attention maps), and reasoning traces (reasoning traces) as environmental perception data, which can be any one of the above-mentioned first, second, and third environmental perception data. In addition, the LLM decision-making module is based on the LLM, taking the VLM output, vehicle state data, and user profile information as inputs, and making complex traffic decisions through multi-modal fusion to obtain vehicle control commands (such as steering angles, acceleration and deceleration decisions), which are any one of the above-mentioned first, second, and third assisted driving decisions. Moreover, the LLM decision-making module has the ability of backward reasoning. The Bad Case detector takes the LLM output, VLM confidence, and vehicle execution feedback (feedback data) as inputs, and uses an algorithm that sets multi-threshold rule functions (such as decision deviation and perception confidence) to comprehensively judge whether there are abnormalities in the assisted driving decisions output by the LLM. The function design of the reflection mechanism module is to trigger the reflection mechanism of LLM→VLM to generate attention guidance backward through the LLM, modify the VLM attention vector A' = f_{LLM}(err_{trace}), and / or trigger the reflection mechanism of VLM→LLM to construct a simulation scenario S^= Sim(VLM, BadCase) based on the VLM analysis of abnormal cases, and the LLM performs strategy re-verification. The function design of the communication protocol and data fusion module is to transmit the VLM output in the JSON+protobuf format, and fuse multi-source information and perform standardization and efficient transmission.

[0138] Please refer to Figure 9 , Figure 9 is the VLM+LLM cross-layer bidirectional reflection technology architecture diagram involved in a complete embodiment of the vehicle assisted driving decision-making method provided by this application example.

[0139] As Figure 9 shown, the vehicle assisted driving decision-making method provided by this application example first makes multi-dimensional judgments through the above-mentioned Bad Case detector. When the judgment of the assisted driving decision output by the LLM is that the decision deviation exceeds the threshold, the judgment of the environmental perception data output by the VLM is that the perception confidence is low, and / or the judgment of the execution effect of the assisted driving decision is that the execution effect is poor, the reflection mechanism is triggered to enter the subsequent cross-layer reflection workflow.

[0140] During the execution of the cross-layer reflection workflow, the LLM-VLM reverse reasoning and the VLM→LLM environment reconstruction can be carried out simultaneously or separately. Among them, when executing the reflection workflow of LLM-VLM reverse reasoning, the LLM first performs decision traceability analysis to generate perceptual attention guidance, and then the VLM adjusts the VLM attention weights according to the perceptual attention guidance to obtain an updated VLM. After that, the updated VLM performs a new round of environment perception. In addition, when executing the reflection workflow of VLM→LLM environment reconstruction, a virtual adversarial scenario is first constructed based on the VLM, and then the LLM performs a decision pressure test based on the virtual adversarial scenario to optimize the LLM decision-making strategy and obtain an improved LLM decision. After that, a new round of decision reasoning is carried out based on the improved LLM decision. The intelligent assisted driving system executes a new assisted driving decision by combining the results of the new round of environment perception and the new round of decision reasoning. And the Bad Case detector still continuously detects the execution of the assisted driving decision by the system to determine whether to trigger the reflection mechanism.

[0141] In some embodiments, in the reflection workflow of VLM→LLM environment reconstruction, during the process of inputting the virtual adversarial scenario into the LLM for pressure testing, the decision output of the LLM in the new extreme scenario (including path planning, game judgment, motion control, etc.) can also be observed. And, perform strategy weakness analysis and recording on the LLM, that is, by continuously recording the performance of the LLM (does an error occur again? Does the decision confidence decrease? And, does the decision delay increase? etc.), generate a decision behavior log and an anomaly diagnosis report, so that the decision behavior log and the anomaly diagnosis report can be used for subsequent strategy correction of the LLM or for future sample annotation training.

[0142] Please refer to Figure 10 , the embodiment of the present application also provides a decision-making device for vehicle assisted driving, which can implement the above-mentioned decision-making method for vehicle assisted driving.

[0143] As Figure 10 shown, the decision-making device for vehicle assisted driving provided by the embodiment of the present application includes an acquisition module 1001, an anomaly recognition module 1002, a reflection module 1003, and a decision module 1004. Among them, The acquisition module 1001 is configured to acquire feedback data and first environment perception data of the vehicle executing the first assisted driving decision; the first assisted driving decision is output by the decision-making large model of the cloud device, and the first environment perception data is output by the environment perception large model of the vehicle, and the cloud device communicates with the vehicle; The anomaly recognition module 1002 is configured to perform anomaly recognition on the first assisted driving decision based on the feedback data and the first environment perception data to obtain an anomaly recognition result; A reflection module 1003, configured to, when the abnormal recognition result indicates that the first assisted driving decision is abnormal, send a decision traceability instruction to a cloud device to trigger the decision-making large model to execute a decision traceability task, and update and optimize the environment perception large model based on a model update guidance signal fed back by the cloud device to obtain an updated environment perception large model; the model update guidance signal is generated by the decision-making large model executing the decision traceability task and output to the environment perception large model; A decision-making module 1004, configured to execute an environment perception task based on the updated environment perception large model to output second environment perception data to the decision-making large model, and receive a second assisted driving decision of the vehicle output by the decision-making large model in combination with the second environment perception data.

[0144] In some embodiments, the reflection module 1003 is further configured to, when the abnormal recognition result indicates that the first assisted driving decision is abnormal, construct an adversarial simulation scenario based on the environment perception large model; and send a stress test instruction to the cloud device based on the adversarial simulation scenario to trigger the decision-making large model to execute a decision stress test task; the decision-making large model is updated and optimized based on executing the decision stress test task to obtain an updated decision-making large model; The decision-making module 1004 is further configured to output third environment perception data to the updated decision-making large model based on the environment perception large model, and receive a third assisted driving decision of the vehicle output by the updated decision-making large model in combination with the third environment perception data.

[0145] In some embodiments, the reflection module 1003 is further configured to perform abnormal analysis on the assisted driving scenario corresponding to the first assisted driving decision to obtain key interference factors in the assisted driving scenario; and construct an adversarial simulation scenario based on the environment perception large model in combination with the key interference factors.

[0146] In some embodiments, the reflection module 1003 is further configured to convert the adversarial simulation scenario into multi-modal adversarial input data; and use a preset heterogeneous data collaborative transmission method to send the multi-modal adversarial input data as a stress test instruction to the cloud device to trigger the decision-making large model to execute a decision stress test task based on the multi-modal adversarial input data.

[0147] In some embodiments, the abnormal recognition module 1002 is further configured to perform multi-dimensional bad case detection based on the feedback data, the model confidence corresponding to the first environment perception data, and the first assisted driving decision to obtain multiple bad case detection results; and perform abnormal recognition on the first assisted driving decision based on the multiple bad case detection results.

[0148] In some embodiments, the anomaly recognition module 1002 is further configured to perform bad case detection on the decision dimension based on the feedback data and the first assisted driving decision to obtain decision deviation data; the multiple bad case detection results include the decision deviation data; perform bad case detection on the perception dimension based on the model confidence corresponding to the first environmental perception data to obtain perception confidence data; the multiple bad case detection results further include the perception confidence data; compare the decision deviation data with a preset deviation threshold, and compare the perception confidence data with a preset confidence threshold to identify anomalies in the first assisted driving decision; Wherein, in the case where the decision deviation data is greater than the deviation threshold, and / or in the case where the perception confidence data is less than the confidence threshold, an anomaly recognition result indicating that the first assisted driving decision is abnormal is obtained.

[0149] In some embodiments, the reflection module 1003 is further configured to obtain a decision traceability trigger instruction sent by the vehicle; wherein, the vehicle sends the decision traceability trigger instruction to the cloud device when it recognizes that the first assisted driving decision is abnormal, and the first assisted driving decision is output by the decision large model of the cloud device; control the decision large model to execute a decision traceability task to generate a model update guidance signal in response to the decision traceability trigger instruction; and send the model update guidance signal to the vehicle for the vehicle to update and optimize the environmental perception large model based on the model update guidance signal to obtain an updated environmental perception large model; The decision module 1004 is further configured to obtain second environmental perception data output by the updated environmental perception large model to the decision large model, and output a second assisted driving decision of the vehicle to the environmental perception large model based on the decision large model in combination with the second environmental perception data.

[0150] In some embodiments, the reflection module 1003 is further configured to obtain a stress test instruction sent by the vehicle; wherein, the vehicle constructs an adversarial simulation scenario based on the environmental perception large model when it recognizes that the first assisted driving decision is abnormal, and sends the stress test instruction to the cloud device based on the adversarial simulation scenario; and trigger the decision large model to execute a decision stress test task for update and optimization to obtain an updated decision large model in response to the stress test instruction; The decision module 1004 is further configured to obtain third environmental perception data output by the environmental perception large model to the updated decision large model, and output a third assisted driving decision of the vehicle to the environmental perception large model based on the updated decision large model in combination with the third environmental perception data.

[0151] It should be noted that the specific implementation of the vehicle assisted driving decision-making device provided in the embodiments of the present application is basically the same as the specific embodiments of the above-mentioned visual language model evaluation method, and will not be repeated here.

[0152] The embodiments of the present application also provide a vehicle assisted driving decision-making device. The vehicle assisted driving decision-making device includes a memory and a processor. The memory stores a computer program, and when the processor executes the computer program, the above-mentioned visual language model evaluation method is implemented.

[0153] In some embodiments, the vehicle assisted driving decision-making device may be any intelligent terminal such as an in-vehicle computer, a tablet computer, a smart phone, a wearable device, etc.

[0154] Please refer to Figure 11 , Figure 11 , which shows the hardware structure of the vehicle assisted driving decision-making device in an embodiment. The vehicle assisted driving decision-making device includes: A processor 1101, which can be implemented in ways such as a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present application; A memory 1102, which can be implemented in forms such as a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 1102 can store an operating system and other application programs. When implementing the technical solutions provided in the embodiments of this specification through software or firmware, the relevant program codes are stored in the memory 1102, and are called by the processor 1101 to execute the vehicle assisted driving decision-making method of the embodiments of the present application; An input / output interface 1103, which is used to implement information input and output; A communication interface 1104, which is used to implement communication interaction between this device and other devices. Communication can be achieved through wired means (such as USB, network cable, etc.) or through wireless means (such as mobile network, WIFI, Bluetooth, etc.); A bus 1105, which transmits information between various components of the device (such as the processor 1101, the memory 1102, the input / output interface 1103, and the communication interface 1104); Among them, the processor 1101, the memory 1102, the input / output interface 1103, and the communication interface 1104 are communicatively connected to each other inside the device through the bus 1105.

[0155] An embodiment of the present application also provides a vehicle, on which a decision-making device for vehicle assisted driving is configured. The decision-making device for vehicle assisted driving includes a memory and a processor. The memory stores a computer program, and when the processor executes the computer program, the above-mentioned decision-making method for vehicle assisted driving is implemented.

[0156] An embodiment of the present application also provides a cloud device, on which a decision-making device for vehicle assisted driving is configured. The decision-making device for vehicle assisted driving includes a memory and a processor. The memory stores a computer program, and when the processor executes the computer program, the above-mentioned decision-making method for vehicle assisted driving is implemented.

[0157] An embodiment of the present application also provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the above-mentioned decision-making method for vehicle assisted driving is implemented.

[0158] As a non-transitory computer-readable storage medium, the memory can be used to store non-transitory software programs and non-transitory computer-executable programs. In addition, the memory may include high-speed random access memory, and may also include non-transitory memory, such as at least one magnetic disk storage device, a flash memory device, or other non-transitory solid-state storage devices. In some embodiments, the memory may optionally include a memory remotely disposed relative to the processor, and these remote memories can be connected to the processor through a network. Examples of the above network include but are not limited to the Internet, an enterprise intranet, a local area network, a mobile communication network, and combinations thereof.

[0159] An embodiment of the present application also provides a computer program product, including a computer program. The steps implemented when the computer program is executed by a processor are substantially the same as the specific embodiments of the above-mentioned decision-making method for vehicle assisted driving, and will not be described in detail here.

[0160] The embodiments described in the embodiments of the present application are for more clearly illustrating the technical solutions of the embodiments of the present application, and do not constitute a limitation on the technical solutions provided by the embodiments of the present application. Those skilled in the art can know that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided by the embodiments of the present application are equally applicable to similar technical problems.

[0161] Those skilled in the art can understand that the technical solutions shown in the figures do not constitute a limitation on the embodiments of the present application, and may include more or fewer steps than shown in the figures, or combine certain steps, or different steps.

[0162] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separated, that is, they may be located in one place or distributed to multiple network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0163] Those of ordinary skill in the art can understand that all or some of the steps in the methods disclosed above, and the functional modules / units in systems and devices, can be implemented as software, firmware, hardware, and appropriate combinations thereof.

[0164] In the specification of this application and the above-mentioned drawings, terms such as "first", "second", "third", "fourth", etc. (if any) are used to distinguish similar objects and do not necessarily describe a specific order or sequence. It should be understood that the data used in this way can be interchanged under appropriate circumstances so that the embodiments of this application described here can be implemented in an order other than those illustrated or described here. In addition, the terms "comprising" and "having" and any variations thereof are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or device that includes a series of steps or units does not necessarily limit to those steps or units clearly listed, but may include other steps or units not clearly listed or inherent to these processes, methods, products, or devices.

[0165] It should be understood that in this application, "at least one (item)" means one or more, and "a plurality" means two or more. "And / or" is used to describe the association relationship of associated objects and indicates that three relationships can exist. For example, "A and / or B" can mean: only A exists, only B exists, and both A and B exist at the same time. Among them, A and B can be singular or plural. The character " / " generally means that the associated objects before and after are in an "or" relationship. "At least one (one)" or similar expressions below refer to any combination of these items, including any combination of single item (one) or plural items (ones). For example, at least one (one) of a, b, or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0166] In several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For example, the above division of units is only a logical function division. In actual implementation, there may be other division methods. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the displayed or discussed coupling, direct coupling, or communication connection between each other can be through some interfaces. The indirect coupling or communication connection of devices or units can be in electrical, mechanical, or other forms.

[0167] The units described above as separate components may or may not be physically separated. The components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed to multiple network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0168] In addition, in each embodiment of the present application, the functional units can be integrated into a processing unit, or each unit can exist physically alone, or two or more units can be integrated into one unit. The above integrated units can be implemented in the form of hardware or in the form of software functional units.

[0169] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, in essence, or the part that contributes to the prior art, or all or part of this technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes multiple instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods in each embodiment of the present application. And the aforementioned storage medium includes: various media that can store programs such as USB flash drives, mobile hard disks, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical discs.

[0170] The preferred embodiments of the embodiments of the present application have been described above with reference to the drawings, but this does not limit the scope of the rights of the embodiments of the present application. Any modifications, equivalent replacements, and improvements made by those skilled in the art without departing from the scope and essence of the embodiments of the present application shall be within the scope of the rights of the embodiments of the present application.

Claims

1. A decision-making method for vehicle assisted driving, characterized in that, The method is applied to a vehicle, and the method includes: Obtaining feedback data and first environmental perception data for the vehicle to execute a first assisted driving decision; the first assisted driving decision is output by a decision-making large model of a cloud device, and the first environmental perception data is output by an environmental perception large model of the vehicle, and the cloud device communicates with the vehicle; Performing anomaly recognition on the first assisted driving decision based on the feedback data and the first environmental perception data to obtain an anomaly recognition result; When the anomaly recognition result indicates that the first assisted driving decision is abnormal, sending a decision traceability instruction to the cloud device to trigger the decision-making large model to execute a decision traceability task, and updating and optimizing the environmental perception large model based on a model update guidance signal fed back by the cloud device to obtain an updated environmental perception large model; the model update guidance signal is generated by the decision-making large model executing the decision traceability task and output to the environmental perception large model; Performing an environmental perception task based on the updated environmental perception large model to output second environmental perception data to the decision-making large model, and receiving a second assisted driving decision of the vehicle output by the decision-making large model in combination with the second environmental perception data.

2. The method according to claim 1, characterized in that, The method further includes: When the anomaly recognition result indicates that the first assisted driving decision is abnormal, constructing an adversarial simulation scenario based on the environmental perception large model; Sending a stress test instruction to the cloud device based on the adversarial simulation scenario to trigger the decision-making large model to execute a decision stress test task; the decision-making large model is updated and optimized based on executing the decision stress test task to obtain an updated decision-making large model; Outputting third environmental perception data to the updated decision-making large model based on the environmental perception large model, and receiving a third assisted driving decision of the vehicle output by the updated decision-making large model in combination with the third environmental perception data.

3. The method according to claim 2, wherein The constructing an adversarial simulation scenario based on the environmental perception large model includes: Performing anomaly analysis on the assisted driving scenario corresponding to the first assisted driving decision to obtain key interference factors in the assisted driving scenario; Constructing an adversarial simulation scenario based on the environmental perception large model in combination with the key interference factors.

4. The method according to claim 2, wherein The sending a stress test instruction to the cloud device based on the adversarial simulation scenario to trigger the decision-making large model to execute a decision stress test task includes: Converting the adversarial simulation scenario into multi-modal adversarial input data; Using a preset heterogeneous data collaborative transmission method to send the multi-modal adversarial input data as a stress test instruction to the cloud device to trigger the decision-making large model to execute a decision stress test task based on the multi-modal adversarial input data.

5. The method according to claim 1, characterized in that The performing anomaly recognition on the first assisted driving decision based on the feedback data and the first environmental perception data includes: Performing multi-dimensional bad case detection based on the feedback data, the model confidence corresponding to the first environmental perception data, and the first assisted driving decision to obtain multiple bad case detection results; Performing anomaly recognition on the first assisted driving decision based on the multiple bad case detection results.

6. The method according to claim 5, characterized in that, Performing multi-dimensional bad case detection based on the feedback data, the model confidence corresponding to the first environmental perception data, and the first assisted driving decision, to obtain multiple bad case detection results, including: Performing bad case detection in the decision dimension based on the feedback data and the first assisted driving decision, to obtain decision deviation data; the multiple bad case detection results include the decision deviation data; Performing bad case detection in the perception dimension based on the model confidence corresponding to the first environmental perception data, to obtain perception confidence data; the multiple bad case detection results further include the perception confidence data; The abnormal identification of the first assisted driving decision based on the multiple bad case detection results includes: Comparing the decision deviation data with a preset deviation threshold, and comparing the perception confidence data with a preset confidence threshold, to perform abnormal identification of the first assisted driving decision; Wherein, when the decision deviation data is greater than the deviation threshold, and / or when the perception confidence data is less than the confidence threshold, an abnormal identification result indicating that the first assisted driving decision is abnormal is obtained.

7. A decision-making method for vehicle assisted driving, characterized in that, The method is applied to a cloud device communicating with a vehicle, and the method includes: Obtaining a decision traceability trigger instruction sent by the vehicle; wherein, when the vehicle identifies that the first assisted driving decision is abnormal, the vehicle sends the decision traceability trigger instruction to the cloud device, and the first assisted driving decision is output by a decision-making large model of the cloud device; Responding to the decision traceability trigger instruction to control the decision-making large model to execute a decision traceability task to generate a model update guidance signal; Sending the model update guidance signal to the vehicle for the vehicle to update and optimize the environmental perception large model based on the model update guidance signal to obtain an updated environmental perception large model; Obtaining second environmental perception data output by the updated environmental perception large model to the decision-making large model, and outputting a second assisted driving decision of the vehicle to the environmental perception large model based on the decision-making large model in combination with the second environmental perception data.

8. The method according to claim 7, wherein The method further includes: Obtaining a stress test instruction sent by the vehicle; wherein, when the vehicle identifies that the first assisted driving decision is abnormal, an adversarial simulation scenario is constructed based on the environmental perception large model, and the stress test instruction is sent to the cloud device based on the adversarial simulation scenario; Responding to the stress test instruction to trigger the decision-making large model to execute a decision stress test task for update and optimization to obtain an updated decision-making large model; Obtaining third environmental perception data output by the environmental perception large model to the updated decision-making large model, and outputting a third assisted driving decision of the vehicle to the environmental perception large model based on the updated decision-making large model in combination with the third environmental perception data.

9. A decision-making device for vehicle assisted driving, characterized in that, The device includes: An acquisition module, configured to acquire feedback data and first environmental perception data of the vehicle executing a first assisted driving decision; the first assisted driving decision is output by a decision-making large model of a cloud device, and the first environmental perception data is output by an environmental perception large model of the vehicle, and the cloud device communicates with the vehicle; An anomaly recognition module, configured to perform anomaly recognition on the first assisted driving decision based on the feedback data and the first environmental perception data to obtain an anomaly recognition result; A reflection module, configured to, when the anomaly recognition result indicates that the first assisted driving decision is abnormal, send a decision traceability instruction to the cloud device to trigger the decision-making large model to execute a decision traceability task, and update and optimize the environmental perception large model based on a model update guidance signal fed back by the cloud device to obtain an updated environmental perception large model; the model update guidance signal is generated by the decision-making large model executing the decision traceability task and output to the environmental perception large model; A decision-making module, configured to perform an environmental perception task based on the updated environmental perception large model to output second environmental perception data of the vehicle to the decision-making large model, and receive a second assisted driving decision of the vehicle output by the decision-making large model in combination with the second environmental perception data.

10. A decision-making device for vehicle assisted driving, characterized in that, The decision-making device for vehicle assisted driving includes a memory and a processor, the memory stores a computer program, and when the processor executes the computer program, it implements the decision-making method for vehicle assisted driving according to any one of claims 1 to 8.

11. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, it implements the decision-making method for vehicle assisted driving according to any one of claims 1 to 8.

12. A computer program product, characterized in that, The computer program product includes a computer program, and when the computer program is executed by a processor, it implements the decision-making method for vehicle assisted driving according to any one of claims 1 to 8.

Images