Intelligent driving system evaluation method and device, electronic equipment and storage medium

Abstract

The invention provides an intelligent driving system evaluation method and device, electronic equipment and a storage medium, and relates to the technical field of intelligent driving, detection data of an intelligent driving system is collected, and the detection data comprises environment data and driving behavior data of each vehicle; performing causal reasoning according to the environment data and the driving behavior data in combination with a preset confrontation scene, and updating a scene key factor of each driving scene; performing user portrait processing according to the environment data and the driving behavior data, and updating the weight of the evaluation dimension decision model; and according to the scene key factors and the weights, combining with the environment data and the driving behavior data, applying an evaluation dimension decision model to carry out personalized scoring on each driving scene, and obtaining a personalized scoring result for optimizing the intelligent driving system. Accurate, dynamic and demand-fitting adaptive evaluation of the intelligent driving system is realized, so that the intelligent driving system is optimized.

Description

Technical Field

[0001] This invention relates to the field of intelligent driving technology, and specifically to an evaluation method, device, electronic device, and storage medium for an intelligent driving system. Background Technology

[0002] With the rapid development of intelligent driving technology, the performance of intelligent driving functions has increasingly become a focus of attention for automakers and users. However, existing intelligent driving evaluation methods rely on preset scenario libraries for fixed-rule evaluation, failing to dynamically identify key influencing factors of each scenario. Furthermore, the use of uniform evaluation standards ignores personalized needs such as driving style (aggressive / conservative) and usage scenario (commuting / long-distance), resulting in a low degree of relevance between evaluation results and real-world user experiences. These shortcomings directly hinder the iterative upgrade of intelligent driving technology and the improvement of user experience. Summary of the Invention

[0003] In view of this, embodiments of the present invention aim to provide an intelligent driving system evaluation method, device, electronic equipment, and storage medium, which can combine complex scenarios and personalized needs to achieve accurate, dynamic, and demand-appropriate adaptive evaluation of the intelligent driving system, so as to optimize the intelligent driving system.

[0004] To achieve the above technical objectives, the embodiments of this specification provide the following technical solutions: In a first aspect, embodiments of this application provide an evaluation method for an intelligent driving system. This method includes: collecting detection data from the intelligent driving system, the detection data including environmental data and driving behavior data for each vehicle; performing causal inference based on the environmental data and the driving behavior data combined with preset adversarial scenarios, and updating the scenario key factors for each driving scenario; performing user profile processing based on the environmental data and the driving behavior data, and updating the weights of the evaluation dimension decision model, wherein the adversarial scenario is at least one driving scenario generated based on the results of the previous round of personalized evaluation, the scenario key factors are used to identify each driving scenario, and the user profile includes feature data representing each user; applying the evaluation dimension decision model to perform personalized scoring on each driving scenario based on the scenario key factors and the weights, combined with the environmental data and the driving behavior data, and obtaining personalized scoring results for optimizing the intelligent driving system.

[0005] In this embodiment, updating the key factors of the scenario can improve the evaluation coverage and accuracy of long-tail scenarios. Adjusting the weights of the evaluation dimension decision model can achieve precise matching of personalized needs. Thus, by combining complex scenarios with personalized needs, we can achieve accurate, dynamic and demand-aligned adaptive evaluation of the intelligent driving system, so as to optimize the intelligent driving system.

[0006] Optionally, the step of performing causal inference based on the environmental data and the driving behavior data in combination with a preset adversarial scenario, and updating the scenario key factors of each driving scenario, includes: applying a pre-trained causal discovery engine to perform causal inference based on the environmental data and the driving behavior data, and obtaining updated scenario key factors for each driving scenario, wherein the causal discovery engine is obtained after training based on the preset adversarial scenario.

[0007] In this embodiment, key influencing factors of long-tail scenarios can be dynamically identified through causal reasoning. Combined with the calculation of key scenario factors weighted by environmental credibility, the evaluation coverage and accuracy of long-tail scenarios such as pedestrians crossing against the light in rainstorms and temporary detours on construction sections can be greatly improved. Scenario adaptation can be achieved, enabling adaptive evaluation of complex scenarios.

[0008] Optionally, the step of processing the user profile based on the environmental data and the driving behavior data, and updating the weights of the evaluation dimension decision model, includes: obtaining an initial user profile; performing intelligent driving behavior analysis based on the environmental data and the driving behavior data combined with the initial user profile, and updating the feature data to obtain an updated user profile; and updating the weights of the evaluation dimension decision model based on the updated user profile.

[0009] In this implementation, user profiles are adjusted through driving behavior analysis, which in turn adjusts the weights of the evaluation dimension decision model. Relying on the dynamic migration mechanism of user profiles driven by driving behavior, the profile weights are updated based on real-time behavioral characteristics such as takeover frequency and steering wheel angle fluctuation rate. This allows the evaluation dimensions to be accurately matched with the user's personalized needs such as driving style (aggressive / conservative) and usage scenario (commuting / long-distance). The user demand matching degree is significantly improved compared with the traditional unified standard solution.

[0010] Optionally, the step of applying the evaluation dimension decision model to personalize the scoring of each driving scenario based on the scenario key factors and the weights, combined with the environmental data and the driving behavior data, includes: determining the detection type of each driving scenario based on the scenario key factors, wherein the detection type is either specialized detection or basic detection, wherein the basic detection is for normal driving scenarios, and the specialized detection is for long-tail scenarios, wherein the long-tail scenarios are driving scenarios that occur infrequently but are dangerous; determining each evaluation dimension based on the environmental data and the driving behavior data using the evaluation dimension decision model corresponding to the weights; and personalizing the scoring of each driving scenario based on each evaluation dimension and the detection type to obtain personalized scoring results, wherein each driving scenario in the personalized scoring results includes multiple evaluation dimensions, each evaluation dimension corresponds to multiple evaluation indicators, and each evaluation indicator corresponds to a score value.

[0011] In this embodiment, the intelligent driving system can be individually scored based on the weights of the updated scenario key factors and evaluation dimension decision model. The personalized evaluation results are generated by combining the scenario key factor β with the dynamic profile. This includes a multi-dimensional performance evaluation of the intelligent driving system in the current scenario, which can solve the shortcomings of existing methods in scenario adaptation and user demand matching, and achieve adaptive evaluation of complex scenarios and personalized needs.

[0012] Optionally, determining the detection type of each driving scenario based on the scenario key factors includes: if the scenario key factor of any driving scenario is greater than or equal to a threshold, then the detection type of the driving scenario is determined to be a special detection; if the scenario key factor of any driving scenario is less than a threshold, then the detection type of the driving scenario is determined to be a basic detection.

[0013] In this embodiment, by classifying and detecting driving scenarios and assigning personalized scores based on key scenario factors, the shortcomings of existing methods in scenario adaptation can be overcome, and adaptive evaluation of complex scenarios and personalized needs can be achieved.

[0014] Optionally, the method further includes: generating a vulnerability heatmap based on the personalized scoring results; obtaining targeting instructions and new adversarial scenarios based on the vulnerability heatmap, wherein the targeting instructions are used to instruct the updating of parameters corresponding to the targeting instructions in the intelligent driving system to optimize the intelligent driving system.

[0015] In this embodiment, vulnerability heatmap analysis of personalized scoring results is used to upgrade and optimize the intelligent driving system, further improving its performance. Simultaneously, new adversarial scenarios can be generated for the next round of personalized evaluation and optimization. Leveraging the bidirectional co-evolutionary architecture of the evaluation dimension decision model and the intelligent driving system, personalized scoring results can feed back into the intelligent driving system to generate targeted instructions. Furthermore, after the intelligent driving system upgrade, the evaluation dimension decision model can iterate through adversarial testing scenarios. The response speed to high-risk vulnerabilities is significantly improved compared to traditional one-way output schemes, resulting in a markedly faster iteration efficiency for the intelligent driving system.

[0016] Optionally, the method further includes: testing the optimized intelligent driving system according to the new adversarial scenario to test the optimization effect of the intelligent driving system; and conducting a next round of evaluation and optimization of the intelligent driving system according to the new adversarial scenario.

[0017] In this embodiment, thanks to the bidirectional co-evolutionary architecture of the evaluation dimension decision model and the intelligent driving system, personalized scoring results can feed back into the intelligent driving system to generate targeted optimization instructions. At the same time, after the intelligent driving system is upgraded, the evaluation dimension decision model can be iterated through adversarial testing scenarios. The response speed to high-risk vulnerabilities is significantly improved compared to the traditional one-way output scheme. Through data closure, the evaluation dimension decision model is iterated with the intelligent driving system upgrade. Meanwhile, the personalized scoring results guide the intelligent driving system to target optimization in a triangular evolutionary mode, breaking the limitations of the traditional one-way output and significantly accelerating the iteration efficiency of the intelligent driving system.

[0018] Secondly, this application also provides an intelligent driving system evaluation device applied to a server. The device includes: a data acquisition module for acquiring detection data of the intelligent driving system, the detection data including environmental data and driving behavior data of each vehicle; a weight update module for performing causal reasoning based on the environmental data and the driving behavior data combined with preset adversarial scenarios, and updating the scenario key factors of each driving scenario; performing user profile processing based on the environmental data and the driving behavior data, and updating the weights of the evaluation dimension decision model, wherein the adversarial scenario is at least one driving scenario generated based on the results of the previous round of personalized evaluation, the scenario key factors are used to identify each driving scenario, and the user profile includes feature data representing each user; and a personalized scoring module for applying the evaluation dimension decision model to perform personalized scoring on each driving scenario based on the scenario key factors and the weights combined with the environmental data and driving behavior data, and obtaining personalized scoring results for optimizing the intelligent driving system.

[0019] Thirdly, embodiments of this application also provide a vehicle, including: a memory for storing executable program code; and a processor for calling and running the executable program code from the memory, causing the processor to perform the aforementioned method.

[0020] Fourthly, embodiments of this application also provide a computer-readable storage medium storing a computer program that, when executed, implements the aforementioned method.

[0021] As can be seen from the above technical solutions, the intelligent driving system evaluation method provided in this specification collects detection data of the intelligent driving system, including environmental data and driving behavior data of each vehicle; performs causal reasoning based on the environmental data and driving behavior data combined with preset adversarial scenarios, and updates the key factors of each driving scenario; processes user profiles based on the environmental data and driving behavior data, and updates the weights of the evaluation dimension decision model, wherein the adversarial scenario is at least one driving scenario generated based on the results of the previous round of personalized evaluation, the key factors of the scenario are used to identify each driving scenario, and the user profile includes feature data representing each user; applies the evaluation dimension decision model to perform personalized scoring on each driving scenario based on the key factors of the scenario and the weights combined with the environmental data and driving behavior data, and obtains personalized scoring results for optimizing the intelligent driving system. By updating the key factors of the scenario, the evaluation coverage and accuracy of long-tail scenarios can be improved, and by adjusting the weights of the evaluation dimension decision model, accurate matching of personalized needs can be achieved. Thus, by combining complex scenarios and personalized needs, an accurate, dynamic, and demand-aligned adaptive evaluation of the intelligent driving system can be achieved, so as to optimize the intelligent driving system.

[0022] The above description is merely an overview of the technical solution disclosed herein. In order to better understand the technical means of this disclosure and to implement it in accordance with the contents of the specification, and to make the above and other objects, features and advantages of this disclosure more apparent and understandable, specific embodiments of this disclosure are described below. Attached Figure Description

[0023] Various other advantages and benefits will become apparent to those skilled in the art upon reading the following detailed description of preferred embodiments. The accompanying drawings are for illustrative purposes only and are not intended to limit the scope of this disclosure. Furthermore, the same reference numerals denote the same parts throughout the drawings.

[0024] Figure 1 The diagram shown is a flowchart illustrating the intelligent driving system evaluation method provided in an embodiment of this application.

[0025] Figure 2 The diagram shown is an example of the intelligent driving system evaluation method provided in this application embodiment.

[0026] Figure 3 The diagram shown is a structural schematic of an intelligent driving system evaluation device provided in an embodiment of this application.

[0027] Figure 4 The diagram shown is a structural schematic of a vehicle provided in an embodiment of this application. Detailed Implementation

[0028] Unless otherwise defined, the technical or scientific terms used in the embodiments of this specification shall have the ordinary meaning understood by one of ordinary skill in the art to which this specification pertains. The terms "first," "second," and similar terms used in the embodiments of this specification do not indicate any order, quantity, or importance, but are merely used to avoid confusion of constituent elements.

[0029] Unless the context otherwise requires, throughout this specification, "a plurality of" means "at least two," and "including" is interpreted as open-ended or encompassing, that is, "including, but not limited to." In the description of this specification, terms such as "one embodiment," "some embodiments," "exemplary embodiment," "example," "specific example," or "some examples" are intended to indicate that a particular feature, structure, material, or characteristic associated with that embodiment or example is included in at least one embodiment or example of this specification. The illustrative representations of the above terms do not necessarily refer to the same embodiment or example.

[0030] Exemplary embodiments of the present disclosure will now be described in more detail with reference to the accompanying drawings. While exemplary embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure may be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the disclosure to those skilled in the art.

[0031] In related technologies, with the rapid development of intelligent driving technology, the performance of intelligent driving systems has increasingly become a focus of attention for automakers and users. However, existing evaluation methods for intelligent driving systems rely on preset scenario libraries for fixed-rule evaluation, resulting in insufficient coverage and accuracy for long-tail scenarios (such as pedestrians crossing against the light in heavy rain or temporary detours on construction sites). This leads to low evaluation precision and limited scenario coverage, making it impossible to accurately assess the performance of intelligent driving systems in special scenarios. User modeling is based solely on initial registration information, resulting in static user profiles that are not dynamically updated in conjunction with real-time driving behavior. This leads to a low degree of matching between evaluation and personalized user needs, causing evaluation weights to be out of touch with actual user requirements and failing to meet the differentiated requirements of users for intelligent driving systems under different driving styles and usage scenarios. Furthermore, the evaluation model and the intelligent driving system have a one-way output relationship, lacking a co-evolution mechanism. Test data cannot feed back into the evaluation model, resulting in slow response to high-risk vulnerabilities. Evaluation standards lag behind the iteration of intelligent driving technology, leading to slow iteration speed of intelligent driving systems and creating a vicious cycle that hinders vulnerability response and system iteration, affecting the iterative upgrade of intelligent driving technology and the improvement of user experience.

[0032] Based on this, in order to address the technical problem that existing technologies rely on preset scenario libraries for fixed-rule evaluation, which restricts the iterative upgrade of intelligent driving technology and the improvement of user experience, this application provides an intelligent driving system evaluation method, such as... Figure 1 As shown, Figure 1 This is a schematic flowchart illustrating an intelligent driving system evaluation method provided in an embodiment of this application. The method is applied to a server, which can be a terminal server, electronic device, or cloud server capable of communicating with various vehicles. The intelligent driving system evaluation method includes: Step S11: Collect detection data from the intelligent driving system, including environmental data and driving behavior data of each vehicle.

[0033] See Figure 1 The system collects detection data through a data acquisition layer. Specifically, it collects environmental data through multi-source sensors on the vehicle and driving behavior data through the in-vehicle terminal. Both types of data are encrypted and uploaded to the cloud server. The cloud server communicates with the multi-source sensors on the vehicle in real time to ensure data timeliness and processing efficiency. The multi-source sensors on the vehicle include LiDAR, cameras, millimeter-wave radar, etc. The collected environmental data includes road environment data, pedestrian and vehicle flow data, and weather data, such as road surface slippage, icy conditions, driving on curves, traffic light conditions at intersections, and traffic density. The cloud server also communicates with the in-vehicle terminal in real time to obtain driving behavior data, including vehicle driving parameter data and user operation data. The detection data includes environmental data and driving behavior data for various driving scenarios. The cloud server can also receive user feedback data based on the current user experience of the intelligent driving system.

[0034] Step S12: Perform causal reasoning based on the environmental data and driving behavior data combined with preset adversarial scenarios, and update the key factors of each driving scenario; perform user profile processing based on the environmental data and driving behavior data, and update the weights of the evaluation dimension decision model, wherein the adversarial scenario is at least one driving scenario generated based on the results of the previous round of personalized evaluation, the key factors of the scenario are used to identify each driving scenario, and the user profile includes feature data representing each user.

[0035] Based on the environmental data and driving behavior data, a pre-defined causal graph for adversarial scenarios is learned. This causal graph includes learned environmental and driving behavior data closely related to the adversarial scenario, accumulating general causal reasoning knowledge to obtain updated key factors for each driving scenario. An adversarial scenario is at least one driving scenario generated based on the results of the previous round of personalized evaluation. For example, if some evaluation indicators are unsatisfactory, such as the presence of high-risk vulnerabilities, at least one driving scenario is generated based on these indicators. This at least one driving scenario is then incorporated into the original driving scenarios and evaluated and further optimized using the previously optimized intelligent driving system.

[0036] User profiling is performed based on the environmental data and driving behavior data. The feature data of each user corresponding to the environmental data and driving behavior data is updated, and the weights of the evaluation dimension decision model are updated based on the updated user profiles. For example, for continuous zero-intervention driving behavior, if the user is determined to prioritize efficiency, the efficiency weight can be increased. For frequent intervention driving behavior, if the user is determined to prioritize safety, the safety weight can be increased.

[0037] It should be noted that if the detection data also includes user feedback data, then causal reasoning and user profiling are performed based on environmental data, driving behavior data, and user feedback data in combination with preset adversarial scenarios to update the weights of the key factors and evaluation dimensions of the decision model for each driving scenario.

[0038] Step S13: Based on the key factors of the scenario and the weights, combined with the environmental data and the driving behavior data, apply the evaluation dimension decision model to perform personalized scoring for each driving scenario, and obtain personalized scoring results for optimizing the intelligent driving system.

[0039] In this embodiment, firstly, all evaluation dimensions are obtained by applying an evaluation dimension decision model with corresponding weights based on environmental data and driving behavior data. Then, for each driving scenario, basic or specialized tests are performed on each evaluation dimension according to the corresponding scenario key factors to obtain personalized scores for each evaluation dimension of each driving scenario. Finally, the values ​​of each evaluation dimension for each driving scenario are weighted to obtain the total personalized score of the intelligent driving system. The personalized score result considers the evaluation of the intelligent driving system from various driving scenarios and different users, and can characterize the performance or existing vulnerabilities or defects of the intelligent driving system. The obtained personalized score result includes the personalized scores for each evaluation dimension of each driving scenario and the total personalized score. Based on this personalized score result, the intelligent driving system can be upgraded. Simultaneously, new adversarial scenarios including at least one driving scenario can be generated for the next round of evaluation and optimization upgrades of the intelligent driving system.

[0040] The intelligent driving system evaluation method of this application collects detection data of the intelligent driving system, including environmental data and driving behavior data of each vehicle; performs causal inference based on the environmental data and driving behavior data in conjunction with preset adversarial scenarios, and updates the key factors of each driving scenario; processes user profiles based on the environmental data and driving behavior data, and updates the weights of the evaluation dimension decision model, wherein the adversarial scenario is at least one driving scenario generated based on the results of the previous round of personalized evaluation, the key factors of the scenario are used to identify each driving scenario, and the user profile includes feature data representing each user; and applies the evaluation dimension decision model to perform personalized scoring on each driving scenario based on the key factors of the scenario and the weights, combined with the environmental data and driving behavior data, to obtain personalized scoring results for optimizing the intelligent driving system. By updating the key factors of the scenario, the evaluation coverage and accuracy of long-tail scenarios can be improved, and by adjusting the weights of the evaluation dimension decision model, accurate matching of personalized needs can be achieved. Thus, by combining complex scenarios with personalized needs, an accurate, dynamic, and demand-aligned adaptive evaluation of the intelligent driving system can be achieved, thereby optimizing the intelligent driving system.

[0041] To more clearly illustrate the technical solutions provided in the embodiments of this application, the following further describes an evaluation method for an intelligent driving system provided in this application.

[0042] Considering that traditional evaluation methods rely on fixed rules and preset scenarios, lack the ability to dynamically identify scenarios, and cannot cope with the complexity and uncertainty of long-tail scenarios, in this embodiment of the application, optionally, the step of performing causal inference based on the environmental data and the driving behavior data combined with preset adversarial scenarios, and updating the scenario key factors of each driving scenario, includes: applying a pre-trained causal discovery engine to perform causal inference based on the environmental data and the driving behavior data, and obtaining updated scenario key factors for each driving scenario, wherein the causal discovery engine is obtained after training based on preset adversarial scenarios.

[0043] In this embodiment, if some evaluation metrics performed poorly in the previous round of personalized evaluation, an adversarial scenario including at least one driving scenario can be generated to optimize those metrics. A single driving scenario can be designed to optimize one or more evaluation metrics. See also... Figure 2The causal discovery engine is trained based on generated adversarial scenarios. During the pre-training phase, the engine learns from a scenario library, and during the online training phase, its parameters are fine-tuned in real time. The causal discovery engine employs a meta-learning graph neural network (GNN) architecture, integrating meta-learning and graph neural network architectures. It acquires general causal reasoning capabilities through pre-training and rapidly adapts to new scenarios for causal relationship mining during the online phase, providing technical support for dynamic scenario evaluation. Considering that adversarial scenarios are generated based on poorly performing evaluation metrics from the previous round of personalized evaluation, and that these scenarios are generally long-tailed scenarios—driving scenarios with low probability of occurrence but inherent dangers—the causal discovery engine learns causal graphs in virtual long-tailed scenarios during the pre-training phase, accumulating general causal reasoning knowledge. During the online phase, it fine-tunes parameters using real-time data, outputting the scenario's key factor β. The formula for calculating the scenario's key factor β is: β = α * GNN (sensor data) + (1-α) * RL (driving behavior data). Here, α represents the environmental credibility weight (higher in rainy / foggy weather, lower in sunny weather), used to dynamically balance the reliability of sensor data and driving results. Sensor data refers to environmental data acquired through sensors, and GNN (sensor data) represents a value obtained after processing the sensor data using GNN. RL stands for Reinforcement Learning, an algorithm that optimizes behavioral strategies through interaction with the environment, used for quantitative analysis of driving results. RL (driving behavior data) represents a value obtained after reinforcement learning on driving behavior data. GNN is a Graph Neural Network, a neural network that processes graph-structured data, used to discover the relationships between scene elements. In the online phase, the causal discovery engine outputs not only the scene key factors for each driving scenario in the added adversarial scenario, but also the scene key factors for each driving scenario in the previous round. The scene key factors for this part of the driving scenario may be affected by the added adversarial scenario, resulting in different values ​​of scene key factors. The scene key factors quantify the challenge level of the scenario to the intelligent driving system, calculated from multimodal data weighted by environmental credibility, providing a quantitative basis for dynamic adjustment of evaluation dimensions. It should be noted that the GNN in the causal discovery engine can be replaced with the Transformer model, which uses a self-attention mechanism to mine causal relationships and is suitable for highway scenarios with larger amounts of data.

[0044] The embodiment of this application uses a causal discovery engine built by meta-learning GNN to dynamically identify key influencing factors in long-tail scenarios. Combined with the calculation of scene key factor β weighted by environmental credibility, it can significantly improve the evaluation coverage and accuracy of long-tail scenarios such as pedestrians crossing against the light in rainstorms and temporary detours in construction sections. It can achieve scene adaptation and adaptive evaluation of complex scenarios.

[0045] Considering that different users have different driving styles in the same driving scenario, their needs for intelligent driving systems may also differ. To achieve personalized evaluation of intelligent driving systems, it is also necessary to consider the needs of different users. Therefore, optionally, the step of processing user profiles based on the environmental data and driving behavior data, and updating the weights of the evaluation dimension decision model, includes: obtaining an initial user profile; performing intelligent driving behavior analysis based on the environmental data and driving behavior data combined with the initial user profile, and updating the feature data to obtain an updated user profile; and updating the weights of the evaluation dimension decision model based on the updated user profile.

[0046] In this embodiment, basic tags (such as "urban commuters" and "long-distance drivers") are generated based on registration information (age, vehicle type, region) using a clustering algorithm to obtain an initial user profile. Intelligent driving behavior analysis is performed based on the environmental data and driving behavior data to obtain multiple features representing each user. Combined with the initial user profile, a Generative Adversarial Network (GAN) is used to transfer driving style features, updating the feature data representing each user to obtain an updated user profile. The weights of the updated evaluation dimension decision model are adjusted based on the updated user profile. For example, when the driving style changes from "conservative" to "aggressive," the safety weight decreases and the efficiency weight increases, achieving dynamic matching between the evaluation dimension and user needs. This embodiment adjusts the user profile through driving behavior analysis, thereby adjusting the weights of the evaluation dimension decision model. Relying on a driving behavior-driven dynamic user profile transfer mechanism, the profile weights are updated based on real-time behavioral characteristics such as takeover frequency and steering wheel angle fluctuation rate, enabling precise matching between the evaluation dimension and personalized needs such as user driving style (aggressive / conservative) and usage scenario (commuting / long-distance). The user need matching degree is significantly improved compared to traditional unified standard solutions. This mechanism, which periodically updates profile weights based on real-time driving behavior characteristics (takeover frequency, steering wheel angle fluctuation rate, etc.), combined with Generative Adversarial Networks (GANs) to achieve driving style feature transfer, solves the problem of static user modeling and provides key support for personalized evaluation. GANs are generative models that use adversarial training between two networks: a generator and a discriminator. This allows the generator to produce realistic data, making it difficult for the discriminator to distinguish between real and generated data. GANs have been widely used in tasks such as image generation, image inpainting, style transfer, and text generation. The generator receives a random noise vector (usually Gaussian or uniformly distributed) and maps it to the data space, making the generated data as close to real data as possible. Its goal is to "deceive" the discriminator, making it unable to distinguish between generated and real data. The discriminator receives input samples and judges whether the samples are real or fake, aiming to distinguish between real and fake samples as accurately as possible. GAN training is a process of mutual competition between the generator and the discriminator, gradually improving the generator's generation quality through adversarial training. By integrating meta-learning and graph neural network architectures, general causal reasoning capabilities are acquired through pre-training, enabling rapid adaptation to new scenarios in causal relationship mining during the online phase, thus providing technical support for dynamic scenario evaluation. It should be noted that the adversarial generative network can be replaced with a federated learning framework, enabling cross-vehicle enterprise user feature sharing while protecting privacy, thereby enhancing user profile richness.

[0047] After obtaining the updated scenario key factors and the weights of the evaluation dimension decision model, it is necessary to combine the two to perform personalized scoring of the intelligent driving system in order to determine the performance of the intelligent driving system. Based on this, optionally, the step of applying the evaluation dimension decision model to perform personalized scoring of each driving scenario based on the scenario key factors and the weights, combined with the environmental data and the driving behavior data, includes: determining the detection type of each driving scenario based on the scenario key factors, wherein the detection type is either specialized detection or basic detection, wherein the basic detection is for normal driving scenarios, and the specialized detection is for long-tail scenarios, where long-tail scenarios are driving scenarios with low frequency of occurrence but inherent dangers; determining each evaluation dimension based on the environmental data and the driving behavior data using the evaluation dimension decision model corresponding to the weights; and performing personalized scoring of each driving scenario based on each evaluation dimension and the detection type to obtain personalized scoring results, wherein each driving scenario in the personalized scoring results includes multiple evaluation dimensions, each evaluation dimension corresponds to multiple evaluation indicators, and each evaluation indicator corresponds to a score value.

[0048] See also Figure 2 In the dynamic evaluation generation layer, the detection type of each driving scenario can be identified based on its key factors. This determines whether a driving scenario is a basic scenario or a long-tail scenario. Basic scenarios are normal driving scenarios that occur frequently, while long-tail scenarios are scenarios that occur infrequently but pose a danger, such as certain extreme scenarios. If a driving scenario is a basic scenario, basic detection is applied; if it is a long-tail scenario, specialized detection is applied. Different types of detection have different weights for each evaluation dimension. The evaluation dimension decision model is applied and updated based on environmental and driving behavior data to determine each evaluation dimension. For example, after the previous intelligent driving system upgrade, adversarial test scenarios (such as extreme weather scenarios) are automatically generated, and evaluation dimensions are updated, such as adding a dimension like "nighttime lane-changing conservatism." If basic scenarios can be iterated periodically, using periodically updated driving scenarios as the basic scenarios, long-tail scenarios can be deeply optimized, such as modifying the evaluation dimensions of a certain extreme scenario, ensuring that personalized evaluation evolves in sync with the capabilities of the intelligent driving system. Personalized scores are then applied to each driving scenario based on each evaluation dimension and detection type to obtain personalized scoring results. For any basic scenario, a personalized score is obtained based on each evaluation dimension and the multiple evaluation dimensions and corresponding weights corresponding to the basic detection type. For any long-tail scenario, a personalized score is obtained based on each evaluation dimension and the multiple evaluation dimensions and corresponding weights corresponding to the specialized detection type. All driving scenarios are iterated over to obtain a personalized score for each driving scenario. Alternatively, the personalized scores for each driving scenario can be weighted to obtain the overall personalized score for the intelligent driving system.

[0049] This application embodiment can personalize the scoring of intelligent driving systems by using the updated scenario key factors and the weights of the evaluation dimension decision model. It combines the scenario key factor β with the dynamic profile to generate personalized evaluation results, including a multi-dimensional performance evaluation of the intelligent driving system in the current scenario. This can solve the shortcomings of existing methods in scenario adaptation and user demand matching, and achieve adaptive evaluation of complex scenarios and personalized needs.

[0050] To accurately determine the detection type of each driving scenario based on key scenario factors, in this embodiment, optionally, determining the detection type of each driving scenario based on the key scenario factors includes: if the key scenario factor of any driving scenario is greater than or equal to a threshold, then the detection type of the driving scenario is determined to be a special detection; if the key scenario factor of any driving scenario is less than the threshold, then the detection type of the driving scenario is determined to be a basic detection. Basic scenarios are normal driving scenarios with high occurrence rates. Intelligent driving systems generally have good control over regular normal driving scenarios, and their impact on the personalized scoring of the intelligent driving system is not significant, resulting in relatively small key scenario factors. Long-tail scenarios are driving scenarios with low occurrence rates but high danger. Due to their low occurrence rate, intelligent driving systems may not be able to control long-tail scenarios well. The performance of an intelligent driving system often depends on its ability to control long-tail scenarios, such as extreme scenarios, effectively. Therefore, the key scenario factors for long-tail scenarios are relatively large. The key scenario factors can be compared with a threshold; if the key scenario factor of any driving scenario is greater than or equal to the threshold, then the detection type of that driving scenario is determined to be a special detection. Similarly, if the key factors of any driving scenario are less than a threshold, the detection type for that driving scenario is determined to be basic detection. This application's embodiments, by classifying and detecting driving scenarios based on key factors and providing personalized scoring, can overcome the shortcomings of existing methods in scenario adaptation and achieve adaptive evaluation for complex scenarios and personalized needs.

[0051] Considering that the personalized evaluation of the intelligent driving system in this application embodiment is for the purpose of optimizing the intelligent driving system so that it can be applied to various long-tail scenarios, based on this, in this application embodiment, optionally, the method further includes: generating a vulnerability heatmap based on the personalized scoring result; obtaining targeted instructions and new adversarial scenarios based on the vulnerability heatmap, wherein the targeted instructions are used to instruct the updating of parameters in the intelligent driving system corresponding to the targeted instructions to optimize the intelligent driving system. See also... Figure 2In the co-evolution layer, vulnerability heatmap analysis is performed based on personalized scoring results. The heatmap displays the scores of each evaluation metric in each driving scenario. If a poorly performing evaluation metric appears in the heatmap, it indicates a vulnerability in the intelligent driving system, requiring further optimization. Targeted instructions can be generated based on these evaluation metrics, including at least one parameter in the intelligent driving system that needs modification. Modifying this parameter allows for optimization and upgrades to the intelligent driving system. Simultaneously, new adversarial scenarios are generated based on poorly performing evaluation metrics in the heatmap. These adversarial scenarios can include multiple driving scenarios, with one or more scenarios generated based on performance characteristics. These adversarial scenarios can be used for further testing of the optimized intelligent driving system, enabling the next round of personalized evaluation and optimization. It should be noted that reinforcement learning can also be used to replace rule generation in adversarial scenario generation, increasing the diversity and challenge of extreme scenarios and providing a more comprehensive test of the intelligent driving system's performance.

[0052] This application embodiment feeds back the evaluation results to the co-evolution layer, driving the iteration of the evaluation dimension decision model and the optimization of the intelligent driving system. It performs vulnerability heatmap analysis on the personalized scoring results to upgrade and optimize the intelligent driving system, further improving its performance. At the same time, it can also generate new adversarial scenarios for the next round of personalized evaluation and optimization. With the bidirectional co-evolutionary architecture of the evaluation dimension decision model and the intelligent driving system, the personalized scoring results can feed back into the intelligent driving system to generate targeted instructions. Meanwhile, after the intelligent driving system is upgraded, the evaluation dimension decision model can be iterated through adversarial test scenarios, which significantly improves the response speed of high-risk vulnerabilities compared to the traditional one-way output scheme, and significantly accelerates the iteration efficiency of the intelligent driving system.

[0053] Considering that acquiring new adversarial scenarios allows for testing of the optimized intelligent driving system and subsequent evaluation and optimization, this application embodiment optionally includes: testing the optimized intelligent driving system based on the new adversarial scenario to test the optimization effect; and conducting further evaluation and optimization of the intelligent driving system based on the new adversarial scenario. (Continue to see...) Figure 2The optimized intelligent driving system can be tested on a test vehicle. Adversarial test scenarios can be automatically generated on the test vehicle based on adversarial scenarios to test the optimized intelligent driving system and acquire test equipment data. In this embodiment, the next round of evaluation and optimization of the intelligent driving system can also be conducted based on the new adversarial scenarios, restarting data collection and forming a closed-loop iteration to ensure continuous adaptation between the evaluation and the capabilities of the intelligent driving system. Each iteration improves the matching degree between the evaluation dimensions and the functions of the intelligent driving system, forming a virtuous cycle of mutual promotion. This embodiment, relying on the bidirectional co-evolutionary architecture of the evaluation dimension decision model and the intelligent driving system, allows personalized scoring results to feed back into the intelligent driving system to generate targeted optimization instructions. Simultaneously, after the intelligent driving system upgrades, the evaluation dimension decision model can be iterated through adversarial test scenarios. The response speed to high-risk vulnerabilities is significantly improved compared to traditional one-way output schemes. Through data closed-loop, the evaluation dimension decision model iterates with the intelligent driving system upgrades, while personalized scoring results guide the targeted optimization of the intelligent driving system in a triangular evolutionary mode, breaking the limitations of traditional one-way output and significantly accelerating the iteration efficiency of the intelligent driving system.

[0054] This application embodiment collects detection data from an intelligent driving system, including environmental data and driving behavior data for each vehicle. It then performs causal inference based on the environmental data and driving behavior data, combined with preset adversarial scenarios, and updates the key factors of each driving scenario. Next, it performs user profiling based on the environmental data and driving behavior data, and updates the weights of the evaluation dimension decision model. The adversarial scenario is at least one driving scenario generated based on the previous round of personalized evaluation results. The key factors of the scenario are used to identify each driving scenario, and the user profile includes feature data representing each user. Based on the key factors of the scenario and the weights, combined with the environmental data and driving behavior data, the evaluation dimension decision model is applied to perform personalized scoring for each driving scenario, obtaining personalized scoring results to optimize the intelligent driving system. Updating the key factors of the scenario improves the evaluation coverage and accuracy for long-tail scenarios, and adjusting the weights of the evaluation dimension decision model enables precise matching of personalized needs. This allows for the combination of complex scenarios and personalized needs to achieve accurate, dynamic, and demand-aligned adaptive evaluation of the intelligent driving system, thereby optimizing the intelligent driving system.

[0055] In one exemplary embodiment of this specification, an intelligent driving system evaluation device is also provided, applied to a cloud server. For example... Figure 3 As shown, the intelligent driving system evaluation device 300 includes: Data acquisition module 301 is used to collect detection data from the intelligent driving system, including environmental data and driving behavior data of each vehicle; The weight update module 302 is used to perform causal reasoning based on the environmental data and the driving behavior data combined with a preset adversarial scenario, and update the scenario key factors of each driving scenario; to perform user profile processing based on the environmental data and the driving behavior data, and update the weights of the evaluation dimension decision model, wherein the adversarial scenario is at least one driving scenario generated based on the results of the previous round of personalized evaluation, the scenario key factors are used to identify each driving scenario, and the user profile includes feature data representing each user; The personalized scoring module 303 is used to apply the evaluation dimension decision model to give personalized scores to each driving scenario based on the key factors of the scenario and the weights, combined with the environmental data and the driving behavior data, and to obtain personalized scoring results for optimizing the intelligent driving system.

[0056] In one specific implementation, the weight update module 302 is used to: apply a pre-trained causal discovery engine to perform causal inference based on the environmental data and the driving behavior data, and obtain updated scene key factors for each driving scenario, wherein the causal discovery engine is obtained after training based on a preset adversarial scenario.

[0057] In some implementations, the weight update module 302 is further configured to: obtain an initial user profile; perform intelligent driving behavior analysis based on the environmental data and the driving behavior data combined with the initial user profile, and update the feature data to obtain an updated user profile; and update the weights of the evaluation dimension decision model based on the updated user profile.

[0058] In some implementations, the weight update module 302 is further configured to: determine the detection type of each driving scenario based on the scenario key factors, wherein the detection type is a special detection or a basic detection, wherein the basic detection is the detection of normal driving scenarios, and the special detection is the detection of long-tail scenarios, wherein the long-tail scenarios are driving scenarios that occur infrequently but are dangerous; determine each evaluation dimension based on the environmental data and the driving behavior data using an evaluation dimension decision model corresponding to the weights; and perform personalized scoring on each driving scenario based on each evaluation dimension and the detection type to obtain personalized scoring results, wherein each driving scenario in the personalized scoring results includes multiple evaluation dimensions, each evaluation dimension corresponds to multiple evaluation indicators, and each evaluation indicator corresponds to a score value.

[0059] In some implementations, the individual scoring module 303 is used to: determine the detection type of the driving scenario as specialized detection if the scenario key factor of any driving scenario is greater than or equal to a threshold; and determine the detection type of the driving scenario as basic detection if the scenario key factor of any driving scenario is less than a threshold.

[0060] In some implementations, the personalized scoring module 303 is further configured to: generate a vulnerability heatmap based on the personalized scoring results; and obtain targeting instructions and new adversarial scenarios based on the vulnerability heatmap, wherein the targeting instructions are used to instruct the updating of parameters corresponding to the targeting instructions in the intelligent driving system to optimize the intelligent driving system.

[0061] In some implementations, the personalized scoring module 303 is also used to: test the optimized intelligent driving system according to the new adversarial scenario to test the optimization effect of the intelligent driving system; and to conduct the next round of evaluation and optimization of the intelligent driving system according to the new adversarial scenario.

[0062] Specific limitations regarding the evaluation device for intelligent driving systems can be found in the limitations regarding the evaluation methods for intelligent driving systems mentioned above, and will not be repeated here. Each module in the aforementioned evaluation device for intelligent driving systems can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device in hardware form, or stored in the memory of a computer device in software form, so that the processor can call and execute the operations corresponding to each module.

[0063] Regarding the apparatus in the above embodiments, the specific manner in which each unit performs its operation has been described in detail in the embodiments concerning the intelligent driving system evaluation method, and will not be elaborated upon here.

[0064] Figure 4 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application.

[0065] For example, such as Figure 4 As shown, the electronic device includes a memory 401 and a processor 402. The memory 401 stores executable program code 4011, and the processor 402 is used to call and execute the executable program code 4011 to perform an intelligent driving system evaluation method.

[0066] This embodiment can divide the electronic device into functional modules according to the above method embodiment. For example, each module can correspond to a separate functional module, or two or more functions can be integrated into one processing module. The integrated module can be implemented in hardware. It should be noted that the module division in this embodiment is illustrative and is only a logical functional division. In actual implementation, there may be other division methods.

[0067] When each functional module is divided according to its corresponding function, the electronic device may include: a data acquisition module, a weight update module, a personalized scoring module, etc.

[0068] It should be noted that all relevant content of each step involved in the above method embodiments can be referenced from the functional description of the corresponding functional module, and will not be repeated here.

[0069] The electronic device provided in this embodiment is used to execute the above-described intelligent driving system evaluation method, and therefore can achieve the same effect as the above-described implementation method.

[0070] When using integrated units, the electronic device may include a processing module and a storage module. The processing module is used to control and manage the operation of the electronic device. The storage module is used to support the processing module in executing program code and data.

[0071] The processing module may be a processor or a controller, which can implement or execute various exemplary logic blocks, modules, and circuits as disclosed in this application. The processor may also be a combination of computing functions, such as a combination of one or more microprocessors, a combination of digital signal processing (DSP) and microprocessors, etc., and the storage module may be a memory.

[0072] This embodiment also provides a computer-readable storage medium (including but not limited to disk storage, CD-ROM, optical storage, etc.) storing computer program code. When the computer program code is run on a computer, the computer executes the above-mentioned related method steps to implement the intelligent driving system evaluation method provided in the above embodiment. The computer-readable storage medium may include, but is not limited to, any type of disk, including floppy disks, optical disks, Digital Video Discs (DVDs), Compact Disc Read-Only Memory (CD-ROMs), microdrives, and magneto-optical disks, read-only memory (ROMs), random access memory (RAMs), erasable programmable read-only memory (EPROMs), electrically erasable programmable read-only memory (EEPROMs), dynamic random access memory (DRAMs), video random access memory (VRAMs), flash memory devices, magnetic cards or optical cards, nanosystems (including molecular memory ICs), or any type of medium or device suitable for storing instructions and / or data.

[0073] This embodiment also provides a computer program product that, when run on a computer, causes the computer to perform the aforementioned related steps to implement the intelligent driving system evaluation method provided in the above embodiment.

[0074] The beneficial effects of the above embodiments can be referred to the beneficial effects of the corresponding methods provided above, and will not be repeated here.

[0075] Through the above description of the embodiments, those skilled in the art will understand that, for the sake of convenience and brevity, only the division of the above functional modules is used as an example. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above.

[0076] In the embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another device, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0077] In the description of this disclosure, it should be understood that if the terms "upper", "lower", "front", "rear", "left" and "right" are used to indicate the orientation or positional relationship based on the orientation or positional relationship shown in the drawings, they are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the position or element referred to must have a specific orientation, or be constructed and operated in a specific orientation, and therefore should not be construed as a limitation of this disclosure.

[0078] It should be noted that, in the embodiments of this application, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes the element.

[0079] The above are merely embodiments of this disclosure and are not intended to limit the scope of this disclosure. Various modifications and variations can be made to this disclosure by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this disclosure should be included within the scope of the claims of this disclosure.

Claims

1. A method for evaluating an intelligent driving system, characterized in that, The method includes: Collect detection data from the intelligent driving system, including environmental data and driving behavior data of each vehicle; Causal reasoning is performed based on the environmental data and driving behavior data in combination with preset adversarial scenarios, and the key factors of each driving scenario are updated; user profile processing is performed based on the environmental data and driving behavior data, and the weights of the evaluation dimension decision model are updated. The adversarial scenario is at least one driving scenario generated based on the results of the previous round of personalized evaluation. The key factors of the scenario are used to identify each driving scenario. The user profile includes feature data that characterizes each user. Based on the key factors and weights of the scenario, combined with the environmental data and driving behavior data, the evaluation dimension decision model is applied to give personalized scores to each driving scenario, and the personalized score results are used to optimize the intelligent driving system.

2. The method according to claim 1, characterized in that, The step of performing causal reasoning based on the environmental data and driving behavior data combined with preset adversarial scenarios, and updating the key scenario factors for each driving scenario, includes: Based on the environmental data and the driving behavior data, a pre-trained causal discovery engine is applied to perform causal inference to obtain updated scene key factors for each driving scenario. The causal discovery engine is obtained after training based on a preset adversarial scenario.

3. The method according to claim 1, characterized in that, The step of processing user profiles based on the environmental data and driving behavior data, and updating the weights of the evaluation dimension decision model, includes: Obtain initial user profile; Based on the environmental data and driving behavior data, combined with the initial user profile, intelligent driving behavior analysis is performed, and the feature data is updated to obtain the updated user profile; The weights of the evaluation dimension decision model are updated based on the updated user profile.

4. The method according to claim 1, characterized in that, The process of applying the evaluation dimension decision model to personalize scores for each driving scenario based on the scenario's key factors and weights, combined with environmental data and driving behavior data, includes: The detection type of each driving scenario is determined based on the key factors of the scenario. The detection type is either special detection or basic detection. The basic detection is for normal driving scenarios, and the special detection is for long-tail scenarios. Long-tail scenarios are driving scenarios that occur infrequently but are dangerous. Each evaluation dimension is determined by applying an evaluation dimension decision model corresponding to the weights based on the environmental data and the driving behavior data. Each driving scenario is individually scored based on the evaluation dimensions and the detection type to obtain a personalized scoring result. Each driving scenario in the personalized scoring result includes multiple evaluation dimensions, each evaluation dimension corresponds to multiple evaluation indicators, and each evaluation indicator corresponds to a score value.

5. The method according to claim 4, characterized in that, The step of determining the detection type of each driving scenario based on the key factors of the scenario includes: If the key factor of any driving scenario is greater than or equal to the threshold, then the detection type of the driving scenario is determined to be a special detection. If the key factor of any driving scenario is less than a threshold, then the detection type of the driving scenario is determined to be basic detection.

6. The method according to claim 1, characterized in that, The method further includes: A vulnerability heatmap is generated based on the personalized scoring results; Based on the vulnerability heatmap, target instructions and new adversarial scenarios are obtained. The target instructions are used to instruct the updating of parameters corresponding to the target instructions in the intelligent driving system to optimize the intelligent driving system.

7. The method according to claim 6, characterized in that, The method further includes: The optimized intelligent driving system was tested according to the new adversarial scenario to test the optimization effect of the intelligent driving system. The next round of evaluation and optimization of the intelligent driving system will be conducted based on the new adversarial scenarios.

8. An evaluation device for an intelligent driving system, characterized in that, The intelligent driving system evaluation device includes: The data acquisition module is used to collect detection data from the intelligent driving system, including environmental data and driving behavior data of each vehicle. The weight update module is used to perform causal reasoning based on the environmental data and the driving behavior data combined with preset adversarial scenarios, and update the key factors of each driving scenario; to perform user profile processing based on the environmental data and the driving behavior data, and update the weights of the evaluation dimension decision model, wherein the adversarial scenario is at least one driving scenario generated based on the results of the previous round of personalized evaluation, the key factors of the scenario are used to identify each driving scenario, and the user profile includes feature data representing each user; The personalized scoring module is used to apply the evaluation dimension decision model to give personalized scores to each driving scenario based on the key factors of the scenario and the weights, combined with the environmental data and the driving behavior data, and to obtain personalized scoring results for optimizing the intelligent driving system.

9. An electronic device, characterized in that, The electronic device includes: Memory, used to store executable program code; A processor for calling and running the executable program code from the memory, such that the processor performs the method as described in any one of claims 1 to 8.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed, implements the method as described in any one of claims 1 to 8.

Images