Vehicle-mounted AI model test method, device and equipment and vehicle

Abstract

The invention provides a vehicle-mounted AI model test method, apparatus and device, and a vehicle. The vehicle-mounted AI model test method comprises the steps of obtaining a test data set of a to-be-tested vehicle-mounted AI model; wherein the vehicle-mounted AI model to be tested comprises any one vehicle-mounted AI model in different vehicle-mounted AI models developed for a vehicle; according to the test data set, performing simulation evaluation on the to-be-tested vehicle-mounted AI model, and outputting a first vehicle-mounted AI model; and deploying the first vehicle-mounted AI model to the vehicle, performing evaluation in a real environment, and outputting a target vehicle-mounted AI model.

Description

Technical Field

[0001] The embodiments of the present disclosure relate to the field of vehicle testing technology, and more specifically, to an in-vehicle AI model testing method, device, equipment, and vehicle. Background Art

[0002] In the related technology, the testing methods provided are usually only applicable to AI (Artificial Intelligence) models developed for a specific function of a vehicle. For example, the testing method for testing AI (Artificial Intelligence) models developed for the driving function of a vehicle is not applicable to AI models developed for the intelligent functions within the vehicle. In other words, the testing method is only applicable to AI models developed for the driving function of the vehicle, and is not applicable to models developed for other vehicle functions. Summary of the invention

[0003] One purpose of the embodiments of the present disclosure is to provide a new technical solution for vehicle-mounted AI model testing.

[0004] According to a first aspect of the present disclosure, a vehicle-mounted AI model testing method is provided, comprising:

[0005] Obtain a test data set of an in-vehicle AI model to be tested; wherein the in-vehicle AI model to be tested includes any in-vehicle AI model among different in-vehicle AI models developed for the vehicle;

[0006] According to the test data set, a simulation evaluation is performed on the in-vehicle AI model to be tested, and a first in-vehicle AI model is output;

[0007] The first in-vehicle AI model is deployed to the vehicle, and is evaluated in a real environment to output a target in-vehicle AI model.

[0008] Optionally, obtaining a test data set of the in-vehicle AI model to be tested includes:

[0009] Constructing a natural environment scene according to the model function of the in-vehicle AI model to be tested;

[0010] Collecting data in the natural environment scene to obtain natural scene data;

[0011] Performing data engineering processing on the natural scene data to generate a candidate test data set;

[0012] According to the set verification items, the test data in the candidate test data set is verified, and the test data set is generated according to the test data that passes the verification.

[0013] Optionally, performing data engineering processing on the natural scene data to generate a candidate test data set includes:

[0014] Performing data pre-processing on the natural scene data to obtain an intermediate test data set;

[0015] Generate the candidate test data set according to the sample generation method and the intermediate test data set;

[0016] Among them, the sample generation method at least includes an adversarial sample generation method, and the adversarial sample generation method is used to generate adversarial test data.

[0017] Optionally, the performing simulation evaluation on the in-vehicle AI model to be tested according to the test data set and outputting a first in-vehicle AI model includes:

[0018] According to the test data set, the on-board AI model to be tested is subjected to model-in-environmental assessment, software-in-environmental assessment and hardware-in-environmental assessment in sequence, and the first on-board AI model is output.

[0019] Optionally, for any environmental assessment link, the method further includes:

[0020] According to the test data set, the vehicle-mounted AI model currently tested in the environmental assessment phase is evaluated to obtain a first evaluation result of the vehicle-mounted AI model currently tested in the environmental assessment phase; wherein the first evaluation result involves performance indicators, functional indicators, and safety indicators;

[0021] If the first evaluation result is that the on-board AI model currently tested in the environmental assessment phase has passed the evaluation, the on-board AI model currently tested in the environmental assessment phase will be used as the on-board AI model to be tested in the next environmental assessment phase.

[0022] Optionally, when the first evaluation result is that the on-board AI model currently tested in the environmental assessment link fails the evaluation, the on-board AI model currently tested in the environmental assessment link is optimized, and the optimized on-board AI model continues to be evaluated based on the test data set.

[0023] Optionally, deploying the first vehicle-mounted AI model to the vehicle, and evaluating it in a real environment, and outputting a target vehicle-mounted AI model includes:

[0024] Deploy the first vehicle-mounted AI model to the vehicle, and conduct an evaluation in a controllable real environment to output a second vehicle-mounted AI model; and,

[0025] The second in-vehicle AI model is deployed to the vehicle, and is evaluated in a real road environment to output the target in-vehicle AI model.

[0026] Optionally, deploying the first vehicle-mounted AI model to the vehicle, and evaluating it in different controllable real environments, and outputting a second vehicle-mounted AI model includes:

[0027] Formulate different test scenarios and determine controllable real environments corresponding to the test scenarios; wherein the different test scenarios include conventional test scenarios and adversarial scenarios;

[0028] Deploying the first in-vehicle AI model to the vehicle, and evaluating the first in-vehicle AI model in different controllable real environments to obtain a second evaluation result; wherein the second evaluation result involves performance indicators, functional indicators, and safety indicators;

[0029] If the second evaluation result is that the first in-vehicle AI model passes the evaluation, the first in-vehicle AI model is used as the second in-vehicle AI model.

[0030] Optionally, the method further comprises:

[0031] When the second evaluation result shows that the first in-vehicle AI model fails the evaluation, the first in-vehicle AI model is optimized, and the optimized first in-vehicle AI model is deployed to the vehicle for evaluation in the controllable real environment.

[0032] Optionally, deploying the second in-vehicle AI model to the vehicle, and evaluating it in a real road environment, and outputting the target in-vehicle AI model includes:

[0033] Formulate different test plans and determine the real road environment corresponding to the test plans;

[0034] Deploying the second in-vehicle AI model to the vehicle, and evaluating the second in-vehicle AI model in different real road environments to obtain a third evaluation result; wherein the third evaluation result involves functional indicators and safety indicators of the second in-vehicle AI model;

[0035] When the third evaluation result shows that the second in-vehicle AI model passes the evaluation, the second in-vehicle AI model is used as the target in-vehicle AI model.

[0036] Optionally, the method further comprises:

[0037] When the third evaluation result shows that the second on-board AI model fails the evaluation, the second on-board AI model is optimized, and the optimized second on-board AI model is deployed to the vehicle and evaluated in the different controllable real environments.

[0038] According to a second aspect of the present disclosure, there is provided a vehicle-mounted AI model testing device, comprising:

[0039] An acquisition module, used to acquire a test data set of an on-board AI model to be tested; wherein the on-board AI model to be tested includes any one of the different on-board AI models developed for the vehicle;

[0040] A first evaluation module, used to perform simulation evaluation on the in-vehicle AI model to be tested according to the test data set, and output a first in-vehicle AI model;

[0041] The second evaluation module is used to deploy the first vehicle-mounted AI model to the vehicle, evaluate it in a real environment, and output a target vehicle-mounted AI model.

[0042] According to a third aspect of the present disclosure, an electronic device is provided, comprising a memory and a processor, wherein the memory is used to store an executable computer program; and the computer program is used to control the processor to execute the method according to the first aspect of the present disclosure.

[0043] According to a fourth aspect of the present disclosure, a vehicle is provided, comprising a memory and a processor, wherein the memory is used to store an executable computer program; and the computer program is used to control the processor to execute the method according to the first aspect of the present disclosure.

[0044] Through the vehicle-mounted AI model testing method of the present embodiment, any vehicle-mounted AI model among the different vehicle-mounted AI models developed for the vehicle can be used as the vehicle-mounted AI model to be tested, and for the vehicle-mounted AI model to be tested, first obtain the test data set of the vehicle-mounted AI model to be tested, and perform simulation evaluation on the vehicle-mounted AI model to be tested according to the test data set, output the first vehicle-mounted AI model, and deploy the first vehicle-mounted AI model to the vehicle, and perform evaluation in a real environment, and output the target vehicle-mounted AI model. Through the vehicle-mounted AI model testing method of the present embodiment, full-cycle and comprehensive test evaluation can be performed for AI models developed for different functions of the vehicle, and it has universality for various AI models developed for different functions of the vehicle.

[0045] Further features and advantages of the present invention will become apparent from the following detailed description of exemplary embodiments of the present invention with reference to the attached drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] The accompanying drawings, which are incorporated in and constitute a part of the specification, illustrate embodiments of the invention and, together with the description, serve to explain the principles of the invention.

[0047] Figure 1A block diagram showing a hardware configuration of an electronic device that can implement an embodiment of the present disclosure;

[0048] Figure 2 is a flowchart of a vehicle-mounted AI model testing method according to an embodiment of the present disclosure;

[0049] Figure 3 is a schematic diagram of an in-vehicle AI model testing method and its corresponding testing framework according to an example of the present disclosure;

[0050] Figure 4 is a flowchart of a data engineering phase according to an example of the present disclosure;

[0051] Figure 5 It is a flow chart of the server-side development and simulation evaluation phase according to an example of the present disclosure;

[0052] Figure 6 is a flow chart of a controllable real environment evaluation phase according to an example of the present disclosure;

[0053] Figure 7 is a flow chart of an actual vehicle deployment evaluation phase according to an example of the present disclosure;

[0054] Figure 8 is a block diagram of a vehicle-mounted AI model testing device according to an embodiment of the present disclosure;

[0055] Fig. 9 is a block diagram of an electronic device according to an embodiment of the present disclosure;

[0056] Fig.10 is a block diagram of a vehicle according to one embodiment of the present disclosure. DETAILED DESCRIPTION

[0057] Various exemplary embodiments of the present invention will now be described in detail with reference to the accompanying drawings. It should be noted that the relative arrangement of components and steps, numerical expressions and numerical values ​​set forth in these embodiments do not limit the scope of the present invention unless otherwise specifically stated.

[0058] The following description of at least one exemplary embodiment is merely illustrative in nature and is in no way intended to limit the invention, its application, or uses.

[0059] Technologies, methods and equipment known to persons of ordinary skill in the relevant art may not be discussed in detail, but where appropriate, the technologies, methods and equipment should be considered part of the specification.

[0060] In all examples shown and discussed herein, any specific values ​​should be interpreted as merely exemplary and not as limiting. Therefore, other examples of the exemplary embodiments may have different values.

[0061] It should be noted that like reference numerals and letters refer to similar items in the following figures, and therefore, once an item is defined in one figure, it need not be further discussed in subsequent figures.

[0062] <Hardware Configuration>

[0063] Figure 1 A block diagram showing a hardware configuration of an electronic device 1000 that can implement an embodiment of the present disclosure is shown.

[0064] The electronic device 1000 may be a portable computer, a desktop computer, a mobile phone, a tablet computer, etc. Figure 1 As shown, the electronic device 1000 may include a processor 1100, a memory 1200, an interface device 1300, a communication device 1400, a display device 1500, an input device 1600, a speaker 1700, a microphone 1800, and the like. Among them, the processor 1100 may be a central processing unit CPU, a microprocessor MCU, and the like. The memory 1200 includes, for example, a ROM (read-only memory), a RAM (random access memory), a non-volatile memory such as a hard disk, and the like. The interface device 1300 includes, for example, a USB interface, a headphone interface, and the like. The communication device 1400 is, for example, capable of wired or wireless communication, and may specifically include Wifi communication, Bluetooth communication, 2G / 3G / 4G / 5G communication, and the like. The display device 1500 is, for example, a liquid crystal display screen, a touch display screen, and the like. The input device 1600 may include, for example, a touch screen, a keyboard, a somatosensory input, and the like. The user may input / output voice information through the speaker 1700 and the microphone 1800.

[0065] Figure 1 The electronic device shown is merely illustrative and does not in any way imply any limitation on the present disclosure, its application or use. In the embodiments of the present disclosure, the memory 1200 of the electronic device 1000 is used to store instructions, and the instructions are used to control the processor 1100 to operate to perform any of the vehicle-mounted AI model testing methods provided in the embodiments of the present disclosure. It should be understood by those skilled in the art that although Figure 1 In the electronic device 1000, multiple devices are shown, but the present disclosure may only involve some of the devices, for example, the electronic device 1000 only involves the processor 1100 and the storage device 1200. A technician can design instructions according to the scheme disclosed in the present disclosure. How instructions control the processor to operate is well known in the art, so it will not be described in detail here.

[0066] <Method Example>

[0067] The present disclosure discloses a vehicle-mounted AI model testing method, such as Figure 2As shown, the vehicle-mounted AI model testing method includes the following steps S2100 to S2300.

[0068] Step S2100, obtaining a test data set of the in-vehicle AI model to be tested.

[0069] The in-vehicle AI model to be tested includes any one of the different in-vehicle AI models developed for the vehicle. For example, the in-vehicle AI model to be tested may be an in-vehicle AI model developed for the driving function of the vehicle; for another example, the in-vehicle AI model to be tested may be an in-vehicle AI model developed for different intelligent functions inside the vehicle, wherein the intelligent functions inside the vehicle include, for example but not limited to, intelligent air conditioning and intelligent seats.

[0070] In this embodiment, step S2100 serves as the data engineering stage, and its main purpose is to construct a data collection scenario based on the model functions of the on-board AI model to be tested, collect and process data based on the scenario, and finally generate a comprehensive and reliable test data set.

[0071] In an optional embodiment, step S2100 of obtaining a test data set for the on-board AI model to be tested may further include: constructing a natural environment scene according to the model function of the on-board AI model to be tested; collecting data in the natural environment scene to obtain natural scene data; performing data engineering processing on the natural scene data to generate a candidate test data set; verifying the test data in the candidate test data set according to the set verification items, and generating the test data set based on the test data that has passed the verification.

[0072] Among them, the verification items are set, for example, but not limited to verification involving coverage, sample balance, adversarial samples, sample quantity, and sample true value quality.

[0073] Among them, the above-mentioned data engineering processing of the natural scene data to generate a candidate test data set includes: performing data pre-processing on the natural scene data to obtain an intermediate test data set; generating the candidate test data set according to a sample generation method and the intermediate test data set; wherein the sample generation method at least includes an adversarial sample generation method, and the adversarial sample generation method is used to generate adversarial test data.

[0074] Among them, the above-mentioned natural environment scenes involve the first type of natural environment scenes and the second type of natural environment scenes.

[0075] The first type of scene can be understood as some common scenes in the natural environment, usually real environment scenes, such as but not limited to controllable real environment scenes and real road environment scenes. The controllable real environment scene can be, for example, a scene such as a closed road or a closed park. The real road environment scene can be, for example, a scene such as a random road, which usually has uncontrollable factors, such as being unable to control other vehicles or other pedestrians to drive on the road. In other words, the vehicle parameters and environmental parameters of the vehicle can be collected when the vehicle is operated in the controllable real environment scene and the real road environment scene respectively, thereby obtaining natural scene data.

[0076] The second type of scenario can be understood as some scenarios with a relatively low probability of occurring in the natural environment or some dangerous scenarios, which usually require simulation construction and can also be called simulation scenarios. In other words, when the vehicle is running in the constructed simulation scenario, the vehicle parameters and environmental parameters of the vehicle can be collected to obtain the natural scene data.

[0077] Specifically, refer to Figure 3 and Figure 4 The data engineering stage S1000 specifically includes the following S1010 to S1070.

[0078] S1010, preliminary preparation for data engineering.

[0079] Specifically, after clarifying the model functions of the in-vehicle AI model to be tested, a data scenario set is constructed to traverse all scenarios within the applicable scope.

[0080] The construction of the scene set can use the equivalence classification method, that is, use the values ​​of each category attribute to distinguish different scenes, and traverse all combinations of category attributes according to the comprehensive experimental design method. Clarify the implementation method of each natural environment scene.

[0081] S1020, determine data content and data type.

[0082] Specifically, determine the input data required for the in-vehicle AI model to be tested, and clarify the data type of each category of data, where the data type includes, for example but not limited to, numerical data, categorical data, text data, image data, or time series data.

[0083] S1030, formulate a plan for collecting natural environment scene data.

[0084] S1040, data collection and data labeling.

[0085] Specifically, according to the data collection scheme determined in S1030, natural scene data is collected in the natural environment scene, and the true value of the data is annotated using manual or algorithmic models.

[0086] It should be noted that when collecting natural scene data, it is necessary to follow relevant data laws and regulations to ensure that the data is reasonable and legal, and to ensure data security.

[0087] S1050, data processing.

[0088] Specifically, the collected natural scene data is subjected to closed-loop data processing, that is, data pre-processing (which can be called data cleaning), data enhancement processing and data generation processing are repeated until the data meets the requirements.

[0089] Among them, data pre-processing includes, for example but not limited to, data desensitization, selection of required information subsets, deletion of duplicate values, missing value processing, outlier processing, consistency processing and data sorting processing.

[0090] Among them, data enhancement processing includes, for example but not limited to, improving data quality (such as image dedistortion, dark light scene image enhancement, etc.) and increasing the number of samples (such as data generation and resampling for small sample categories).

[0091] The data generation is used to obtain the first type of test data and the second type of test data.

[0092] The first type of test data is ordinary test data. The generation of the first type of test data is as follows: based on different types of data, different data generation methods are considered. The specific process is to reconstruct the virtual environment on the server side based on part of the collected data, and adjust the relevant scene attribute parameters in combination with context semantics to generate new test data.

[0093] The second type of test data is adversarial test data, which is data generated based on an adversarial sample generation method, and can also be understood as interference data generated by a certain disturbance of the test data. The generation of the adversarial test data is as follows: According to the characteristics of the data, an adversarial sample generation method is selected to generate adversarial test data. The adversarial sample generation method can be an algorithm based on gradient generation of adversarial samples (Fast Gradient Sign Method, FGSM), and of course other algorithms, which are not limited in this embodiment.

[0094] S1060, Dataset quality check.

[0095] Specifically, check whether the test data set finally obtained meets the test requirements of the in-vehicle AI model to be tested, for example, the test data set can be checked from multiple dimensions such as coverage, sample balance, adversarial samples, sample quantity, sample true value quality, etc. If there are problems, data supplementation collection (S1040) or reprocessing (S1050) is required.

[0096] S1070, output the test data set.

[0097] According to the above S1010 to S1070, the test data set required for testing the in-vehicle AI model to be tested can be obtained.

[0098] Step S2200: Perform simulation evaluation on the in-vehicle AI model to be tested according to the test data set, and output a first in-vehicle AI model.

[0099] In this embodiment, step S2200 serves as the server-side development and simulation evaluation stage, and its main purpose is to simulate and evaluate the vehicle-mounted AI model to be tested based on the test data set and output the first vehicle-mounted AI model.

[0100] In an optional embodiment, this step S2200 performs simulation evaluation on the on-board AI model to be tested according to the test data set, and outputting the first on-board AI model may further include: performing model-in-environment evaluation, software-in-environment evaluation and hardware-in-environment evaluation on the on-board AI model to be tested in sequence according to the test data set, and outputting the first on-board AI model.

[0101] Of course, during the actual testing process, additional personnel in-environment evaluation and / or vehicle in-environment evaluation may be performed on the in-vehicle AI model to be tested according to the actual testing tasks.

[0102] Generally, the so-called in-environmental assessment corresponds to the open-loop assessment. In-environmental assessment means that the vehicle-mounted AI model is in this assessment process, and the vehicle-mounted AI model can control the vehicle according to the collected sensing data. For example, if the vehicle-mounted AI model detects a pedestrian crossing the road in a traffic simulation environment, the vehicle-mounted AI model can control the vehicle to stop in the traffic simulation environment.

[0103] Specifically, refer to Figure 3 and Figure 5 ,The server side development and simulation evaluation phase S2000 specifically includes the following S2010 to S2050.

[0104] S2010, Model-in-the-Environment Assessment.

[0105] Model in-the-environment evaluation refers to the evaluation of the on-board AI model using a test data set during the development of the on-board AI model on the server side. Specifically, the test data set is used on the server side to calculate the mathematical and statistical indicators of the on-board AI model to be tested, and the performance of the on-board AI model to be tested is measured based on the calculated mathematical and statistical indicators. Among them, mathematical and statistical indicators include, but are not limited to, accuracy, recall rate, mean square error, mean absolute error, intersection over union ratio, computational complexity, and resource consumption.

[0106] S2020, software in the environment evaluation.

[0107] The software environmental assessment is to convert the on-board AI model that has passed the environmental assessment from the original programming language into a common format such as C++ required for embedded deployment or cloud deployment, use the test data set to calculate various mathematical and statistical indicators, and analyze whether the performance of the on-board AI model in the corresponding format after conversion and compilation is consistent with the performance of the on-board AI model before conversion.

[0108] S2030, hardware in the environment evaluation.

[0109] The hardware-in-the-environment evaluation is to import the on-board AI model that has passed the evaluation in step S202 on the vehicle system-level (SoC) chip side, input the test data set on the chip side, transmit the output of the on-board AI model on the test data set back, calculate various mathematical and statistical indicators on the server side, analyze whether the on-board AI model running on the SoC chip side is affected by the chip platform architecture, and evaluate the performance level of the model deployed on the SoC chip side.

[0110] S2040, personnel in-environment assessment.

[0111] The personnel in-the-environment assessment phase is an assessment phase for testing some related functions that require on-board personnel to operate. If there are no related functions, this assessment phase can be ignored. Specifically, the on-board AI model that has passed the evaluation in step S2030 is loaded into the simulation software as the control system. The tester operates according to the functional scenario, analyzes whether the on-board AI model's response to the given personnel operation input meets expectations, and quantifies the evaluation results according to the functional evaluation indicators.

[0112] S2050, vehicle environmental assessment.

[0113] The vehicle in-environmental assessment phase is aimed at some functional scenarios that require vehicle systems. The complete vehicle system is embedded in a simulation loop for simulation testing, and the performance and functions of the on-board AI model that has passed the evaluation in step S2030 or step S2040 are analyzed to see whether they meet the predetermined goals.

[0114] In this embodiment, for any one of the environmental assessment links, it specifically includes: evaluating the on-board AI model currently tested in the environmental assessment link according to the test data set, and obtaining a first evaluation result of the on-board AI model currently tested in the environmental assessment link; wherein, the first evaluation result involves performance indicators, functional indicators and safety indicators; when the first evaluation result is that the on-board AI model currently tested in the environmental assessment link fails the evaluation, the on-board AI model currently tested in the environmental assessment link is optimized, and the optimized on-board AI model is continued to be evaluated according to the test data set; when the first evaluation result is that the on-board AI model currently tested in the environmental assessment link passes the evaluation, the on-board AI model currently tested in the environmental assessment link is used as the on-board AI model to be tested in the next environmental assessment link.

[0115] Specifically, refer to Figure 3 and Figure 5 , the specific assessment process for any environmental assessment link S2001 includes the following S2111~S2117.

[0116] S2111, select a test data set.

[0117] The selected test data set may include the entire test data set of the above step S2100 or a part of the test data set.

[0118] It should be noted that the selected test data set includes normal test data and adversarial test data, and the adversarial test data is used for adversarial evaluation.

[0119] S2112, determine the test platform and test related equipment.

[0120] S2113, Develop test methods and test standards.

[0121] S2114, conduct relevant simulation evaluation tests.

[0122] In this step S2114, the vehicle-mounted AI model in the current environmental assessment phase can be evaluated based on the test data set, and a first evaluation result can be output. It should be noted that the evaluation at least includes adversarial evaluation.

[0123] Among them, the first evaluation result involves the quantitative scores corresponding to the performance indicators, functional indicators and safety indicators respectively, and the first evaluation result also involves the total score of each indicator. Usually, the performance indicator is the mathematical and statistical indicator of the vehicle-mounted AI model, such as but not limited to accuracy, recall rate, mean square error, mean absolute error, intersection-over-union ratio, computational complexity, resource consumption, and functional indicators can also be proposed based on the corresponding tasks. Safety indicators can be obtained based on adversarial detection.

[0124] S2115, standard determination.

[0125] In this step S2115, the vehicle-mounted AI model currently in the environmental assessment link is judged according to the first evaluation result. If the first evaluation result shows that the vehicle-mounted AI model currently in the environmental assessment link fails the evaluation, for example, if the quantitative scores corresponding to the functional indicators and the safety indicators respectively fail, the process proceeds to S2116. On the contrary, if the first evaluation result shows that the vehicle-mounted AI model currently in the environmental assessment link passes the evaluation, the process proceeds to S2117.

[0126] S2116, iteratively tune the model.

[0127] In this step S2116, when the first evaluation result is that the on-board AI model currently in the environmental assessment link fails the evaluation, the on-board AI model currently in the environmental assessment link will be subjected to targeted analysis, debugging and optimization will be performed to overcome the model problems exposed in the current environmental assessment link, adjust the model parameters, and then re-enter S2114.

[0128] S2117, output the evaluation results and the optimal model.

[0129] In this step S2117, when the first evaluation result shows that the on-board AI model of the current in-environmental evaluation link has passed the evaluation, the first evaluation result and the on-board AI model finally outputted by the current evaluation link are output to prepare for the next in-environmental evaluation link, so as to ensure that the on-board AI model can achieve the expected performance and functional level in each evaluation link.

[0130] Step S2300: deploy the first in-vehicle AI model to the vehicle, evaluate it in a real environment, and output a target in-vehicle AI model.

[0131] In this embodiment, this step S2300 deploys the first vehicle-mounted AI model to the vehicle and performs an evaluation in a real environment, and outputting the target vehicle-mounted AI model may further include steps S2310 to S2320:

[0132] Step S2310: deploy the first vehicle-mounted AI model to the vehicle, evaluate it in a controllable real environment, and output a second vehicle-mounted AI model.

[0133] Among them, the controllable real environment can be understood as a closed real environment, such as a closed road or a closed park.

[0134] In this embodiment, step S2310 serves as a controllable real environment evaluation stage, which deploys the first vehicle-mounted AI model output by step S2200 to an actual vehicle, and selects different controllable real environments to verify whether the function and performance of the first vehicle-mounted AI model on the actual vehicle meet the expected level.

[0135] In an optional embodiment, this step S2310 deploys the first vehicle-mounted AI model to the vehicle and evaluates it in a controllable environment. Outputting the second vehicle-mounted AI model may further include: formulating different test scenarios and determining the controllable environment corresponding to the test scenarios; wherein the different test scenarios include conventional test scenarios and adversarial scenarios; deploying the first vehicle-mounted AI model to the vehicle, and evaluating the first vehicle-mounted AI model in different controllable real environments to obtain a second evaluation result; wherein the second evaluation result involves performance indicators, functional indicators and safety indicators; when the second evaluation result is that the first vehicle-mounted AI model passes the evaluation, the first vehicle-mounted AI model is used as the second vehicle-mounted AI model.

[0136] It should be noted that adversarial scenarios are scenarios that easily cause AI models to make errors. They are used to evaluate the stability and security of AI models. They are mainly tested by considering the characteristics of AI models that are easily disturbed. In layman's terms, adversarial scenarios can be understood as making some artificial adjustments to the scene for the AI ​​model, such as posting some interference patterns on traffic signs and then testing the AI ​​model.

[0137] It should be noted that when the second evaluation result shows that the first vehicle-mounted AI model fails the evaluation, the first vehicle-mounted AI model is optimized, and the optimized first vehicle-mounted AI model is deployed to the vehicle for evaluation in the different controllable real environments.

[0138] Specifically, refer to Figure 3 and Figure 6 The controllable real environment evaluation stage S3000 specifically includes the following S3010~S3070.

[0139] S3010, general test scenario and adversarial scenario settings.

[0140] In this step S3010, all application scenarios applicable to the in-vehicle AI model to be tested are identified, and appropriate test environments are selected based on these scenarios. Among them, adversarial scenarios are used for safety testing of in-vehicle AI models in actual vehicles.

[0141] S3020, Development of test plans and test plans.

[0142] In this step S3020, an adaptive test plan is formulated to specify the test vehicle, test equipment, test links, test parameters, test indicators and test personnel. A test plan is formulated to ensure that problems that arise during the test process can be handled in a timely manner.

[0143] S3030, Develop test standards.

[0144] In this step S3030, evaluation indicators are designed, and based on relevant prior knowledge, quantitative scoring standards for the evaluation indicators are determined.

[0145] Among them, the evaluation indicators include performance indicators, functional indicators and safety indicators.

[0146] Among them, performance indicators can also be called mathematical and statistical indicators, which are used to characterize the average performance of on-board AI models running on actual vehicles.

[0147] Among them, functional indicators are used to evaluate the actual functions and human-computer interaction of the in-vehicle AI model.

[0148] Among them, safety indicators are used to evaluate the ability of on-board AI models to defend against adversarial scenarios.

[0149] S3040, model + post-processing algorithm imported into real vehicle test.

[0150] In this step S3040, the on-board AI model to be tested and the post-processing algorithm of the statistical indicators are imported into the actual vehicle, and the relevant algorithms are run during the operation of the vehicle to output the second evaluation result.

[0151] Among them, the second evaluation result involves quantitative scores corresponding to performance indicators, functional indicators and safety indicators respectively.

[0152] S3050, standard determination.

[0153] In this step S3050, the vehicle-mounted AI model at the current stage is judged according to the second evaluation result. If the second evaluation result shows that the vehicle-mounted AI model at the current stage fails the evaluation, for example, if the quantitative score corresponding to the safety index fails, the vehicle-mounted AI model is judged to fail the evaluation, and the process proceeds to S3060. On the contrary, if the second evaluation result shows that the vehicle-mounted AI model at the current stage passes the evaluation, the process proceeds to S3070.

[0154] S3060, iteratively tune the model.

[0155] In this step S3060, when the second evaluation result shows that the on-board AI model of the current stage fails the evaluation, the on-board AI model of the current stage is subjected to targeted analysis, debugging and optimization are performed to overcome the model problems exposed at the current stage, adjust the model parameters, and then re-enter S3040.

[0156] S3070, output the optimal model of the controllable environment evaluation link.

[0157] In this step S3070, when the second evaluation result shows that the on-board AI model of the current in-circuit evaluation phase has passed the evaluation, the second evaluation result and the on-board AI model finally outputted through the current stage are output to enter the evaluation process for actual vehicle deployment.

[0158] After executing step S2310 to deploy the first vehicle-mounted AI model to the vehicle and conduct an evaluation in a controllable real environment and output the second vehicle-mounted AI model, proceed to:

[0159] Step S2320: deploy the second in-vehicle AI model to the vehicle, evaluate it in a real road environment, and output the target in-vehicle AI model.

[0160] The real road environment refers to a random real environment, such as a random road. The road usually has uncontrollable factors, such as being unable to control other vehicles or other pedestrians from driving on the road.

[0161] In this embodiment, step S2320 serves as the actual vehicle deployment and evaluation stage, which is to deploy the second vehicle-mounted AI model output by step S2310 to the actual vehicle and conduct tests in different actual environments. Usually, the actual environment is complex, random, and uncontrollable, and it can test whether the model can operate according to the expected functions in a real environment without external intervention.

[0162] In an optional embodiment, this step S2320 deploys the second on-board AI model to the vehicle and evaluates it in a real road environment. Outputting the target on-board AI model may further include: formulating different test schemes to determine the real road environment corresponding to the test schemes; deploying the second on-board AI model to the vehicle, and evaluating the second on-board AI model in different real road environments to obtain a third evaluation result; wherein the third evaluation result involves functional indicators and safety indicators of the second on-board AI model; if the third evaluation result is that the second on-board AI model passes the evaluation, the second on-board AI model is used as the target on-board AI model.

[0163] It should be noted that when the third evaluation result shows that the second vehicle-mounted AI model fails the evaluation, the second vehicle-mounted AI model is optimized, and the optimized second vehicle-mounted AI model is deployed to the vehicle and evaluated in the different controllable real environments.

[0164] Specifically, refer to Figure 3 and Figure 7 The actual vehicle deployment evaluation stage S4000 includes the following S4010 to S4080:

[0165] S4010, Development of test plans and test plans.

[0166] In this step S4010, according to relevant laws and regulations, the test vehicle, test equipment, test links, test parameters, test indicators and test personnel are clearly defined, and a test plan is formulated to ensure that problems that arise during the test process can be handled in a timely manner.

[0167] S4020, test road selection is complex, random and uncontrollable.

[0168] In this step S4020, the test is conducted in an open field where the test is allowed according to the test plan. The test environment is pre-analyzed to clarify the coverage scenarios, ensure that general scenarios are covered, and do not conduct tests on adversarial scenarios.

[0169] S4030, determining the coverage of the test road for general scenarios.

[0170] In this step S4030, the scenes involved in the selected site are checked, all sites are traversed to extract the complete set of test scenes, unreasonable scenes are eliminated, summarized and merged, scenes with high similarity are removed, and the complete set of test scenes is constructed.

[0171] If the coverage is qualified, the process proceeds to step S4040 to formulate the test standards. If the coverage is unqualified, the process proceeds to step S4030 to supplement the test sites to add new test scenarios.

[0172] S4040, Establishing test standards.

[0173] At the current stage, the evaluation indicators formulated mainly include functional indicators, which evaluate the application effect and human-computer interaction of the in-vehicle AI model. Formulate quantitative scoring rules for the evaluation indicators and clarify the corresponding relationship between the functional description and the evaluation results. At the same time, the formulated evaluation indicators also include safety indicators. This is because it is necessary to consider whether the in-vehicle AI model has an impact on the safety level of the vehicle during operation, and conduct subjective evaluation through dimensions such as resource occupation and model output interference.

[0174] S4050, the real vehicle deployment model is tested on the test road.

[0175] In this step S4050, during the actual operation of the vehicle, the corresponding scenario function is implemented through the on-board AI model, and the third evaluation result is output.

[0176] Among them, the third evaluation result involves quantitative scores corresponding to functional indicators and safety indicators respectively.

[0177] S4060, standard determination.

[0178] In this step S4060, the on-board AI model of the current stage is judged according to the third evaluation result. When the third evaluation result shows that the on-board AI model of the current stage fails the evaluation, for example, when the quantitative score corresponding to the safety indicator fails, the process proceeds to S4070. On the contrary, when the third evaluation result shows that the on-board AI model of the current stage passes the evaluation, the process proceeds to S4080.

[0179] S4070, iteratively tune the model.

[0180] In this step S4070, when the third evaluation result shows that the on-board AI model of the current stage fails the evaluation, the on-board AI model of the current stage is subjected to targeted analysis, debugging and optimization are performed to overcome the model problems exposed in the current stage, adjust the model parameters, and then re-enter the controllable real environment evaluation stage of S3000.

[0181] It should be noted that after the model is improved, it is necessary to enter the S3000 stage for evaluation in a controllable real environment to analyze whether the exposed problems have been resolved.

[0182] S4080, model evaluation results given real vehicle deployment.

[0183] In this step S4080, when the third evaluation result shows that the on-board AI model of the current environmental evaluation phase has passed the evaluation, the third evaluation result and the final model that has passed the evaluation of the current phase are output, and then the on-board AI model of the actual vehicle is deployed and mass-produced.

[0184] According to the disclosed embodiment, any one of the different on-board AI models developed for the vehicle can be used as the on-board AI model to be tested, and for the on-board AI model to be tested, first obtain a test data set of the on-board AI model to be tested, and perform simulation evaluation on the on-board AI model to be tested based on the test data set, output a first on-board AI model, and deploy the first on-board AI model to the vehicle, and perform evaluation in a real environment, and output a target on-board AI model. Through the on-board AI model testing method of the disclosed embodiment, full-cycle and all-round testing and evaluation can be performed for AI models developed for different functions of the vehicle, and it has universality for various AI models developed for different functions of the vehicle.

[0185] Compared with the prior art, on the one hand, the in-vehicle AI model testing method proposed in the embodiment of the present disclosure can perform full-cycle and comprehensive testing and evaluation on AI models developed with different functions. From the model development stage to the actual vehicle deployment stage, corresponding testing and evaluation processes and experimental methods are designed, which can provide useful information on performance, safety, cost-effectiveness and feasibility for the development and deployment of in-vehicle AI models.

[0186] On the other hand, the disclosed embodiments formulate a data engineering preparation stage based on the data-driven characteristics of the AI ​​model, and formulate a closed-loop standardized process for acquiring test data including collection, processing, generation, and quality inspection. This can more effectively acquire data with more comprehensive coverage and better detect the AI ​​model in the testing phase.

[0187] On the other hand, the in-vehicle AI model testing method proposed in the embodiment of the present disclosure has the ability to supervise the in-vehicle AI model's defense attack level, evaluate the model from the perspective of robustness and security, and can provide an adversarial defense level check for the AI ​​model, thereby helping to optimize the model and making an effective contribution to the security of the in-vehicle AI model.

[0188] <Device Example>

[0189] The present disclosure also provides a vehicle-mounted AI model testing device 8000, such as Figure 8 As shown, the vehicle-mounted AI model testing device 8000 includes an acquisition module 8100, a first evaluation module 8200, and a second evaluation module 8300.

[0190] An acquisition module 8100 is used to acquire a test data set of an in-vehicle AI model to be tested; wherein the in-vehicle AI model to be tested includes any in-vehicle AI model among different in-vehicle AI models developed for the vehicle;

[0191] A first evaluation module 8200 is used to perform simulation evaluation on the in-vehicle AI model to be tested according to the test data set, and output a first in-vehicle AI model;

[0192] The second evaluation module 8300 is used to deploy the first vehicle-mounted AI model to the vehicle, evaluate it in a real environment, and output a target vehicle-mounted AI model.

[0193] In one embodiment, the acquisition module 8100 is specifically used to construct a natural environment scene according to the model function of the on-board AI model to be tested; perform data collection in the natural environment scene to obtain natural scene data; perform data engineering processing on the natural scene data to generate a candidate test data set; verify the test data in the candidate test data set according to the set verification items, and generate the test data set based on the test data that has passed the verification.

[0194] In one embodiment, the acquisition module 8100 is specifically used to perform data pre-processing on the natural scene data to obtain an intermediate test data set; generate the candidate test data set according to the sample generation method and the intermediate test data set; wherein the sample generation method at least includes an adversarial sample generation method, and the adversarial sample generation method is used to generate adversarial test data.

[0195] In one embodiment, the first evaluation module 8200 is specifically used to perform model-in-environment evaluation, software-in-environment evaluation and hardware-in-environment evaluation on the on-board AI model to be tested in sequence according to the test data set, and output the first on-board AI model.

[0196] In one embodiment, the first evaluation module 8200 is specifically used to evaluate the on-board AI model currently tested in the environmental assessment link according to the test data set, and obtain a first evaluation result of the on-board AI model currently tested in the environmental assessment link; wherein, the first evaluation result involves performance indicators, functional indicators and safety indicators; when the first evaluation result is that the on-board AI model currently tested in the environmental assessment link has passed the evaluation, the on-board AI model currently tested in the environmental assessment link will be used as the on-board AI model to be tested in the next environmental assessment link.

[0197] In one embodiment, the first evaluation module 8200 is specifically used to optimize the on-board AI model currently tested in the environmental assessment link when the first evaluation result is that the on-board AI model currently tested in the environmental assessment link fails the evaluation, and continue to evaluate the optimized on-board AI model according to the test data set.

[0198] In one embodiment, the second evaluation module 8300 is specifically used to deploy the first vehicle-mounted AI model to the vehicle, evaluate it in a controllable real environment, and output a second vehicle-mounted AI model; and deploy the second vehicle-mounted AI model to the vehicle, evaluate it in a real road environment, and output the target vehicle-mounted AI model.

[0199] In one embodiment, the second evaluation module 8300 is specifically used to formulate different test scenarios and determine the controllable environment corresponding to the test scenarios; wherein the different test scenarios include at least adversarial scenarios; deploy the first vehicle-mounted AI model to the vehicle, and evaluate the first vehicle-mounted AI model in different controllable real environments to obtain a second evaluation result; wherein the second evaluation result involves performance indicators, functional indicators and safety indicators; when the second evaluation result is that the first vehicle-mounted AI model passes the evaluation, the first vehicle-mounted AI model is used as the second vehicle-mounted AI model.

[0200] In one embodiment, the second evaluation module 8300 is specifically used to optimize the first vehicle-mounted AI model when the second evaluation result shows that the first vehicle-mounted AI model fails the evaluation, and deploy the optimized first vehicle-mounted AI model to the vehicle for evaluation in the different controllable real environments.

[0201] In one embodiment, the second evaluation module 8300 is specifically used to formulate different test schemes and determine the real road environment corresponding to the test schemes; deploy the second on-board AI model to the vehicle, and evaluate the second on-board AI model in the different real road environments to obtain a third evaluation result; wherein the third evaluation result involves functional indicators and safety indicators of the second on-board AI model; if the third evaluation result is that the second on-board AI model passes the evaluation, the second on-board AI model is used as the target on-board AI model.

[0202] In one embodiment, the second evaluation module 8300 is specifically used to optimize the second vehicle-mounted AI model when the third evaluation result is that the second vehicle-mounted AI model fails the evaluation, and deploy the optimized second vehicle-mounted AI model to the vehicle, and evaluate it in the different controllable real environments.

[0203] According to the disclosed embodiment, any one of the different on-board AI models developed for the vehicle can be used as the on-board AI model to be tested, and for the on-board AI model to be tested, first obtain a test data set of the on-board AI model to be tested, and perform simulation evaluation on the on-board AI model to be tested based on the test data set, output a first on-board AI model, and deploy the first on-board AI model to the vehicle, and perform evaluation in a real environment, and output a target on-board AI model. Through the on-board AI model testing method of the disclosed embodiment, full-cycle and all-round testing and evaluation can be performed for AI models developed for different functions of the vehicle, and it has universality for various AI models developed for different functions of the vehicle.

[0204] <Electronic equipment>

[0205] In this embodiment, an electronic device 9000 is also provided. Fig. 9 As shown, it includes a memory 9200 and a processor 9100.

[0206] The memory 9200 is used to store executable computer programs; the computer programs are used to control the processor 9100 to execute any one of the vehicle-mounted AI model testing methods provided in this embodiment.

[0207] In this embodiment, the electronic device 9000 may be Figure 1 The electronic device 1000 shown may also be a device with other hardware structures, which is not limited here.

[0208] In another embodiment, the electronic device 9000 may include the above-mentioned vehicle-mounted AI model testing device 8000.

[0209] <Vehicle>

[0210] In this embodiment, a vehicle 100 is also provided. Fig.10 As shown, it includes a memory 120 and a processor 110 .

[0211] The memory 120 is used to store an executable computer program; the computer program is used to control the processor 110 to execute any one of the vehicle-mounted AI model testing methods provided in this embodiment.

[0212] In this embodiment, the server may first perform simulation evaluation on the on-board AI model, and deploy the on-board AI model that passes the simulation evaluation to the vehicle 100 for evaluation in a real environment.

[0213] <Computer Readable Storage Medium>

[0214] In this embodiment, a computer-readable storage medium is further provided, on which a computer program is stored. When the computer program is executed by a processor, the method of any embodiment of the present disclosure is implemented.

[0215] The present invention may be a system, a method and / or a computer program product. The computer program product may include a computer-readable storage medium carrying computer-readable program instructions for causing a processor to implement various aspects of the present invention.

[0216] A computer-readable storage medium may be a tangible device that can hold and store instructions used by an instruction execution device. A computer-readable storage medium may be, for example, but not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. More specific examples of computer-readable storage media (a non-exhaustive list) include: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disk read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanical encoding device, such as a punch card or a raised structure in a groove on which instructions are stored, and any suitable combination of the foregoing. As used herein, a computer-readable storage medium is not to be interpreted as a transient signal per se, such as a radio wave or other freely propagating electromagnetic wave, an electromagnetic wave propagating through a waveguide or other transmission medium (e.g., a light pulse through a fiber optic cable), or an electrical signal transmitted through a wire.

[0217] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to each computing / processing device, or downloaded to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. The network can include copper transmission cables, optical fiber transmissions, wireless transmissions, routers, firewalls, switches, gateway computers, and / or edge servers. The network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions for storage in the computer-readable storage medium in each computing / processing device.

[0218] The computer program instructions for performing the operation of the present invention may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-related instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages, such as Smalltalk, C++, etc., and conventional procedural programming languages, such as "C" language or similar programming languages. Computer-readable program instructions may be executed entirely on a user's computer, partially on a user's computer, as an independent software package, partially on a user's computer, partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., using an Internet service provider to connect via the Internet). In some embodiments, an electronic circuit, such as a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA), may be personalized by utilizing the state information of the computer-readable program instructions, and the electronic circuit may execute the computer-readable program instructions, thereby realizing various aspects of the present invention.

[0219] Various aspects of the present invention are described herein with reference to the flow charts and / or block diagrams of the methods, devices (systems) and computer program products according to embodiments of the present invention. It should be understood that each box of the flow chart and / or block diagram and the combination of each box in the flow chart and / or block diagram can be implemented by computer-readable program instructions.

[0220] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, thereby producing a machine, so that when these instructions are executed by the processor of the computer or other programmable data processing device, a device that implements the functions / actions specified in one or more boxes in the flowchart and / or block diagram is generated. These computer-readable program instructions can also be stored in a computer-readable storage medium, and these instructions cause the computer, programmable data processing device, and / or other equipment to work in a specific manner, so that the computer-readable medium storing the instructions includes a manufactured product, which includes instructions for implementing various aspects of the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0221] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device so that a series of operating steps are performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to implement the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0222] The flow charts and block diagrams in the accompanying drawings show the possible architecture, functions and operations of the systems, methods and computer program products according to multiple embodiments of the present invention. In this regard, each box in the flow chart or block diagram can represent a part of a module, a program segment or an instruction, and a part of the module, a program segment or an instruction contains one or more executable instructions for realizing the specified logical function. In some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flow chart, and the combination of the boxes in the block diagram and / or flow chart can be implemented by a dedicated hardware-based system that performs the specified function or action, or can be implemented by a combination of dedicated hardware and computer instructions. It is well known to those skilled in the art that it is equivalent to implement it by hardware, implement it by software, and implement it by combining software and hardware.

[0223] Embodiments of the present invention have been described above, and the above description is exemplary, not exhaustive, and is not limited to the disclosed embodiments. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The selection of terms used herein is intended to best explain the principles of the embodiments, practical applications, or technical improvements to the technology in the marketplace, or to enable other persons of ordinary skill in the art to understand the embodiments disclosed herein. The scope of the present invention is defined by the appended claims.

Claims

1. A vehicle-mounted AI model testing method, characterized in that: include: Obtain a test data set of an in-vehicle AI model to be tested; wherein the in-vehicle AI model to be tested includes any in-vehicle AI model among different in-vehicle AI models developed for the vehicle; According to the test data set, a simulation evaluation is performed on the in-vehicle AI model to be tested, and a first in-vehicle AI model is output; The first in-vehicle AI model is deployed to the vehicle, and is evaluated in a real environment to output a target in-vehicle AI model.

2. The method according to claim 1, characterized in that The step of obtaining a test data set of the vehicle-mounted AI model to be tested includes: Constructing a natural environment scene according to the model function of the in-vehicle AI model to be tested; Collecting data in the natural environment scene to obtain natural scene data; Performing data engineering processing on the natural scene data to generate a candidate test data set; According to the set verification items, the test data in the candidate test data set is verified, and the test data set is generated according to the test data that passes the verification.

3. The method according to claim 2, characterized in that The performing data engineering processing on the natural scene data to generate a candidate test data set includes: Performing data pre-processing on the natural scene data to obtain an intermediate test data set; Generate the candidate test data set according to the sample generation method and the intermediate test data set; Among them, the sample generation method at least includes an adversarial sample generation method, and the adversarial sample generation method is used to generate adversarial test data.

4. The method according to claim 1, characterized in that: The step of performing simulation evaluation on the in-vehicle AI model to be tested according to the test data set and outputting a first in-vehicle AI model comprises: According to the test data set, the on-board AI model to be tested is subjected to model-in-environmental assessment, software-in-environmental assessment and hardware-in-environmental assessment in sequence, and the first on-board AI model is output.

5. The method according to claim 4, characterized in that For any environmental assessment link, the method further includes: According to the test data set, the vehicle-mounted AI model currently tested in the environmental assessment phase is evaluated to obtain a first evaluation result of the vehicle-mounted AI model currently tested in the environmental assessment phase; wherein the first evaluation result involves performance indicators, functional indicators, and safety indicators; If the first evaluation result is that the on-board AI model currently tested in the environmental assessment phase has passed the evaluation, the on-board AI model currently tested in the environmental assessment phase will be used as the on-board AI model to be tested in the next environmental assessment phase.

6. The method according to claim 5, characterized in that The method further comprises: When the first evaluation result is that the on-board AI model currently tested in the environmental assessment link fails the evaluation, the on-board AI model currently tested in the environmental assessment link is optimized, and the optimized on-board AI model continues to be evaluated based on the test data set.

7. The method according to claim 1, characterized in that The step of deploying the first vehicle-mounted AI model to the vehicle, performing an evaluation in a real environment, and outputting a target vehicle-mounted AI model includes: Deploy the first vehicle-mounted AI model to the vehicle, and conduct an evaluation in a controllable real environment to output a second vehicle-mounted AI model; and, The second in-vehicle AI model is deployed to the vehicle, and is evaluated in a real road environment to output the target in-vehicle AI model.

8. The method according to claim 7, characterized in that The step of deploying the first vehicle-mounted AI model to the vehicle, performing an evaluation in a controllable real environment, and outputting a second vehicle-mounted AI model includes: Formulate different test scenarios and determine controllable real environments corresponding to the test scenarios; wherein the different test scenarios include conventional test scenarios and adversarial scenarios; Deploying the first in-vehicle AI model to the vehicle, and evaluating the first in-vehicle AI model in different controllable real environments to obtain a second evaluation result; wherein the second evaluation result involves performance indicators, functional indicators, and safety indicators; If the second evaluation result is that the first in-vehicle AI model passes the evaluation, the first in-vehicle AI model is used as the second in-vehicle AI model.

9. The method according to claim 8, characterized in that The method further comprises: When the second evaluation result shows that the first in-vehicle AI model fails the evaluation, the first in-vehicle AI model is optimized, and the optimized first in-vehicle AI model is deployed to the vehicle for evaluation in the controllable real environment.

10. The method according to claim 7, characterized in that The step of deploying the second vehicle-mounted AI model to the vehicle, performing an evaluation in a real road environment, and outputting the target vehicle-mounted AI model includes: Formulate different test plans and determine the real road environment corresponding to the test plans; Deploying the second in-vehicle AI model to the vehicle, and evaluating the second in-vehicle AI model in different real road environments to obtain a third evaluation result; wherein the third evaluation result involves functional indicators and safety indicators of the second in-vehicle AI model; When the third evaluation result shows that the second in-vehicle AI model passes the evaluation, the second in-vehicle AI model is used as the target in-vehicle AI model.

11. The method according to claim 10, characterized in that The method further comprises: When the third evaluation result shows that the second on-board AI model fails the evaluation, the second on-board AI model is optimized, and the optimized second on-board AI model is deployed to the vehicle and evaluated in the controllable real environment.

12. A vehicle-mounted AI model testing device, characterized in that: include: An acquisition module, used to acquire a test data set of an on-board AI model to be tested; wherein the on-board AI model to be tested includes any one of the different on-board AI models developed for the vehicle; A first evaluation module, used to perform simulation evaluation on the in-vehicle AI model to be tested according to the test data set, and output a first in-vehicle AI model; The second evaluation module is used to deploy the first vehicle-mounted AI model to the vehicle, evaluate it in a real environment, and output a target vehicle-mounted AI model.

13. An electronic device, characterized in that: The method comprises a memory and a processor, wherein the memory is used to store an executable computer program; and the computer program is used to control the processor to execute the method according to any one of claims 1 to 11.

14. A vehicle, characterized in that: The method comprises a memory and a processor, wherein the memory is used to store an executable computer program; and the computer program is used to control the processor to execute the method according to any one of claims 1 to 11.

Images