Machine learning model evaluation method, data processing method and related device

Abstract

A machine learning model evaluation method, a data processing method, and a related device. The machine learning model evaluation method can be used in the field of autonomous driving in artificial intelligence, and comprises: processing a plurality of pieces of fragment data in an evaluation sample by means of a machine learning model, so as to obtain a plurality of prediction labels; and determining a parameter value of at least one evaluation index. The evaluation sample comprises description data of a traffic scenario within a first duration, the fragment data comprises description data of the traffic scenario within a first sub-duration, and the first duration comprises the first sub-duration; and the at least one evaluation index indicates stability and / or accuracy corresponding to the plurality of prediction labels, and the accuracy is obtained on the basis of truth values corresponding to the plurality of pieces of fragment data. "Constructing evaluation indexes at the granularity of traffic scenarios" can more accurately reflect the performance of the machine learning model applied in a vehicle driving process, and simulated traffic scenarios do not need to be established, reducing the computer resource consumption and improving the evaluation efficiency.

Description

Machine learning model evaluation methods, data processing methods, and related equipment

[0001] This application claims priority to the Chinese patent application filed with the State Intellectual Property Office on August 15, 2023, with application number 202311033757.6 and application name “Evaluation method, data processing method and related equipment for machine learning models”, the entire contents of which are incorporated by reference into this application. Technical Field

[0002] The present application relates to the field of artificial intelligence, and in particular to a method for evaluating a machine learning model, a data processing method, and related equipment. Background Art

[0003] Artificial Intelligence (AI) is the theory, methods, techniques, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, to perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results. In other words, AI is a branch of computer science that seeks to understand the essence of intelligence and produce new intelligent machines that can respond in a manner similar to human intelligence. AI studies the design principles and implementation methods of various intelligent machines, enabling them to possess the capabilities of perception, reasoning, and decision-making. Applying machine learning models to autonomous driving is a common application of AI.

[0004] Currently, in order to evaluate the trained machine learning model, a simulated traffic environment can be established, so that vehicles equipped with machine learning models can travel in the simulated traffic environment, thereby evaluating the performance of multiple machine learning models.

[0005] However, since establishing a simulated traffic environment consumes a lot of computer resources and the process of establishing a simulated traffic environment is time-consuming, the evaluation efficiency is low.

[0006] Summary of the Invention

[0007] In order to solve the above technical problems, the present application provides a machine learning model evaluation method, data processing method and related equipment. The evaluation indicators used are constructed at the granularity of traffic scenes. Since the vehicle driving process is continuous, "constructing evaluation indicators at the granularity of traffic scenes" can more accurately reflect the performance of the machine learning model applied to the vehicle driving process; in addition, the solution provided by the present application does not require the establishment of a simulated traffic scene, which reduces the computer resources consumed by the evaluation process of the machine learning model and improves the evaluation efficiency.

[0008] The technical solutions provided in this application are as follows:

[0009] In a first aspect, an embodiment of the present application provides a method for evaluating a machine learning model, which can be used in the field of autonomous driving in artificial intelligence. The method includes: a first device obtains a first evaluation sample, the first evaluation sample includes descriptive data of a first traffic scene within a first time length, the first evaluation sample includes multiple first fragment data, each first fragment data includes descriptive data of the first traffic scene within a first sub-time length, and the first time length may include one or more first sub-time lengths; exemplarily, the aforementioned descriptive data may be specifically expressed as video, image, scene description file or other types of data, etc. For example, the scene description file may include the position, speed, acceleration or other status information of each traffic participant in the first traffic scene, etc.

[0010] The first device processes multiple first fragment data through a first machine learning model to obtain multiple first prediction labels corresponding one-to-one to the multiple first fragment data; and then determines a first parameter value of at least one evaluation indicator based on the multiple first prediction labels.

[0011] Among them, at least one evaluation indicator indicates the stability of the first machine learning model when generating multiple first prediction labels, and / or, at least one evaluation indicator indicates the accuracy of the multiple first prediction labels generated by the first machine learning model in the first traffic scene, and the accuracy of the multiple first prediction labels is obtained based on the multiple first prediction labels and the true values ​​corresponding to the multiple first fragment data.

[0012] For example, the first evaluation indicator can represent the number of jumps of multiple first prediction labels. The smaller the number of jumps of multiple first prediction labels, the better the stability of the multiple first prediction labels. For another example, the first evaluation indicator can represent the jump frequency of multiple first prediction labels. The jump frequency of multiple first prediction labels can be the ratio between the number of jumps of multiple first prediction labels and the total number of multiple first prediction labels. The lower the jump frequency of multiple first prediction labels, the better the stability of the multiple first prediction labels. The "accuracy of multiple first prediction labels" can adopt the accuracy of multiple first prediction labels. The accuracy of first prediction labels can represent the ratio between the number of first prediction labels that are consistent with the true value in multiple first prediction labels and the total number of multiple first prediction labels.

[0013] In this implementation, a first machine learning model is used to generate multiple first prediction labels for descriptive data of a certain traffic scene, and then the performance of the first machine learning model in the aforementioned traffic scene is evaluated based on the multiple first prediction labels. The evaluation indicators used indicate the stability of the first machine learning model when generating the multiple first prediction labels, and / or at least one evaluation indicator indicates the accuracy of the multiple first prediction labels generated by the first machine learning model in the first traffic scene. That is, the evaluation indicators used in this application are constructed at the granularity of the traffic scene. Since the vehicle driving process is continuous, "constructing evaluation indicators at the granularity of the traffic scene" can more accurately reflect the performance of the machine learning model applied to the vehicle driving process; in addition, the solution provided by this application does not require the establishment of a simulated traffic scene, which reduces the computer resources consumed by the evaluation process of the machine learning model and improves the evaluation efficiency.

[0014] In a possible implementation of the first aspect, the first evaluation sample can be obtained based on road test data, and the road test data includes descriptive data of traffic scenes collected when the autonomous driving vehicle is driving on the road; optionally, the road test data may include descriptive data of traffic scenes collected when the autonomous driving vehicle is driving on a real road. The "autonomous driving vehicle" can be configured as a fully or partially autonomous driving mode. In this implementation, since the first machine learning model is configured in the vehicle to realize the autonomous driving mode of the vehicle, the behavior of "using road test data to evaluate the performance of the first machine learning model" makes the "process of evaluating the first machine learning model" and the "actual application process of the first machine learning model" more similar, so that the obtained evaluation results are more accurate.

[0015] In one possible implementation of the first aspect, if the first parameter value of the at least one evaluation indicator obtained based on the first evaluation sample does not satisfy the second condition, it indicates that the first machine learning model has failed the evaluation in the first traffic scenario. This implementation clarifies the criteria for determining whether the first machine learning model has failed the evaluation in a specific traffic scenario, thereby reducing the implementation difficulty of this solution.

[0016] In a possible implementation of the first aspect, when the stability of the first machine learning model in generating multiple prediction labels corresponding to the first evaluation sample does not meet the first sub-condition, and / or when the accuracy of the multiple prediction labels corresponding to the first evaluation sample does not meet the second sub-condition, it is deemed that the first parameter value of at least one evaluation indicator does not meet the second condition.

[0017] For example, when the number of jumps of the multiple prediction labels corresponding to the first evaluation sample is greater than or equal to a preset number, it can be regarded that the stability of the multiple first prediction labels does not meet the first sub-condition; or, when the jump frequency of the multiple prediction labels corresponding to the first evaluation sample is greater than or equal to a preset value, it can be regarded that the stability of the multiple first prediction labels does not meet the first sub-condition.

[0018] When the accuracy of the multiple predicted labels corresponding to the first evaluation sample is less than or equal to the preset accuracy, it can be considered that the accuracy of the multiple first predicted labels does not meet the second sub-condition.

[0019] In a possible implementation of the first aspect, after the first device determines the first parameter value of at least one evaluation indicator based on multiple first prediction labels, the method also includes: the first device processes the first training sample through the second machine learning model to obtain a second prediction label corresponding to the first training sample, and the similarity between the first training sample and the second evaluation sample meets the first condition, wherein the first training sample and the second evaluation sample both include descriptive data of the traffic scene, and the second evaluation sample is an evaluation sample in which the parameter value of at least one evaluation indicator obtained when evaluating the first machine learning model does not meet the second condition; it should be noted that the "second evaluation sample" and the "first evaluation sample" have similar meanings, and both are evaluation samples in the evaluation sample set of the first machine learning model. The difference is that the "first evaluation sample" is any evaluation sample used when evaluating the first machine learning model, and the parameter value of at least one evaluation indicator obtained after evaluating the first machine learning model through the "second evaluation sample" does not meet the second condition.

[0020] The first device processes the second training sample through the second machine learning model to obtain a second prediction label corresponding to the second training sample, where the second training sample is different from the first training sample; and processes the second training sample through the first machine learning model to obtain a first prediction label corresponding to the second training sample.

[0021] The first device updates the second machine learning model according to a first loss term and a second loss term, wherein the first loss term indicates the similarity between the second predicted label corresponding to the first training sample and the true value corresponding to the first training sample, and the second loss term indicates the similarity between the second predicted label corresponding to the second training sample and the first predicted label corresponding to the second training sample.

[0022] In this implementation, since the similarity between the first training sample and the second evaluation sample satisfies the first condition, and the parameter value of at least one evaluation indicator obtained when the first machine learning model is evaluated using the second evaluation sample does not satisfy the second condition, that is, when the first machine learning model is evaluated by the second evaluation sample, the performance of the first machine learning model is not good, then the first loss term indicates the similarity between the second predicted label corresponding to the first training sample and the true value corresponding to the first training sample, which is beneficial to improving the performance of the second machine learning model in the traffic scene corresponding to the first evaluation sample; the second training sample is different from the first training sample, that is, the first machine learning model has better performance in the traffic scene corresponding to the second training sample, and the second loss term indicates the similarity between the second predicted label corresponding to the second training sample and the first predicted label corresponding to the second training sample, which is beneficial to enable the second machine learning model to learn the excellent performance of the first machine learning model in the traffic scene corresponding to the second training sample; therefore, the trained second machine learning model not only has better performance in the traffic scene corresponding to the second evaluation sample, but also has better performance in the traffic scene corresponding to the second training sample, that is, a second machine learning model that can perform well in all traffic scenes can be obtained.

[0023] In a possible implementation of the first aspect, the method further includes: the first device extracts features from the second evaluation sample through the first machine learning model to obtain feature information of the second evaluation sample; illustratively, the feature information of the second evaluation sample may include feature information of each fragment data in at least one fragment data included in the second evaluation sample. The first device may determine a first training sample set from a training sample set based on the feature information of the second evaluation sample and the feature information of the training sample of the first machine learning model, wherein the training sample set includes multiple training samples of the first machine learning model, the first training sample set includes a first training sample, and the similarity between the feature information of the first training sample and the feature information of the second evaluation sample satisfies a first condition.

[0024] In this implementation, a manual method is adopted to classify the training samples according to different traffic scenarios, which may not be consistent with the different traffic scenarios understood by the machine learning model; based on the feature information of each second evaluation sample and the similarity between the feature information of each training sample, multiple first training samples are obtained from the training sample set, that is, the multiple first training samples obtained are samples that the first machine learning model considers to be similar to the second evaluation samples, so that from the perspective of the machine learning model, training samples that the first machine learning model is not good at processing can be found to increase the probability that each first training sample is a training sample that the first machine learning model is not good at processing, so that the second machine learning model learns the true value corresponding to the first training sample when processing the first training sample, which is also conducive to obtaining a second machine learning model with better performance.

[0025] In a possible implementation of the first aspect, the similarity between the feature information of the first training sample and the feature information of the second evaluation sample can be obtained based on the first similarity between the feature information of the first training sample and the feature information of each fragment data of at least one fragment data included in the second evaluation sample, that is, the similarity between the feature information of the first training sample and the feature information of the second evaluation sample can be obtained based on at least one first similarity that corresponds one-to-one to at least one fragment data included in the second evaluation sample. Exemplarily, the similarity between the feature information of the first training sample and the feature information of the second evaluation sample can be the average value of the aforementioned at least one first similarity, or the similarity between the feature information of the first training sample and the feature information of the second evaluation sample can be the median value of the aforementioned at least one first similarity.

[0026] In a possible implementation of the first aspect, the evaluation sample set of the first machine learning model includes multiple evaluation samples, the target evaluation sample is any one of the evaluation sample set of the first machine learning model, the target evaluation sample is obtained based on driving data, the driving data includes descriptive data of the traffic scene and the original label corresponding to the descriptive data of the aforementioned traffic scene, the target evaluation sample includes the descriptive data of the aforementioned traffic scene and the true value corresponding to the descriptive data of the aforementioned traffic scene.

[0027] In the case where the driving data is a negative sample, the true value in the target evaluation sample is obtained by modifying the original label in the driving data. "The driving data is a negative sample" can mean that the action indicated by the original label in the driving data is not suitable for execution in the traffic scene. In this implementation, since when a driving sample is a negative sample, the action indicated by the original label corresponding to the descriptive data of the traffic scene is not suitable for the traffic scene, the original label is modified to obtain the true value in the target evaluation sample, thereby ensuring that the actions executed by the true value in the evaluation sample are all actions that are adapted to the traffic scene, which is conducive to improving the accuracy of the evaluation results.

[0028] In one possible implementation of the first aspect, when the driving data is a negative sample, the driving data includes descriptive data of the traffic scene within a second time period, and the target evaluation sample includes descriptive data of the traffic scene within a third time period, where the second time period is greater than the third time period. For example, the descriptive data of the target traffic scene within the second time period included in the target driving data can be deleted from the descriptive data within the first time period, thereby obtaining the descriptive data of the target traffic scene within the third time period included in the target evaluation sample, where the descriptive data within the first time period includes descriptive data within a preset time period from the back of the second time period.

[0029] In this implementation, since when the target driving data is a negative sample, the data in the second half of the description data of the traffic scene included in the target driving data within the second time length may lead the first machine learning model to output an erroneous prediction label, resulting in the inability to accurately reflect the performance of the first machine learning model itself. The second time length is greater than the third time length, that is, the data in the second half of the description data of the traffic scene included in the target driving data within the second time length is cut off, which is conducive to the obtained evaluation results more accurately reflecting the true performance of the first machine learning model.

[0030] In one possible implementation of the first aspect, the task types performed by the first machine learning model include one or more of the following: determining whether to cut in, determining whether to yield, or determining whether to limit the speed. This implementation provides multiple task types that can be performed by the first machine learning model, expanding the application scenarios of this solution and increasing its implementation flexibility.

[0031] In a second aspect, an embodiment of the present application provides a data processing method that can be used in the field of autonomous driving in artificial intelligence. The method includes: a third device processes a first training sample through a second machine learning model to obtain a second predicted label corresponding to the first training sample. The third device processes a second training sample through a second machine learning model to obtain a second predicted label corresponding to the second training sample, wherein both the second training sample and the first training sample include descriptive data of a traffic scene, and the second training sample and the first training sample correspond to different traffic scenes; the second training sample is processed through the first machine learning model to obtain a first predicted label corresponding to the second training sample. The third device updates the second machine learning model based on a first loss term and a second loss term, wherein the first loss term indicates the similarity between the second predicted label corresponding to the first training sample and the true value corresponding to the first training sample, and the second loss term indicates the similarity between the second predicted label corresponding to the second training sample and the first predicted label corresponding to the second training sample.

[0032] In this implementation, since the second training sample and the first training sample correspond to different traffic scenarios, the first loss term indicates the similarity between the second predicted label corresponding to the first training sample and the true value corresponding to the first training sample, which is conducive to improving the performance of the second machine learning model in the traffic scenario corresponding to the first training sample; the second loss term indicates the similarity between the second predicted label corresponding to the second training sample and the first predicted label corresponding to the second training sample, which is conducive to enabling the second machine learning model to learn the excellent performance of the first machine learning model in the traffic scenario corresponding to the second training sample; therefore, when the first machine learning model performs poorly in the traffic scenario corresponding to the first training sample, the trained second machine learning model not only performs well in the traffic scenario corresponding to the first training sample, but also performs well in the traffic scenario corresponding to the second training sample, that is, a second machine learning model that can perform well in all traffic scenarios can be obtained.

[0033] In a possible implementation of the second aspect, the first training sample is determined based on the second evaluation sample, and the second evaluation sample is an evaluation sample in which the parameter value of at least one evaluation indicator obtained when evaluating the first machine learning model does not meet the second condition.

[0034] In the second aspect of this application, the third device can also be used to execute the steps performed by the first device in the first aspect and various possible implementation methods of the first aspect. The specific implementation methods, meanings of terms and beneficial effects of the steps in the second aspect can all be referred to the first aspect and will not be repeated here.

[0035] In a third aspect, an embodiment of the present application provides an evaluation device for a machine learning model, which can be used in the field of autonomous driving of artificial intelligence. The device includes: an acquisition module for acquiring a first evaluation sample, the first evaluation sample including descriptive data of a first traffic scene within a first time length, the first evaluation sample including multiple first segment data, each first segment data including descriptive data of the first traffic scene within a first sub-time length, and the first time length including the first sub-time length; a processing module for processing the multiple first segment data through a first machine learning model to obtain multiple first prediction labels corresponding one-to-one to the multiple first segment data; a determination module for determining a first parameter value of at least one evaluation indicator based on the multiple first prediction labels; wherein, at least one evaluation indicator indicates the stability of the first machine learning model when generating multiple first prediction labels, and / or, at least one evaluation indicator indicates the accuracy of the multiple first prediction labels generated by the first machine learning model in the first traffic scene, and the accuracy of the multiple first prediction labels is obtained based on the multiple first prediction labels and the true values ​​corresponding to the multiple first segment data.

[0036] In the third aspect of this application, the evaluation device of the machine learning model is also used to execute the steps performed by the first device in the first aspect and various possible implementation methods of the first aspect. The specific implementation methods of the steps in the third aspect, the meanings of the terms and the beneficial effects brought about can all be referred to the first aspect and will not be repeated here.

[0037] In a fourth aspect, an embodiment of the present application provides a data processing device that can be used in the field of autonomous driving of artificial intelligence, the device comprising: a processing module for processing a first training sample through a second machine learning model to obtain a second prediction label corresponding to the first training sample; a processing module for processing a second training sample through a second machine learning model to obtain a second prediction label corresponding to the second training sample, wherein both the second training sample and the first training sample include description data of a traffic scene, and the second training sample and the first training sample correspond to different traffic scenes; the processing module is further used to process the second training sample through the first machine learning model to obtain a first prediction label corresponding to the second training sample; an updating module for updating the second machine learning model according to a first loss term and a second loss term, wherein the first loss term indicates the similarity between the second prediction label corresponding to the first training sample and the true value corresponding to the first training sample, and the second loss term indicates the similarity between the second prediction label corresponding to the second training sample and the first prediction label corresponding to the second training sample.

[0038] In the fourth aspect of this application, the data processing device is also used to execute the steps performed by the third device in the second aspect and various possible implementation methods of the second aspect. The specific implementation methods, meanings of terms and beneficial effects brought about by the steps in the fourth aspect can all be referred to the second aspect and will not be repeated here.

[0039] In a fifth aspect, the present application provides a device that may include a memory, a processor, and a bus system, wherein the memory is used to store programs and the processor is used to execute programs in the memory, including the following steps: the bus system is used to connect the memory and the processor so that the memory and the processor can communicate.

[0040] In the fifth aspect of this application, the processor in the device can also be used to execute the steps performed by the first device in each possible implementation of the first aspect, or can also be used to execute the steps performed by the third device in each possible implementation of the second aspect. For details, please refer to the first and second aspects and will not be repeated here.

[0041] In a sixth aspect, the present application provides a computer-readable storage medium, which stores a computer program. When the computer-readable storage medium runs on a computer, it enables the computer to execute the evaluation method of the machine learning model of the first aspect mentioned above, or enables the computer to execute the data processing method of the second aspect mentioned above.

[0042] In a seventh aspect, the present application provides a circuit system, which includes a processing circuit, and the processing circuit is configured to execute the evaluation method of the machine learning model of the first aspect, or to execute the data processing method of the second aspect mentioned above.

[0043] In an eighth aspect, the present application provides a computer program product which, when running on a computer, enables the computer to execute the machine learning model evaluation method of the first aspect, or enables the computer to execute the data processing method of the second aspect.

[0044] In a ninth aspect, the present application provides a chip system, which includes a processor for supporting the implementation of the functions involved in the above aspects, such as sending or processing the data and / or information involved in the above methods. In one possible design, the chip system also includes a memory, which is used to store program instructions and data necessary for the server or communication device. The chip system can be composed of a chip or can include a chip and other discrete devices. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] FIG1a is a schematic diagram of the structure of an artificial intelligence main framework provided in an embodiment of the present application;

[0046] FIG1b is a flow chart of a method for evaluating a machine learning model according to an embodiment of the present application;

[0047] FIG2 is another flow chart of the method for evaluating a machine learning model provided in an embodiment of the present application;

[0048] FIG3 is a schematic diagram of obtaining evaluation samples based on driving data according to an embodiment of the present application;

[0049] FIG4 is another flow chart of the method for evaluating a machine learning model provided in an embodiment of the present application;

[0050] FIG5 is a schematic diagram of obtaining a first training sample set according to an embodiment of the present application;

[0051] FIG6 is a schematic diagram of training a second machine learning model according to an embodiment of the present application;

[0052] FIG7 is a schematic diagram of obtaining a first training sample set and training a second machine learning model using the first training sample set according to an embodiment of the present application;

[0053] FIG8 is a schematic diagram of the structure of a device for evaluating a machine learning model according to an embodiment of the present application;

[0054] FIG9 is a schematic structural diagram of a data processing device provided in an embodiment of the present application;

[0055] FIG10 is a schematic structural diagram of a device provided in an embodiment of the present application;

[0056] FIG11 is a schematic diagram of the structure of a chip provided in an embodiment of the present application. DETAILED DESCRIPTION

[0057] The terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequential order. It should be understood that the terms used in this way can be interchangeable under appropriate circumstances, and this is merely a way of distinguishing the objects of the same attributes when describing them in the embodiments of the present application. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, so that the process, method, system, product or equipment comprising a series of units need not be limited to those units, but may include other units that are not clearly listed or inherent to these processes, methods, products or equipment.

[0058] The embodiments of the present application are described below in conjunction with the accompanying drawings. Those skilled in the art will appreciate that, with the development of technology and the emergence of new scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0059] First, let's describe the overall workflow of an AI system. See Figure 1a, which shows a schematic diagram of the AI ​​framework. This AI framework will be explained from two perspectives: the "intelligent information chain" (horizontal axis) and the "IT value chain" (vertical axis). The "intelligent information chain" reflects the entire process from data acquisition to processing. For example, it could be the general process of intelligent information perception, intelligent information representation and formation, intelligent reasoning, intelligent decision-making, and intelligent execution and output. Throughout this process, data undergoes a condensed journey from "data-information-knowledge-wisdom." The "IT value chain," spanning the underlying infrastructure of human intelligence, information (provided and processed by technology), and the system's industrial ecosystem, reflects the value that AI brings to the information technology industry.

[0060] (1) Infrastructure

[0061] The infrastructure provides computing power for AI systems, enabling communication with the outside world and providing support through a basic platform. Communication with the outside world is achieved through sensors; computing power is provided by smart chips, which can specifically adopt hardware acceleration chips such as central processing units (CPUs), embedded neural network processing units (NPUs), graphics processing units (GPUs), application-specific integrated circuits (ASICs), or field programmable gate arrays (FPGAs); the basic platform includes related platform guarantees and support such as distributed computing frameworks and networks, and can include cloud storage and computing, and interconnected networks. For example, sensors communicate with the outside world to obtain data, which is then provided to the smart chips in the distributed computing system provided by the basic platform for calculation.

[0062] (2) Data

[0063] Data above the infrastructure layer represents data sources for AI. This data includes graphics, images, voice, and text, as well as IoT data from traditional devices. This includes business data from existing systems and sensor data such as force, displacement, liquid level, temperature, and humidity.

[0064] (3) Data processing

[0065] Data processing generally includes data training, machine learning, deep learning, search, reasoning, decision-making, etc.

[0066] Among them, machine learning and deep learning can symbolize and formalize data for intelligent information modeling, extraction, preprocessing, and training.

[0067] Reasoning refers to the process of simulating human intelligent reasoning in computers or intelligent systems, using formalized information to perform machine thinking and solve problems based on reasoning control strategies. Typical functions are search and matching.

[0068] Decision-making refers to the process of making decisions after intelligent information is reasoned, and usually provides functions such as classification, sorting, and prediction.

[0069] (4) General ability

[0070] After the data has undergone the data processing mentioned above, some general capabilities can be further formed based on the results of the data processing, such as algorithms or a general system, for example, translation, text analysis, computer vision processing, speech recognition, image recognition, etc.

[0071] (5) Smart products and industry applications

[0072] Smart products and industry applications refer to the products and applications of artificial intelligence systems in various fields. They are the encapsulation of the overall artificial intelligence solution, which productizes intelligent information decision-making and realizes practical application. Its application areas mainly include: smart terminals, smart manufacturing, smart transportation, smart homes, smart medical care, smart security, autonomous driving, smart cities, etc.

[0073] The method provided in this application can be applied to the field of autonomous driving using artificial intelligence technology to evaluate the performance of a first machine learning model in the autonomous driving field. Exemplarily, the first machine learning model is a machine learning model to be deployed in a vehicle. Alternatively, the method can be applied to the vehicle's intelligent driving decision-making system. Because the current approach involves jointly evaluating multiple machine learning models deployed in a vehicle in a simulated environment, it is impossible to accurately evaluate the performance of a single machine learning model.

[0074] Therefore, the present application provides a method for evaluating a machine learning model, which is used to evaluate a first machine learning model to be deployed in a vehicle. Specifically, please refer to Figure 1b, which is a flow chart of a method for evaluating a machine learning model provided in an embodiment of the present application. Among them, 101, the first device obtains a first evaluation sample, the first evaluation sample includes descriptive data of a first traffic scene within a first time length, the first evaluation sample includes multiple first segment data, each first segment data includes descriptive data of the first traffic scene within a first sub-time length, and the first time length includes a first sub-time length.

[0075] Exemplarily, the first device is a device used to evaluate the first machine learning model, which can be a terminal device or an edge device, or a cloud device, such as a server or a server cluster, etc. This application does not limit the product form of the first device.

[0076] For example, the description data may be specifically expressed as a video, an image, a scene description file, or other types of data. The specific form of the description data may be determined based on actual circumstances and is not limited here. For example, the scene description file may include the position, speed, acceleration, or other status information of each traffic participant in the first traffic scene, etc., which is not limited in the embodiments of the present application.

[0077] Exemplarily, the first traffic scenario may be: the vehicle goes straight at an unprotected intersection and other traffic participants turn left, the vehicle goes straight at the intersection and other traffic participants turn right, going straight at the intersection corresponds to other traffic participants crossing diagonally, the vehicle turns left at an unprotected intersection corresponds to other traffic participants going straight, the vehicle turns left at the intersection, the vehicle turns left at the intersection and other traffic participants cross (cross), the vehicle turns right at the intersection, the vehicle turns right at the intersection and other traffic participants cross, the vehicle enters the intersection and other traffic participants cross, the vehicle makes a U-turn, the vehicle crosses the intersection at a gap, the vehicle merges at a gap at an intersection, the vehicle encounters other traffic participants while crossing the intersection, the vehicle merges when entering the ramp auxiliary road, the vehicle crosses the intersection and has no interaction with other traffic participants or other traffic scenarios, etc. The specific traffic scenarios corresponding to the multiple evaluation samples included in the evaluation sample set of the first machine learning model can be flexibly determined in combination with the task type performed by the first machine learning model and the specific application scenario, and are not limited in the embodiments of the present application.

[0078] The first duration may include one or more first sub-durations. For example, the first duration is 10 seconds and the first sub-duration is 2 seconds; or the first duration is 20 seconds and the first sub-duration is 5 seconds. Alternatively, the first duration and the first sub-duration may be of other lengths. The examples provided here are merely for facilitating understanding of the relationship between the "first duration" and the "first sub-duration" and are not intended to limit this solution.

[0079] 102. The first device processes the plurality of first fragment data through a first machine learning model to obtain a plurality of first prediction labels corresponding one-to-one to the plurality of first fragment data.

[0080] 103. The first device determines a first parameter value of at least one evaluation indicator based on multiple first prediction labels; wherein, at least one evaluation indicator indicates the stability of the first machine learning model when generating multiple first prediction labels, and / or, at least one evaluation indicator indicates the accuracy of the multiple first prediction labels generated by the first machine learning model in the first traffic scene, and the accuracy of the multiple first prediction labels is obtained based on the multiple first prediction labels and the true values ​​corresponding to the multiple first fragment data.

[0081] Exemplarily, at least one evaluation indicator may include a first evaluation indicator and a second evaluation indicator, and the first evaluation indicator indicates the stability of the above-mentioned multiple first prediction labels generated by the first machine learning model. For example, the first evaluation indicator may represent the number of jumps of the multiple first prediction labels, and the smaller the number of jumps of the multiple first prediction labels, the better the stability of the multiple first prediction labels; for another example, the first evaluation indicator may represent the jump frequency of the multiple first prediction labels, and the jump frequency of the multiple first prediction labels may be the ratio between the number of jumps of the multiple first prediction labels and the total number of the multiple first prediction labels. The lower the jump frequency of the multiple first prediction labels, the better the stability of the multiple first prediction labels.

[0082] The second evaluation metric indicates the accuracy of multiple first prediction labels generated by the first machine learning model in the first traffic scenario, where the accuracy of the multiple first prediction labels is based on the multiple first prediction labels and the true values ​​corresponding to the multiple first fragments of data. The "accuracy of the multiple first prediction labels" can be represented by the accuracy rate of the multiple first prediction labels, which can represent the ratio of the number of first prediction labels in the multiple first prediction labels that are consistent with the true value to the total number of the multiple first prediction labels.

[0083] In the embodiments of the present application, the evaluation indicators used in the present application are constructed at the granularity of traffic scenes. Since the vehicle driving process is continuous, "constructing evaluation indicators at the granularity of traffic scenes" can more accurately reflect the performance of the machine learning model applied to the vehicle driving process; in addition, the solution provided by the present application does not require the establishment of a simulated traffic scene, which reduces the computer resources consumed by the evaluation process of the machine learning model.

[0084] In conjunction with the above description, the following describes the specific implementation process of the method provided in the embodiment of the present application. Specifically, please refer to Figure 2, which is another flow chart of the evaluation method of the machine learning model provided in the embodiment of the present application. The evaluation method of the machine learning model provided in the embodiment of the present application may include:

[0085] 201. Obtain a first evaluation sample, the first evaluation sample including descriptive data of a first traffic scene within a first duration, the first evaluation sample including multiple first segment data, each first segment data including descriptive data of the first traffic scene within a first sub-duration, and the first duration including a first sub-duration.

[0086] In the embodiment of the present application, step 201 is an optional step. The first device may obtain one or more first evaluation samples from the evaluation sample set of the first machine learning model. The evaluation sample set of the first machine learning model may include multiple evaluation samples, and each first evaluation sample may be any one of the multiple evaluation samples.

[0087] The first evaluation sample includes descriptive data of the first traffic scene within the first duration. The first evaluation sample includes multiple first fragments of data. The first duration may include one or more first sub-durations. For example, the first duration is 10 seconds and the first sub-duration is 2 seconds; for another example, the first duration is 20 seconds and the first sub-duration is 5 seconds. Alternatively, the first duration and the first sub-duration may be of other lengths. The examples here are only for the convenience of understanding the relationship between the "first duration" and the "first sub-duration" and are not used to limit this solution.

[0088] For example, the description data may be specifically expressed as a video, an image, a scene description file, or other types of data. The specific form of the description data may be determined based on actual circumstances and is not limited here. For example, the scene description file may include the position, speed, acceleration, or other status information of each traffic participant in the first traffic scene, etc., which is not limited in the embodiments of the present application.

[0089] For example, the first evaluation sample may include a video of the first traffic scene within 15 seconds, the first evaluation sample includes 5 first segment data, each first segment data includes a video of the first traffic scene within 3 seconds, and the 5 first segment data can constitute the first evaluation sample. For another example, the first evaluation sample may include images of the first traffic scene at multiple time points within 16 seconds, the first evaluation sample includes 4 first segment data, each first segment data includes images of the first traffic scene at multiple time points within 4 seconds, and the 4 first segment data can constitute the first evaluation sample. For another example, the first evaluation sample may include a scene description file of the first traffic scene at multiple time points within 20 seconds, the first evaluation sample includes 5 first segment data, the 5 first segment data each include a scene description file of a certain length, the 5 first segment data include durations of 5 seconds, 6 seconds, 3 seconds, 4 seconds and 2 seconds respectively, the 5 first segment data can constitute the first evaluation sample, and so on. It should be noted that the examples here are only for the convenience of understanding the concepts of "first evaluation sample" and "first segment data" and are not used to limit this solution.

[0090] Optionally, the task performed by the first machine learning model may be a classification task. For example, the task type performed by the first machine learning model may be any one or more of the following: determining whether to cut in, determining whether to yield, determining whether to limit the speed, or other classification tasks, etc., which are not limited in the embodiments of the present application. Providing multiple task types that can be performed by the first machine learning model expands the application scenarios of the present solution and improves the implementation flexibility of the present solution.

[0091] Optionally, an embodiment of the present application further discloses a method for obtaining an evaluation sample set of the first machine learning model. The evaluation sample set of the first machine learning model may include multiple evaluation samples, each of the multiple evaluation samples is obtained based on a piece of driving data, that is, multiple evaluation samples in the first machine learning model can be obtained based on multiple driving data.

[0092] In one scenario, the aforementioned "driving data" may be road test data, which includes descriptive data of traffic scenes collected while the autonomous vehicle is driving on a road; alternatively, the road test data may include descriptive data of traffic scenes collected while the autonomous vehicle is driving on a real road. The "autonomous vehicle" may be configured in a fully or partially autonomous driving mode. For example, when the autonomous vehicle is in a partially autonomous driving mode, while controlling itself, the autonomous vehicle may also determine the current state of the vehicle and its surroundings through human operation, determine the possible behavior of at least one traffic participant in the surrounding environment, and determine the confidence level corresponding to the probability of other traffic participants performing the possible behavior, and control the autonomous vehicle based on the determined information. When the autonomous vehicle is in a fully autonomous driving mode, the autonomous vehicle may also be set to operate without human interaction. In this embodiment of the present application, since the first machine learning model is configured in the vehicle to implement the vehicle's autonomous driving mode, the act of "using road test data to evaluate the performance of the first machine learning model" makes the "process of evaluating the first machine learning model" more similar to the "actual application process of the first machine learning model", making the obtained evaluation results more accurate.

[0093] In another case, the aforementioned "driving data" may be traffic scene description data collected while a manually controlled vehicle is driving on a road. It should be noted that the meaning of "traffic scene description data" can be understood in conjunction with the above descriptions of "traffic scene," "first evaluation sample," and "first data segment," and will not be further elaborated here.

[0094] For the convenience of description, in this application, any evaluation sample in the aforementioned evaluation sample set is referred to as a "target evaluation sample". The target evaluation sample may include descriptive data of the target traffic scene within a certain time length. The target evaluation sample may include one or more target segment data. Each of the aforementioned one or more target segment data may include descriptive data of the target traffic scene within a certain sub-time length. The aforementioned one or more target segment data may constitute a target evaluation sample.

[0095] Exemplarily, the target evaluation sample is obtained based on the target driving data (i.e., one of multiple driving data), and the target driving data may include the descriptive data of the target traffic scene within the second time length and the original label corresponding to the descriptive data of the target traffic scene within the second time length; the target evaluation sample includes the descriptive data of the target traffic scene within the third time length and the true value corresponding to the descriptive data of the target traffic scene within the third time length. The "true value corresponding to the descriptive data of the target traffic scene within the third time length" can also be called the "expected label corresponding to the descriptive data of the target traffic scene within the third time length". The "true value corresponding to the descriptive data of the target traffic scene within the third time length" can be understood as the true value corresponding to each target segment data in the multiple target segment data included in the target evaluation sample, or, it can also be called the expected label corresponding to each target segment data in the multiple target segment data included in the target evaluation sample.

[0096] The meaning of the information indicated by the “original label corresponding to the descriptive data of the traffic scene” is related to the type of task performed by the first machine learning model. For example, if the task performed by the first machine learning model is to determine whether to cut in, then the “original label corresponding to the descriptive data of the traffic scene” may indicate whether the vehicle cuts in or gives way in the traffic scene. For another example, if the task performed by the first machine learning model is to determine whether to give way, then the “original label corresponding to the descriptive data of the traffic scene” may indicate whether the vehicle gives way or does not give way in the traffic scene. For another example, if the task performed by the first machine learning model is to determine whether to limit the speed, then the “original label corresponding to the descriptive data of the traffic scene” may indicate whether the vehicle limits the speed or does not limit the speed in the traffic scene, etc. The specific meaning of the information indicated by the aforementioned “original label” may be determined in combination with the actual application scenario of the first machine learning model, and is not limited in the embodiments of the present application.

[0097] Regarding the specific implementation method of obtaining target evaluation samples based on target driving data, it should be noted that the execution device of "obtaining target evaluation samples based on target driving data" and the first device can be the same device, that is, the evaluation sample set of the first machine learning model is determined based on multiple driving data on the first device; or, the execution device of "obtaining target evaluation samples based on target driving data" and the first device can be different devices, that is, the evaluation sample set of the first machine learning model is determined based on multiple driving data on other devices, and then the evaluation sample set of the first machine learning model is sent to the first device by other devices.

[0098] For example, in one case, the "original label corresponding to the description data of the target traffic scene within the second duration" can be directly determined as the "true value corresponding to the description data of the target traffic scene within the third duration", with the second duration being equal to the third duration. Alternatively, if the target driving data is a positive sample, the original label can be directly determined as the true value, with the second duration being equal to the third duration.

[0099] In another case, when the target driving data is a negative sample, the above-mentioned true value in the target evaluation sample is obtained by modifying the original label in the target driving data. The meaning of "the true value corresponding to the description data of the traffic scene" is similar to the meaning of "the original label corresponding to the description data of the traffic scene", and will not be repeated here. In the embodiment of the present application, since when the target driving sample is a negative sample, the action pointed to by the original label corresponding to the description data of the traffic scene is not suitable for the traffic scene, the original label is modified to obtain the true value in the target evaluation sample, thereby ensuring that the actions performed by the true value in the evaluation sample are all actions adapted to the traffic scene, which is conducive to improving the accuracy of the evaluation results.

[0100] For example, if the task performed by the first machine learning model is to determine whether to cut in, the "original label corresponding to the descriptive data of the target traffic scene" in the target driving data indicates that the vehicle cuts in under the traffic scene. When the target driving data is a negative sample, the "true value corresponding to the descriptive data of the target traffic scene" in the target evaluation sample can indicate that the vehicle yields under the traffic scene.

[0101] For another example, if the task performed by the first machine learning model is to determine whether to change lanes, the "original label corresponding to the descriptive data of the target traffic scene" in the target driving data indicates that the vehicle changes lanes in the traffic scene. When the target driving data is a negative sample, the "true value corresponding to the descriptive data of the target traffic scene" in the target evaluation sample may indicate that the vehicle does not change lanes in the traffic scene.

[0102] For another example, if the task performed by the first machine learning model is to determine whether to limit the speed, the "original label corresponding to the descriptive data of the target traffic scene" in the target driving data indicates that the vehicle is not speed-limited in the traffic scene. When the target driving data is a negative sample, the "true value corresponding to the descriptive data of the target traffic scene" in the target evaluation sample can indicate that the vehicle is speed-limited in the traffic scene, and so on. The examples here are only for the convenience of understanding this solution and are not used to limit this solution.

[0103] For example, there may be labeling information corresponding to each piece of driving data in the plurality of driving data, and the labeling information corresponding to each piece of driving data indicates whether the driving data is a positive sample or a negative sample. "The target driving data is a negative sample" may indicate that the action indicated by the original label in the target driving data is not suitable for execution in the target traffic scenario, and "the target driving data is a positive sample" may indicate that the action indicated by the original label in the target driving data is suitable for execution in the target traffic scenario.

[0104] The “labeling information corresponding to each driving data” may be manually labeled; or, the “labeling information corresponding to each driving data” may be generated by a machine learning model that has performed a training operation.

[0105] Optionally, in the case where the target driving data is a negative sample, the second time period is greater than the third time period, that is, in the case where the target driving data is a negative sample, not only can the original label in the target driving data be modified to obtain the true value in the target evaluation sample, but the description data of the target traffic scene included in the target driving data within the second time period can also be deleted, thereby obtaining the description data of the target traffic scene included in the target evaluation sample within the third time period, and the description data within the first time period includes the description data within the preset time period from the back to the front within the second time period. For example, the second time period includes 20 seconds, the first time period can include the last 5 seconds of the aforementioned 20 seconds, and the third time period can include the first 15 seconds of the aforementioned 20 seconds. It should be understood that the example here is only for the convenience of understanding this solution and is not used to limit this solution.

[0106] In order to understand this solution more intuitively, please refer to Figure 3, which is a schematic diagram of an evaluation sample obtained based on driving data provided in an embodiment of the present application. Here, the task of the first machine learning model is to determine whether the self-vehicle is overtaking as an example. As shown in Figure 3, the target traffic scene includes the self-vehicle and the game car, and the game car is other traffic participants in the game with the self-vehicle. The target driving data includes the descriptive data of the target traffic scene from t0 to t15, and the original label corresponding to the descriptive data of the target traffic scene from t0 to t15: overtaking; from t0 to t15 represents 15 time steps. The descriptive data of the target traffic scene from t0 to t15 includes the driving trajectory of the self-vehicle from t0 to t15 and the driving trajectory of the game car from t0 to t15. As shown in Figure 3, there is an intersection between the driving trajectory of the self-vehicle and the driving trajectory of the game car. If the target driving data is a negative sample, the target evaluation sample includes the descriptive data of the target traffic scene from t0 to t10, and the true value corresponding to the descriptive data of the target traffic scene from t0 to t10: yield; from t0 to t10 represents the first 10 time steps of the 15 time steps from t0 to t15. If the target driving data is a positive sample, the target evaluation sample may include the descriptive data of the target traffic scene from t0 to t15, and the true value corresponding to the descriptive data of the target traffic scene from t0 to t15: cut in; that is, the target driving data is directly determined as the target evaluation sample. It should be understood that the example in Figure 3 is only for the convenience of understanding this solution and is not used to limit this solution.

[0107] In an embodiment of the present application, since when the target driving data is a negative sample, the data in the second half of the description data of the traffic scene included in the target driving data within the second time length may lead the first machine learning model to output an erroneous prediction label, resulting in the inability to accurately reflect the performance of the first machine learning model itself. The second time length is greater than the third time length, that is, the data in the second half of the description data of the traffic scene included in the target driving data within the second time length is cut off, which is conducive to the obtained evaluation results more accurately reflecting the true performance of the first machine learning model.

[0108] Optionally, after determining the type of task performed by the first machine learning model, at least one dimension related to the type of task performed by the first machine learning model can also be determined, and at least one traffic scenario related to the type of task performed by the first machine learning model can be determined based on the aforementioned at least one dimension. Exemplarily, the aforementioned at least one dimension may include: the type of road, the behavior type of the vehicle, the behavior type of other traffic participants in a game relationship with the vehicle, the decision type of the vehicle, or other dimensions, etc., which are not exhaustive here. The type of road may include unprotected intersections, gaps, ramps, or other road types. The behavior type of the vehicle may include going straight, turning left, turning right, or other behavior types. The behavior type of other traffic participants in a game relationship with the vehicle may include cutting in, merging, crossing, or other behavior types, etc. The decision type of the vehicle is related to the task type performed by the first machine learning model. If the task type performed by the first machine learning model is to determine whether to cut in, the decision type of the vehicle may include cutting in and giving way; if the task type performed by the first machine learning model is to determine whether to give way, the decision type of the vehicle may include giving way and not giving way; if the task type performed by the first machine learning model is to determine whether to limit the speed, the decision type of the vehicle may include speed limit or no speed limit, etc. It should be noted that the examples here are only for the convenience of understanding this solution and are not used to limit this solution.

[0109] The evaluation sample set of the first machine learning model includes multiple evaluation samples corresponding to each of the at least one traffic scenario mentioned above, and the difference between the number of evaluation samples corresponding to each traffic scenario can be within a preset range. Each evaluation sample in the evaluation sample set of the first machine learning model includes descriptive data of a traffic scenario within a certain duration and a true value corresponding to the aforementioned descriptive data. Therefore, "which traffic scenario does the target evaluation sample correspond to" can be understood as the target evaluation sample including the descriptive data of the traffic scenario.

[0110] For example, the "at least one traffic scenario" determined based on the above "at least one dimension" may include any one or more of the following traffic scenarios: the vehicle goes straight in an unprotected intersection and other traffic participants turn left, the vehicle goes straight in an intersection and other traffic participants turn right, the vehicle goes straight at the intersection and other traffic participants cross diagonally, the vehicle turns left in an unprotected intersection and other traffic participants go straight, the vehicle turns left at the intersection, the vehicle turns left at the intersection and other traffic participants cross, the vehicle turns right at the intersection, the vehicle turns right at the intersection and other traffic participants cross, the vehicle enters the intersection and other traffic participants cross, the vehicle makes a U-turn (U-turn) turn), the vehicle crosses the intersection at a gap, the vehicle merges at a gap, the vehicle encounters other traffic participants while crossing the intersection, the vehicle merges when entering the ramp auxiliary road, the vehicle crosses the intersection without interacting with other traffic participants or other traffic scenarios, etc. The traffic scenarios to which the multiple evaluation samples included in the evaluation sample set of the specific first machine learning model correspond can be flexibly determined in combination with the task type performed by the first machine learning model and the specific application scenario, and are not limited in the embodiments of the present application.

[0111] 202. Process the plurality of first fragment data using a first machine learning model to obtain a plurality of first prediction labels corresponding one-to-one to the plurality of first fragment data.

[0112] In this embodiment of the present application, step 202 is an optional step. The first device may input the multiple first segment data included in each first evaluation sample into a first machine learning model, process the multiple first segment data included in each first evaluation sample through the first machine learning model, and obtain multiple first prediction labels output by the first machine learning model that correspond one-to-one to the multiple first segment data.

[0113] Among them, the meaning of the information indicated by the "first prediction label" is related to the type of task performed by the first machine learning model. The meaning of the "first prediction label" is similar to that of the above-mentioned "original label". Please refer to the above description and will not be repeated here.

[0114] 203. Determine a first parameter value of at least one evaluation indicator based on multiple first prediction labels, wherein the at least one evaluation indicator indicates the stability of the first machine learning model when generating multiple first prediction labels, and / or the at least one evaluation indicator indicates the accuracy of the multiple first prediction labels generated by the first machine learning model in the first traffic scene, and the accuracy of the multiple first prediction labels is obtained based on the multiple first prediction labels and the true values ​​corresponding to the multiple first fragment data.

[0115] In this embodiment of the present application, step 203 is an optional step. The first device can determine a first parameter value of at least one evaluation indicator corresponding to each first evaluation sample based on multiple first prediction labels corresponding to multiple first fragment data included in each first evaluation sample.

[0116] The at least one evaluation indicator may include a first evaluation indicator and a second evaluation indicator, where the first evaluation indicator indicates the stability of the at least one evaluation indicator generated by the first machine learning model.

[0117] For example, the first evaluation indicator can represent the number of jumps of multiple first prediction labels. The smaller the number of jumps of the multiple first prediction labels, the better the stability of the multiple first prediction labels; exemplarily, the multiple first prediction labels include 5 first prediction labels that correspond one-to-one to the 5 first fragment data included in the first evaluation sample. The task performed by the first machine learning model is to determine whether to grab the line. The 5 first prediction labels are grab the line, grab the line, give way, grab the line, and grab the line, respectively. The number of jumps of the 5 first prediction labels is 2.

[0118] For another example, the first evaluation index can represent the jumping frequency of multiple first prediction labels. The jumping frequency of multiple first prediction labels can be the ratio between the number of jumps of multiple first prediction labels and the total number of multiple first prediction labels. The lower the jumping frequency of multiple first prediction labels, the better the stability of the multiple first prediction labels. Exemplarily, the multiple first prediction labels include 5 first prediction labels that correspond one-to-one to the 5 first fragment data included in the first evaluation sample. The task performed by the first machine learning model is to determine whether to rush in. The 5 first prediction labels are rush in, rush in, give way, rush in, rush in, and rush in, respectively. The jumping frequency of the 5 first prediction labels can be two-fifths, etc. It should be noted that the above examples of the "first evaluation index" are only for the convenience of understanding this solution and are not used to limit this solution.

[0119] The second evaluation metric indicates the accuracy of multiple first prediction labels generated by the first machine learning model in the first traffic scenario, where the accuracy of the multiple first prediction labels is based on the multiple first prediction labels and the true values ​​corresponding to the multiple first fragments of data. The multiple true values ​​corresponding one-to-one to the multiple first fragments of data included in the first evaluation sample can be the same; for example, if the true value corresponding to a first evaluation sample is "rushing", then the true value corresponding to each of the multiple first fragments of data included in the aforementioned first evaluation sample is "rushing".

[0120] The “accuracy of multiple first prediction labels” may adopt the accuracy rate of multiple first prediction labels, and the accuracy rate of multiple first prediction labels may represent the ratio between the number of first prediction labels in the multiple first prediction labels that are consistent with the true value and the total number of the multiple first prediction labels.

[0121] Exemplarily, the multiple first prediction labels include 5 first prediction labels that correspond one-to-one to the 5 first fragment data included in the first evaluation sample. The task performed by the first machine learning model is to determine whether to rush the line. The 5 true values ​​that correspond one-to-one to the 5 first fragment data included in the first evaluation sample are all rush the line. The 5 first prediction labels are rush the line, rush the line, give way, rush the line, and rush the line respectively. The accuracy of the 5 first prediction labels can be four-fifths, or eighty percent. It should be understood that the examples here are only to prove the feasibility of this solution and are not used to limit this solution.

[0122] Optionally, at least one evaluation indicator may also indicate the safety of the first machine learning model generating multiple first prediction labels; optionally, the "safety of multiple first prediction labels" may be the distance between the vehicle and other traffic participants. For example, after obtaining multiple first prediction labels corresponding one-to-one to multiple first segment data, for the convenience of description, any one of the multiple first segment data will be referred to as target segment data. The first device may deduce the trajectory of the vehicle based on the target segment data and the target prediction label corresponding to the target segment data, and obtain the first minimum distance between the vehicle and other traffic participants in the target segment data. The first device performs the aforementioned operation on each of the multiple first segment data, and can obtain multiple first minimum distances corresponding one-to-one to the multiple first segment data. The first device may use the smallest of the multiple first minimum distances as the final distance corresponding to the multiple first prediction labels, and then evaluate the safety of the multiple first prediction labels generated by the first machine learning model based on the aforementioned final distance, that is, evaluate whether the multiple first prediction labels generated by the first machine learning model are safe.

[0123] Optionally, when the first parameter value of at least one of the above-mentioned evaluation indicators does not meet the second condition, it may represent that the first machine learning model has failed the evaluation in the first traffic scenario; the judgment conditions for whether the first machine learning model has failed the evaluation in a certain traffic scenario are clarified, thereby reducing the difficulty of implementing this solution.

[0124] Exemplarily, after determining the first parameter value of at least one evaluation indicator corresponding to each first evaluation sample, the first device can determine whether the first parameter value of at least one evaluation indicator corresponding to the aforementioned first evaluation sample meets the second condition, and the first device can perform the aforementioned operation on each evaluation sample in the evaluation sample set of the first machine learning model; if the parameter value of at least one evaluation indicator corresponding to each evaluation sample in the evaluation sample set of the first machine learning model meets the second condition, it can be determined that the first machine learning model has passed the evaluation; if the parameter value of at least one evaluation indicator corresponding to any evaluation sample in the evaluation sample set of the first machine learning model does not meet the second condition, it can be deemed that the first machine learning model has failed the evaluation.

[0125] Optionally, when the stability of the first machine learning model when generating multiple first prediction labels does not meet the first sub-condition, and / or when the accuracy of the multiple first prediction labels does not meet the second sub-condition, it is deemed that the parameter value of at least one evaluation indicator does not meet the second condition, and / or when the security of the multiple first prediction labels does not meet the third sub-condition, it is deemed that the parameter value of at least one evaluation indicator does not meet the second condition.

[0126] For example, when the number of jumps of the multiple first prediction labels is greater than or equal to a preset number, it can be considered that the stability of the multiple first prediction labels does not meet the first sub-condition; for example, the preset number can be 2, 3, 4, or another value. Alternatively, when the jump frequency of the multiple first prediction labels is greater than or equal to a preset value, it can be considered that the stability of the multiple first prediction labels does not meet the first sub-condition, etc.

[0127] When the accuracy of multiple first prediction labels is less than or equal to the preset accuracy, it can be considered that the accuracy of multiple first prediction labels does not meet the second sub-condition; for example, the preset accuracy can be ninety percent, eighty percent or other values, etc. The examples here are only for the convenience of understanding this solution and are not used to limit this solution.

[0128] When the final distance corresponding to multiple first prediction labels is less than or equal to the distance threshold, it can be regarded that the safety of multiple first prediction labels does not meet the third sub-condition; in one implementation, the aforementioned distance threshold may include a distance threshold, for example, the distance threshold may be 1 meter, 1.2 meters or other values, etc.; in another implementation, the aforementioned distance threshold may include multiple distance values ​​corresponding to multiple speed ranges, that is, when the current speed of the vehicle belongs to the first speed range among the multiple speed ranges, the first distance value corresponding to the first speed range can be obtained; when the current speed of the vehicle belongs to the second speed range among the multiple speed ranges, the second distance value corresponding to the first speed range can be obtained. The design of the specific distance value can be determined in combination with actual conditions, and is not limited in the embodiments of the present application.

[0129] Optionally, after the first device uses each evaluation sample in the above-mentioned evaluation sample set to evaluate the first machine learning model, it can determine that the parameter value of at least one evaluation indicator corresponding to each second evaluation sample in at least one second evaluation sample does not meet the second condition, and then obtain the type of traffic scene corresponding to each second evaluation sample, that is, determine in which traffic scenes the first machine learning model fails the evaluation.

[0130] It should be noted that the "second evaluation sample" and the "first evaluation sample" have similar meanings, and both are evaluation samples in the evaluation sample set of the first machine learning model. The difference is that the "first evaluation sample" is any evaluation sample used when evaluating the first machine learning model, and the parameter value of at least one evaluation indicator obtained after evaluating the first machine learning model through the "second evaluation sample" does not meet the second condition.

[0131] In an embodiment of the present application, a scheme for evaluating a single machine learning model is provided, which can obtain the evaluation results of a single machine learning model more accurately; and the evaluation indicators used in this application are constructed at the granularity of traffic scenes. Since the vehicle driving process is continuous, "constructing evaluation indicators at the granularity of traffic scenes" can more accurately reflect the performance of the machine learning model applied to the vehicle driving process; in addition, the scheme provided by this application does not require the establishment of a simulated traffic scene, which reduces the computer resources consumed by the evaluation process of the machine learning model.

[0132] Optionally, the first device may also perform a training operation on the aforementioned first machine learning model based on the aforementioned at least one second evaluation sample in a subsequent step. The detailed implementation method will be described in the subsequent steps.

[0133] To understand this solution more intuitively, please refer to Figure 4, which is another flow chart of the evaluation method of the machine learning model provided in the embodiment of the present application. In Figure 4, the evaluation samples of the first machine learning model are all road test data as an example. As shown in Figure 4, the road test data set includes multiple road test data, and the multiple road test data include negative road test data and positive road test data. After determining the task type performed by the first machine learning model, at least one traffic scenario corresponding to the aforementioned task type can be determined, and then multiple road test data in the road test data set can be screened based on the aforementioned at least one traffic scenario to obtain multiple road test data for constructing the evaluation sample set.

[0134] The true value of each negative road test data in the multiple filtered road test data can be labeled, that is, the original label in each negative road test data is modified to obtain the true value in the evaluation sample; as shown in Figure 4, each evaluation sample may include: the identification information of the evaluation sample, the time period corresponding to the evaluation sample, the true value included in the evaluation sample, the descriptive data of the traffic scene included in the evaluation sample, the type of the aforementioned traffic scene, and the type of task performed by the first machine learning model and other information.

[0135] In Figure 4, at least one indicator used to evaluate the first machine learning model includes: the accuracy of multiple first prediction labels corresponding to the evaluation samples, and the number of jumps obtained based on the multiple first prediction labels corresponding to the evaluation samples.

[0136] After determining the two evaluation indicators and the evaluation sample set, the first machine learning model can be evaluated to generate parameter values ​​for the two evaluation indicators corresponding to each evaluation sample. It should be noted that the detailed implementation of the steps involved in Figure 4 can be found in the description of the above steps and will not be repeated here. The example in Figure 4 is only for the convenience of understanding this solution and is not intended to limit this solution.

[0137] 204. Perform feature extraction on the second evaluation sample through the first machine learning model to obtain feature information of the second evaluation sample, wherein the second evaluation sample is an evaluation sample in which the parameter value of at least one evaluation indicator obtained when the first machine learning model is evaluated does not meet the second condition.

[0138] In this embodiment of the present application, step 204 is an optional step. After obtaining one or more second evaluation samples through steps 201 to 203, the second device may input each of the at least one fragment data included in each second evaluation sample into the first machine learning model, and perform feature extraction on each of the fragment data included in each second evaluation sample using the first machine learning model to obtain feature information of each second evaluation sample; illustratively, the feature information of each second evaluation sample may include feature information of each of the at least one fragment data included in each second evaluation sample.

[0139] It should be noted that the "second device" and the "first device" can be the same device or different devices, and can be flexibly determined based on actual conditions, and are not limited in the embodiments of this application.

[0140] 205. Based on the feature information of the second evaluation sample and the feature information of the training sample of the first machine learning model, determine the first training sample set from the training sample set, wherein the training sample set includes multiple training samples of the first machine learning model, the first training sample set includes the first training sample, the first training sample and the second evaluation sample both include descriptive data of the traffic scene, and the similarity between the feature information of the first training sample and the feature information of the second evaluation sample meets the first condition.

[0141] In the embodiment of the present application, step 205 is an optional step. The second device can also obtain a training sample set of the first machine learning model, and the training sample set includes multiple training samples of the first machine learning model. For example, each training sample can include descriptive data of a certain traffic scene within a certain sub-duration; the meaning of "training sample" is similar to that of "segment data", which can be understood by referring to the above description and will not be repeated here. The second device inputs each training sample in the training sample set of the first machine learning model into the first machine learning model, and extracts features from each of the aforementioned training samples through the first machine learning model to obtain feature information of each training sample.

[0142] The second device can determine a first training sample set from the training sample set based on the feature information of each second evaluation sample and the feature information of each training sample of the first machine learning model, where the first training sample set includes multiple first training samples, and the similarity between the feature information of each first training sample and the feature information of a certain second evaluation sample in the at least one second evaluation sample satisfies a first condition. Exemplarily, the first condition can be that the similarity between the feature information of the training sample and the feature information of a certain second evaluation sample in the at least one second evaluation sample is greater than or equal to a similarity threshold.

[0143] In order to further understand the present solution, the specific implementation process of determining one or more first training samples from the training sample set of the first machine learning model based on the feature information of any second evaluation sample is described here. For example, in one implementation method, after the second device obtains the feature information of any second evaluation sample (for the convenience of distinction, any second evaluation sample will be referred to as the third evaluation sample later), it also obtains the feature information of each fragment data included in the third evaluation sample. The second device can generate a first similarity between the feature information of the target training sample and the feature information of each fragment data in the third evaluation sample, and obtain at least one first similarity corresponding one-to-one to at least one fragment data included in the third evaluation sample; wherein, the target training sample is any training sample in the training sample set of the first machine learning model.

[0144] Exemplarily, the above-mentioned "first similarity" can be obtained by calculating the cosine similarity between the feature information of the target training sample and the feature information of each fragment data in the third evaluation sample. Alternatively, the above-mentioned "cosine similarity" can also be replaced by Euclidean distance, L1 distance, L2 distance or other similarity calculation methods, etc. The examples here are only for the convenience of understanding this solution and are not used to limit this solution.

[0145] The second device can determine whether the similarity between the feature information of the target training sample and the feature information of the third evaluation sample meets the first condition based on at least one first similarity that corresponds one-to-one to at least one fragment data included in the third evaluation sample; if the judgment result is yes, the target training sample is determined to be the first training sample; if the judgment result is no, it is determined that the target training sample is not the first training sample.

[0146] Exemplarily, in one case, the second device may average the at least one first similarity to obtain an average value, and determine whether the average value is greater than or equal to the first similarity threshold. If the judgment result is yes, the target training sample is determined to be the first training sample; if the judgment result is no, it is determined that the target training sample is not the first training sample.

[0147] In another case, the second device can obtain the median value of the above-mentioned at least one first similarity, and determine whether the median value is greater than or equal to the second similarity threshold. If the judgment result is yes, the target training sample is determined to be the first training sample; if the judgment result is no, it is determined that the target training sample is not the first training sample, and so on. The second device can also use other methods to implement "based on the above-mentioned at least one first similarity, determine whether to determine the target training sample as the first training sample". The example here is only to prove the feasibility of this solution and is not used to limit this solution.

[0148] The second device can perform the above operation on each training sample in the training sample set of the first machine learning model to determine whether the similarity between the feature information of each training sample and the feature information of the third evaluation sample meets the first condition, so as to filter out at least one first training sample from the training sample set of the first machine learning model based on the feature information of the third evaluation sample.

[0149] The second device can repeat the above operation on each second evaluation sample in at least one second evaluation sample to filter out at least one first training sample from the training sample set of the first machine learning model based on each second evaluation sample in at least one second evaluation sample, that is, to obtain all first training samples in the first training sample set.

[0150] In one implementation, after obtaining the feature information of the third evaluation sample, the second device also obtains the feature information of each data segment included in the third evaluation sample. The second device may cluster the feature information of at least one data segment included in the third evaluation sample to obtain a category center corresponding to the feature information of the third evaluation sample.

[0151] The second device can generate a second similarity between the feature information of the target training sample and the aforementioned category center, and the "second similarity between the feature information of the target training sample and the aforementioned category center" is regarded as the similarity between the feature information of the target training sample and the feature information of the third evaluation sample; exemplarily, the aforementioned "second similarity" can be obtained by calculating the cosine similarity between the feature information of the target training sample and the category center, or the aforementioned "cosine similarity" can also be replaced by Euclidean distance, L1 distance, L2 distance or other similarity calculation methods, etc. The examples here are only for the convenience of understanding this solution and are not used to limit this solution.

[0152] The second device may determine whether the second similarity between the feature information of the target training sample and the center of the category satisfies the first condition; if the determination result is yes, the target training sample is determined to be the first training sample; if the determination result is no, the target training sample is determined not to be the first training sample. For example, the second device may determine whether the second similarity between the feature information of the target training sample and the center of the category is greater than or equal to a third similarity threshold; if the determination result is yes, the first condition is determined to be satisfied; if the determination result is no, the first condition is determined not to be satisfied.

[0153] The second device can perform the above operation on each training sample in the training sample set of the first machine learning model to determine whether the similarity between the feature information of each training sample and the feature information of the third evaluation sample meets the first condition, so as to filter out at least one first training sample from the training sample set of the first machine learning model based on the feature information of the third evaluation sample.

[0154] The second device can repeat the above operation on each second evaluation sample in at least one second evaluation sample to filter out at least one first training sample from the training sample set of the first machine learning model based on each second evaluation sample in at least one second evaluation sample, that is, to obtain all first training samples in the first training sample set.

[0155] To understand this solution more intuitively, please refer to Figure 5. Figure 5 is a schematic diagram of obtaining the first training sample set provided in an embodiment of the present application. As shown in Figure 5, after obtaining one or more second evaluation samples, the feature information of each fragment data in the multiple fragment data included in each second evaluation sample can be obtained through the first machine learning model; after obtaining the training sample set of the first machine learning model, the feature information of each training sample in the training sample set can be obtained through the first machine learning model; and then based on the feature information of each fragment data in the multiple fragment data included in each second evaluation sample, as well as the feature information of each training sample, similarity matching is performed in the dimension of feature information to obtain the first training sample set from the training sample set of the first machine learning model. The specific implementation method of the aforementioned steps can be found in the above description and will not be elaborated here. It should be understood that the example in Figure 5 is only for the convenience of understanding this solution and is not used to limit this solution.

[0156] If a manual method is used to classify training samples according to different traffic scenarios, it may not be consistent with the different traffic scenarios understood by the machine learning model. In an embodiment of the present application, a plurality of first training samples are obtained from the training sample set based on the feature information of each second evaluation sample and the similarity between the feature information of each training sample. That is, the plurality of first training samples obtained are samples that the first machine learning model considers to be similar to the second evaluation sample. In this way, from the perspective of the machine learning model, training samples that the first machine learning model is not good at processing can be found, so as to increase the probability that each first training sample is a training sample that the first machine learning model is not good at processing, thereby improving the prediction accuracy of the second machine learning model when processing the first training sample, and obtaining a second machine learning model with better performance.

[0157] It should be noted that steps 204 and 205 are optional steps. If steps 204 and 205 are not performed, after obtaining at least one second evaluation sample, the second device can also determine which traffic scene description data each second evaluation sample includes, thereby being able to determine at least one traffic scene corresponding to at least one second prediction sample, that is, determine all traffic scenes involved in at least one second prediction sample; thereby all training samples in the training sample set of the first machine learning model corresponding to the aforementioned at least one traffic scene are determined as the first training sample set.

[0158] 206. Process the first training sample through the second machine learning model to obtain a second prediction label corresponding to the first training sample.

[0159] In the embodiment of the present application, step 206 is an optional step. After obtaining the first training sample set, the third device may obtain one or more first training samples from the first training sample set, input each first training sample into the second machine learning model, and process each first training sample using the second machine learning model to obtain a second predicted label corresponding to each first training sample.

[0160] Exemplarily, the type of task performed by the "second machine learning model" is the same as the type of task performed by the "first machine learning model", the difference being that the first machine learning model is a machine learning model that has performed a training operation, while the second machine learning model has not yet performed a training operation; the meaning of the "second prediction label" is similar to that of the "first prediction label", and both can be found in the description in the above steps, which will not be repeated here.

[0161] It should be noted that the "third device", "second device" and "first device" can be the same device; or, the "third device", "second device" and "first device" can all be different devices; or, any two of the "third device", "second device" and "first device" can be the same device, and the remaining devices can be independent devices, etc. The specific configuration can be flexibly determined based on actual conditions and is not limited in the embodiments of this application.

[0162] 207. Process the second training sample using a second machine learning model to obtain a second prediction label corresponding to the second training sample, where the second training sample is different from the first training sample.

[0163] In the embodiment of the present application, step 207 is an optional step. The third device may also obtain a second training sample set. For example, the second training sample included in the second training sample set and the first training sample included in the first training sample set may be different training samples in the training sample set of the first machine learning model.

[0164] The third device can obtain one or more second training samples from the second training sample set, input each second training sample into the second machine learning model, process each second training sample through the second machine learning model, and obtain a second prediction label corresponding to each second training sample.

[0165] It should be noted that since steps 201 to 205 are optional steps, if steps 201 to 205 are performed, the first training sample set can be obtained through steps 201 to 205, and then the second training sample set can be determined based on the first training sample set. If steps 201 to 205 are not performed, other methods can also be used to divide the training sample set of the first machine learning model into the first training sample set and the second training sample set. For example, in the process of deploying the first machine learning model on an autonomous driving vehicle to perform inference through the first machine learning model, it can be determined in which traffic scenarios the first machine learning model performs better, then the traffic scenarios corresponding to the training samples in the second training sample set can all be traffic scenarios in which the first machine learning model performs well, and then all training samples in the training sample set of the first machine learning model other than the second training sample set are determined as the first training sample set; it should be noted that other methods can also be used to determine the first training sample set and the second training sample set. The example here is only to prove the feasibility of this solution and is not used to limit this solution.

[0166] 208. Process the second training sample using the first machine learning model to obtain a first prediction label corresponding to the second training sample.

[0167] In the embodiment of the present application, step 208 is an optional step. The third device may also input each second training sample into the first machine learning model, process each second training sample through the first machine learning model, and obtain a first prediction label corresponding to each second training sample.

[0168] 209. According to the first loss term and the second loss term, the second machine learning model is trained until the convergence condition is met, thereby obtaining a second machine learning model that has performed a training operation, wherein the first loss term indicates the similarity between the second predicted label corresponding to the first training sample and the true value corresponding to the first training sample, and the second loss term indicates the similarity between the second predicted label corresponding to the second training sample and the first predicted label corresponding to the second training sample.

[0169] In the embodiment of the present application, step 209 is an optional step. It should be noted that although steps 201 to 209 are all optional steps, when steps 201 to 203 are not performed, steps 206 to 209 may be required steps; when steps 206 to 209 are not performed, steps 201 to 203 are required steps. After obtaining the second predicted label corresponding to the first training sample, the second predicted label corresponding to the second training sample, and the first predicted label corresponding to the second training sample, the third device may generate a value for a first loss term based on the second predicted label corresponding to the first training sample and the true value corresponding to the first training sample, where the goal of training using the first loss term includes improving the similarity between the second predicted label corresponding to the first training sample and the true value corresponding to the first training sample; and generate a value for a second loss term based on the second predicted label corresponding to the second training sample and the first predicted label corresponding to the second training sample, where the goal of training using the second loss term includes improving the similarity between the second predicted label corresponding to the second training sample and the first predicted label corresponding to the second training sample. The third device can generate a function value of the loss function based on the value of the first loss term and the value of the second loss term; illustratively, the value of the first loss term and the value of the second loss term can be weighted summed to obtain the function value of the loss function.

[0170] For example, the first loss term can adopt the cross entropy loss function, 0-1 loss function, L1 loss function or other types of loss functions, etc., and the second loss term can also adopt the cross entropy loss function, 0-1 loss function, L1 loss function or other types of loss functions, etc. It should be noted that the examples here are only to prove the feasibility of this solution and are not used to limit this solution.

[0171] The meaning of “the true value corresponding to the first training sample” is similar to that of “the true value corresponding to the segment data”, which can be understood by referring to the description in the above steps and will not be elaborated here.

[0172] The third device can calculate the gradient of the function value of the loss function and use the backpropagation algorithm to update the parameters of the second machine learning model to implement training of the second machine learning model; the third device can repeatedly execute steps 206 to 209 to implement iterative training of the second machine learning model until the convergence conditions are met, and a second machine learning model that has performed the training operation is obtained.

[0173] Exemplarily, the above-mentioned convergence condition may be that the convergence condition of the loss function is satisfied, or that the second machine learning model is trained a preset number of times, etc.

[0174] For a more intuitive understanding of this solution, please refer to Figures 6 and 7. First, refer to Figure 6, which is a schematic diagram of training the second machine learning model provided in an embodiment of the present application. As shown in Figure 6, after obtaining the first training sample set and the second training sample set, the first training sample and the true value corresponding to the first training sample can be obtained, the first training sample can be input into the second machine learning model, and the second prediction label corresponding to the first training sample can be generated by the second machine learning model. Then, based on the second prediction label corresponding to the first training sample and the true value corresponding to the first training sample, the value of the first loss term is generated.

[0175] The second training sample is input into the first machine learning model and the second machine learning model respectively, the first prediction label corresponding to the second training sample is generated by the first machine learning model, the second prediction label corresponding to the second training sample is generated by the second machine learning model, and the value of the second loss item is generated according to the second prediction label corresponding to the second training sample and the first prediction label corresponding to the second training sample.

[0176] The value of the first loss term and the value of the second loss term are added to obtain the function value of the loss function, and the second machine learning model is trained based on the function value of the loss function. It should be understood that the example in Figure 6 is only for the convenience of understanding this solution and is not used to limit this solution.

[0177] Referring again to FIG. 7 , FIG. 7 is a schematic diagram illustrating obtaining a first training sample set and using the first training sample set to train a second machine learning model, as provided in an embodiment of the present application. As shown in FIG. 7 , similarity matching in the feature information dimension can be performed based on each second evaluation sample and the training sample set of the first machine learning model to obtain the first training sample set. The specific implementation process of the aforementioned steps can be found in the description of steps 204 and 205 and will not be repeated here.

[0178] Then, the first training sample in the first training sample set is input into the second machine learning model to generate the value of the first loss term; the second training sample in the second training sample set is input into the first machine learning model and the second machine learning model respectively to generate the value of the second loss term; the value of the first loss term and the value of the second loss term are added to obtain the function value of the loss function, and the parameters of the second machine learning model are updated based on the back propagation algorithm to realize the training of the second machine learning model. The specific implementation process of the above steps can be found in the description of steps 206 to 209, which will not be repeated here. It should be understood that the example in Figure 7 is only for the convenience of understanding this solution and is not used to limit this solution.

[0179] In an embodiment of the present application, since the similarity between the first training sample and the second evaluation sample satisfies the first condition, and the parameter value of the at least one evaluation indicator obtained when the first machine learning model is evaluated using the second evaluation sample does not satisfy the second condition, that is, when the first machine learning model is evaluated by the second evaluation sample, the performance of the first machine learning model is not good, then the first loss term indicates the similarity between the second predicted label corresponding to the first training sample and the true value corresponding to the first training sample, which is beneficial to improving the performance of the second machine learning model in the traffic scene corresponding to the second evaluation sample; the second training sample is different from the first training sample, that is, the first machine learning model has better performance in the traffic scene corresponding to the second training sample, and the second loss term indicates the similarity between the second predicted label corresponding to the second training sample and the first predicted label corresponding to the second training sample, which is beneficial to enable the second machine learning model to learn the excellent performance of the first machine learning model in the traffic scene corresponding to the second training sample; therefore, the trained second machine learning model not only has better performance in the traffic scene corresponding to the second evaluation sample, but also has better performance in the traffic scene corresponding to the second training sample, that is, a second machine learning model that can perform well in each traffic scene can be obtained.

[0180] Optionally, after executing step 209, step 201 can be re-entered to evaluate the second machine learning model that has performed the training operation through steps 201 to 203. If the second machine learning model fails the evaluation, the third machine learning model can be trained again based on the second machine learning model. The above steps can be repeated at least once until the trained machine learning model passes the evaluation, that is, the target machine learning model that has performed the training operation is obtained.

[0181] The target machine learning model that has performed the training operation can be applied to the autonomous driving vehicle. Optionally, the target machine learning model that has performed the training operation can be applied to the autonomous driving decision system of the autonomous driving vehicle.

[0182] On the basis of the embodiments corresponding to FIG. 1 b to FIG. 7 , in order to better implement the above-mentioned solutions of the embodiments of the present application, related devices for implementing the above-mentioned solutions are also provided below. Please refer to Figure 8 specifically, which is a structural diagram of the evaluation device of the machine learning model provided in an embodiment of the present application. The evaluation device 800 of the machine learning model includes: an acquisition module 801, used to obtain a first evaluation sample, the first evaluation sample includes descriptive data of the first traffic scene within a first time length, the first evaluation sample includes multiple first segment data, each first segment data includes descriptive data of the first traffic scene within a first sub-time length, and the first time length includes the first sub-time length; a processing module 802, used to process the multiple first segment data through the first machine learning model to obtain multiple first prediction labels corresponding one-to-one to the multiple first segment data; a determination module 803, used to determine a first parameter value of at least one evaluation indicator based on the multiple first prediction labels; wherein, at least one evaluation indicator indicates the stability of the first machine learning model when generating multiple first prediction labels, and / or, at least one evaluation indicator indicates the accuracy of the multiple first prediction labels generated by the first machine learning model in the first traffic scene, and the accuracy of the multiple first prediction labels is obtained based on the multiple first prediction labels and the true values ​​corresponding to the multiple first segment data.

[0183] Optionally, the first evaluation sample is obtained based on road test data, and the road test data includes descriptive data of traffic scenes collected when the autonomous driving vehicle is driving on the road.

[0184] Optionally, when the first parameter value of at least one evaluation indicator does not meet the second condition, it means that the first machine learning model fails the evaluation in the first traffic scenario.

[0185] Optionally, the processing module 802 is further used to process the first training sample through the second machine learning model to obtain a second prediction label corresponding to the first training sample, and the similarity between the first training sample and the second evaluation sample meets the first condition, wherein the first training sample and the second evaluation sample both include description data of the traffic scene, and the second evaluation sample is an evaluation sample in which the parameter value of at least one evaluation indicator obtained when the first machine learning model is evaluated does not meet the second condition; the processing module 802 is further used to process the second training sample through the second machine learning model to obtain a second prediction label corresponding to the second training sample, and the second The training sample is different from the first training sample; the processing module 802 is also used to process the second training sample through the first machine learning model to obtain a first prediction label corresponding to the second training sample; the evaluation device 800 of the machine learning model also includes: an updating module 804, used to update the second machine learning model according to the first loss term and the second loss term, wherein the first loss term indicates the similarity between the second prediction label corresponding to the first training sample and the true value corresponding to the first training sample, and the second loss term indicates the similarity between the second prediction label corresponding to the second training sample and the first prediction label corresponding to the second training sample.

[0186] Optionally, the evaluation device 800 of the machine learning model also includes: a feature extraction module 805, which is used to extract features of the second evaluation sample through the first machine learning model to obtain feature information of the second evaluation sample; a determination module 803, which is also used to determine the first training sample set from the training sample set based on the feature information of the second evaluation sample and the feature information of the training sample of the first machine learning model, wherein the training sample set includes multiple training samples of the first machine learning model, the first training sample set includes the first training sample, and the similarity between the feature information of the first training sample and the feature information of the second evaluation sample meets the first condition.

[0187] Optionally, the evaluation sample set of the first machine learning model includes multiple evaluation samples, the target evaluation sample is any one of the evaluation sample set of the first machine learning model, the target evaluation sample is obtained based on the driving data, the driving data includes the description data of the traffic scene and the original label corresponding to the description data of the traffic scene, the target evaluation sample includes the description data of the traffic scene and the true value corresponding to the description data of the traffic scene; wherein, when the driving data is a negative sample, the true value in the target evaluation sample is obtained by modifying the original label in the driving data.

[0188] Optionally, when the driving data is a negative sample, the driving data includes descriptive data of the traffic scene within a second time length, and the target evaluation sample includes descriptive data of the traffic scene within a third time length, and the second time length is greater than the third time length.

[0189] Optionally, the task types performed by the first machine learning model include one or more of the following: determining whether to cut in line, determining whether to give way, or determining whether to limit the speed.

[0190] It should be noted that the information interaction, execution process, etc. between the modules / units in the evaluation device 800 of the machine learning model are based on the same concept as the various method embodiments corresponding to Figures 1b to 7 in this application. For specific contents, please refer to the description in the method embodiments shown above in this application, and will not be repeated here.

[0191] Please continue to refer to Figure 9, which is a structural diagram of a data processing device provided in an embodiment of the present application. The data processing device 900 includes: a processing module 901, which is used to process the first training sample through the second machine learning model to obtain a second predicted label corresponding to the first training sample; the processing module 901 is also used to process the second training sample through the second machine learning model to obtain a second predicted label corresponding to the second training sample, wherein the second training sample and the first training sample both include description data of the traffic scene, and the second training sample and the first training sample correspond to different traffic scenes; the processing module 901 is also used to process the second training sample through the first machine learning model to obtain a first predicted label corresponding to the second training sample; an updating module 902 is used to update the second machine learning model according to a first loss term and a second loss term, wherein the first loss term indicates the similarity between the second predicted label corresponding to the first training sample and the true value corresponding to the first training sample, and the second loss term indicates the similarity between the second predicted label corresponding to the second training sample and the first predicted label corresponding to the second training sample.

[0192] Optionally, the first training sample is determined based on a second evaluation sample, where the second evaluation sample is an evaluation sample in which a parameter value of at least one evaluation indicator obtained when evaluating the first machine learning model does not meet the second condition.

[0193] Optionally, the data processing device 900 also includes: an acquisition module 903, used to obtain a first evaluation sample, the first evaluation sample includes descriptive data of the first traffic scene within a first time length, the first evaluation sample includes multiple first segment data, each first segment data includes descriptive data of the first traffic scene within a first sub-time length, and the first time length includes the first sub-time length; the processing module 901 is also used to process the multiple first segment data through the first machine learning model to obtain multiple first prediction labels corresponding one-to-one to the multiple first segment data; the data processing device 900 also includes: a determination module 904, used to determine the first parameter value of at least one evaluation indicator based on the multiple first prediction labels; wherein, when the first parameter value of at least one evaluation indicator meets the second condition, it represents that the first machine learning model passes the evaluation in the first traffic scene, at least one evaluation indicator indicates the stability of the first machine learning model when generating multiple first prediction labels, and / or, at least one evaluation indicator indicates the accuracy of the multiple first prediction labels generated by the first machine learning model in the first traffic scene, and the accuracy of the multiple first prediction labels is obtained based on the multiple first prediction labels and the true values ​​corresponding to the multiple first segment data.

[0194] It should be noted that the information interaction, execution process, etc. between the modules / units in the data processing device 900 are based on the same concept as the various method embodiments corresponding to Figures 1b to 7 in this application. For specific contents, please refer to the description in the method embodiments shown above in this application, and will not be repeated here.

[0195] The present application also provides a device. Please refer to Figure 10, which is a schematic structural diagram of a device provided in an embodiment of the present application. The device 1000 includes: a receiver 1001, a transmitter 1002, a processor 1003, and a memory 1004 (wherein the number of processors 1003 in the device 1000 may be one or more, and Figure 10 uses one processor as an example). The processor 1003 may include an application processor 10031 and a communication processor 10032. In some embodiments of the present application, the receiver 1001, the transmitter 1002, the processor 1003, and the memory 1004 may be connected via a bus or other means.

[0196] The memory 1004 may include a read-only memory and a random access memory, and provides instructions and data to the processor 1003. A portion of the memory 1004 may also include non-volatile random access memory (NVRAM). The memory 1004 stores processor and operation instructions, executable modules, or data structures, or subsets or extended sets thereof. The operation instructions may include various operation instructions for implementing various operations.

[0197] Processor 1003 controls the operation of the execution device. In specific applications, the various components of the execution device are coupled together via a bus system. In addition to a data bus, the bus system may also include a power bus, a control bus, and a status signal bus. However, for clarity, all bus systems are referred to as a bus system in the figure.

[0198] The methods disclosed in the above embodiments of the present application can be applied to or implemented by processor 1003. Processor 1003 can be an integrated circuit chip with signal processing capabilities. During implementation, each step of the above method can be completed by hardware integrated logic circuits or software instructions in processor 1003. The above processor 1003 can be a general-purpose processor, a digital signal processor (DSP), a microprocessor, or a microcontroller, and can further include an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. The processor 1003 can implement or execute the various methods, steps, and logic block diagrams disclosed in the embodiments of the present application. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the methods disclosed in conjunction with the embodiments of the present application can be directly implemented as being executed by a hardware decoding processor, or can be executed by a combination of hardware and software modules in the decoding processor. The software module can be located in a storage medium well-known in the art, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, etc. The storage medium is located in memory 1004, and processor 1003 reads the information in memory 1004 and, in conjunction with its hardware, completes the steps of the above method.

[0199] Receiver 1001 can be used to receive input digital or character information and generate signal input related to executing device-related settings and function control. Transmitter 1002 can be used to output digital or character information through the first interface. Transmitter 1002 can also be used to send instructions to the disk pack through the first interface to modify data in the disk pack. Transmitter 1002 can also include a display device such as a display screen.

[0200] In the embodiment of the present application, the processor 1003 can be used to execute the method executed by the first device, the second device, and / or the third device in the embodiments corresponding to Figures 1b to 7. It should be noted that the specific manner in which the processor 1003 executes the aforementioned steps is based on the same concept as the various method embodiments corresponding to Figures 1b to 7 in the present application, and the technical effects brought about are the same as the various method embodiments corresponding to Figures 1b to 7 in the present application. For specific details, please refer to the description of the method embodiments shown above in the present application, and no further details will be given here.

[0201] A computer-readable storage medium is also provided in an embodiment of the present application. The computer-readable storage medium stores a program, which, when executed on a computer, enables the computer to execute the steps performed by the first device, the second device and / or the third device in the method described in the embodiments shown in Figures 1b to 7 above.

[0202] An embodiment of the present application also provides a computer program product, which includes a program. When the aforementioned program is run on a computer, the computer executes the steps performed by the first device, the second device and / or the third device in the method described in the embodiments shown in Figures 1b to 7 above.

[0203] A circuit system is also provided in an embodiment of the present application, which includes a processing circuit, and the processing circuit is configured to execute the steps performed by the first device, the second device and / or the third device in the method described in the embodiments shown in Figures 1b to 7 above.

[0204] The evaluation device, data processing device or equipment of the machine learning model provided in the embodiments of the present application can be specifically a chip, and the chip includes: a processing unit and a communication unit, wherein the processing unit can be, for example, a processor, and the communication unit can be, for example, an input / output interface, a pin or a circuit. The processing unit can execute the computer-executable instructions stored in the storage unit so that the chip executes the method described in the embodiments shown in Figures 1b to 7 above. Optionally, the storage unit is a storage unit within the chip, such as a register, a cache, etc. The storage unit can also be a storage unit located outside the chip within the wireless access device, such as a read-only memory (ROM) or other types of static storage devices that can store static information and instructions, a random access memory (RAM), etc.

[0205] Specifically, see Figure 11, which is a schematic diagram of the structure of a chip provided in an embodiment of the present application. The chip can be represented as a neural network processor NPU 110. NPU 110 is mounted on the host CPU (host CPU) as a coprocessor and assigned tasks by the host CPU. The core of the NPU is arithmetic circuit 1103, which is controlled by controller 1104 to extract matrix data from memory and perform multiplication operations.

[0206] In some implementations, the arithmetic circuit 1103 includes multiple processing units (PEs). In some implementations, the arithmetic circuit 1103 is a two-dimensional systolic array. The arithmetic circuit 1103 may also be a one-dimensional systolic array or other electronic circuitry capable of performing mathematical operations such as multiplication and addition. In some implementations, the arithmetic circuit 1103 is a general-purpose matrix processor.

[0207] For example, assume there are input matrix A, weight matrix B, and output matrix C. The arithmetic circuit retrieves the corresponding data of matrix B from weight memory 1102 and caches it on each PE in the arithmetic circuit. The arithmetic circuit retrieves the data of matrix A from input memory 1101 and performs a matrix operation on matrix B. The partial or final matrix result is stored in accumulator 1108.

[0208] Unified memory 1106 is used to store input and output data. Weight data is directly transferred to weight memory 1102 through the Direct Memory Access Controller (DMAC) 1105. Input data is also transferred to unified memory 1106 through the DMAC.

[0209] BIU stands for Bus Interface Unit, i.e., bus interface unit 1110 , which is used for interaction between the AXI bus, DMAC, and instruction fetch buffer (IFB) 1109 .

[0210] The bus interface unit 1110 (BIU) is used for the instruction fetch memory 1109 to obtain instructions from the external memory, and is also used for the storage unit access controller 1105 to obtain the original data of the input matrix A or the weight matrix B from the external memory.

[0211] DMAC is mainly used to move input data in the external memory DDR to the unified memory 1106 or to move weight data to the weight memory 1102 or to move input data to the input memory 1101.

[0212] The vector calculation unit 1107 includes multiple operation processing units. When necessary, it further processes the output of the operation circuit, such as vector multiplication, vector addition, exponential operation, logarithmic operation, size comparison, etc. It is mainly used for non-convolutional / fully connected layer network calculations in neural networks, such as batch normalization, pixel-level summation, and upsampling of feature planes.

[0213] In some implementations, the vector calculation unit 1107 can store the processed output vector to the unified memory 1106. For example, the vector calculation unit 1107 can apply a linear function and / or a nonlinear function to the output of the operation circuit 1103, such as linear interpolation of the feature plane extracted by the convolution layer, or accumulate a vector of values ​​to generate an activation value. In some implementations, the vector calculation unit 1107 generates a normalized value, a pixel-level summed value, or both. In some implementations, the processed output vector can be used as an activation input to the operation circuit 1103, such as for use in subsequent layers in a neural network.

[0214] An instruction fetch buffer 1109 connected to the controller 1104 is used to store instructions used by the controller 1104;

[0215] Unified memory 1106, input memory 1101, weight memory 1102, and instruction fetch memory 1109 are all on-chip memories. External memories are private to the NPU hardware architecture.

[0216] Among them, the operations of each layer in each machine learning model shown in Figure 1b to Figure 7 can be performed by the operation circuit 1103 or the vector calculation unit 1107.

[0217] The processor mentioned in any of the above places can be a general-purpose central processing unit, a microprocessor, an ASIC, or one or more integrated circuits for controlling the execution of the program of the above-mentioned first aspect method.

[0218] It should also be noted that the device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separate, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed across multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the present embodiment. In addition, in the drawings of the device embodiments provided in this application, the connection relationship between the modules indicates that there is a communication connection between them, which can be specifically implemented as one or more communication buses or signal lines.

[0219] Through the description of the above embodiments, those skilled in the art can clearly understand that the present application can be implemented by means of software plus necessary general-purpose hardware, and of course can also be implemented by dedicated hardware including application-specific integrated circuits, dedicated CLUs, dedicated memories, dedicated components, etc. In general, all functions performed by computer programs can be easily implemented with corresponding hardware, and the specific hardware structures used to implement the same function can also be diverse, such as analog circuits, digital circuits, or dedicated circuits. However, for the present application, software program implementation is a better implementation method in most cases. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which is stored in a readable storage medium, such as a computer's floppy disk, USB flash drive, mobile hard disk, ROM, RAM, disk or optical disk, etc., and includes a number of instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to execute the methods described in each embodiment of the present application.

[0220] In the above embodiments, all or part of the embodiments may be implemented by software, hardware, firmware, or any combination thereof. When implemented by software, all or part of the embodiments may be implemented in the form of a computer program product.

[0221] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from a website, a computer, a server, or a data center by wired (such as coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (such as infrared, wireless, microwave, etc.) mode to another website, a computer, a server, or a data center. The computer-readable storage medium can be any available medium that a computer can store or a data storage device such as a server or a data center that includes one or more available media integrations. The available medium can be a magnetic medium, (such as a floppy disk, a hard disk, a magnetic tape), an optical medium (such as a DVD), or a semiconductor medium (such as a solid-state drive (SSD)).

Claims

1. A method for evaluating a machine learning model, characterized in that: The method comprises: Obtain a first evaluation sample, the first evaluation sample comprising description data of a first traffic scene within a first duration, the first evaluation sample comprising a plurality of first segment data, each of the first segment data comprising description data of the first traffic scene within a first sub-duration, and the first duration including the first sub-duration; Processing the plurality of first segment data by a first machine learning model to obtain a plurality of first prediction labels corresponding one-to-one to the plurality of first segment data; Determining a first parameter value of at least one evaluation indicator according to the multiple first prediction labels; Among them, the at least one evaluation indicator indicates the stability of the first machine learning model when generating the multiple first prediction labels, and / or, the at least one evaluation indicator indicates the accuracy of the multiple first prediction labels generated by the first machine learning model in the first traffic scene, and the accuracy of the multiple first prediction labels is based on the multiple first prediction labels and the true values ​​corresponding to the multiple first fragment data.

2. The method according to claim 1, characterized in that The first evaluation sample is obtained based on road test data, and the road test data includes descriptive data of traffic scenes collected when the autonomous driving vehicle is driving on the road.

3. The method according to claim 1 or 2, characterized in that: When the first parameter value of the at least one evaluation indicator does not satisfy the second condition, it means that the first machine learning model fails the evaluation in the first traffic scenario.

4. The method according to claim 1 or 2, characterized in that: After determining the first parameter value of at least one evaluation indicator according to the plurality of first prediction labels, the method further includes: The first training sample is processed by the second machine learning model to obtain a second prediction label corresponding to the first training sample, and the similarity between the first training sample and the second evaluation sample satisfies the first condition, wherein the first training sample and the second evaluation sample both include description data of the traffic scene, and the second evaluation sample is an evaluation sample in which the parameter value of the at least one evaluation indicator obtained when the first machine learning model is evaluated does not satisfy the second condition; Processing the second training sample by the second machine learning model to obtain a second prediction label corresponding to the second training sample, where the second training sample is different from the first training sample; Processing the second training sample by the first machine learning model to obtain a first prediction label corresponding to the second training sample; The second machine learning model is updated according to a first loss term and a second loss term, wherein the first loss term indicates the similarity between the second predicted label corresponding to the first training sample and the true value corresponding to the first training sample, and the second loss term indicates the similarity between the second predicted label corresponding to the second training sample and the first predicted label corresponding to the second training sample.

5. The method according to claim 4, characterized in that The method further comprises: Extracting features of the second evaluation sample by using the first machine learning model to obtain feature information of the second evaluation sample; Based on the feature information of the second evaluation sample and the feature information of the training sample of the first machine learning model, a first training sample set is determined from the training sample set, wherein the training sample set includes multiple training samples of the first machine learning model, the first training sample set includes the first training sample, and the similarity between the feature information of the first training sample and the feature information of the second evaluation sample satisfies the first condition.

6. The method according to claim 1 or 2, characterized in that: The evaluation sample set of the first machine learning model includes multiple evaluation samples, the target evaluation sample is any one of the evaluation sample set of the first machine learning model, the target evaluation sample is obtained based on driving data, the driving data includes description data of the traffic scene and original labels corresponding to the description data of the traffic scene, and the target evaluation sample includes description data of the traffic scene and true values ​​corresponding to the description data of the traffic scene; Wherein, in the case where the driving data is a negative sample, the true value in the target evaluation sample is obtained by modifying the original label in the driving data.

7. The method according to claim 6, characterized in that In the case where the driving data is a negative sample, the driving data includes descriptive data of the traffic scene within a second time length, and the target evaluation sample includes descriptive data of the traffic scene within a third time length, and the second time length is greater than the third time length.

8. The method according to claim 1 or 2, characterized in that: The task types performed by the first machine learning model include one or more of the following: determining whether to cut in, determining whether to give way, or determining whether to limit the speed.

9. A data processing method, characterized in that: The method comprises: Processing the first training sample by a second machine learning model to obtain a second prediction label corresponding to the first training sample; Processing the second training sample by the second machine learning model to obtain a second prediction label corresponding to the second training sample, wherein the second training sample and the first training sample both include description data of the traffic scene, and the second training sample and the first training sample correspond to different traffic scenes; Processing the second training sample by the first machine learning model to obtain a first prediction label corresponding to the second training sample; The second machine learning model is updated according to a first loss term and a second loss term, wherein the first loss term indicates the similarity between the second predicted label corresponding to the first training sample and the true value corresponding to the first training sample, and the second loss term indicates the similarity between the second predicted label corresponding to the second training sample and the first predicted label corresponding to the second training sample.

10. The method according to claim 9, characterized in that The similarity between the first training sample and the second evaluation sample satisfies the first condition, and the second evaluation sample is an evaluation sample in which the parameter value of at least one evaluation indicator obtained when evaluating the first machine learning model does not satisfy the second condition.

11. The method according to claim 10, characterized in that Before processing the first training sample by the second machine learning model, the method further includes: Obtain a first evaluation sample, the first evaluation sample comprising description data of a first traffic scene within a first duration, the first evaluation sample comprising a plurality of first segment data, each of the first segment data comprising description data of the first traffic scene within a first sub-duration, and the first duration including the first sub-duration; Processing the plurality of first segment data by a first machine learning model to obtain a plurality of first prediction labels corresponding one-to-one to the plurality of first segment data; Determining a first parameter value of the at least one evaluation indicator according to the multiple first prediction labels; Among them, when the first parameter value of the at least one evaluation indicator satisfies the second condition, it represents that the first machine learning model passes the evaluation in the first traffic scenario, and the at least one evaluation indicator indicates the stability of the first machine learning model when generating the multiple first prediction labels, and / or the at least one evaluation indicator indicates the accuracy of the multiple first prediction labels generated by the first machine learning model in the first traffic scenario, and the accuracy of the multiple first prediction labels is obtained based on the multiple first prediction labels and the true values ​​corresponding to the multiple first fragment data.

12. A device for evaluating a machine learning model, characterized in that: The device comprises: An acquisition module, used to acquire a first evaluation sample, wherein the first evaluation sample includes description data of a first traffic scene within a first duration, the first evaluation sample includes a plurality of first segment data, each of the first segment data includes description data of the first traffic scene within a first sub-duration, and the first duration includes the first sub-duration; A processing module, configured to process the plurality of first segment data by using a first machine learning model to obtain a plurality of first prediction labels corresponding one-to-one to the plurality of first segment data; A determination module, configured to determine a first parameter value of at least one evaluation indicator according to the plurality of first prediction labels; Among them, the at least one evaluation indicator indicates the stability of the first machine learning model when generating the multiple first prediction labels, and / or, the at least one evaluation indicator indicates the accuracy of the multiple first prediction labels generated by the first machine learning model in the first traffic scene, and the accuracy of the multiple first prediction labels is based on the multiple first prediction labels and the true values ​​corresponding to the multiple first fragment data.

13. The device according to claim 12, characterized in that The first evaluation sample is obtained based on road test data, and the road test data includes descriptive data of traffic scenes collected when the autonomous driving vehicle is driving on the road.

14. The device according to claim 12 or 13, characterized in that When the first parameter value of the at least one evaluation indicator does not satisfy the second condition, it means that the first machine learning model fails the evaluation in the first traffic scenario.

15. The device according to claim 12 or 13, characterized in that The processing module is further used to process the first training sample through the second machine learning model to obtain a second prediction label corresponding to the first training sample, and the similarity between the first training sample and the second evaluation sample satisfies the first condition, wherein the first training sample and the second evaluation sample both include description data of the traffic scene, and the second evaluation sample is an evaluation sample in which the parameter value of the at least one evaluation indicator obtained when the first machine learning model is evaluated does not satisfy the second condition; The processing module is further configured to process a second training sample by using the second machine learning model to obtain a second prediction label corresponding to the second training sample, where the second training sample is different from the first training sample; The processing module is further configured to process the second training sample by using the first machine learning model to obtain a first prediction label corresponding to the second training sample; The device also includes: an updating module, used to update the second machine learning model according to a first loss term and a second loss term, wherein the first loss term indicates the similarity between the second predicted label corresponding to the first training sample and the true value corresponding to the first training sample, and the second loss term indicates the similarity between the second predicted label corresponding to the second training sample and the first predicted label corresponding to the second training sample.

16. The device according to claim 15, characterized in that The device also includes: A feature extraction module, used to extract features of the second evaluation sample by using the first machine learning model to obtain feature information of the second evaluation sample; The determination module is also used to determine a first training sample set from a training sample set based on the feature information of the second evaluation sample and the feature information of the training sample of the first machine learning model, wherein the training sample set includes multiple training samples of the first machine learning model, the first training sample set includes the first training sample, and the similarity between the feature information of the first training sample and the feature information of the second evaluation sample satisfies the first condition.

17. The device according to claim 12 or 13, characterized in that The evaluation sample set of the first machine learning model includes multiple evaluation samples, the target evaluation sample is any one of the evaluation sample set of the first machine learning model, the target evaluation sample is obtained based on driving data, the driving data includes description data of the traffic scene and original labels corresponding to the description data of the traffic scene, and the target evaluation sample includes description data of the traffic scene and true values ​​corresponding to the description data of the traffic scene; Wherein, in the case where the driving data is a negative sample, the true value in the target evaluation sample is obtained by modifying the original label in the driving data.

18. The device according to claim 17, characterized in that In the case where the driving data is a negative sample, the driving data includes descriptive data of the traffic scene within a second time length, and the target evaluation sample includes descriptive data of the traffic scene within a third time length, and the second time length is greater than the third time length.

19. The device according to claim 12 or 13, characterized in that The task types performed by the first machine learning model include one or more of the following: determining whether to cut in, determining whether to give way, or determining whether to limit the speed.

20. A data processing device, characterized in that: The device comprises: A processing module, configured to process the first training sample by using a second machine learning model to obtain a second prediction label corresponding to the first training sample; The processing module is further used to process the second training sample through the second machine learning model to obtain a second prediction label corresponding to the second training sample, wherein the second training sample and the first training sample both include description data of the traffic scene, and the second training sample and the first training sample correspond to different traffic scenes; The processing module is further configured to process the second training sample by using the first machine learning model to obtain a first prediction label corresponding to the second training sample; An updating module is used to update the second machine learning model according to a first loss term and a second loss term, wherein the first loss term indicates the similarity between the second predicted label corresponding to the first training sample and the true value corresponding to the first training sample, and the second loss term indicates the similarity between the second predicted label corresponding to the second training sample and the first predicted label corresponding to the second training sample.

21. The device according to claim 20, characterized in that The similarity between the first training sample and the second evaluation sample satisfies the first condition, and the second evaluation sample is an evaluation sample in which the parameter value of at least one evaluation indicator obtained when evaluating the first machine learning model does not satisfy the second condition.

22. The device according to claim 21, characterized in that The device also includes: An acquisition module, used to acquire a first evaluation sample, wherein the first evaluation sample includes description data of a first traffic scene within a first duration, the first evaluation sample includes a plurality of first segment data, each of the first segment data includes description data of the first traffic scene within a first sub-duration, and the first duration includes the first sub-duration; The processing module is further used to process the plurality of first segment data by using a first machine learning model to obtain a plurality of first prediction labels corresponding one-to-one to the plurality of first segment data; The device further includes: a determination module, configured to determine a first parameter value of the at least one evaluation indicator according to the plurality of first prediction labels; Among them, when the first parameter value of the at least one evaluation indicator satisfies the second condition, it represents that the first machine learning model passes the evaluation in the first traffic scenario, and the at least one evaluation indicator indicates the stability of the first machine learning model when generating the multiple first prediction labels, and / or the at least one evaluation indicator indicates the accuracy of the multiple first prediction labels generated by the first machine learning model in the first traffic scenario, and the accuracy of the multiple first prediction labels is obtained based on the multiple first prediction labels and the true values ​​corresponding to the multiple first fragment data.

23. A device, characterized in that The method comprises a processor, wherein the processor is coupled to a memory, wherein the memory stores program instructions, and when the program instructions stored in the memory are executed by the processor, the method according to any one of claims 1 to 11 is implemented.

24. A computer-readable storage medium comprising a program, which, when executed on a computer, causes the computer to execute the method according to any one of claims 1 to 11.

25. A circuit system, characterized in that: The circuit system comprises a processing circuit configured to perform the method of any one of claims 1 to 11.

Images