Evaluation method of machine learning model, data processing method and related equipment

Abstract

The embodiment of the invention discloses an evaluation method of a machine learning model, a data processing method and related equipment, the method can be used in the field of automatic driving in artificial intelligence, and the method comprises the following steps: processing a plurality of fragment data in an evaluation sample through the machine learning model to obtain a plurality of prediction tags, determining a parameter value of at least one evaluation index; the evaluation sample comprises description data of the traffic scene in a first duration, the fragment data comprises description data of the traffic scene in a first sub-duration, and the first duration comprises the first sub-duration; the at least one evaluation index indicates stability and / or accuracy corresponding to the plurality of prediction labels, and the accuracy is obtained based on true values corresponding to the plurality of fragment data; the performance of the machine learning model applied to the vehicle driving process can be reflected more accurately by constructing the evaluation index on the granularity of the traffic scene, the simulated traffic scene does not need to be established, the consumed computer resources are reduced, and the evaluation efficiency is improved.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence, and in particular to a machine learning model evaluation method, a data processing method, and related equipment. Background Art

[0002] Artificial Intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. In other words, artificial intelligence is a branch of computer science that attempts to understand the essence of intelligence and produce a new type of intelligent machine that can respond in a similar way to human intelligence. Artificial intelligence is to study the design principles and implementation methods of various intelligent machines so that machines have the functions of perception, reasoning and decision-making. Applying machine learning models to the field of autonomous driving is a common application of artificial intelligence.

[0003] At present, in order to evaluate the trained machine learning model, a simulated traffic environment can be established, so that vehicles equipped with machine learning models can travel in the simulated traffic environment, thereby evaluating the performance of multiple machine learning models.

[0004] However, since establishing a simulated traffic environment consumes a lot of computer resources and the process of establishing a simulated traffic environment is time-consuming, the evaluation efficiency is low. Summary of the invention

[0005] In order to solve the above technical problems, the present application provides a machine learning model evaluation method, a data processing method and related equipment. The evaluation indicators adopted are constructed at the granularity of traffic scenes. Since the vehicle driving process is continuous, "constructing evaluation indicators at the granularity of traffic scenes" can more accurately reflect the performance of the machine learning model applied to the vehicle driving process; in addition, the solution provided by the present application does not require the establishment of a simulated traffic scene, which reduces the computer resources consumed by the evaluation process of the machine learning model and improves the evaluation efficiency.

[0006] The technical solutions provided by this application are as follows:

[0007] In a first aspect, an embodiment of the present application provides a method for evaluating a machine learning model, which can be used in the field of autonomous driving in artificial intelligence. The method includes: a first device obtains a first evaluation sample, the first evaluation sample includes description data of a first traffic scene within a first time length, the first evaluation sample includes multiple first fragment data, each first fragment data includes description data of the first traffic scene within a first sub-time length, and the first time length may include one or more first sub-time lengths; exemplarily, the aforementioned description data may be specifically expressed as video, image, scene description file or other types of data, etc. For example, the scene description file may include the position, speed, acceleration or other status information of each traffic participant in the first traffic scene, etc.

[0008] The first device processes the multiple first fragment data through the first machine learning model to obtain multiple first prediction labels corresponding to the multiple first fragment data; and then determines the first parameter value of at least one evaluation indicator based on the multiple first prediction labels.

[0009] Among them, at least one evaluation indicator indicates the stability of the first machine learning model when generating multiple first prediction labels, and / or, at least one evaluation indicator indicates the accuracy of the multiple first prediction labels generated by the first machine learning model in the first traffic scene, and the accuracy of the multiple first prediction labels is based on the multiple first prediction labels and the true values ​​corresponding to the multiple first fragment data.

[0010] For example, the first evaluation index can represent the number of jumps of multiple first prediction labels. The smaller the number of jumps of multiple first prediction labels, the better the stability of the multiple first prediction labels. For another example, the first evaluation index can represent the jump frequency of multiple first prediction labels. The jump frequency of multiple first prediction labels can be the ratio between the number of jumps of multiple first prediction labels and the total number of multiple first prediction labels. The lower the jump frequency of multiple first prediction labels, the better the stability of multiple first prediction labels. The "accuracy of multiple first prediction labels" can adopt the accuracy of multiple first prediction labels. The accuracy of first prediction labels can represent the ratio between the number of first prediction labels that are consistent with the true value and the total number of multiple first prediction labels.

[0011] In this implementation, a first machine learning model is used to generate multiple first prediction labels for descriptive data of a certain traffic scene, and then the performance of the first machine learning model in the aforementioned traffic scene is evaluated based on the multiple first prediction labels. The adopted evaluation indicators indicate the stability of the first machine learning model when generating the multiple first prediction labels, and / or, at least one evaluation indicator indicates the accuracy of the multiple first prediction labels generated by the first machine learning model in the first traffic scene. That is, the evaluation indicators adopted in this application are constructed at the granularity of the traffic scene. Since the vehicle driving process is continuous, "constructing evaluation indicators at the granularity of the traffic scene" can more accurately reflect the performance of the machine learning model applied to the vehicle driving process; in addition, the solution provided in this application does not require the establishment of a simulated traffic scene, which reduces the computer resources consumed by the evaluation process of the machine learning model and improves the evaluation efficiency.

[0012] In a possible implementation of the first aspect, the first evaluation sample can be obtained based on road test data, and the road test data includes description data of traffic scenes collected when the autonomous driving vehicle is driving on the road; optionally, the road test data may include description data of traffic scenes collected when the autonomous driving vehicle is driving on a real road. The "autonomous driving vehicle" can be configured as a fully or partially autonomous driving mode. In this implementation, since the first machine learning model is configured in the vehicle to realize the autonomous driving mode of the vehicle, the behavior of "using road test data to evaluate the performance of the first machine learning model" makes the "process of evaluating the first machine learning model" and the "actual application process of the first machine learning model" more similar, so that the obtained evaluation results are more accurate.

[0013] In a possible implementation of the first aspect, when the first parameter value of the at least one evaluation indicator obtained based on the first evaluation sample does not meet the second condition, it means that the first machine learning model has failed the evaluation in the first traffic scene. In this implementation, the judgment condition that the first machine learning model has failed the evaluation in a certain traffic scene is clarified, which reduces the difficulty of implementing this solution.

[0014] In a possible implementation of the first aspect, when the stability of the first machine learning model when generating multiple prediction labels corresponding to the first evaluation sample does not meet the first sub-condition, and / or when the accuracy of the multiple prediction labels corresponding to the first evaluation sample does not meet the second sub-condition, it is deemed that the first parameter value of at least one evaluation indicator does not meet the second condition.

[0015] For example, when the number of jumps of multiple prediction labels corresponding to the first evaluation sample is greater than or equal to a preset number of times, it can be regarded that the stability of the multiple first prediction labels does not meet the first sub-condition; or, when the jump frequency of the multiple prediction labels corresponding to the first evaluation sample is greater than or equal to a preset value, it can be regarded that the stability of the multiple first prediction labels does not meet the first sub-condition.

[0016] When the accuracy of the multiple predicted labels corresponding to the first evaluation sample is less than or equal to the preset accuracy, it can be regarded that the accuracy of the multiple first predicted labels does not meet the second sub-condition.

[0017] In a possible implementation of the first aspect, after the first device determines the first parameter value of at least one evaluation indicator based on multiple first prediction labels, the method also includes: the first device processes the first training sample through the second machine learning model to obtain a second prediction label corresponding to the first training sample, and the similarity between the first training sample and the second evaluation sample satisfies the first condition, wherein the first training sample and the second evaluation sample both include description data of the traffic scene, and the second evaluation sample is an evaluation sample in which the parameter value of at least one evaluation indicator obtained when evaluating the first machine learning model does not meet the second condition; it should be noted that the "second evaluation sample" and the "first evaluation sample" have similar meanings, and both are evaluation samples in the evaluation sample set of the first machine learning model. The difference is that the "first evaluation sample" is any evaluation sample used when evaluating the first machine learning model, and the parameter value of at least one evaluation indicator obtained after evaluating the first machine learning model through the "second evaluation sample" does not meet the second condition.

[0018] The first device processes the second training sample through the second machine learning model to obtain a second prediction label corresponding to the second training sample, where the second training sample is different from the first training sample; and processes the second training sample through the first machine learning model to obtain a first prediction label corresponding to the second training sample.

[0019] The first device updates the second machine learning model according to a first loss term and a second loss term, wherein the first loss term indicates the similarity between a second predicted label corresponding to the first training sample and a true value corresponding to the first training sample, and the second loss term indicates the similarity between a second predicted label corresponding to the second training sample and the first predicted label corresponding to the second training sample.

[0020] In this implementation, since the similarity between the first training sample and the second evaluation sample satisfies the first condition, and the parameter value of at least one evaluation indicator obtained when the first machine learning model is evaluated using the second evaluation sample does not satisfy the second condition, that is, when the first machine learning model is evaluated by the second evaluation sample, the performance of the first machine learning model is not good, then the first loss term indicates the similarity between the second prediction label corresponding to the first training sample and the true value corresponding to the first training sample, which is beneficial to improving the performance of the second machine learning model in the traffic scene corresponding to the first evaluation sample; the second training sample is different from the first training sample, that is, the first machine learning model has better performance in the traffic scene corresponding to the second training sample, and the second loss term indicates the similarity between the second prediction label corresponding to the second training sample and the first prediction label corresponding to the second training sample, which is beneficial to enable the second machine learning model to learn the excellent performance of the first machine learning model in the traffic scene corresponding to the second training sample; therefore, the trained second machine learning model not only has better performance in the traffic scene corresponding to the second evaluation sample, but also has better performance in the traffic scene corresponding to the second training sample, that is, a second machine learning model that can perform well in various traffic scenes can be obtained.

[0021] In a possible implementation of the first aspect, the method further includes: the first device extracts features from the second evaluation sample through the first machine learning model to obtain feature information of the second evaluation sample; illustratively, the feature information of the second evaluation sample may include feature information of each fragment data in at least one fragment data included in the second evaluation sample. The first device may determine a first training sample set from a training sample set based on the feature information of the second evaluation sample and the feature information of the training sample of the first machine learning model, wherein the training sample set includes multiple training samples of the first machine learning model, the first training sample set includes a first training sample, and the similarity between the feature information of the first training sample and the feature information of the second evaluation sample satisfies a first condition.

[0022] In this implementation, a manual method is adopted to classify the training samples according to different traffic scenarios, which may not be consistent with the different traffic scenarios understood by the machine learning model; based on the feature information of each second evaluation sample and the similarity between the feature information of each training sample, multiple first training samples are obtained from the training sample set, that is, the multiple first training samples obtained are samples that the first machine learning model considers to be similar to the second evaluation samples, so that from the perspective of the machine learning model, training samples that the first machine learning model is not good at processing can be found, so as to increase the probability that each first training sample is a training sample that the first machine learning model is not good at processing, so that the second machine learning model learns the true value corresponding to the first training sample when processing the first training sample, which is also conducive to obtaining a second machine learning model with better performance.

[0023] In a possible implementation of the first aspect, the similarity between the feature information of the first training sample and the feature information of the second evaluation sample can be obtained based on the first similarity between the feature information of the first training sample and the feature information of each fragment data of at least one fragment data included in the second evaluation sample, that is, the similarity between the feature information of the first training sample and the feature information of the second evaluation sample can be obtained based on at least one first similarity corresponding one-to-one to at least one fragment data included in the second evaluation sample. Exemplarily, the similarity between the feature information of the first training sample and the feature information of the second evaluation sample can be the average value of the aforementioned at least one first similarity, or the similarity between the feature information of the first training sample and the feature information of the second evaluation sample can be the median value of the aforementioned at least one first similarity.

[0024] In a possible implementation of the first aspect, the evaluation sample set of the first machine learning model includes multiple evaluation samples, the target evaluation sample is any one of the evaluation sample set of the first machine learning model, the target evaluation sample is obtained based on driving data, the driving data includes description data of the traffic scene and original labels corresponding to the description data of the aforementioned traffic scene, and the target evaluation sample includes the description data of the aforementioned traffic scene and the true value corresponding to the description data of the aforementioned traffic scene.

[0025] Among them, when the driving data is a negative sample, the true value in the target evaluation sample is obtained by modifying the original label in the driving data, and "driving data is a negative sample" can represent that the action indicated by the original label in the driving data is not suitable for execution in the traffic scene. In this implementation, since when a driving sample is a negative sample, the action pointed to by the original label corresponding to the description data of the traffic scene is not suitable for the traffic scene, the original label is modified to obtain the true value in the target evaluation sample, thereby ensuring that the actions performed by the true value in the evaluation sample are all actions adapted to the traffic scene, which is conducive to improving the accuracy of the evaluation results.

[0026] In a possible implementation of the first aspect, when the driving data is a negative sample, the driving data includes description data of the traffic scene within a second time period, the target evaluation sample includes description data of the traffic scene within a third time period, and the second time period is greater than the third time period. Exemplarily, the description data within the first time period included in the description data of the target traffic scene within the second time period included in the target driving data can be deleted to obtain the description data of the target traffic scene within the third time period included in the target evaluation sample, and the description data within the first time period includes the description data within a preset time period from the back to the front within the second time period.

[0027] In this implementation, because when the target driving data is a negative sample, the data in the second half of the description data of the traffic scene included in the target driving data within the second time length may lead the first machine learning model to output an erroneous prediction label, resulting in the inability to accurately reflect the performance of the first machine learning model itself. The second time length is greater than the third time length, that is, the data in the second half of the description data of the traffic scene included in the target driving data within the second time length is cut off, which is conducive to the obtained evaluation results to more accurately reflect the true performance of the first machine learning model.

[0028] In a possible implementation of the first aspect, the task types performed by the first machine learning model include one or more of the following: determining whether to cut in, determining whether to give way, or determining whether to limit the speed. In this implementation, multiple task types that can be performed by the first machine learning model are provided, which expands the application scenarios of this solution and improves the implementation flexibility of this solution.

[0029] In a second aspect, an embodiment of the present application provides a data processing method that can be used in the field of autonomous driving in artificial intelligence, the method comprising: a third device processes a first training sample through a second machine learning model to obtain a second prediction label corresponding to the first training sample. The third device processes a second training sample through a second machine learning model to obtain a second prediction label corresponding to the second training sample, wherein both the second training sample and the first training sample include description data of a traffic scene, and the second training sample and the first training sample correspond to different traffic scenes; the second training sample is processed through the first machine learning model to obtain a first prediction label corresponding to the second training sample. The third device updates the second machine learning model according to a first loss term and a second loss term, wherein the first loss term indicates the similarity between the second prediction label corresponding to the first training sample and the true value corresponding to the first training sample, and the second loss term indicates the similarity between the second prediction label corresponding to the second training sample and the first prediction label corresponding to the second training sample.

[0030] In this implementation, since the second training sample and the first training sample correspond to different traffic scenes, the first loss term indicates the similarity between the second prediction label corresponding to the first training sample and the true value corresponding to the first training sample, which is conducive to improving the performance of the second machine learning model in the traffic scene corresponding to the first training sample; the second loss term indicates the similarity between the second prediction label corresponding to the second training sample and the first prediction label corresponding to the second training sample, which is conducive to enabling the second machine learning model to learn the excellent performance of the first machine learning model in the traffic scene corresponding to the second training sample; therefore, when the first machine learning model performs poorly in the traffic scene corresponding to the first training sample, the trained second machine learning model not only has better performance in the traffic scene corresponding to the first training sample, but also has better performance in the traffic scene corresponding to the second training sample, that is, a second machine learning model that can perform well in various traffic scenes can be obtained.

[0031] In a possible implementation of the second aspect, the first training sample is determined based on the second evaluation sample, and the second evaluation sample is an evaluation sample in which the parameter value of at least one evaluation indicator obtained when evaluating the first machine learning model does not meet the second condition.

[0032] In the second aspect of the present application, the third device can also be used to execute the steps performed by the first device in the first aspect and various possible implementation methods of the first aspect. The specific implementation methods of the steps in the second aspect, the meanings of the terms and the beneficial effects brought about can all be referred to the first aspect and will not be repeated here.

[0033] In a third aspect, an embodiment of the present application provides an evaluation device for a machine learning model, which can be used in the field of autonomous driving of artificial intelligence, the device comprising: an acquisition module, used to acquire a first evaluation sample, the first evaluation sample comprising descriptive data of a first traffic scene within a first time length, the first evaluation sample comprising multiple first segment data, each first segment data comprising descriptive data of the first traffic scene within a first sub-time length, the first time length including the first sub-time length; a processing module, used to process the multiple first segment data through a first machine learning model to obtain multiple first prediction labels corresponding one-to-one to the multiple first segment data; a determination module, used to determine a first parameter value of at least one evaluation indicator based on the multiple first prediction labels; wherein, at least one evaluation indicator indicates the stability of the first machine learning model when generating multiple first prediction labels, and / or, at least one evaluation indicator indicates the accuracy of the multiple first prediction labels generated by the first machine learning model in the first traffic scene, and the accuracy of the multiple first prediction labels is obtained based on the multiple first prediction labels and the true values ​​corresponding to the multiple first segment data.

[0034] In the third aspect of the present application, the evaluation device of the machine learning model is also used to execute the steps performed by the first device in the first aspect and various possible implementation methods of the first aspect. The specific implementation methods of the steps in the third aspect, the meanings of the terms and the beneficial effects brought about can all be referred to the first aspect and will not be repeated here.

[0035] In a fourth aspect, an embodiment of the present application provides a data processing device that can be used in the field of autonomous driving of artificial intelligence, the device comprising: a processing module, used to process a first training sample through a second machine learning model to obtain a second prediction label corresponding to the first training sample; the processing module, also used to process the second training sample through the second machine learning model to obtain a second prediction label corresponding to the second training sample, wherein the second training sample and the first training sample both include description data of traffic scenes, and the second training sample and the first training sample correspond to different traffic scenes; the processing module, also used to process the second training sample through the first machine learning model to obtain a first prediction label corresponding to the second training sample; an updating module, used to update the second machine learning model according to a first loss term and a second loss term, wherein the first loss term indicates the similarity between the second prediction label corresponding to the first training sample and the true value corresponding to the first training sample, and the second loss term indicates the similarity between the second prediction label corresponding to the second training sample and the first prediction label corresponding to the second training sample.

[0036] In the fourth aspect of the present application, the data processing device is also used to execute the steps performed by the third device in the second aspect and various possible implementation methods of the second aspect. The specific implementation methods of the steps in the fourth aspect, the meanings of the terms and the beneficial effects brought about can all be referred to the second aspect and will not be repeated here.

[0037] In a fifth aspect, the present application provides a device that may include a memory, a processor, and a bus system, wherein the memory is used to store programs and the processor is used to execute programs in the memory, including the following steps: the bus system is used to connect the memory and the processor so that the memory and the processor can communicate.

[0038] In the fifth aspect of the present application, the processor in the device can also be used to execute the steps performed by the first device in each possible implementation of the first aspect, or can also be used to execute the steps performed by the third device in each possible implementation of the second aspect. For details, please refer to the first aspect and the second aspect, which will not be repeated here.

[0039] In a sixth aspect, the present application provides a computer-readable storage medium, which stores a computer program. When the computer-readable storage medium is run on a computer, it enables the computer to execute the evaluation method of the machine learning model of the first aspect, or enables the computer to execute the data processing method of the second aspect.

[0040] In a seventh aspect, the present application provides a circuit system, which includes a processing circuit, and the processing circuit is configured to execute the evaluation method of the machine learning model of the first aspect, or to execute the data processing method of the second aspect.

[0041] In an eighth aspect, the present application provides a computer program product which, when running on a computer, enables the computer to execute the machine learning model evaluation method of the first aspect, or enables the computer to execute the data processing method of the second aspect.

[0042] In a ninth aspect, the present application provides a chip system, which includes a processor for supporting the implementation of the functions involved in the above aspects, for example, sending or processing the data and / or information involved in the above methods. In one possible design, the chip system also includes a memory, which is used to store program instructions and data necessary for a server or communication device. The chip system can be composed of a chip, or it can include a chip and other discrete devices. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Figure 1a A schematic diagram of the structure of the artificial intelligence main framework provided in the embodiment of the present application;

[0044] Figure 1bA flowchart of a method for evaluating a machine learning model provided in an embodiment of the present application;

[0045] Figure 2 Another flowchart of the evaluation method of the machine learning model provided in the embodiment of the present application;

[0046] Figure 3 A schematic diagram of obtaining evaluation samples based on driving data provided in an embodiment of the present application;

[0047] Figure 4 Another flowchart of the evaluation method of the machine learning model provided in the embodiment of the present application;

[0048] Figure 5 A schematic diagram of obtaining a first training sample set provided in an embodiment of the present application;

[0049] Figure 6 A schematic diagram of training a second machine learning model provided in an embodiment of the present application;

[0050] Figure 7 A schematic diagram of obtaining a first training sample set and training a second machine learning model using the first training sample set provided in an embodiment of the present application;

[0051] Figure 8 A schematic diagram of the structure of a device for evaluating a machine learning model provided in an embodiment of the present application;

[0052] Fig. 9 A schematic diagram of the structure of a data processing device provided in an embodiment of the present application;

[0053] Fig.10 A schematic diagram of the structure of a device provided in an embodiment of the present application;

[0054] Fig.11 A schematic diagram of the structure of a chip provided in an embodiment of the present application. DETAILED DESCRIPTION

[0055] The terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and need not be used to describe a specific order or sequential order. It should be understood that the terms used in this way can be interchangeable under appropriate circumstances, which is only to describe the distinction mode adopted by the objects of the same attributes when describing in the embodiments of the present application. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, so that the process, method, system, product or equipment comprising a series of units need not be limited to those units, but may include other units that are not clearly listed or inherent to these processes, methods, products or equipment.

[0056] The embodiments of the present application are described below in conjunction with the accompanying drawings. Those skilled in the art will appreciate that, with the development of technology and the emergence of new scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0057] First, the overall workflow of the artificial intelligence system is described. Figure 1a , Figure 1a The figure shows a structural diagram of the main framework of artificial intelligence. The following is an explanation of the above artificial intelligence theme framework from the two dimensions of "intelligent information chain" (horizontal axis) and "IT value chain" (vertical axis). Among them, the "intelligent information chain" reflects a series of processes from data acquisition to processing. For example, it can be a general process of intelligent information perception, intelligent information representation and formation, intelligent reasoning, intelligent decision-making, intelligent execution and output. In this process, the data has undergone a condensation process of "data-information-knowledge-wisdom". The "IT value chain" reflects the value that artificial intelligence brings to the information technology industry from the underlying infrastructure of human intelligence, information (providing and processing technology implementation) to the industrial ecology process of the system.

[0058] (1) Infrastructure

[0059] The infrastructure provides computing power support for the artificial intelligence system, enables communication with the outside world, and is supported by the basic platform. It communicates with the outside world through sensors; computing power is provided by smart chips, which can specifically use hardware acceleration chips such as central processing units (CPU), embedded neural network processing units (NPU), graphics processing units (GPU), application specific integrated circuits (ASIC) or field programmable gate arrays (FPGA); the basic platform includes distributed computing frameworks and networks and other related platform guarantees and support, which can include cloud storage and computing, interconnected networks, etc. For example, sensors communicate with the outside world to obtain data, and these data are provided to the smart chips in the distributed computing system provided by the basic platform for calculation.

[0060] (2) Data

[0061] The data on the upper layer of the infrastructure is used to represent the data sources in the field of artificial intelligence. The data involves graphics, images, voice, text, and IoT data of traditional devices, including business data of existing systems and perception data such as force, displacement, liquid level, temperature, and humidity.

[0062] (3) Data processing

[0063] Data processing usually includes data training, machine learning, deep learning, search, reasoning, decision-making and other methods.

[0064] Among them, machine learning and deep learning can symbolize and formalize data for intelligent information modeling, extraction, preprocessing, and training.

[0065] Reasoning refers to the process of simulating human intelligent reasoning in computers or intelligent systems, using formalized information to perform machine thinking and solve problems based on reasoning control strategies. Typical functions are search and matching.

[0066] Decision-making refers to the process of making decisions after intelligent information is reasoned, usually providing functions such as classification, sorting, and prediction.

[0067] (4) General ability

[0068] After the data has undergone the data processing mentioned above, some general capabilities can be further formed based on the results of the data processing, such as an algorithm or a general system, for example, translation, text analysis, computer vision processing, speech recognition, image recognition, etc.

[0069] (5) Smart products and industry applications

[0070] Smart products and industry applications refer to the products and applications of artificial intelligence systems in various fields. They are the encapsulation of the overall artificial intelligence solution, which productizes intelligent information decision-making and realizes practical application. Its application areas mainly include: smart terminals, smart manufacturing, smart transportation, smart homes, smart medical care, smart security, autonomous driving, smart cities, etc.

[0071] The method provided in this application can be applied to the field of autonomous driving of artificial intelligence technology, and is used to evaluate the performance of a first machine learning model in the field of autonomous driving. Exemplarily, the aforementioned first machine learning model is a machine learning model to be deployed in a vehicle; optionally, it can be applied to the intelligent driving decision-making system of the vehicle. Since the current method is to jointly evaluate multiple machine learning models deployed in the vehicle in a simulated environment, it is impossible to accurately obtain the evaluation status of a certain machine learning model.

[0072] Therefore, the present application provides a method for evaluating a machine learning model, which is used to evaluate a first machine learning model to be deployed in a vehicle. Figure 1b , Figure 1b A flow chart of an evaluation method for a machine learning model provided in an embodiment of the present application. In which, 101, a first device obtains a first evaluation sample, the first evaluation sample includes description data of a first traffic scene within a first duration, the first evaluation sample includes a plurality of first segment data, each first segment data includes description data of the first traffic scene within a first sub-duration, and the first duration includes a first sub-duration.

[0073] Exemplarily, the first device is a device used to evaluate the first machine learning model, which may be a terminal device or an edge device, or a cloud device such as a server or a server cluster, etc. The product form of the first device is not limited in this application.

[0074] For example, the above-mentioned description data can be specifically expressed as a video, an image, a scene description file, or other types of data, etc. The specific form of the description data can be determined according to the actual situation and is not limited here. For example, the scene description file can include the position, speed, acceleration, or other state information of each traffic participant in the first traffic scene, etc., which is not limited in the embodiments of the present application.

[0075] Exemplarily, the first traffic scenario may be: the vehicle goes straight at an unprotected intersection and other traffic participants turn left, the vehicle goes straight at the intersection and other traffic participants turn right, the vehicle goes straight at the intersection and other traffic participants cross diagonally, the vehicle turns left at an unprotected intersection and other traffic participants go straight, the vehicle turns left at the intersection, the vehicle turns left at the intersection and other traffic participants cross, the vehicle turns right at the intersection, the vehicle turns right at the intersection and other traffic participants cross, the vehicle enters the intersection and other traffic participants cross, the vehicle makes a U-turn, the vehicle crosses the intersection at a gap, the vehicle merges at a gap intersection, the vehicle encounters other traffic participants while crossing the intersection, the vehicle merges when entering the ramp, the vehicle crosses the intersection without interacting with other traffic participants, or other traffic scenarios, etc. The specific traffic scenarios corresponding to the multiple evaluation samples included in the evaluation sample set of the first machine learning model can be flexibly determined in combination with the task type performed by the first machine learning model and the specific application scenario, and are not limited in the embodiments of the present application.

[0076] The first duration may include one or more first sub-durations. For example, the first duration is 10 seconds and the first sub-duration is 2 seconds; for another example, the first duration is 20 seconds and the first sub-duration is 5 seconds, or the first duration and the first sub-duration may be other lengths, etc. The examples here are only for the convenience of understanding the relationship between the "first duration" and the "first sub-duration" and are not used to limit this solution.

[0077] 102. The first device processes the multiple first segment data through the first machine learning model to obtain multiple first prediction labels corresponding to the multiple first segment data.

[0078] 103. The first device determines a first parameter value of at least one evaluation indicator based on multiple first prediction labels; wherein, at least one evaluation indicator indicates the stability of the first machine learning model when generating multiple first prediction labels, and / or, at least one evaluation indicator indicates the accuracy of the multiple first prediction labels generated by the first machine learning model in the first traffic scene, and the accuracy of the multiple first prediction labels is obtained based on the multiple first prediction labels and the true values ​​corresponding to the multiple first fragment data.

[0079] Exemplarily, at least one evaluation indicator may include a first evaluation indicator and a second evaluation indicator, and the first evaluation indicator indicates the stability of the above-mentioned multiple first prediction labels generated by the first machine learning model. For example, the first evaluation indicator may represent the number of jumps of the multiple first prediction labels, and the smaller the number of jumps of the multiple first prediction labels, the better the stability of the multiple first prediction labels; for another example, the first evaluation indicator may represent the jump frequency of the multiple first prediction labels, and the jump frequency of the multiple first prediction labels may be the ratio between the number of jumps of the multiple first prediction labels and the total number of the multiple first prediction labels, and the lower the jump frequency of the multiple first prediction labels, the better the stability of the multiple first prediction labels.

[0080] The second evaluation indicator indicates the accuracy of multiple first prediction labels generated by the first machine learning model in the first traffic scene, and the accuracy of the multiple first prediction labels is obtained based on the multiple first prediction labels and the true values ​​corresponding to the multiple first fragment data. The "accuracy of multiple first prediction labels" can adopt the accuracy rate of multiple first prediction labels, and the accuracy rate of the first prediction label can represent the ratio between the number of first prediction labels consistent with the true value in the multiple first prediction labels and the total number of the multiple first prediction labels.

[0081] In the embodiments of the present application, the evaluation indicators used in the present application are constructed at the granularity of traffic scenes. Since the vehicle driving process is continuous, "constructing evaluation indicators at the granularity of traffic scenes" can more accurately reflect the performance of the machine learning model applied to the vehicle driving process; in addition, the solution provided by the present application does not require the establishment of a simulated traffic scene, thereby reducing the computer resources consumed by the evaluation process of the machine learning model.

[0082] In combination with the above description, the following describes the specific implementation process of the method provided in the embodiment of the present application. Figure 2 , Figure 2 Another flowchart of the evaluation method of the machine learning model provided in the embodiment of the present application is provided. The evaluation method of the machine learning model provided in the embodiment of the present application may include:

[0083] 201. Obtain a first evaluation sample, the first evaluation sample including descriptive data of a first traffic scene within a first duration, the first evaluation sample including a plurality of first segment data, each first segment data including descriptive data of the first traffic scene within a first sub-duration, and the first duration including a first sub-duration.

[0084] In the embodiment of the present application, step 201 is an optional step. The first device may obtain one or more first evaluation samples from the evaluation sample set of the first machine learning model, the evaluation sample set of the first machine learning model may include multiple evaluation samples, and each first evaluation sample may be any one of the multiple evaluation samples.

[0085] The first evaluation sample includes the description data of the first traffic scene within the first duration, and the first evaluation sample includes multiple first fragment data. The first duration may include one or more first sub-durations. For example, the first duration is 10 seconds, and the first sub-duration is 2 seconds; for another example, the first duration is 20 seconds, and the first sub-duration is 5 seconds, or the first duration and the first sub-duration may be other lengths, etc. The examples given here are only for the convenience of understanding the relationship between the "first duration" and the "first sub-duration", and are not used to limit this solution.

[0086] For example, the above-mentioned description data can be specifically expressed as a video, an image, a scene description file, or other types of data, etc. The specific form of the description data can be determined according to the actual situation and is not limited here. For example, the scene description file can include the position, speed, acceleration, or other state information of each traffic participant in the first traffic scene, etc., which is not limited in the embodiments of the present application.

[0087] For example, the first evaluation sample may include a video of the first traffic scene within 15 seconds, the first evaluation sample includes 5 first fragment data, each of the first fragment data includes a video of the first traffic scene within 3 seconds, and the 5 first fragment data can constitute the first evaluation sample. For another example, the first evaluation sample may include images of the first traffic scene at multiple time points within 16 seconds, the first evaluation sample includes 4 first fragment data, each of the first fragment data includes images of the first traffic scene at multiple time points within 4 seconds, and the 4 first fragment data can constitute the first evaluation sample. For another example, the first evaluation sample may include a scene description file of the first traffic scene at multiple time points within 20 seconds, the first evaluation sample includes 5 first fragment data, the 5 first fragment data all include a scene description file of a certain length, the 5 first fragment data include lengths of 5 seconds, 6 seconds, 3 seconds, 4 seconds and 2 seconds respectively, the 5 first fragment data can constitute the first evaluation sample, and so on. It should be noted that the examples here are only for the convenience of understanding the concepts of "first evaluation sample" and "first fragment data", and are not used to limit this solution.

[0088] Optionally, the task performed by the first machine learning model can be a classification task. For example, the task type performed by the first machine learning model can be any one or more of the following: determining whether to cut in, determining whether to give way, determining whether to limit the speed or other classification tasks, etc., which are not limited in the embodiments of the present application. A variety of task types that can be performed by the first machine learning model are provided, which expands the application scenarios of the present solution and improves the implementation flexibility of the present solution.

[0089] Optionally, the embodiment of the present application also discloses a method for obtaining an evaluation sample set of the first machine learning model. The evaluation sample set of the first machine learning model may include multiple evaluation samples, each of the multiple evaluation samples is obtained based on a driving data, that is, multiple evaluation samples in the first machine learning model can be obtained based on multiple driving data.

[0090] In one case, the above-mentioned "driving data" may be road test data, which includes description data of traffic scenes collected when the autonomous driving vehicle is driving on the road; alternatively, the road test data may include description data of traffic scenes collected when the autonomous driving vehicle is driving on a real road. The "autonomous driving vehicle" may be configured as a fully or partially autonomous driving mode. For example, when the autonomous driving vehicle is in a partially autonomous driving mode, the autonomous driving vehicle may determine the current state of the vehicle and its surrounding environment through human operation while controlling itself, determine the possible behavior of at least one traffic participant in the surrounding environment, and determine the confidence level corresponding to the possibility of other traffic participants performing possible behaviors, and control the autonomous driving vehicle based on the determined information; when the autonomous driving vehicle is in a fully autonomous driving mode, the autonomous driving vehicle may also be set to operate without human interaction. In the embodiment of the present application, since the first machine learning model is configured in the vehicle to realize the autonomous driving mode of the vehicle, the behavior of "using road test data to evaluate the performance of the first machine learning model" makes the "process of evaluating the first machine learning model" more similar to the "actual application process of the first machine learning model", so that the evaluation result obtained is more accurate.

[0091] In another case, the above-mentioned "driving data" may be the descriptive data of the traffic scene collected when a manually controlled vehicle is driving on the road. It should be noted that the meaning of "descriptive data of the traffic scene" can be understood in combination with the above-mentioned descriptions of "traffic scene", "first evaluation sample" and "first fragment data", which will not be elaborated here.

[0092] For the convenience of description, in this application, any evaluation sample in the aforementioned evaluation sample set is referred to as a "target evaluation sample". The target evaluation sample may include descriptive data of the target traffic scene within a certain time length. The target evaluation sample may include one or more target segment data. Each of the aforementioned one or more target segment data may include descriptive data of the target traffic scene within a certain sub-time length. The aforementioned one or more target segment data may constitute a target evaluation sample.

[0093] Exemplarily, the target evaluation sample is obtained based on the target driving data (i.e., one of multiple driving data), and the target driving data may include the descriptive data of the target traffic scene within the second time length and the original label corresponding to the descriptive data of the target traffic scene within the second time length; the target evaluation sample includes the descriptive data of the target traffic scene within the third time length and the true value corresponding to the descriptive data of the target traffic scene within the third time length, and the "true value corresponding to the descriptive data of the target traffic scene within the third time length" may also be referred to as the "expected label corresponding to the descriptive data of the target traffic scene within the third time length", and the "true value corresponding to the descriptive data of the target traffic scene within the third time length" may be understood as the true value corresponding to each target segment data among the multiple target segment data included in the target evaluation sample, or, it may also be referred to as the expected label corresponding to each target segment data among the multiple target segment data included in the target evaluation sample.

[0094] The meaning of the information indicated by the “original label corresponding to the description data of the traffic scene” is related to the type of task performed by the first machine learning model. For example, the task performed by the first machine learning model is to determine whether to cut in, then the “original label corresponding to the description data of the traffic scene” may indicate whether the vehicle cuts in or gives way in the traffic scene. For another example, the task performed by the first machine learning model is to determine whether to give way, then the “original label corresponding to the description data of the traffic scene” may indicate whether the vehicle gives way or does not give way in the traffic scene. For another example, the task performed by the first machine learning model is to determine whether to limit the speed, then the “original label corresponding to the description data of the traffic scene” may indicate whether the vehicle has a speed limit or no speed limit in the traffic scene, etc. The specific meaning of the information indicated by the aforementioned “original label” may be determined in combination with the actual application scenario of the first machine learning model, and is not limited in the embodiments of the present application.

[0095] Regarding the specific implementation method of obtaining target evaluation samples based on target driving data, it should be noted that the execution device of "obtaining target evaluation samples based on target driving data" and the first device can be the same device, that is, on the first device, based on multiple driving data, the evaluation sample set of the first machine learning model is determined; or, the execution device of "obtaining target evaluation samples based on target driving data" and the first device can be different devices, that is, on other devices, based on multiple driving data, the evaluation sample set of the first machine learning model is determined, and then the other devices send the evaluation sample set of the first machine learning model to the first device.

[0096] For example, in one case, the “original label corresponding to the description data of the target traffic scene in the second duration” can be directly determined as the “true value corresponding to the description data of the target traffic scene in the third duration”, and the second duration is equal to the third duration. Optionally, when the target driving data is a positive sample, the original label can be directly determined as the true value, and the second duration is equal to the third duration.

[0097] In another case, when the target driving data is a negative sample, the above true value in the target evaluation sample is obtained by modifying the original label in the target driving data. The meaning of "the true value corresponding to the description data of the traffic scene" is similar to the meaning of "the original label corresponding to the description data of the traffic scene", which will not be repeated here. In the embodiment of the present application, when the target driving sample is a negative sample, the action pointed to by the original label corresponding to the description data of the traffic scene is not suitable for the traffic scene, then the original label is modified to obtain the true value in the target evaluation sample, thereby ensuring that the actions performed by the true value in the evaluation sample are all actions adapted to the traffic scene, which is beneficial to improving the accuracy of the evaluation results.

[0098] For example, if the task performed by the first machine learning model is to determine whether to cut in, the "original label corresponding to the description data of the target traffic scene" in the target driving data indicates that the vehicle cuts in under the traffic scene. When the target driving data is a negative sample, the "true value corresponding to the description data of the target traffic scene" in the target evaluation sample may indicate that the vehicle gives way under the traffic scene.

[0099] For another example, if the task performed by the first machine learning model is to determine whether to change lanes, the "original label corresponding to the description data of the target traffic scene" in the target driving data indicates that the vehicle changes lanes in the traffic scene. When the target driving data is a negative sample, the "true value corresponding to the description data of the target traffic scene" in the target evaluation sample may indicate that the vehicle does not change lanes in the traffic scene.

[0100] For another example, if the task performed by the first machine learning model is to determine whether to limit the speed, the "original label corresponding to the description data of the target traffic scene" in the target driving data indicates that the vehicle is not speed-limited in the traffic scene. When the target driving data is a negative sample, the "true value corresponding to the description data of the target traffic scene" in the target evaluation sample may indicate that the vehicle is speed-limited in the traffic scene, and so on. The examples given here are only for the convenience of understanding this solution and are not used to limit this solution.

[0101] Exemplarily, there may be labeling information corresponding to each driving data in the plurality of driving data, and the labeling information corresponding to each driving data indicates whether the driving data is a positive sample or a negative sample. "The target driving data is a negative sample" may represent that the action indicated by the original label in the target driving data is not suitable for execution in the target traffic scene, and "the target driving data is a positive sample" may represent that the action indicated by the original label in the target driving data is suitable for execution in the target traffic scene.

[0102] The “labeling information corresponding to each driving data” may be manually labeled; or, the “labeling information corresponding to each driving data” may also be generated by a machine learning model that has performed a training operation.

[0103] Optionally, in the case where the target driving data is a negative sample, the second time period is greater than the third time period, that is, in the case where the target driving data is a negative sample, not only can the original label in the target driving data be modified to obtain the true value in the target evaluation sample, but the description data in the first time period included in the description data of the target traffic scene included in the target driving data in the second time period can also be deleted to obtain the description data of the target traffic scene included in the target evaluation sample in the third time period, and the description data in the first time period includes the description data in the preset time period from back to front in the second time period. Exemplarily, the second time period includes 20 seconds, the first time period can include the last 5 seconds of the aforementioned 20 seconds, and the third time period can include the first 15 seconds of the aforementioned 20 seconds. It should be understood that the example here is only for the convenience of understanding this solution and is not used to limit this solution.

[0104] For a more intuitive understanding of this solution, please refer to Figure 3 , Figure 3 A schematic diagram of obtaining evaluation samples based on driving data provided in an embodiment of the present application. Here, the task of the first machine learning model is to determine whether the vehicle is engaged in a robbing behavior. Figure 3 As shown, the target traffic scene includes the ego vehicle and the game car, and the game car is another traffic participant in the game with the ego vehicle. The target driving data includes the description data of the target traffic scene from t0 to t15, and the original label corresponding to the description data of the target traffic scene from t0 to t15: grab the line; from t0 to t15 represents 15 time steps. The description data of the target traffic scene from t0 to t15 includes the driving trajectory of the ego vehicle from t0 to t15 and the driving trajectory of the game car from t0 to t15, as shown in Figure 3As shown, there is an intersection between the driving trajectory of the ego car and the driving trajectory of the game car. If the target driving data is a negative sample, the target evaluation sample includes the descriptive data of the target traffic scene from t0 to t10, and the true value corresponding to the descriptive data of the target traffic scene from t0 to t10: give way; from t0 to t10 represents the first 10 time steps of the 15 time steps from t0 to t15. If the target driving data is a positive sample, the target evaluation sample may include the descriptive data of the target traffic scene from t0 to t15, and the true value corresponding to the descriptive data of the target traffic scene from t0 to t15: grab the line; that is, directly determine the target driving data as the target evaluation sample, it should be understood that, Figure 3 The examples are only for facilitating the understanding of this solution and are not intended to limit this solution.

[0105] In an embodiment of the present application, since when the target driving data is a negative sample, the data in the second half of the description data of the traffic scene included in the target driving data within the second time length may lead the first machine learning model to output an erroneous prediction label, resulting in the inability to accurately reflect the performance of the first machine learning model itself. The second time length is greater than the third time length, that is, the data in the second half of the description data of the traffic scene included in the target driving data within the second time length is cut off, which is conducive to the obtained evaluation results to more accurately reflect the actual performance of the first machine learning model.

[0106] Optionally, after determining the type of task performed by the first machine learning model, at least one dimension related to the type of task performed by the first machine learning model can also be determined, and at least one traffic scenario related to the type of task performed by the first machine learning model can be determined based on the aforementioned at least one dimension. Exemplarily, the aforementioned at least one dimension may include: the type of road, the behavior type of the vehicle, the behavior type of other traffic participants in a game relationship with the vehicle, the decision type of the vehicle, or other dimensions, etc., which are not exhaustive here. The type of road may include unprotected intersections, gaps, ramps, or other road types. The behavior type of the vehicle may include going straight, turning left, turning right, or other behavior types. The behavior types of other traffic participants in a game relationship with the vehicle may include cut in, merge in, cross, or other behavior types, etc. The decision type of the vehicle itself is related to the type of task performed by the first machine learning model. If the type of task performed by the first machine learning model is to determine whether to cut in, the decision type of the vehicle itself may include cutting in and giving way; if the type of task performed by the first machine learning model is to determine whether to give way, the decision type of the vehicle itself may include giving way and not giving way; if the type of task performed by the first machine learning model is to determine whether to limit the speed, the decision type of the vehicle itself may include limiting the speed or not limiting the speed, etc. It should be noted that the examples given here are only for the convenience of understanding this scheme and are not used to limit this scheme.

[0107] The evaluation sample set of the first machine learning model includes multiple evaluation samples corresponding to each traffic scene in the at least one traffic scene mentioned above, and the difference between the number of evaluation samples corresponding to each traffic scene can be within a preset range. Each evaluation sample in the evaluation sample set of the first machine learning model includes the description data of a traffic scene within a certain length of time and the true value corresponding to the aforementioned description data, then "which traffic scene does the target evaluation sample correspond to" can be understood as the target evaluation sample including the description data of the traffic scene.

[0108] Exemplarily, the "at least one traffic scenario" determined based on the above-mentioned "at least one dimension" may include any one or more of the following traffic scenarios: the vehicle goes straight in an unprotected intersection and other traffic participants turn left, the vehicle goes straight in an intersection and other traffic participants turn right, the vehicle goes straight in an intersection and other traffic participants cross diagonally, the vehicle turns left in an unprotected intersection and other traffic participants go straight, the vehicle turns left at the intersection, the vehicle turns left at the intersection and other traffic participants cross, the vehicle turns right at the intersection, the vehicle turns right at the intersection and other traffic participants cross, the vehicle enters the intersection and other traffic participants cross, the vehicle makes a U-turn (U-turn) turn), the vehicle crosses the intersection at a gap, the vehicle merges at the intersection at a gap, the vehicle encounters other traffic participants while crossing the intersection, the vehicle merges when entering the ramp auxiliary road, the vehicle crosses the intersection without interacting with other traffic participants or other traffic scenarios, etc. The traffic scenarios to which the multiple evaluation samples included in the evaluation sample set of the specific first machine learning model correspond can be flexibly determined in combination with the task type performed by the first machine learning model and the specific application scenario, and are not limited in the embodiments of the present application.

[0109] 202. Process the multiple first segment data through the first machine learning model to obtain multiple first prediction labels corresponding to the multiple first segment data.

[0110] In the embodiment of the present application, step 202 is an optional step. The first device can input the multiple first segment data included in each first evaluation sample into the first machine learning model, and process the multiple first segment data included in each first evaluation sample through the first machine learning model to obtain multiple first prediction labels output by the first machine learning model that correspond one to one to the multiple first segment data.

[0111] Among them, the meaning of the information indicated by the "first prediction label" is related to the type of task performed by the first machine learning model. The meaning of the "first prediction label" is similar to the meaning of the above-mentioned "original label". Please refer to the above description and will not be repeated here.

[0112] 203. Determine a first parameter value of at least one evaluation indicator based on the multiple first prediction labels, wherein the at least one evaluation indicator indicates the stability of the first machine learning model when generating the multiple first prediction labels, and / or the at least one evaluation indicator indicates the accuracy of the multiple first prediction labels generated by the first machine learning model in the first traffic scene, and the accuracy of the multiple first prediction labels is obtained based on the multiple first prediction labels and true values ​​corresponding to the multiple first fragment data.

[0113] In the embodiment of the present application, step 203 is an optional step. The first device can determine a first parameter value of at least one evaluation indicator corresponding to the first evaluation sample according to a plurality of first prediction labels corresponding one-to-one to a plurality of first segment data included in each first evaluation sample.

[0114] The at least one evaluation indicator may include a first evaluation indicator and a second evaluation indicator, and the first evaluation indicator indicates the stability of the at least one evaluation indicator generated by the first machine learning model.

[0115] For example, the first evaluation indicator may represent the number of jumps of multiple first prediction labels. The smaller the number of jumps of the multiple first prediction labels, the better the stability of the multiple first prediction labels. Exemplarily, the multiple first prediction labels include 5 first prediction labels that correspond one-to-one to the 5 first fragment data included in the first evaluation sample. The task performed by the first machine learning model is to determine whether to rush the line. The 5 first prediction labels are rush the line, rush the line, give way, rush the line, and rush the line, respectively. The number of jumps of the 5 first prediction labels is 2.

[0116] For another example, the first evaluation index may represent the jumping frequency of multiple first prediction labels, and the jumping frequency of multiple first prediction labels may be the ratio between the number of jumps of multiple first prediction labels and the total number of multiple first prediction labels. The lower the jumping frequency of multiple first prediction labels, the better the stability of the multiple first prediction labels. Exemplarily, the multiple first prediction labels include 5 first prediction labels corresponding one-to-one to the 5 first fragment data included in the first evaluation sample, and the task performed by the first machine learning model is to determine whether to rush in, and the 5 first prediction labels are rush in, rush in, give way, rush in, rush in, and the jumping frequency of the 5 first prediction labels may be two-fifths, etc.; it should be noted that the above examples of the "first evaluation index" are only for the convenience of understanding this scheme and are not used to limit this scheme.

[0117] The second evaluation index indicates the accuracy of multiple first prediction labels generated by the first machine learning model in the first traffic scene, and the accuracy of the multiple first prediction labels is obtained based on the multiple first prediction labels and the true values ​​corresponding to the multiple first fragment data. The multiple true values ​​corresponding one by one to the multiple first fragment data included in the first evaluation sample can be the same; for example, the true value corresponding to a first evaluation sample is rushing, and the true value corresponding to each of the multiple first fragment data included in the aforementioned first evaluation sample is rushing.

[0118] The “accuracy of multiple first prediction labels” may adopt the accuracy rate of multiple first prediction labels, and the accuracy rate of multiple first prediction labels may represent the ratio between the number of first prediction labels that are consistent with the true value and the total number of multiple first prediction labels.

[0119] Exemplarily, the multiple first prediction labels include 5 first prediction labels that correspond one-to-one to the 5 first fragment data included in the first evaluation sample. The task performed by the first machine learning model is to determine whether to rush the line. The 5 true values ​​that correspond one-to-one to the 5 first fragment data included in the first evaluation sample are all rush the line. The 5 first prediction labels are rush the line, rush the line, give way, rush the line, and rush the line, respectively. The accuracy rate of the 5 first prediction labels can be four-fifths, or eighty percent. It should be understood that the examples given here are only to prove the feasibility of this solution and are not used to limit this solution.

[0120] Optionally, at least one evaluation indicator may also indicate the safety of the first machine learning model generating multiple first prediction labels; optionally, the "safety of multiple first prediction labels" may be the distance between the vehicle and other traffic participants. Exemplarily, after obtaining multiple first prediction labels corresponding to multiple first segment data one by one, for the convenience of description, any one of the multiple first segment data will be referred to as target segment data in the following. The first device may deduce the trajectory of the vehicle based on the target segment data and the target prediction label corresponding to the target segment data, and obtain the first minimum distance between the vehicle and other traffic participants in the target segment data. The first device performs the aforementioned operation on each of the multiple first segment data, and can obtain multiple first minimum distances corresponding to the multiple first segment data one by one. The first device may use the smallest first minimum distance among the multiple first minimum distances as the final distance corresponding to the multiple first prediction labels, and then evaluate the safety of the first machine learning model generating multiple first prediction labels based on the aforementioned final distance, that is, evaluate whether the first machine learning model generates multiple first prediction labels safely.

[0121] Optionally, when the first parameter value of at least one of the above-mentioned evaluation indicators does not meet the second condition, it may represent that the first machine learning model fails the evaluation in the first traffic scenario; the judgment conditions for whether the first machine learning model fails the evaluation in a certain traffic scenario are clarified, thereby reducing the difficulty of implementing this solution.

[0122] Exemplarily, after determining the first parameter value of at least one evaluation indicator corresponding to each first evaluation sample, the first device can determine whether the first parameter value of at least one evaluation indicator corresponding to the aforementioned first evaluation sample satisfies the second condition, and the first device can perform the aforementioned operation on each evaluation sample in the evaluation sample set of the first machine learning model; if the parameter value of at least one evaluation indicator corresponding to each evaluation sample in the evaluation sample set of the first machine learning model satisfies the second condition, it can be determined that the first machine learning model has passed the evaluation; if the parameter value of at least one evaluation indicator corresponding to any evaluation sample in the evaluation sample set of the first machine learning model does not satisfy the second condition, it can be deemed that the first machine learning model has failed the evaluation.

[0123] Optionally, when the stability of the first machine learning model when generating multiple first prediction labels does not meet the first sub-condition, and / or when the accuracy of the multiple first prediction labels does not meet the second sub-condition, it is deemed that the parameter value of at least one evaluation indicator does not meet the second condition, and / or when the security of the multiple first prediction labels does not meet the third sub-condition, it is deemed that the parameter value of at least one evaluation indicator does not meet the second condition.

[0124] Exemplarily, when the number of jumps of the multiple first prediction labels is greater than or equal to a preset number, it can be considered that the stability of the multiple first prediction labels does not meet the first sub-condition; for example, the preset number can be 2, 3, 4 or other values. Alternatively, when the jump frequency of the multiple first prediction labels is greater than or equal to a preset value, it can be considered that the stability of the multiple first prediction labels does not meet the first sub-condition, etc.

[0125] When the accuracy of multiple first prediction labels is less than or equal to the preset accuracy, it can be considered that the accuracy of the multiple first prediction labels does not meet the second sub-condition; for example, the preset accuracy may be ninety percent, eighty percent or other values, etc. The examples here are only for the convenience of understanding this solution and are not used to limit this solution.

[0126] When the final distance corresponding to multiple first prediction labels is less than or equal to the distance threshold, it can be regarded that the safety of the multiple first prediction labels does not meet the third sub-condition; in one implementation, the aforementioned distance threshold may include a distance threshold, for example, the distance threshold may be 1 meter, 1.2 meters or other numerical values, etc.; in another implementation, the aforementioned distance threshold may include multiple distance values ​​corresponding to multiple speed ranges, that is, when the current speed of the vehicle belongs to the first speed range among the multiple speed ranges, the first distance value corresponding to the first speed range can be obtained; when the current speed of the vehicle belongs to the second speed range among the multiple speed ranges, the second distance value corresponding to the first speed range can be obtained. The design of the specific distance value can be determined in combination with the actual situation, and is not limited in the embodiments of the present application.

[0127] Optionally, after the first device uses each evaluation sample in the above-mentioned evaluation sample set to evaluate the first machine learning model, it can determine that the parameter value of at least one evaluation indicator corresponding to each second evaluation sample in at least one second evaluation sample does not meet the second condition, and then obtain the type of traffic scene corresponding to each second evaluation sample, that is, determine in which traffic scenes the first machine learning model has failed the evaluation.

[0128] It should be noted that the "second evaluation sample" and the "first evaluation sample" have similar meanings, and both are evaluation samples in the evaluation sample set of the first machine learning model. The difference is that the "first evaluation sample" is any evaluation sample used when evaluating the first machine learning model, and the parameter value of at least one evaluation indicator obtained after evaluating the first machine learning model through the "second evaluation sample" does not meet the second condition.

[0129] In an embodiment of the present application, a scheme for evaluating a single machine learning model is provided, which can obtain the evaluation results of a single machine learning model more accurately; and the evaluation indicators used in the present application are constructed at the granularity of traffic scenes. Since the vehicle driving process is continuous, "constructing evaluation indicators at the granularity of traffic scenes" can more accurately reflect the performance of the machine learning model applied to the vehicle driving process; in addition, the scheme provided by the present application does not require the establishment of a simulated traffic scene, thereby reducing the computer resources consumed by the evaluation process of the machine learning model.

[0130] Optionally, the first device may also perform a training operation on the aforementioned first machine learning model based on the aforementioned at least one second evaluation sample in a subsequent step. The detailed implementation method will be described in the subsequent steps.

[0131] For a more intuitive understanding of this solution, please refer to Figure 4 , Figure 4 Another flowchart of the method for evaluating the machine learning model provided in an embodiment of the present application. Figure 4 In the example, the evaluation samples of the first machine learning model are all road test data. Figure 4 As shown, the drive test data set includes multiple drive test data, and the multiple drive test data include negative drive test data and positive drive test data. After determining the task type performed by the first machine learning model, at least one traffic scenario corresponding to the aforementioned task type can be determined, and then multiple drive test data sets in the drive test data set are screened based on the aforementioned at least one traffic scenario to obtain multiple drive test data for constructing an evaluation sample set.

[0132] Each negative drive test data in the filtered multiple drive test data can be labeled with a true value, that is, the original label in each negative drive test data is modified to obtain the true value in the evaluation sample; Figure 4 As shown, each evaluation sample may include: identification information of the evaluation sample, the time period corresponding to the evaluation sample, the true value included in the evaluation sample, descriptive data of the traffic scene included in the evaluation sample, the type of the aforementioned traffic scene, and the type of task performed by the first machine learning model and other information.

[0133] Figure 4 At least one indicator used when evaluating the first machine learning model includes: the accuracy of multiple first prediction labels corresponding to the evaluation samples, and the number of jumps obtained based on the multiple first prediction labels corresponding to the evaluation samples.

[0134] After determining the two evaluation indicators and the evaluation sample set, the first machine learning model can be evaluated to generate parameter values ​​of the two evaluation indicators corresponding to each evaluation sample. It should be noted that: Figure 4 The detailed implementation of the steps involved can be found in the description of the above steps and will not be repeated here. Figure 4 The examples are only for facilitating the understanding of this solution and are not intended to limit this solution.

[0135] 204. Perform feature extraction on the second evaluation sample through the first machine learning model to obtain feature information of the second evaluation sample, wherein the second evaluation sample is an evaluation sample in which the parameter value of at least one evaluation indicator obtained when the first machine learning model is evaluated does not satisfy the second condition.

[0136] In the embodiment of the present application, step 204 is an optional step. After obtaining one or more second evaluation samples through steps 201 to 203, the second device can input each fragment data of at least one fragment data included in each second evaluation sample into the first machine learning model, and perform feature extraction on each fragment data included in each second evaluation sample through the first machine learning model to obtain feature information of each second evaluation sample; illustratively, the feature information of each second evaluation sample can include feature information of each fragment data of at least one fragment data included in each second evaluation sample.

[0137] It should be noted that the "second device" and the "first device" can be the same device or different devices, which can be flexibly determined based on actual conditions and is not limited in the embodiments of the present application.

[0138] 205. Based on the feature information of the second evaluation sample and the feature information of the training sample of the first machine learning model, determine the first training sample set from the training sample set, wherein the training sample set includes multiple training samples of the first machine learning model, the first training sample set includes the first training sample, the first training sample and the second evaluation sample both include description data of the traffic scene, and the similarity between the feature information of the first training sample and the feature information of the second evaluation sample meets the first condition.

[0139] In the embodiment of the present application, step 205 is an optional step. The second device can also obtain a training sample set of the first machine learning model, and the training sample set includes multiple training samples of the first machine learning model. Exemplarily, each training sample can include descriptive data of a certain traffic scene within a certain sub-duration; the meaning of "training sample" is similar to the meaning of "segment data", which can be understood by referring to the above description and will not be repeated here. The second device inputs each training sample in the training sample set of the first machine learning model into the first machine learning model, and extracts features of each of the aforementioned training samples through the first machine learning model to obtain feature information of each training sample.

[0140] The second device can determine a first training sample set from the training sample set based on the feature information of each second evaluation sample and the feature information of each training sample of the first machine learning model, wherein the first training sample set includes multiple first training samples, and the similarity between the feature information of each first training sample and the feature information of one of the at least one second evaluation sample satisfies the first condition. Exemplarily, the first condition can be that the similarity between the feature information of the training sample and the feature information of one of the at least one second evaluation sample is greater than or equal to a similarity threshold.

[0141] In order to further understand the present solution, the specific implementation process of determining one or more first training samples from the training sample set of the first machine learning model based on the feature information of any second evaluation sample is described here. Exemplarily, in one implementation method, after the second device obtains the feature information of any second evaluation sample (for the convenience of distinction, any second evaluation sample will be referred to as the third evaluation sample later), it also obtains the feature information of each fragment data included in the third evaluation sample. The second device can generate a first similarity between the feature information of the target training sample and the feature information of each fragment data in the third evaluation sample, and obtain at least one first similarity that corresponds one-to-one to at least one fragment data included in the third evaluation sample; wherein the target training sample is any training sample in the training sample set of the first machine learning model.

[0142] Exemplarily, the above-mentioned "first similarity" can be obtained by calculating the cosine similarity between the feature information of the target training sample and the feature information of each fragment data in the third evaluation sample, or the above-mentioned "cosine similarity" can also be replaced by Euclidean distance, L1 distance, L2 distance or other similarity calculation methods, etc. The examples given here are only for the convenience of understanding this scheme and are not used to limit this scheme.

[0143] The second device can determine whether the similarity between the feature information of the target training sample and the feature information of the third evaluation sample meets the first condition based on at least one first similarity that corresponds one-to-one to at least one fragment data included in the third evaluation sample; if the judgment result is yes, the target training sample is determined to be the first training sample; if the judgment result is no, it is determined that the target training sample is not the first training sample.

[0144] Exemplarily, in one case, the second device may average the at least one first similarity to obtain an average value, and determine whether the average value is greater than or equal to the first similarity threshold; if the determination result is yes, the target training sample is determined to be the first training sample; if the determination result is no, it is determined that the target training sample is not the first training sample.

[0145] In another case, the second device can obtain the median value of the above-mentioned at least one first similarity, and determine whether the median value is greater than or equal to the second similarity threshold. If the judgment result is yes, the target training sample is determined to be the first training sample; if the judgment result is no, it is determined that the target training sample is not the first training sample, and so on. The second device can also use other methods to implement "based on the above-mentioned at least one first similarity, determine whether to determine the target training sample as the first training sample". The example here is only to prove the feasibility of this solution and is not used to limit this solution.

[0146] The second device can perform the above operation on each training sample in the training sample set of the first machine learning model to determine whether the similarity between the feature information of each training sample and the feature information of the third evaluation sample meets the first condition, so as to filter out at least one first training sample from the training sample set of the first machine learning model based on the feature information of the third evaluation sample.

[0147] The second device can repeat the above operation on each second evaluation sample in at least one second evaluation sample to filter out at least one first training sample from the training sample set of the first machine learning model based on each second evaluation sample in at least one second evaluation sample, that is, to obtain all first training samples in the first training sample set.

[0148] In one implementation, after the second device obtains the feature information of the third evaluation sample, it also obtains the feature information of each segment data included in the third evaluation sample. The second device can cluster the feature information of at least one segment data included in the third evaluation sample to obtain the category center corresponding to the feature information of the third evaluation sample.

[0149] The second device can generate a second similarity between the feature information of the target training sample and the aforementioned category center, and the "second similarity between the feature information of the target training sample and the aforementioned category center" is regarded as the similarity between the feature information of the target training sample and the feature information of the third evaluation sample; illustratively, the aforementioned "second similarity" can be obtained by calculating the cosine similarity between the feature information of the target training sample and the category center, or the aforementioned "cosine similarity" can also be replaced by Euclidean distance, L1 distance, L2 distance or other similarity calculation methods, etc. The examples given here are only for the convenience of understanding this solution and are not used to limit this solution.

[0150] The second device may determine whether the second similarity between the feature information of the target training sample and the center of the category satisfies the first condition; if the determination result is yes, the target training sample is determined to be the first training sample; if the determination result is no, the target training sample is determined not to be the first training sample. Exemplarily, the second device may determine whether the second similarity between the feature information of the target training sample and the center of the category is greater than or equal to a third similarity threshold; if the determination result is yes, the first condition is determined to be satisfied; if the determination result is no, the first condition is determined not to be satisfied.

[0151] The second device can perform the above operation on each training sample in the training sample set of the first machine learning model to determine whether the similarity between the feature information of each training sample and the feature information of the third evaluation sample meets the first condition, so as to filter out at least one first training sample from the training sample set of the first machine learning model based on the feature information of the third evaluation sample.

[0152] The second device can repeat the above operation on each second evaluation sample in at least one second evaluation sample to filter out at least one first training sample from the training sample set of the first machine learning model based on each second evaluation sample in at least one second evaluation sample, that is, to obtain all first training samples in the first training sample set.

[0153] For a more intuitive understanding of this solution, please refer to Figure 5 , Figure 5 A schematic diagram of obtaining a first training sample set provided in an embodiment of the present application, such as Figure 5As shown, after obtaining one or more second evaluation samples, the feature information of each of the multiple fragment data included in each second evaluation sample can be obtained through the first machine learning model; after obtaining the training sample set of the first machine learning model, the feature information of each training sample in the training sample set can be obtained through the first machine learning model; then based on the feature information of each of the multiple fragment data included in each second evaluation sample, and the feature information of each training sample, similarity matching is performed in the dimension of feature information to achieve the acquisition of the first training sample set from the training sample set of the first machine learning model. The specific implementation method of the aforementioned steps can be found in the above description, which will not be repeated here. It should be understood that Figure 5 The examples are only for facilitating the understanding of this solution and are not intended to limit this solution.

[0154] If a manual method is used to classify training samples according to different traffic scenarios, it may not be consistent with the different traffic scenarios understood by the machine learning model. In an embodiment of the present application, a plurality of first training samples are obtained from a training sample set based on the feature information of each second evaluation sample and the similarity between the feature information of each training sample. That is, the plurality of first training samples obtained are samples that the first machine learning model considers to be similar to the second evaluation sample. Thus, from the perspective of the machine learning model, training samples that the first machine learning model is not good at processing can be found, so as to increase the probability that each first training sample is a training sample that the first machine learning model is not good at processing, thereby helping to improve the prediction accuracy of the second machine learning model when processing the first training samples, and helping to obtain a second machine learning model with better performance.

[0155] It should be noted that steps 204 and 205 are optional steps. If steps 204 and 205 are not performed, after acquiring at least one second evaluation sample, the second device can also determine which traffic scene description data each second evaluation sample includes, thereby being able to determine at least one traffic scene corresponding to at least one second prediction sample, that is, determine all traffic scenes involved in at least one second prediction sample; thereby all training samples in the training sample set of the first machine learning model that correspond to the aforementioned at least one traffic scene are determined as the first training sample set.

[0156] 206. Process the first training sample through the second machine learning model to obtain a second prediction label corresponding to the first training sample.

[0157] In the embodiment of the present application, step 206 is an optional step. After obtaining the first training sample set, the third device may obtain one or more first training samples from the first training sample set, input each first training sample into the second machine learning model, and process each first training sample through the second machine learning model to obtain a second prediction label corresponding to each first training sample.

[0158] Exemplarily, the type of task performed by the "second machine learning model" is the same as the type of task performed by the "first machine learning model", with the difference that the first machine learning model is a machine learning model that has performed training operations, while the second machine learning model has not yet performed training operations; the meaning of the "second prediction label" is similar to that of the "first prediction label", and both can refer to the description in the above steps, which will not be repeated here.

[0159] It should be noted that the "third device", "second device" and "first device" may be the same device; or, the "third device", "second device" and "first device" may all be different devices; or, any two of the "third device", "second device" and "first device" may be the same device, and the remaining devices may be independent devices, etc. The specific configuration may be flexibly determined based on actual conditions, and is not limited in the embodiments of the present application.

[0160] 207. Process the second training sample through the second machine learning model to obtain a second prediction label corresponding to the second training sample, and the second training sample is different from the first training sample.

[0161] In the embodiment of the present application, step 207 is an optional step. The third device may also obtain a second training sample set. For example, the second training sample included in the second training sample set and the first training sample included in the first training sample set may be different training samples in the training sample set of the first machine learning model.

[0162] The third device can obtain one or more second training samples from the second training sample set, input each second training sample into the second machine learning model, process each second training sample through the second machine learning model, and obtain a second prediction label corresponding to each second training sample.

[0163] It should be noted that since steps 201 to 205 are optional steps, if steps 201 to 205 are performed, the first training sample set can be obtained through steps 201 to 205, and then the second training sample set can be determined based on the first training sample set. If steps 201 to 205 are not performed, other methods can also be used to divide the training sample set of the first machine learning model into the first training sample set and the second training sample set. Exemplarily, in the process of deploying the first machine learning model on an autonomous driving vehicle to perform reasoning through the first machine learning model, it can be determined in which traffic scenarios the first machine learning model performs better, then the traffic scenarios corresponding to the training samples in the second training sample set can all be traffic scenarios where the first machine learning model performs well, and then all training samples other than the second training sample set in the training sample set of the first machine learning model are determined as the first training sample set; it should be noted that other methods can also be used to determine the first training sample set and the second training sample set, and the example here is only to prove the feasibility of this solution, and is not used to limit this solution.

[0164] 208. Process the second training sample through the first machine learning model to obtain a first prediction label corresponding to the second training sample.

[0165] In the embodiment of the present application, step 208 is an optional step. The third device may also input each second training sample into the first machine learning model, process each second training sample through the first machine learning model, and obtain a first prediction label corresponding to each second training sample.

[0166] 209. According to the first loss term and the second loss term, the second machine learning model is trained until the convergence condition is met to obtain a second machine learning model that has performed a training operation, wherein the first loss term indicates the similarity between the second predicted label corresponding to the first training sample and the true value corresponding to the first training sample, and the second loss term indicates the similarity between the second predicted label corresponding to the second training sample and the first predicted label corresponding to the second training sample.

[0167] In the embodiment of the present application, step 209 is an optional step. It should be noted that although steps 201 to 209 are all optional steps, when steps 201 to 203 are not performed, steps 206 to 209 may be mandatory steps; when steps 206 to 209 are not performed, steps 201 to 203 are mandatory steps. After obtaining the second predicted label corresponding to the first training sample, the second predicted label corresponding to the second training sample, and the first predicted label corresponding to the second training sample, the third device may generate the value of the first loss term according to the second predicted label corresponding to the first training sample and the true value corresponding to the first training sample, and the goal of training using the first loss term includes improving the similarity between the second predicted label corresponding to the first training sample and the true value corresponding to the first training sample; and generate the value of the second loss term according to the second predicted label corresponding to the second training sample and the first predicted label corresponding to the second training sample, and the goal of training using the second loss term includes improving the similarity between the second predicted label corresponding to the second training sample and the first predicted label corresponding to the second training sample. The third device may generate a function value of the loss function according to the value of the first loss term and the value of the second loss term; illustratively, the value of the first loss term and the value of the second loss term may be weightedly summed to obtain the function value of the loss function.

[0168] For example, the first loss term can adopt a cross entropy loss function, a 0-1 loss function, an L1 loss function, or other types of loss functions, and the second loss term can also adopt a cross entropy loss function, a 0-1 loss function, an L1 loss function, or other types of loss functions, etc. It should be noted that the examples given here are only to prove the feasibility of this solution and are not used to limit this solution.

[0169] The meaning of “the true value corresponding to the first training sample” is similar to the meaning of “the true value corresponding to the segment data”, which can be understood by referring to the description in the above steps and will not be elaborated here.

[0170] The third device can calculate the gradient of the function value of the loss function and use the back-propagation algorithm to update the parameters of the second machine learning model to implement training of the second machine learning model; the third device can repeatedly execute steps 206 to 209 to implement iterative training of the second machine learning model until the convergence conditions are met, thereby obtaining a second machine learning model that has performed the training operation.

[0171] Exemplarily, the above-mentioned convergence condition may be that a convergence condition of a loss function is satisfied, or that the second machine learning model is trained for a preset number of times, etc.

[0172] For a more intuitive understanding of this solution, please refer to Figure 6 and Figure 7 , see first Figure 6 , Figure 6 A schematic diagram of training a second machine learning model provided in an embodiment of the present application. Figure 6 As shown, after obtaining the first training sample set and the second training sample set, the first training sample and the true value corresponding to the first training sample can be obtained, the first training sample can be input into the second machine learning model, and the second prediction label corresponding to the first training sample can be generated by the second machine learning model, and then based on the second prediction label corresponding to the first training sample and the true value corresponding to the first training sample, the value of the first loss item is generated.

[0173] The second training sample is input into the first machine learning model and the second machine learning model respectively, a first prediction label corresponding to the second training sample is generated by the first machine learning model, a second prediction label corresponding to the second training sample is generated by the second machine learning model, and a value of the second loss item is generated according to the second prediction label corresponding to the second training sample and the first prediction label corresponding to the second training sample.

[0174] The value of the first loss term and the value of the second loss term are added to obtain the function value of the loss function, and the second machine learning model is trained based on the function value of the loss function. It should be understood that Figure 6 The examples are only for facilitating the understanding of this solution and are not intended to limit this solution.

[0175] See also Figure 7 , Figure 7 A schematic diagram of obtaining a first training sample set and training a second machine learning model using the first training sample set provided in an embodiment of the present application. Figure 7 As shown, similarity matching in the feature information dimension can be achieved based on each second evaluation sample and the training sample set of the first machine learning model to obtain the first training sample set. The specific implementation process of the aforementioned steps can be found in the description of steps 204 and 205 and will not be repeated here.

[0176] Then, the first training sample in the first training sample set is input into the second machine learning model to generate the value of the first loss term; the second training sample in the second training sample set is respectively input into the first machine learning model and the second machine learning model to generate the value of the second loss term; the value of the first loss term and the value of the second loss term are added to obtain the function value of the loss function, and the parameters of the second machine learning model are updated based on the back propagation algorithm to realize the training of the second machine learning model. The specific implementation process of the aforementioned steps can refer to the description in steps 206 to 209, which will not be repeated here. It should be understood that Figure 7 The examples are only for facilitating the understanding of this solution and are not intended to limit this solution.

[0177] In an embodiment of the present application, since the similarity between the first training sample and the second evaluation sample satisfies the first condition, and the parameter value of the at least one evaluation indicator obtained when the first machine learning model is evaluated using the second evaluation sample does not satisfy the second condition, that is, when the first machine learning model is evaluated by the second evaluation sample, the performance of the first machine learning model is not good, then the first loss term indicates the similarity between the second prediction label corresponding to the first training sample and the true value corresponding to the first training sample, which is beneficial to improving the performance of the second machine learning model in the traffic scene corresponding to the second evaluation sample; the second training sample is different from the first training sample, that is, the first machine learning model has better performance in the traffic scene corresponding to the second training sample, and the second loss term indicates the similarity between the second prediction label corresponding to the second training sample and the first prediction label corresponding to the second training sample, which is beneficial to enable the second machine learning model to learn the excellent performance of the first machine learning model in the traffic scene corresponding to the second training sample; therefore, the trained second machine learning model not only has better performance in the traffic scene corresponding to the second evaluation sample, but also has better performance in the traffic scene corresponding to the second training sample, that is, a second machine learning model that can perform well in various traffic scenes can be obtained.

[0178] Optionally, after executing step 209, step 201 can be re-entered to evaluate the second machine learning model that has performed the training operation through steps 201 to 203. If the second machine learning model fails the evaluation, the third machine learning model can be trained again based on the second machine learning model. The aforementioned steps can be repeated at least once until the trained machine learning model passes the evaluation, and the target machine learning model that has performed the training operation is obtained.

[0179] The target machine learning model that has performed the training operation can be applied to an autonomous driving vehicle. Optionally, the target machine learning model that has performed the training operation can be applied to an autonomous driving decision system of an autonomous driving vehicle.

[0180] exist Figure 1b to Figure 7 On the basis of the corresponding embodiments, in order to better implement the above solutions of the embodiments of the present application, the following also provides related devices for implementing the above solutions. Figure 8 , Figure 8A structural schematic diagram of an evaluation device for a machine learning model provided in an embodiment of the present application, the evaluation device 800 for a machine learning model includes: an acquisition module 801, used to acquire a first evaluation sample, the first evaluation sample includes descriptive data of a first traffic scene within a first time length, the first evaluation sample includes multiple first segment data, each first segment data includes descriptive data of the first traffic scene within a first sub-time length, and the first time length includes the first sub-time length; a processing module 802, used to process the multiple first segment data through a first machine learning model to obtain multiple first prediction labels corresponding one-to-one to the multiple first segment data; a determination module 803, used to determine a first parameter value of at least one evaluation indicator based on the multiple first prediction labels; wherein, at least one evaluation indicator indicates the stability of the first machine learning model when generating multiple first prediction labels, and / or, at least one evaluation indicator indicates the accuracy of the multiple first prediction labels generated by the first machine learning model in the first traffic scene, and the accuracy of the multiple first prediction labels is obtained based on the multiple first prediction labels and the true values ​​corresponding to the multiple first segment data.

[0181] Optionally, the first evaluation sample is obtained based on road test data, and the road test data includes descriptive data of traffic scenes collected when the autonomous driving vehicle is driving on the road.

[0182] Optionally, when the first parameter value of at least one evaluation indicator does not satisfy the second condition, it means that the first machine learning model fails the evaluation in the first traffic scenario.

[0183] Optionally, the processing module 802 is further used to process the first training sample through the second machine learning model to obtain a second prediction label corresponding to the first training sample, and the similarity between the first training sample and the second evaluation sample satisfies the first condition, wherein the first training sample and the second evaluation sample both include description data of the traffic scene, and the second evaluation sample is an evaluation sample in which the parameter value of at least one evaluation indicator obtained when the first machine learning model is evaluated does not satisfy the second condition; the processing module 802 is further used to process the second training sample through the second machine learning model to obtain a second prediction label corresponding to the second training sample, and the second The training sample is different from the first training sample; the processing module 802 is also used to process the second training sample through the first machine learning model to obtain a first prediction label corresponding to the second training sample; the evaluation device 800 of the machine learning model also includes: an updating module 804, which is used to update the second machine learning model according to the first loss term and the second loss term, wherein the first loss term indicates the similarity between the second prediction label corresponding to the first training sample and the true value corresponding to the first training sample, and the second loss term indicates the similarity between the second prediction label corresponding to the second training sample and the first prediction label corresponding to the second training sample.

[0184] Optionally, the evaluation device 800 of the machine learning model also includes: a feature extraction module 805, which is used to extract features of the second evaluation sample through the first machine learning model to obtain feature information of the second evaluation sample; a determination module 803, which is also used to determine a first training sample set from the training sample set based on the feature information of the second evaluation sample and the feature information of the training sample of the first machine learning model, wherein the training sample set includes multiple training samples of the first machine learning model, the first training sample set includes a first training sample, and the similarity between the feature information of the first training sample and the feature information of the second evaluation sample satisfies the first condition.

[0185] Optionally, the evaluation sample set of the first machine learning model includes multiple evaluation samples, the target evaluation sample is any one of the evaluation sample set of the first machine learning model, the target evaluation sample is obtained based on driving data, the driving data includes descriptive data of the traffic scene and original labels corresponding to the descriptive data of the traffic scene, the target evaluation sample includes descriptive data of the traffic scene and true values ​​corresponding to the descriptive data of the traffic scene; wherein, when the driving data is a negative sample, the true value in the target evaluation sample is obtained by modifying the original label in the driving data.

[0186] Optionally, when the driving data is a negative sample, the driving data includes descriptive data of the traffic scene within a second time length, and the target evaluation sample includes descriptive data of the traffic scene within a third time length, and the second time length is greater than the third time length.

[0187] Optionally, the task types performed by the first machine learning model include one or more of the following: determining whether to cut in, determining whether to give way, or determining whether to limit the speed.

[0188] It should be noted that the information interaction, execution process, etc. between the modules / units in the evaluation device 800 of the machine learning model are the same as those in the present application. Figure 1b to Figure 7 The corresponding method embodiments are based on the same concept. For specific contents, please refer to the description in the method embodiments shown above in this application, which will not be repeated here.

[0189] Please continue reading Fig. 9 , Fig. 9A structural schematic diagram of a data processing device provided in an embodiment of the present application, the data processing device 900 includes: a processing module 901, which is used to process the first training sample through a second machine learning model to obtain a second prediction label corresponding to the first training sample; the processing module 901 is also used to process the second training sample through the second machine learning model to obtain a second prediction label corresponding to the second training sample, wherein the second training sample and the first training sample both include description data of the traffic scene, and the second training sample and the first training sample correspond to different traffic scenes; the processing module 901 is also used to process the second training sample through the first machine learning model to obtain a first prediction label corresponding to the second training sample; an updating module 902 is used to update the second machine learning model according to a first loss term and a second loss term, wherein the first loss term indicates the similarity between the second prediction label corresponding to the first training sample and the true value corresponding to the first training sample, and the second loss term indicates the similarity between the second prediction label corresponding to the second training sample and the first prediction label corresponding to the second training sample.

[0190] Optionally, the first training sample is determined based on a second evaluation sample, and the second evaluation sample is an evaluation sample in which a parameter value of at least one evaluation indicator obtained when evaluating the first machine learning model does not satisfy a second condition.

[0191] Optionally, the data processing device 900 also includes: an acquisition module 903, used to acquire a first evaluation sample, the first evaluation sample includes descriptive data of the first traffic scene within a first time length, the first evaluation sample includes multiple first segment data, each first segment data includes descriptive data of the first traffic scene within a first sub-time length, and the first time length includes the first sub-time length; the processing module 901 is also used to process the multiple first segment data through a first machine learning model to obtain multiple first prediction labels corresponding one to one to the multiple first segment data; the data processing device 900 also includes: a determination module 904, used to determine a first parameter value of at least one evaluation indicator according to the multiple first prediction labels; wherein, when the first parameter value of at least one evaluation indicator satisfies the second condition, it represents that the first machine learning model passes the evaluation in the first traffic scene, at least one evaluation indicator indicates the stability of the first machine learning model when generating multiple first prediction labels, and / or, at least one evaluation indicator indicates the accuracy of the multiple first prediction labels generated by the first machine learning model in the first traffic scene, and the accuracy of the multiple first prediction labels is obtained based on the multiple first prediction labels and the true values ​​corresponding to the multiple first segment data.

[0192] It should be noted that the information interaction, execution process, etc. between the modules / units in the data processing device 900 are the same as those in the present application. Figure 1b to Figure 7The corresponding method embodiments are based on the same concept. For specific contents, please refer to the description in the method embodiments shown above in this application, which will not be repeated here.

[0193] The present application also provides a device, see Fig.10 , Fig.10 A schematic diagram of a structure of a device provided in an embodiment of the present application, wherein the device 1000 includes: a receiver 1001, a transmitter 1002, a processor 1003, and a memory 1004 (wherein the number of processors 1003 in the device 1000 may be one or more, Fig.10 In the example of FIG. 1003 , the processor 1003 may include an application processor 10031 and a communication processor 10032. In some embodiments of the present application, the receiver 1001, the transmitter 1002, the processor 1003 and the memory 1004 may be connected via a bus or other means.

[0194] The memory 1004 may include a read-only memory and a random access memory, and provides instructions and data to the processor 1003. A portion of the memory 1004 may also include a non-volatile random access memory (NVRAM). The memory 1004 stores processor and operation instructions, executable modules or data structures, or subsets thereof, or extended sets thereof, wherein the operation instructions may include various operation instructions for implementing various operations.

[0195] The processor 1003 controls the operation of the execution device. In a specific application, the various components of the execution device are coupled together through a bus system, wherein the bus system includes not only a data bus but also a power bus, a control bus, and a status signal bus, etc. However, for the sake of clarity, various buses are referred to as bus systems in the figure.

[0196] The method disclosed in the above embodiment of the present application can be applied to the processor 1003, or implemented by the processor 1003. The processor 1003 can be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by the hardware integrated logic circuit or software instructions in the processor 1003. The above processor 1003 can be a general processor, a digital signal processor (digital signal processing, DSP), a microprocessor or a microcontroller, and can further include an application specific integrated circuit (application specific integrated circuit, ASIC), a field programmable gate array (field-programmable gate array, FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The processor 1003 can implement or execute the various methods, steps and logic block diagrams disclosed in the embodiments of the present application. The general processor can be a microprocessor or the processor can also be any conventional processor, etc. The steps of the method disclosed in the embodiment of the present application can be directly embodied as a hardware decoding processor to execute, or the hardware and software modules in the decoding processor are combined to execute. The software module may be located in a storage medium mature in the art, such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory, or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory 1004, and the processor 1003 reads the information in the memory 1004 and completes the steps of the above method in combination with its hardware.

[0197] The receiver 1001 can be used to receive input digital or character information and generate signal input related to the relevant settings and function control of the execution device. The transmitter 1002 can be used to output digital or character information through the first interface; the transmitter 1002 can also be used to send instructions to the disk group through the first interface to modify the data in the disk group; the transmitter 1002 can also include a display device such as a display screen.

[0198] In the embodiment of the present application, the processor 1003 can be used to execute Figure 1b to Figure 7 The method executed by the first device, the second device and / or the third device in the corresponding embodiment. It should be noted that the specific manner in which the processor 1003 executes the above steps is the same as that in the present application. Figure 1b to Figure 7 The corresponding method embodiments are based on the same concept, and the technical effects they bring are the same as those in this application. Figure 1b to Figure 7 The corresponding method embodiments are the same. For specific contents, please refer to the description in the method embodiments shown above in this application, which will not be repeated here.

[0199] The present application also provides a computer-readable storage medium in which a program is stored. When the program is run on a computer, the computer executes the above-mentioned Figure 1b to Figure 7 The illustrated embodiments describe steps performed by the first device, the second device and / or the third device in the method.

[0200] The present application also provides a computer program product, which includes a program. When the program is run on a computer, the computer executes the above-mentioned Figure 1b to Figure 7 The illustrated embodiments describe steps performed by the first device, the second device and / or the third device in the method.

[0201] The present application also provides a circuit system, which includes a processing circuit, wherein the processing circuit is configured to perform the above Figure 1b to Figure 7 The illustrated embodiments describe steps performed by the first device, the second device and / or the third device in the method.

[0202] The evaluation device, data processing device or equipment of the machine learning model provided in the embodiment of the present application can be specifically a chip, and the chip includes: a processing unit and a communication unit, the processing unit can be, for example, a processor, and the communication unit can be, for example, an input / output interface, a pin or a circuit. The processing unit can execute the computer execution instructions stored in the storage unit to enable the chip to execute the above Figure 1b to Figure 7 Optionally, the storage unit is a storage unit in the chip, such as a register, a cache, etc. The storage unit may also be a storage unit located outside the chip in the wireless access device, such as a read-only memory (ROM) or other types of static storage devices that can store static information and instructions, a random access memory (RAM), etc.

[0203] For details, please refer to Fig.11 , Fig.11 A schematic diagram of the structure of a chip provided in an embodiment of the present application, the chip can be expressed as a neural network processor NPU 110, NPU 110 is mounted on the host CPU (Host CPU) as a coprocessor, and the host CPU assigns tasks. The core part of the NPU is the operation circuit 1103, which is controlled by the controller 1104 to extract matrix data from the memory and perform multiplication operations.

[0204] In some implementations, the operation circuit 1103 includes multiple processing units (Process Engine, PE) inside. In some implementations, the operation circuit 1103 is a two-dimensional systolic array. The operation circuit 1103 can also be a one-dimensional systolic array or other electronic circuits capable of performing mathematical operations such as multiplication and addition. In some implementations, the operation circuit 1103 is a general-purpose matrix processor.

[0205] For example, assume there is an input matrix A, a weight matrix B, and an output matrix C. The operation circuit takes the corresponding data of matrix B from the weight memory 1102 and caches it on each PE in the operation circuit. The operation circuit takes the matrix A data from the input memory 1101 and performs matrix operations with matrix B, and the partial results or final results of the matrix are stored in the accumulator 1108.

[0206] The unified memory 1106 is used to store input data and output data. The weight data is directly transferred to the weight memory 1102 through the direct memory access controller (DMAC) 1105. The input data is also transferred to the unified memory 1106 through the DMAC.

[0207] BIU stands for Bus Interface Unit, i.e., bus interface unit 1110 , which is used for interaction between AXI bus, DMAC and instruction fetch buffer (IFB) 1109 .

[0208] The bus interface unit 1110 (BIU for short) is used for the instruction fetch memory 1109 to obtain instructions from the external memory, and is also used for the storage unit access controller 1105 to obtain the original data of the input matrix A or the weight matrix B from the external memory.

[0209] DMAC is mainly used to transfer input data in the external memory DDR to the unified memory 1106 or to transfer weight data to the weight memory 1102 or to transfer input data to the input memory 1101.

[0210] The vector calculation unit 1107 includes multiple operation processing units, and further processes the output of the operation circuit when necessary, such as vector multiplication, vector addition, exponential operation, logarithmic operation, size comparison, etc. It is mainly used for non-convolutional / fully connected layer network calculations in neural networks, such as Batch Normalization, pixel-level summation, upsampling of feature planes, etc.

[0211] In some implementations, the vector calculation unit 1107 can store the processed output vector to the unified memory 1106. For example, the vector calculation unit 1107 can apply a linear function and / or a nonlinear function to the output of the operation circuit 1103, such as linear interpolation of the feature plane extracted by the convolution layer, and then, for example, a vector of accumulated values ​​to generate an activation value. In some implementations, the vector calculation unit 1107 generates a normalized value, a pixel-level summed value, or both. In some implementations, the processed output vector can be used as an activation input to the operation circuit 1103, for example, for use in a subsequent layer in a neural network.

[0212] An instruction fetch buffer 1109 connected to the controller 1104 is used to store instructions used by the controller 1104;

[0213] Unified memory 1106, input memory 1101, weight memory 1102 and instruction fetch memory 1109 are all on-chip memories. External memories are private to the NPU hardware architecture.

[0214] in, Figure 1b to Figure 7 The operations of each layer in each machine learning model shown can be performed by the operation circuit 1103 or the vector calculation unit 1107.

[0215] The processor mentioned in any of the above places may be a general-purpose central processing unit, a microprocessor, an ASIC, or one or more integrated circuits for controlling the execution of the program of the above-mentioned first aspect method.

[0216] It should also be noted that the device embodiments described above are merely schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed over multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. In addition, in the drawings of the device embodiments provided by the present application, the connection relationship between the modules indicates that there is a communication connection between them, which may be specifically implemented as one or more communication buses or signal lines.

[0217] Through the description of the above implementation mode, the technicians in the relevant field can clearly understand that the present application can be implemented by means of software plus necessary general hardware, and of course, it can also be implemented by special hardware including special integrated circuits, special CLUs, special memories, special components, etc. In general, all functions completed by computer programs can be easily implemented by corresponding hardware, and the specific hardware structure used to implement the same function can also be various, such as analog circuits, digital circuits or special circuits. However, for the present application, software program implementation is a better implementation mode in more cases. Based on such an understanding, the technical solution of the present application is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which is stored in a readable storage medium, such as a computer floppy disk, U disk, mobile hard disk, ROM, RAM, disk or optical disk, etc., including a number of instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment of the present application.

[0218] In the above embodiments, all or part of the embodiments may be implemented by software, hardware, firmware or any combination thereof. When implemented by software, all or part of the embodiments may be implemented in the form of a computer program product.

[0219] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium, for example, the computer instructions may be transmitted from a website site, a computer, a server, or a data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, server, or data center. The computer-readable storage medium may be any available medium that a computer can store or a data storage device such as a server or a data center that includes one or more available media integrations. The available medium may be a magnetic medium, (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state drive (SSD)), etc.

Claims

1. A method for evaluating a machine learning model, characterized in that: The method comprises: Obtain a first evaluation sample, the first evaluation sample comprising description data of a first traffic scene within a first duration, the first evaluation sample comprising a plurality of first segment data, each of the first segment data comprising description data of the first traffic scene within a first sub-duration, the first duration including the first sub-duration; Processing the plurality of first segment data by a first machine learning model to obtain a plurality of first prediction labels corresponding one-to-one to the plurality of first segment data; Determining a first parameter value of at least one evaluation indicator according to the multiple first prediction labels; Among them, the at least one evaluation indicator indicates the stability of the first machine learning model when generating the multiple first prediction labels, and / or, the at least one evaluation indicator indicates the accuracy of the multiple first prediction labels generated by the first machine learning model in the first traffic scene, and the accuracy of the multiple first prediction labels is based on the multiple first prediction labels and the true values ​​corresponding to the multiple first fragment data.

2. The method according to claim 1, characterized in that The first evaluation sample is obtained based on road test data, and the road test data includes descriptive data of traffic scenes collected when the autonomous driving vehicle is driving on the road.

3. The method according to claim 1 or 2, characterized in that: When the first parameter value of the at least one evaluation indicator does not satisfy the second condition, it means that the first machine learning model fails the evaluation in the first traffic scenario.

4. The method according to claim 1 or 2, characterized in that: After determining the first parameter value of at least one evaluation indicator according to the plurality of first prediction labels, the method further includes: The first training sample is processed by the second machine learning model to obtain a second prediction label corresponding to the first training sample, and the similarity between the first training sample and the second evaluation sample satisfies the first condition, wherein the first training sample and the second evaluation sample both include description data of the traffic scene, and the second evaluation sample is an evaluation sample in which the parameter value of the at least one evaluation indicator obtained when the first machine learning model is evaluated does not satisfy the second condition; Processing the second training sample by the second machine learning model to obtain a second prediction label corresponding to the second training sample, where the second training sample is different from the first training sample; Processing the second training sample by the first machine learning model to obtain a first prediction label corresponding to the second training sample; The second machine learning model is updated according to a first loss term and a second loss term, wherein the first loss term indicates the similarity between the second predicted label corresponding to the first training sample and the true value corresponding to the first training sample, and the second loss term indicates the similarity between the second predicted label corresponding to the second training sample and the first predicted label corresponding to the second training sample.

5. The method according to claim 4, characterized in that The method further comprises: Extracting features of the second evaluation sample by using the first machine learning model to obtain feature information of the second evaluation sample; Based on the feature information of the second evaluation sample and the feature information of the training sample of the first machine learning model, a first training sample set is determined from the training sample set, wherein the training sample set includes multiple training samples of the first machine learning model, the first training sample set includes the first training sample, and the similarity between the feature information of the first training sample and the feature information of the second evaluation sample satisfies the first condition.

6. The method according to claim 1 or 2, characterized in that: The evaluation sample set of the first machine learning model includes multiple evaluation samples, the target evaluation sample is any one of the evaluation sample set of the first machine learning model, the target evaluation sample is obtained based on driving data, the driving data includes description data of the traffic scene and original labels corresponding to the description data of the traffic scene, and the target evaluation sample includes description data of the traffic scene and true values ​​corresponding to the description data of the traffic scene; Wherein, in the case where the driving data is a negative sample, the true value in the target evaluation sample is obtained by modifying the original label in the driving data.

7. The method according to claim 6, characterized in that In the case where the driving data is a negative sample, the driving data includes descriptive data of the traffic scene within a second time length, and the target evaluation sample includes descriptive data of the traffic scene within a third time length, and the second time length is greater than the third time length.

8. The method according to claim 1 or 2, characterized in that: The task types performed by the first machine learning model include one or more of the following: determining whether to cut in, determining whether to give way, or determining whether to limit the speed.

9. A data processing method, characterized in that: The method comprises: Processing the first training sample by a second machine learning model to obtain a second prediction label corresponding to the first training sample; Processing the second training sample by the second machine learning model to obtain a second prediction label corresponding to the second training sample, wherein the second training sample and the first training sample both include description data of the traffic scene, and the second training sample and the first training sample correspond to different traffic scenes; Processing the second training sample by the first machine learning model to obtain a first prediction label corresponding to the second training sample; The second machine learning model is updated according to a first loss term and a second loss term, wherein the first loss term indicates the similarity between the second predicted label corresponding to the first training sample and the true value corresponding to the first training sample, and the second loss term indicates the similarity between the second predicted label corresponding to the second training sample and the first predicted label corresponding to the second training sample.

10. The method according to claim 9, characterized in that The similarity between the first training sample and the second evaluation sample satisfies the first condition, and the second evaluation sample is an evaluation sample in which the parameter value of at least one evaluation indicator obtained when evaluating the first machine learning model does not satisfy the second condition.

11. The method according to claim 10, characterized in that Before processing the first training sample by the second machine learning model, the method further includes: Obtain a first evaluation sample, the first evaluation sample comprising description data of a first traffic scene within a first duration, the first evaluation sample comprising a plurality of first segment data, each of the first segment data comprising description data of the first traffic scene within a first sub-duration, the first duration including the first sub-duration; Processing the plurality of first segment data by a first machine learning model to obtain a plurality of first prediction labels corresponding one-to-one to the plurality of first segment data; Determining a first parameter value of the at least one evaluation indicator according to the multiple first prediction labels; Among them, when the first parameter value of the at least one evaluation indicator satisfies the second condition, it represents that the first machine learning model passes the evaluation in the first traffic scenario, and the at least one evaluation indicator indicates the stability of the first machine learning model when generating the multiple first prediction labels, and / or the at least one evaluation indicator indicates the accuracy of the multiple first prediction labels generated by the first machine learning model in the first traffic scenario, and the accuracy of the multiple first prediction labels is obtained based on the multiple first prediction labels and the true values ​​corresponding to the multiple first fragment data.

12. A device for evaluating a machine learning model, characterized in that: The device comprises: An acquisition module, used to acquire a first evaluation sample, wherein the first evaluation sample includes description data of a first traffic scene within a first duration, the first evaluation sample includes a plurality of first segment data, each of the first segment data includes description data of the first traffic scene within a first sub-duration, and the first duration includes the first sub-duration; A processing module, configured to process the plurality of first segment data by using a first machine learning model to obtain a plurality of first prediction labels corresponding one-to-one to the plurality of first segment data; A determination module, configured to determine a first parameter value of at least one evaluation indicator according to the plurality of first prediction labels; Among them, the at least one evaluation indicator indicates the stability of the first machine learning model when generating the multiple first prediction labels, and / or, the at least one evaluation indicator indicates the accuracy of the multiple first prediction labels generated by the first machine learning model in the first traffic scene, and the accuracy of the multiple first prediction labels is based on the multiple first prediction labels and the true values ​​corresponding to the multiple first fragment data.

13. The device according to claim 12, characterized in that The first evaluation sample is obtained based on road test data, and the road test data includes descriptive data of traffic scenes collected when the autonomous driving vehicle is driving on the road.

14. The device according to claim 12 or 13, characterized in that When the first parameter value of the at least one evaluation indicator does not satisfy the second condition, it means that the first machine learning model fails the evaluation in the first traffic scenario.

15. The device according to claim 12 or 13, characterized in that The processing module is further used to process the first training sample through the second machine learning model to obtain a second prediction label corresponding to the first training sample, and the similarity between the first training sample and the second evaluation sample satisfies the first condition, wherein the first training sample and the second evaluation sample both include description data of the traffic scene, and the second evaluation sample is an evaluation sample in which the parameter value of the at least one evaluation indicator obtained when the first machine learning model is evaluated does not satisfy the second condition; The processing module is further configured to process a second training sample by using the second machine learning model to obtain a second prediction label corresponding to the second training sample, where the second training sample is different from the first training sample; The processing module is further configured to process the second training sample by using the first machine learning model to obtain a first prediction label corresponding to the second training sample; The device also includes: an updating module, used to update the second machine learning model according to a first loss term and a second loss term, wherein the first loss term indicates the similarity between the second predicted label corresponding to the first training sample and the true value corresponding to the first training sample, and the second loss term indicates the similarity between the second predicted label corresponding to the second training sample and the first predicted label corresponding to the second training sample.

16. The device according to claim 15, characterized in that The device also includes: A feature extraction module, used to extract features of the second evaluation sample by using the first machine learning model to obtain feature information of the second evaluation sample; The determination module is also used to determine a first training sample set from a training sample set based on the feature information of the second evaluation sample and the feature information of the training sample of the first machine learning model, wherein the training sample set includes multiple training samples of the first machine learning model, the first training sample set includes the first training sample, and the similarity between the feature information of the first training sample and the feature information of the second evaluation sample satisfies the first condition.

17. The device according to claim 12 or 13, characterized in that The evaluation sample set of the first machine learning model includes multiple evaluation samples, the target evaluation sample is any one of the evaluation sample set of the first machine learning model, the target evaluation sample is obtained based on driving data, the driving data includes description data of the traffic scene and original labels corresponding to the description data of the traffic scene, and the target evaluation sample includes description data of the traffic scene and true values ​​corresponding to the description data of the traffic scene; Wherein, in the case where the driving data is a negative sample, the true value in the target evaluation sample is obtained by modifying the original label in the driving data.

18. The device according to claim 17, characterized in that In the case where the driving data is a negative sample, the driving data includes descriptive data of the traffic scene within a second time length, and the target evaluation sample includes descriptive data of the traffic scene within a third time length, and the second time length is greater than the third time length.

19. The device according to claim 12 or 13, characterized in that The task types performed by the first machine learning model include one or more of the following: determining whether to cut in, determining whether to give way, or determining whether to limit the speed.

20. A data processing device, characterized in that: The device comprises: A processing module, configured to process the first training sample by using a second machine learning model to obtain a second prediction label corresponding to the first training sample; The processing module is further used to process the second training sample through the second machine learning model to obtain a second prediction label corresponding to the second training sample, wherein the second training sample and the first training sample both include description data of the traffic scene, and the second training sample and the first training sample correspond to different traffic scenes; The processing module is further configured to process the second training sample by using the first machine learning model to obtain a first prediction label corresponding to the second training sample; An updating module is used to update the second machine learning model according to a first loss term and a second loss term, wherein the first loss term indicates the similarity between the second predicted label corresponding to the first training sample and the true value corresponding to the first training sample, and the second loss term indicates the similarity between the second predicted label corresponding to the second training sample and the first predicted label corresponding to the second training sample.

21. The device according to claim 20, characterized in that The similarity between the first training sample and the second evaluation sample satisfies the first condition, and the second evaluation sample is an evaluation sample in which the parameter value of at least one evaluation indicator obtained when evaluating the first machine learning model does not satisfy the second condition.

22. The device according to claim 21, characterized in that The device also includes: An acquisition module, used to acquire a first evaluation sample, wherein the first evaluation sample includes description data of a first traffic scene within a first duration, the first evaluation sample includes a plurality of first segment data, each of the first segment data includes description data of the first traffic scene within a first sub-duration, and the first duration includes the first sub-duration; The processing module is further used to process the plurality of first segment data by using a first machine learning model to obtain a plurality of first prediction labels corresponding one-to-one to the plurality of first segment data; The device further includes: a determination module, configured to determine a first parameter value of the at least one evaluation indicator according to the plurality of first prediction labels; Among them, when the first parameter value of the at least one evaluation indicator satisfies the second condition, it represents that the first machine learning model passes the evaluation in the first traffic scenario, and the at least one evaluation indicator indicates the stability of the first machine learning model when generating the multiple first prediction labels, and / or the at least one evaluation indicator indicates the accuracy of the multiple first prediction labels generated by the first machine learning model in the first traffic scenario, and the accuracy of the multiple first prediction labels is obtained based on the multiple first prediction labels and the true values ​​corresponding to the multiple first fragment data.

23. A device, characterized in that The method comprises a processor, wherein the processor is coupled to a memory, wherein the memory stores program instructions, and when the program instructions stored in the memory are executed by the processor, the method according to any one of claims 1 to 11 is implemented.

24. A computer-readable storage medium comprising a program, which, when executed on a computer, causes the computer to execute the method according to any one of claims 1 to 11.

25. A circuit system, characterized in that: The circuit system comprises a processing circuit configured to perform the method of any one of claims 1 to 11.

Images