Perception model evaluation method and device, storage medium and electronic device

Abstract

The present application discloses a method and device for evaluating a perception model, a storage medium, and an electronic device. The method comprises: inputting road data collected by a vehicle in a plurality of continuous frame segments into a perception model to be evaluated, and obtaining a plurality of corresponding model prediction results, wherein the road data includes a scene label for indicating a driving scene of the vehicle in a continuous frame segment; matching the model prediction results and the true value data corresponding to the road data, and evaluating the perception model to be evaluated to obtain an evaluation result; determining the scene label carried by the road data corresponding to the evaluation result that meets the preset conditions as the target scene label, obtaining training data containing the target scene label, and training the perception model to be evaluated. The present application solves the technical problem that the evaluation method of the perception algorithm provided by the related art has low iterative optimization efficiency.

Description

Technical Field

[0001] The present application relates to the field of intelligent driving technology, and more specifically, to a perception model evaluation method and device, a storage medium, and an electronic device. Background Art

[0002] In intelligent driving systems, the perception system is the basis for subsequent decision-making and control, including detection, classification, positioning and tracking of surrounding objects. Evaluation of perception algorithms is a key step in evaluating algorithm performance and ensuring algorithm security and reliability, which is of great significance to the development and application of algorithms.

[0003] However, when evaluating perception algorithms, the related technologies usually evaluate the performance of the perception algorithm model. When the performance of the perception algorithm model does not meet the requirements in the face of complex road conditions, it may be necessary to obtain a large amount of training data to train the model to improve the performance of the model. In other words, the evaluation method of the perception algorithm provided by the related technologies has the problem of low iterative optimization efficiency.

[0004] To address the above-mentioned problems, no effective solution has been proposed yet. Summary of the invention

[0005] The embodiments of the present application provide a method and device for evaluating a perception model, a storage medium, and an electronic device, so as to at least solve the technical problem that the evaluation method of the perception algorithm provided by the related art has low iterative optimization efficiency.

[0006] According to one aspect of an embodiment of the present application, a method for evaluating a perception model is provided, comprising: inputting road data collected by a vehicle within a plurality of continuous frame segments into a perception model to be evaluated, and obtaining a plurality of corresponding model prediction results, wherein the road data includes a scene label for indicating a driving scene of the vehicle within a continuous frame segment; matching the model prediction results and true value data corresponding to the road data, and evaluating the perception model to be evaluated to obtain an evaluation result; determining the scene label carried by the road data corresponding to the evaluation result that meets a preset condition as a target scene label, obtaining training data including the target scene label, and training the perception model to be evaluated.

[0007] According to another aspect of an embodiment of the present application, a perception model evaluation device is also provided, including: a prediction unit, used to input road data collected by the vehicle in multiple continuous frame segments into the perception model to be evaluated, and obtain corresponding multiple model prediction results, wherein the road data includes a scene label for indicating the driving scene of the vehicle in a continuous frame segment; a model evaluation unit, used to match the model prediction results and true value data corresponding to the road data, and evaluate the perception model to be evaluated to obtain an evaluation result; a data acquisition unit, used to determine the scene label carried by the road data corresponding to the evaluation result that meets the preset conditions as the target scene label, obtain training data containing the target scene label and train the perception model to be evaluated.

[0008] Optionally, the above-mentioned model evaluation unit includes: a category evaluation module, which is used to obtain model prediction results and true value data corresponding to road data collected by multiple continuous frame segments to evaluate the perception model to be evaluated, and obtain category evaluation results, wherein the category evaluation results are used to indicate evaluation results corresponding to multiple categories of perception targets; a model evaluation module, which is used to determine the category of the perception target to be evaluated based on the category evaluation results, and use the road data collected by multiple continuous frame segments to evaluate the perception model to be evaluated on the dimension of the category of the perception target to be evaluated to obtain an evaluation result.

[0009] Optionally, the above-mentioned data acquisition unit includes: a single-segment evaluation module, which is used to evaluate the perception model to be evaluated with the model prediction results and true value data corresponding to each of the road data collected from multiple continuous frame segments, and obtain the evaluation results corresponding to each of the multiple continuous frame segments; a result comparison module, which is used to determine the evaluation results that meet preset conditions among the evaluation results corresponding to each of the multiple continuous frame segments as the evaluation results to be determined; and a label determination module, which is used to determine the repeated scene labels in the scene labels carried by the road data corresponding to the evaluation results to be determined as the target scene labels.

[0010] Optionally, the above-mentioned data acquisition unit also includes: a data selection module, used to select target road data collected in a single frame from road data containing target scene labels; a single frame evaluation module, used to determine the model prediction results and true value data corresponding to the target road data, and match the model prediction results and true value data corresponding to the target road data to obtain a single frame evaluation result.

[0011] Optionally, the above-mentioned model evaluation unit also includes: a distance determination module, which is used to determine the distance between at least one reference obstacle in the road data and the vehicle according to the true value data corresponding to the road data, wherein the reference obstacle is used to indicate the static obstacles and dynamic obstacles around the vehicle; a distance interval division module, which is used to divide at least one reference obstacle into at least one reference obstacle set according to the distance between at least one reference obstacle and the vehicle and a preset distance threshold; an interval evaluation module, which is used to evaluate the perception model to be evaluated in turn according to the road data corresponding to at least one reference obstacle set to obtain an evaluation result.

[0012] Optionally, the above-mentioned model evaluation unit also includes: a matching module, used to match the model prediction result corresponding to the reference obstacle with the true value data corresponding to the reference obstacle; a first error calculation module, used to calculate the difference between the model prediction result and the distance in the true value data to obtain an absolute distance error; a second error calculation module, used to calculate the ratio between the absolute distance error and the distance in the true value data to obtain a relative distance error; a third error calculation module, used to calculate the difference between the speed in the model prediction result and the true value data to obtain an absolute speed error.

[0013] Optionally, the above-mentioned data acquisition unit also includes: a data determination module, used to determine the target category and the target distance threshold interval; a data acquisition module, used to obtain target road data containing a target scene label and a target category within the target distance threshold interval, and determine the target road data as training data; a model training module, used to use the training data to train the perception model to be evaluated.

[0014] Optionally, the above-mentioned prediction unit includes: a data screening module, which is used to screen the reference road data to determine the road data that meets the preset data conditions, wherein the reference road data is the road data continuously collected by the vehicle; a data cutting module, which is used to cut the road data that meets the preset data conditions in the reference road data to obtain the road data in multiple continuous frame segments that are consistent in time; a data labeling module, which is used to perform scene recognition on the road data to obtain the corresponding scene label, and add the scene label to the road data to obtain updated road data.

[0015] Optionally, the above-mentioned model evaluation unit also includes: a truth value acquisition module, which is used to input the road data collected by the vehicle in multiple continuous frame segments into the truth value model to obtain two-dimensional truth value data and three-dimensional truth value data corresponding to the road data; a truth value association module, which is used to associate the two-dimensional truth value data with the three-dimensional truth value data to obtain the truth value data corresponding to the road data.

[0016] According to another aspect of the embodiments of the present application, a computer-readable storage medium is provided, in which a computer program is stored, wherein the computer program is configured to execute the above-mentioned perceptual model evaluation method when running.

[0017] According to another aspect of the embodiments of the present application, a computer program product is provided, the computer program product comprising a computer program / instruction, the computer instruction being stored in a computer-readable storage medium. A processor of a computer device reads the computer program / instruction from the computer-readable storage medium, and the processor executes the computer program / instruction, so that the computer device executes the above-mentioned perceptual model evaluation method.

[0018] According to another aspect of an embodiment of the present application, there is also provided an electronic device, including a memory and a processor, wherein the memory stores a computer program, and the processor is configured to execute the above-mentioned perceptual model evaluation method through the computer program.

[0019] In an embodiment of the present application, the road data collected by the vehicle in multiple continuous frame segments are input into the perception model to be evaluated to obtain multiple corresponding model prediction results, wherein the road data includes a scene label for indicating the driving scene of the vehicle in a continuous frame segment; the model prediction results and the true value data corresponding to the road data are matched, and the perception model to be evaluated is evaluated to obtain the evaluation results; the scene label carried by the road data corresponding to the evaluation result that meets the preset conditions is determined as the target scene label, and the training data containing the target scene label is obtained and the perception model to be evaluated is trained. In the process of evaluating the perception algorithm, the sensor data collected by the vehicle on the road can be labeled to obtain road data containing scene labels, and the prediction results of the perception model and the corresponding true value data are obtained using the road data to obtain the evaluation results of the perception model. The road data is divided into scenes based on the scene labels to conduct multi-scene evaluations, which more clearly shows the shortcomings of the perception model in which driving scenarios. The scene label carried by the road data corresponding to the evaluation results that meet the preset conditions is determined as the target scene label, and the perception model is updated and trained using the training data corresponding to the target scene label to improve the performance of the perception model under the target scene label. Therefore, in the process of training the perception model, a large amount of training data is not required, but the road data in the scene with weak performance is obtained in a targeted manner. The above method solves the technical problem that the evaluation method of the perception algorithm provided by the related technology has low iterative optimization efficiency. The optimization efficiency of the perception model is improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0021] Figure 1 is a schematic diagram of a hardware environment of an optional perception model evaluation method according to an embodiment of the present application;

[0022] Figure 2 is a flow chart of an optional perceptual model evaluation method according to an embodiment of the present application;

[0023] Figure 3 is a schematic diagram of an optional perceptual model evaluation method according to an embodiment of the present application;

[0024] Figure 4 is a schematic diagram of another optional perceptual model evaluation method according to an embodiment of the present application;

[0025] Figure 5 is a schematic diagram of another optional perceptual model evaluation method according to an embodiment of the present application;

[0026] Figure 6 is a schematic diagram of another optional perceptual model evaluation method according to an embodiment of the present application;

[0027] Figure 7 is a schematic diagram of another optional perceptual model evaluation method according to an embodiment of the present application;

[0028] Figure 8 is a schematic diagram of an optional perception model evaluation device according to an embodiment of the present application;

[0029] Fig. 9 is a schematic diagram of an optional electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0030] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present application.

[0031] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0032] According to one aspect of the embodiments of the present application, a method for evaluating a perception model is provided. As an optional implementation, the method for evaluating a perception model can be applied to, but is not limited to, Figure 1 The perception model evaluation system in the hardware environment shown. Optionally, the perception model evaluation method described above can be applied to a vehicle terminal. Figure 1 A side view of a vehicle terminal 101 is shown, which is disposed on a travel surface 113 and is capable of traversing the travel surface 113. The vehicle terminal 101 includes an onboard navigation system 103, a computer-readable storage device or medium (memory) 102 including a digitized road map 104, a spatial monitoring system 117, a vehicle controller 109, a GPS (global positioning system) sensor 110, an HMI (human / machine interface) device 111, and also includes an autonomous controller 112 and a telematics controller 114. The vehicle terminal 101 may include, but is not limited to, commercial vehicles, industrial vehicles, agricultural vehicles, passenger vehicles, all-terrain vehicles, personal mobile devices, robots, and similar forms of mobile platforms to achieve the purpose of the present application.

[0033] In one embodiment, the spatial monitoring system 117 includes: one or more spatial sensors and systems arranged to monitor the visible area 105 in front of the vehicle terminal 101; and a spatial monitoring controller 118. The spatial sensors used to monitor the visible area 105 include, for example, a lidar sensor 106, a radar sensor 107, a camera 108, etc. The placement of the spatial sensors allows the spatial monitoring controller 118 to monitor traffic flow, including approaching vehicles, intersections, lane markings, and other objects around the vehicle terminal 101. The spatial sensors of the spatial monitoring system 117 may include object positioning sensing devices. The lidar sensor 106 uses pulsed and reflected laser beams to measure the range or distance to an object. The radar sensor 107 uses radio waves to determine the range, angle and / or speed of an object. The camera 108 includes an image sensor, a lens, and a camera controller.

[0034] The camera 108 is advantageously mounted and positioned on the vehicle terminal 101 in a position that allows capturing images of the visible area 105, wherein at least part of the visible area 105 includes a portion of the travel surface 113 in front of the vehicle terminal 101 and including the trajectory of the vehicle terminal 101. The visible area 105 may also include the surrounding environment. Other cameras (not shown) may also be used, for example, including a second camera disposed on the rear portion or side portion of the vehicle terminal 101 to monitor the rear of the vehicle terminal 101 and one of the right or left sides of the vehicle terminal 101. The camera 108 may collect road information to generate a road image sequence during the driving of the vehicle terminal 101.

[0035] The autonomous controller 112 is configured to implement autonomous driving or advanced driver assistance system (ADAS) vehicle functionality. Such functionality may include a vehicle onboard control system capable of providing a certain level of driving automation. Driving automation may include a series of dynamic driving and vehicle operations. Driving automation may include simultaneous automatic control of vehicle driving functions (including steering, acceleration and braking), wherein the driver abandons control of the vehicle for a period of time during the journey. Driving automation may include simultaneous automatic control of vehicle driving functions (including steering, acceleration and braking), wherein the driver abandons control of the vehicle terminal 101 for the entire journey. Driving automation includes hardware and a controller configured to monitor the spatial environment in various driving modes to perform various driving tasks during dynamic vehicle operations. By way of non-limiting example, autonomous vehicle functions include adaptive cruise control (ACC) operations, lane guidance and lane keeping operations, lane change operations, steering assistance operations, object avoidance operations, parking assistance operations, vehicle braking operations, vehicle speed and acceleration operations, vehicle lateral motion operations, for example, as part of lane guidance, lane keeping and lane change operations, etc.

[0036] The autonomous controller can be equipped with an operating system and an automatic driving system. The operating system can be responsible for the management of the hardware resources (including sensors, system buses, networks, etc.) of the vehicle terminal 101 and the scheduling of computing resources. The automatic driving system can implement various algorithms required for automatic driving, including positioning, environmental perception, path planning and control, etc., and can make decisions in situations such as driving on curves, driving in a straight line, driving on complex roads, and changing lanes. The autonomous controller can execute step S1 when the vehicle is driving, input the road data collected by the vehicle in multiple continuous frame segments into the perception model to be evaluated, and obtain corresponding multiple model prediction results, wherein the road data includes a scene label for indicating the driving scene of the vehicle in a continuous frame segment.

[0037] The vehicle terminal 101 may include a telematics controller 114, which includes a wireless telematics communication system capable of performing off-vehicle communications (including communications with a communication network 115 having wireless and wired communication capabilities). Optionally or additionally, the telematics controller 114 directly performs off-vehicle communications by communicating with an off-board server 116 via the communication network 115. The server side may execute step S2, matching the model prediction results and true value data corresponding to the road data, and evaluating the perception model to be evaluated to obtain the evaluation results; step S3, determining the scene label carried by the road data corresponding to the evaluation result that meets the preset conditions as the target scene label, obtaining training data containing the target scene label, and training the perception model to be evaluated.

[0038] In an optional embodiment, the vehicle terminal 101 can collect road images in real time using sensors such as cameras 108 through the space monitoring system 117. The road images can contain surrounding road condition data, such as movement information of surrounding vehicles, pedestrians and other traffic participants, and location information of static obstacles such as roadblocks and road fences. The collected road image sequence can be stored in the memory 102. Since the storage space of the memory 102 is limited, based on storage efficiency considerations, images with higher priority can be stored in the memory 102, and a large amount of low-priority data can be stored in the non-record server 116 via the communication network 115 using the telematics controller 114.

[0039] In an optional embodiment, the perception model evaluation method of the present application can be applied to the vehicle side, where the perception model can be tested and evaluated on an actual vehicle, and real-time feedback can be provided in this way. The performance of the perception model can be verified directly in the actual operating environment of the vehicle. The road data of multiple continuous frame fragments collected by the vehicle can be input into the on-board perception model to be evaluated to obtain the model prediction results, and the model prediction results and the true value data are matched to evaluate the perception model to obtain the evaluation results. The performance of the perception model in the current vehicle driving scenario can be determined based on the evaluation results. When the evaluation results of the perception model do not meet the preset conditions, it is determined that the perception effect of the perception model in the current vehicle driving scenario is not good, and data with labels for the current vehicle driving scenario can be obtained to train and update the perception model. In this way, the perception model is updated in a targeted manner to improve the efficiency of updating and iterating the perception model.

[0040] In an optional embodiment, the perception model evaluation method of the present application can be applied to the server side, and the perception model can be tested and evaluated on the server side or in the cloud, and the perception model of multiple scenarios can be evaluated. Road data with scene labels can be obtained, and the road data with scene labels can be input into the perception model. The prediction result of the perception model can be obtained and matched with the true value to evaluate the perception model. The scene label corresponding to the evaluation result that meets the preset conditions is determined as the target scene label, and the road data of the scene corresponding to the target scene label is obtained to update the perception model. In multiple scenarios, scenes with weak performance of the perception model can be quickly identified, and training can be performed to improve the training efficiency of the perception model.

[0041] In an embodiment of the present application, the road data collected by the vehicle in multiple continuous frame segments are input into the perception model to be evaluated to obtain multiple corresponding model prediction results, wherein the road data includes a scene label for indicating the driving scene of the vehicle in a continuous frame segment; the model prediction results and the true value data corresponding to the road data are matched, and the perception model to be evaluated is evaluated to obtain the evaluation results; the scene label carried by the road data corresponding to the evaluation result that meets the preset conditions is determined as the target scene label, and the training data containing the target scene label is obtained and the perception model to be evaluated is trained. In the process of evaluating the perception algorithm, the sensor data collected by the vehicle on the road can be labeled to obtain road data containing scene labels, and the prediction results of the perception model and the corresponding true value data are obtained using the road data to obtain the evaluation results of the perception model. The road data is divided into scenes based on the scene labels to conduct multi-scene evaluations, which more clearly shows the shortcomings of the perception model in which driving scenarios. The scene label carried by the road data corresponding to the evaluation results that meet the preset conditions is determined as the target scene label, and the perception model is updated and trained using the training data corresponding to the target scene label to improve the performance of the perception model under the target scene label. Therefore, in the process of training the perception model, a large amount of training data is not required, but the road data in the scene with weak performance is obtained in a targeted manner. The above method solves the technical problem that the evaluation method of the perception algorithm provided by the related technology has low iterative optimization efficiency. The optimization efficiency of the perception model is improved.

[0042] In an optional embodiment, if Figure 2 As shown, the evaluation method of the above perception model includes the following steps:

[0043] S202, inputting road data collected by the ego vehicle in a plurality of continuous frame segments into the perception model to be evaluated, and obtaining a plurality of corresponding model prediction results, wherein the road data includes a scene label for indicating a driving scene of the ego vehicle in a continuous frame segment;

[0044] It should be noted that, during the driving process of the vehicle, the camera sensor can be used to collect road information and collect the road information as a road image. As a visual method, the road image can include a large amount of traffic environment information, such as traffic signal signs, road lane line markings, and obstacles ahead. Multiple frames of road images can be determined as road data.

[0045] In an optional embodiment, a monocular camera may be used to collect images of the road on which the vehicle is traveling. When the vehicle is traveling, the scene within the collection range may be photographed at a certain collection frequency, and a road image may be generated based on the camera's internal and external parameters. The road data may be a plurality of road images collected by the vehicle in a series of continuous time frames, or may be road images collected by the vehicle under specific conditions. For example, a sequence of road images collected by the vehicle under specific weather conditions may be obtained, and the road images collected in multiple time frames when the vehicle is traveling under specific weather conditions may be used as road data.

[0046] It should be noted that the road data can be data acquired in real time during vehicle driving, or can be road data collected during vehicle driving and stored in a server. The road data can be two-dimensional data or three-dimensional data, and its form is not limited to images. Point cloud data can also be used as road data.

[0047] It should be noted that the perception model needs to work reliably in various traffic scenarios. By evaluating the perception model, the effect of the perception model can be quantitatively measured to intuitively determine the effectiveness of the perception model in different scenarios. The perception model can perform different tasks, such as 2D or 3D object detection, semantic segmentation, and lane line detection. Different perception models can be used as perception models to be evaluated using the above method.

[0048] In an optional implementation, inputting road data into the perception model to be evaluated can be a re-injection of the perception model to be evaluated, and the sensor data (including images, radar signals, etc.) collected historically by the vehicle can be re-input into the perception model to be evaluated to simulate the real-world driving scene. The model prediction results output by the perception model to be evaluated can be determined according to the specific tasks performed by the perception model, such as environmental perception results (obstacle detection, traffic sign recognition, lane line detection, etc.), vehicle status information (vehicle speed, position, acceleration, etc.) or driving behavior (steering, braking, acceleration, etc.).

[0049] It should be noted that scene labels can be weather labels (sunny, rainy, snowy, etc.), light labels (night, day, dusk, etc.), road scene labels (city, tunnel, elevated, etc.), etc. How to obtain road data with scene labels will be described in detail below and will not be repeated here.

[0050] S204, matching the model prediction result and the true value data corresponding to the road data, and evaluating the perception model to be evaluated to obtain an evaluation result;

[0051] It should be noted that in the process of evaluating the perception model, true value data is more important. True value data can be data obtained through high-precision sensors and post-processing algorithms that can accurately reflect the environment around the vehicle. True value data can be data obtained by combining two-dimensional data and three-dimensional data, which can carry two-dimensional information and three-dimensional information, such as combined data of image data and point cloud data.

[0052] In an optional implementation, before evaluating the perception model to be evaluated, it is necessary to match the true value data and the model prediction results. The specific matching method will be described in detail below and will not be repeated here.

[0053] In an optional implementation, the evaluation result may be an output index such as recall rate, precision rate, accuracy rate, etc., or may be an error in the distance or speed between the true value data and the model prediction result.

[0054] S206, determining the scene label carried by the road data corresponding to the evaluation result that meets the preset conditions as the target scene label, obtaining training data containing the target scene label and training the perception model to be evaluated.

[0055] It should be noted that after determining the target scene label, the target scene label can be sent to the technician, or sent to the parameter update model. The technician can find the road data marked with the target scene label, and use the road data containing the target scene label as training data to iteratively update the perception model to be evaluated.

[0056] In an optional implementation, after determining the target scene label, road data containing the target scene label may be determined from pre-stored road data, and the perception model to be evaluated may be updated using the road data containing the target scene label.

[0057] In an embodiment of the present application, the road data collected by the vehicle in multiple continuous frame segments are input into the perception model to be evaluated to obtain multiple corresponding model prediction results, wherein the road data includes a scene label for indicating the driving scene of the vehicle in a continuous frame segment; the model prediction results and the true value data corresponding to the road data are matched, and the perception model to be evaluated is evaluated to obtain the evaluation results; the scene label carried by the road data corresponding to the evaluation result that meets the preset conditions is determined as the target scene label, and the training data containing the target scene label is obtained and the perception model to be evaluated is trained. In the process of evaluating the perception algorithm, the sensor data collected by the vehicle on the road can be labeled to obtain road data containing scene labels, and the prediction results of the perception model and the corresponding true value data are obtained using the road data to obtain the evaluation results of the perception model. The road data is divided into scenes based on the scene labels to conduct multi-scene evaluations, which more clearly shows the shortcomings of the perception model in which driving scenarios. The scene label carried by the road data corresponding to the evaluation results that meet the preset conditions is determined as the target scene label, and the perception model is updated and trained using the training data corresponding to the target scene label to improve the performance of the perception model under the target scene label. Therefore, in the process of training the perception model, a large amount of training data is not required, but the road data in the scene with weak performance is obtained in a targeted manner. The above method solves the technical problem that the evaluation method of the perception algorithm provided by the related technology has low iterative optimization efficiency. The optimization efficiency of the perception model is improved.

[0058] In an optional embodiment, the above-mentioned evaluation of the perception model to be evaluated to obtain evaluation results includes: obtaining model prediction results and true value data corresponding to road data collected by multiple continuous frame segments to evaluate the perception model to be evaluated, and obtaining category evaluation results, wherein the category evaluation results are used to indicate evaluation results corresponding to multiple categories of perception targets; determining the category of the perception target to be evaluated based on the category evaluation results, and using the road data collected by multiple continuous frame segments to evaluate the perception model to be evaluated on the dimension of the category of the perception target to be evaluated, and obtaining evaluation results.

[0059] It should be noted that the road data of multiple continuous frame segments can be obtained, and the road data of each continuous frame segment is continuous frame data under a scene tag. The road data of multiple continuous frame segments can be road data containing multiple scene tags, or road data of different sections under the same scene tag.

[0060] In an optional implementation, the road data collected by multiple data frame segments can be input into the prediction model as an evaluation set to obtain model prediction results, and the number of model prediction results is the same as the number of road data of continuous frame segments. Using road data of multiple continuous frame segments for evaluation can increase the diversity and richness of scene labels contained in the evaluation set, so as to analyze the perception model to be evaluated in multiple scenarios.

[0061] In an optional embodiment, the obtained model prediction results are matched with the true value data, and the evaluation results corresponding to the evaluation set are calculated according to preset evaluation parameters. The evaluation results can be category evaluation results, and the category evaluation results can indicate the categories of all objects and participants around the self-vehicle of specific categories in different scenarios. The categories of objects and participants can be obstacles or traffic participants. Obstacles can be static objects that hinder the vehicle's driving path, such as trees, buildings, walls, rocks or other vehicles parked in non-driving areas. Traffic participants can be dynamic objects, such as other vehicles, motorcycles, bicycles, pedestrians, animals, etc., which can move around the self-vehicle and may affect the vehicle's driving path. For the category of the category evaluation results, it can be set according to actual needs, such as identifying large categories such as traffic participants, and obtaining category evaluation results of static obstacles or category evaluation results of dynamic traffic participants. Small categories such as pedestrians and vehicles can also be identified to obtain category evaluation results of pedestrians or category evaluation results of vehicles. Through the evaluation results of perception targets of different categories, it can be determined which category of perception targets the perception model to be evaluated is not effective for.

[0062] In an optional implementation, after obtaining the category evaluation result, the category evaluation result can be compared with a preset threshold, and the category of the perceived target corresponding to the result less than the preset threshold in the category evaluation result is determined as the category of the perceived target to be evaluated. The category of the perceived target to be evaluated can be a perceived target with poor model perception effect under road data collected using multiple continuous frame segments.

[0063] Through the above implementation of the present application, the perception model to be evaluated can be evaluated using road data collected from multiple continuous frame segments, and the perception model to be evaluated can be evaluated using road data corresponding to a variety of scene labels to obtain category evaluation results. And based on the category evaluation results, it is determined which specific type of perception target the perception model to be evaluated has a poor perception effect on, so as to facilitate subsequent analysis. The diversity of perception model evaluation is improved, and accurate categories are analyzed, which improves the accuracy of perception model evaluation, thereby making the subsequent data screening process more accurate.

[0064] In an optional embodiment, the scene label carried by the road data corresponding to the evaluation result that meets the preset conditions is determined as the target scene label, including: evaluating the perception model to be evaluated using the model prediction results and true value data corresponding to each of the road data collected from multiple consecutive frame segments to obtain the evaluation results corresponding to each of the multiple consecutive frame segments; determining the evaluation result that meets the preset conditions among the evaluation results corresponding to each of the multiple consecutive frame segments as the evaluation result to be determined; and determining the repeated scene label in the scene label carried by the road data corresponding to the evaluation result to be determined as the target scene label.

[0065] It should be noted that after obtaining the model prediction results corresponding to the road data collected by multiple continuous frame segments, the road data collected by the multiple continuous frame segments can be directly input into the perception model to be evaluated, so that the perception model to be evaluated can obtain the evaluation results corresponding to each continuous frame segment.

[0066] In an optional embodiment, after evaluating the road data collected by multiple continuous frame segments, a category evaluation result can be obtained, which can indicate the category of the perceived target to be evaluated. The road data collected by multiple continuous frame segments can be input into the perceived model to be evaluated again, but this evaluation can only focus on the category of the perceived target to be evaluated, and output the evaluation result based on the category of the perceived target to be evaluated.

[0067] In an optional implementation, after the above-mentioned evaluation based on multiple data segments, if the evaluation results indicate that the overall indicators of the pedestrian category are poor, the pedestrian category can be determined as the perception target category to be evaluated, and a single data segment can be evaluated, and the evaluation results of the pedestrian category on each continuous frame segment can be screened. The road data corresponding to the results that meet the preset conditions (such as less than the preset threshold) after the single data segment evaluation is extracted, and the scene labels contained in the road data are determined to determine which scene label has the poor effect of the perception model on the pedestrian category.

[0068] In an optional implementation, repeated scene tags among the screened scene tags may be determined as target scene tags, or all screened scene tags may be determined as target scene tags. For example, after evaluating the pedestrian category, if the evaluation result indicates that the evaluation result meets the preset conditions in the night and rainy scenes, then the night and rainy days may be determined as target scene tags.

[0069] Through the above implementation of the present application, after the category of the perceived target to be evaluated is determined by the evaluation of multiple data segments, a single data segment can be used to determine in which scenarios the evaluation effect of the perceived target category to be evaluated is poor, so as to more accurately determine the perceived weaknesses of the perceived model to be evaluated. This improves the accuracy of the model evaluation and makes the training data obtained subsequently more targeted, thereby improving the efficiency of the iterative update of the model.

[0070] In an optional embodiment, after the scene label carried by the road data corresponding to the evaluation result that meets the preset conditions is determined as the target scene label, it includes: selecting the target road data collected by a single frame from the road data containing the target scene label; determining the model prediction result and the true value data corresponding to the target road data, and matching the model prediction result and the true value data corresponding to the target road data to obtain a single frame evaluation result.

[0071] In an optional implementation, the road data collected in a single frame can be input into the perception model to be evaluated to evaluate the performance of the model. It should be noted that the road data collected in a single frame can be selected from the road data containing the target scene label after the single data segment evaluation, or can be selected from the segment containing the perception target category to be evaluated after the multiple data segments evaluation.

[0072] In an optional implementation, evaluation items can be set as needed when using single-frame data for evaluation. For example, the ability of the perception model to be evaluated to detect small targets can be specifically evaluated, and the data between the vehicle and the specified target can be determined in the single-frame data and detected.

[0073] Figure 3 is a schematic diagram of an optional perceptual model evaluation method according to an embodiment of the present application; Figure 3 As shown, after obtaining road data of multiple continuous frame segments, a multi-data segment evaluation can be performed, and the perception targets to be evaluated, such as pedestrians, vehicles, etc., can be determined based on the multi-data segment evaluation. After determining the perception target to be evaluated, the road data of the continuous frame segments containing the perception target to be evaluated can be extracted, and each continuous frame segment can be input into the perception model to be evaluated to obtain the evaluation results corresponding to each continuous frame segment. The ones that meet the preset conditions are determined as the evaluation results to be determined, and the corresponding target scene labels are determined. After that, a single frame of road data containing the perception target to be evaluated and the target scene label can be found and input into the perception model to be evaluated to perform a single-frame evaluation on the perception model to determine whether the perception model has poor detection effect on the perception target to be evaluated in the scene corresponding to the target scene label, so as to achieve multiple verifications.

[0074] Through the above implementation of the present application, the target scene label and the perceived target to be evaluated can be determined after the multi-data segment evaluation and the single data segment evaluation, and the perception model can be evaluated again through a single frame of road data containing the perceived target to be evaluated and the target scene label to detect the effect of the perception model, and a complete evaluation result can be formed with the multi-data segment evaluation and the single data segment evaluation, and verified to obtain an accurate perception model evaluation result. The accuracy of the perception model evaluation is improved.

[0075] It should be noted that the above process is the overall process of evaluating the perception model to be evaluated. The following will explain in detail how to evaluate the road data of each continuous frame segment in the specific evaluation process.

[0076] In an optional embodiment, the model prediction results and true value data corresponding to the road data are matched, and the perception model to be evaluated is evaluated to obtain an evaluation result, including: determining the distance between at least one reference obstacle in the road data and the vehicle according to the true value data corresponding to the road data, wherein the reference obstacle is used to indicate static obstacles and dynamic obstacles around the vehicle; according to the distance between at least one reference obstacle and the vehicle and a preset distance threshold, at least one reference obstacle is divided into at least one reference obstacle set; according to the road data corresponding to at least one reference obstacle set, the perception model to be evaluated is evaluated in turn to obtain an evaluation result.

[0077] As mentioned above, the perception model can detect and identify static and dynamic obstacles around the ego vehicle. The true value data can be a combination of two-dimensional data and three-dimensional data. Therefore, the true value data contains depth information, which can indicate the distance between the reference obstacle and the ego vehicle.

[0078] In an optional implementation, the reference obstacle can be segmented according to the distance between the reference obstacle and the vehicle to obtain different segmented intervals. For example, it can be divided into five intervals of 0~10m, 10~30m, 30~60m, 60~100m, and 100~150m, and the perception model is evaluated within the five intervals. For example, when evaluating vehicle category targets, the distance between the reference obstacle (surrounding vehicle) and the vehicle of each vehicle category in the continuous frame road data can be obtained, and it can be divided into five distance intervals according to the distance between the vehicle and the surrounding vehicles. The general evaluation results can be calculated for the perception targets of the vehicle category first.

[0079] In an optional implementation, after determining the category to be evaluated, the recall rate, precision rate, multiple object tracking accuracy (Multiple Object Tracking Accuracy, MOTA), multiple object tracking precision (Multiple Object Tracking Precision, MOTP), etc. can be calculated. For example, the following formula can be used to calculate the recall rate:

[0080]

[0081] To calculate the accuracy, you can use the following formula:

[0082]

[0083] To calculate MOTA, you can use the following formula:

[0084]

[0085] To calculate MOTP, you can use the following formula:

[0086]

[0087] TP stands for true positive, i.e., the target correctly detected and tracked by the perception model. In a given frame, if the intersection / intersection ratio of the target box predicted by the perception model and the true value data exceeds a certain threshold, it can be considered correct.

[0088] FP stands for false positive, i.e., the perception model mistakenly detects background or other non-target objects as targets.

[0089] FN represents false negative examples, i.e., actual targets that the perception model fails to detect. They are targets that actually exist in the ground-truth data, but are not detected by the perception model in a given frame.

[0090] ID represents ID switching error, i.e. the perception model mistakenly changes the ID of the target during tracking.

[0091] GT stands for true value, that is, true value data.

[0092] d ti represents the positioning error of the i-th matching pair in the t-th frame, c t represents the number of matching pairs in the t-th frame.

[0093] In an optional embodiment, the matching of the model prediction result corresponding to the road data with the true value data includes: matching the model prediction result corresponding to the reference obstacle with the true value data corresponding to the reference obstacle;

[0094] The above-mentioned evaluation of the perception model to be evaluated obtains an evaluation result, including at least one of the following:

[0095] 1) Calculate the difference between the model prediction result and the distance in the true value data to obtain the absolute distance error;

[0096] 2) Calculate the ratio between the absolute distance error and the distance in the true value data to obtain the relative distance error;

[0097] 3) Calculate the difference between the model prediction result and the speed in the true data to obtain the absolute speed error.

[0098] In an optional implementation, after a category of targets has been evaluated using the above-mentioned general indicators, they can be evaluated by distance segment. The set of errors between the perceived target (reference obstacle) and the matching target in the true data can be calculated, and the measurement can be performed in segments according to the distance. By evaluating the distance in segments, the performance changes of the perception model in different distance ranges can be captured more finely, thereby more accurately evaluating the error performance of the perception model. It can also distinguish between distant targets and close targets, so the depth error of distant targets is larger. By evaluating the targets in segments, the influence of the long-distance error on the overall performance can be avoided.

[0099] In an optional embodiment, the model can be evaluated using an absolute distance error, a relative distance error, and an absolute speed error. The absolute distance error can be a set of absolute lateral and longitudinal distance errors, including a lateral error indicating the lateral distance between the center of mass of the vehicle and the true position, and a longitudinal error indicating the deviation of the true position of the vehicle along the desired trajectory. The relative distance error can be a set of relative lateral and longitudinal distance errors, including a relative lateral error indicating the lateral distance of the vehicle relative to the center line of the lane or the desired trajectory, and a relative longitudinal error indicating the deviation of the front and rear positions of the vehicle on the desired trajectory. The absolute speed error can be a set of absolute lateral and longitudinal speed errors, including a lateral speed error indicating the difference between the actual lateral speed of the vehicle and the desired lateral speed, and a longitudinal speed error indicating the difference between the actual longitudinal speed of the vehicle and the desired longitudinal speed.

[0100] The error can be calculated using the following formula:

[0101]

[0102] Among them, d i represents the true value data, represents the predicted value of the perception model, and M can represent the number of perceived targets in the current distance segment.

[0103] In an optional implementation, after obtaining the error set of the target in the segmented interval, indicators such as mean and variance may also be calculated, such as the root mean square error:

[0104]

[0105] In an optional implementation, when performing segmented evaluation, scene tags may also be combined for evaluation. Figure 4 FIG. 1 is a schematic diagram of another optional perceptual model evaluation method according to an embodiment of the present application; as mentioned above, the scene label may include multiple types. Figure 4 As shown, it is an example that multiple continuous frame segments can be divided according to preset scene labels. For a continuous frame segment, it can contain multiple scene labels. For example, for a segment, it can be sunny, night, and city. It can be divided into sunny, night, and city according to the labels, and the segments with the same labels are evaluated. For example, a sunny day may contain 8 segments, and the 8 segments can be evaluated in segments. And mixed evaluation of multiple scene labels can be performed, such as forming an evaluation group with the segments corresponding to the rainy day, night, and tunnel labels, and performing segmented evaluation.

[0106] Through the above implementation of the present application, the pre-detected data set can be freely set according to the scene label, and can be divided according to the distance during the evaluation. After determining the general indicators, the error and other indicators can be calculated for the targets of different segmented distances to evaluate the perception model. The performance of the perception model at different distances can be more accurately indicated, so as to optimize the model in a targeted manner.

[0107] In an optional embodiment, the above-mentioned acquisition of training data containing target scene labels and training of the perception model to be evaluated includes: determining the target category and the target distance threshold interval; acquiring target road data containing the target scene labels and the target category within the target distance threshold interval, and determining the target road data as training data; and using the training data to train the perception model to be evaluated.

[0108] In an optional implementation, multiple rounds of evaluation can be used to determine in which categories, distance intervals, and scene labels the perception model does not work well. For example, if the current perception model does not work well for pedestrians in a dark scene within the range of 10 to 30 meters, the number of target road data in the training set of the current perception model that contains target scene labels and target categories within the target distance threshold interval can be detected. When it is determined that the number of qualified road data in the acquired training set is less than the quantity threshold, the target road data that contains the target scene label and the target category within the target distance threshold interval can be acquired in the storage and used as training data to update the perception model.

[0109] Through the above implementation of the present application, when it is determined that the perception model has poor prediction effect on the data of those dimensions, the number of data that meet the requirements in the data contained in the current training set can be determined, and when the number is insufficient, it can be supplemented in a targeted manner. In this way, with the same amount of data, the perception model can be iteratively updated in a targeted manner to improve the performance of the perception model.

[0110] In an optional embodiment, before the road data collected by the vehicle in multiple continuous frame segments is input into the perception model to be evaluated, the method includes: performing data screening on the reference road data to determine the road data that meets the preset data conditions, wherein the reference road data is the road data continuously collected by the vehicle; cutting the road data in the reference road data that meets the preset data conditions to obtain the road data in multiple continuous frame segments that are consistent in time; performing scene recognition on the road data to obtain the corresponding scene label, and adding the scene label to the road data to obtain the updated road data.

[0111] In an optional implementation, a true value vehicle can be used for road sampling, and reference road data can be collected through a variety of high-precision sensors carried by the true value vehicle. Sensors include laser radar, cameras, millimeter wave radar, GPS, etc. The reference road data can be stored in the form of rosbag data packets, which can carry accurate timestamps and ensure the integrity of the reference road data. After obtaining the reference road data, data screening can be performed on it, and high-value data can be selected from it. Reference indicators can be set in advance for high-value data, such as vehicle interaction, pedestrian crossing, lane change and other conditions can be set as reference indicators. The reference road data can be divided into continuous frame segments and sliced.

[0112] In an optional implementation, after the data is expanded, scene tags can be added to the slice segments, and scene recognition can be performed on the slice segments to determine their scene tags and mark them. After the road data is acquired, it can be stored in the cloud or a storage device.

[0113] Through the above implementation of the present application, reference road data can be obtained through true value vehicle road sampling, and the reference road data can be processed to obtain marked road data, which is convenient for evaluating the perception model, and when the perception model is updated, training data can be obtained more quickly. By ensuring the accuracy of the data, the accuracy of subsequent perception model evaluation is improved.

[0114] In an optional embodiment, before the above-mentioned matching of the model prediction results and the true value data corresponding to the road data, it includes: inputting the road data collected by the vehicle in multiple consecutive frame segments into the true value model to obtain two-dimensional true value data and three-dimensional true value data corresponding to the road data; associating the two-dimensional true value data with the three-dimensional true value data to obtain the true value data corresponding to the road data.

[0115] In an optional implementation, after obtaining the two-dimensional road data and the three-dimensional road data, they can be input into the truth model to determine the two-dimensional truth data and the three-dimensional truth data. The two-dimensional truth data and the three-dimensional truth data can also be combined to obtain the truth data of the 2D and 3D joint matching.

[0116] In an optional implementation, the truth box of the 3D truth data can be projected onto the image of the 2D truth data, and objects that are not completely blocked can be screened, and the 2D truth data and the 3D truth data whose intersection over union (IoU) is greater than a preset threshold can be associated. The IoU can be the ratio of the intersection area to the union area of ​​the two bounding boxes.

[0117] The remaining two-dimensional truth data and three-dimensional truth data can be sorted according to the intersection-and-union ratio, and then the Euclidean distances between the two-dimensional truth data and the three-dimensional truth data can be calculated respectively. If the three-dimensional truth data has the smallest Euclidean distance to a certain two-dimensional truth data and the Euclidean distance between the three-dimensional truth data and the two-dimensional truth data is also the smallest, the two can be matched.

[0118] For the remaining two-dimensional truth data and three-dimensional truth data with an intersection-and-union ratio of 0, the data of the previous n consecutive frames can be used to analyze them, and the two-dimensional truth data and the three-dimensional truth data with the most association times can be combined. After the truth data are combined, the truth data can include target category, two-dimensional truth box, two-dimensional target ID, three-dimensional target ID, two-dimensional target box xmin, ymin, xmax, ymax, target head or tail longitudinal distance, target head or tail lateral distance, target side longitudinal distance, target side lateral distance, target longitudinal speed, target lateral speed target, two-dimensional truth box x, y, z, rz, l, w, h and information, target speed information, target three-dimensional projection box information and 8 corner point coordinate information of the target's three-dimensional rectangle on the two-dimensional image.

[0119] In an optional embodiment, when matching the true value data with the model prediction results, a cost matrix can be constructed, and the rows of the cost matrix can represent the model prediction results, and the columns can represent the true value data, so that each element in the matrix can represent the matching cost between the corresponding prediction results and the true value data. The matching cost can be an intersection ratio, a Euclidean distance, etc. The matrix can be initialized, and the minimum value of each row is subtracted from the minimum value of the row, and the same operation is performed on each column. Use the minimum number of horizontal and vertical lines to cover all zero elements in the matrix. These zero elements represent the possible best match. If the number of lines covering the zero elements is equal to the dimension of the matrix (i.e., the number of rows or columns), it means that the best match has been found. If it is not equal, it means that further adjustment is required. If the best match is not found, the cost matrix needs to be adjusted. Find the smallest element that is not covered, subtract the minimum value from each element in the row where the element is located, and add the minimum value to each element in the column where the element is located, and then repeat the above process. Until the best match is found. The best match will be represented by the index of the row and column, and these indexes point to the zero elements in the cost matrix.

[0120] It should be noted that in the process of matching the true value data and the model prediction results, the large depth error of the distant target may lead to inaccurate matching. However, by matching the combined two-dimensional and three-dimensional data, the matching can be performed within the two-dimensional frame, avoiding the influence of depth information.

[0121] Through the above-mentioned implementation mode of the present application, true value data of the combination of two-dimensional and three-dimensional information can be obtained, so as to improve the matching accuracy in the subsequent matching process of the true value data and the model prediction results, and by performing evaluation based on the true value data of the combination of two-dimensional and three-dimensional information, more model evaluation indicators (such as depth information required) can be obtained, thereby improving the accuracy of model evaluation.

[0122] Figure 5 is a schematic diagram of another optional perceptual model evaluation method according to an embodiment of the present application; Figure 5 As shown in the figure, the collected road data can be screened and processed to obtain true value data. When evaluating the model, the road data can be obtained and fed back into the perception model to obtain the model prediction results. The model evaluation results can be obtained based on the model prediction results and the true value data. According to the scene labels with poor results in the model evaluation results, data is screened again to obtain the training set of the perception model, and the perception model is updated and iterated.

[0123] In an optional implementation, a variety of sensor data can be obtained by using the true value vehicle road acquisition, and the obtained multiple sensor data can be stored in the form of rosbag data packets, and then after data screening, high-value scene data is selected for expansion, and the high-value scene data is sliced ​​into continuous frame segments, and the segment duration is consistent. In the data expansion stage, the slice segments will first be added with scene labels. The scene label addition mainly uses the scene recognition model to mark the road scene, weather, lighting conditions and other scenes of the screened data slices. Each data slice segment corresponds to a set of scene labels. The expanded image data and point cloud data will then be timestamped and stored in the cloud. The expanded data of the data slices stored in the cloud are processed by frame extraction, and the extracted frames are regarded as key frames. These key frames are manually sent for labeling. The returned manual labeling results can be mapped back to the original slice data segments, and will be used for the subsequent training of the cloud-based true value large model. The data expanded by the data slice segments and the key frame data returned by the frame extraction and labeling will be stored in the object storage, which is convenient for the subsequent acquisition of cloud-based true value model training data and the acquisition of scene labels and other data.

[0124] In an optional implementation, in order to solve the problem that the 3D true value box of the 3D continuous frame true value has a serious offset in the box position and size, and the point cloud is sparse at distant targets and may not be able to output the true value, after the 2D and 3D continuous frame true values ​​are generated, the 2D and 3D continuous frame true values ​​can be jointly annotated. This solves the problem that the vehicle-side algorithm model has a large error in measuring the distance of distant targets in pure 3D space, which makes it impossible to associate with the true value during evaluation, because through joint matching annotation, association can be performed under the 2D box.

[0125] Figure 6 is a schematic diagram of another optional perceptual model evaluation method according to an embodiment of the present application; Figure 6 As shown, on the one hand, S602 is executed to obtain the area of ​​the 2D truth box; S604, the 3D truth box point cloud xyz determines the distance before and after; S606, the 2D truth box border coordinates determine the relative position; S608 is executed according to S602-S606, and the inclusion relationship is comprehensively judged to determine the occlusion; on the other hand, S610 is executed to obtain the 2D continuous frame truth; S612, the 3D continuous frame truth is obtained; S614, the 3D target truth box is projected to the 2D image; S616 is executed in combination with S608 to filter out objects that are not completely occluded; on the other hand, S618, key frames are manually labeled; S620 is executed in combination with S610 and S616, three rounds of matching; S622, the detection result of the perception algorithm model to be evaluated; S624 is executed in combination with step S620 to associate the detection results.

[0126] In an optional implementation, the unified perspective of the joint truth matching is the 2D image perspective. First, based on the camera intrinsic parameters and the joint calibration parameters of the camera and lidar, the 3D target truth frame is projected to the 2D image. This step projects the 3D frame to the 2D image. Because of the maximum circumscribed rectangle, the projected 2D frame will expand. Secondly, the 2D frame projected onto the 2D image by the 3D target truth frame needs to be screened. Due to the characteristics of the point cloud, the cloud-based truth model can output targets that are completely invisible in the 2D image based on the point cloud data. Those truth targets that are completely invisible in the 2D image need to be eliminated because the framework of the entire joint matching annotation process is set within the 2D image range.

[0127] Optionally, in the first round of matching, the target category is first used to divide the 2D truth box and the 3D projection box, and the matching process is performed within the same category of the 2D truth box and the 3D projection box. The first round of matching is mainly based on IoU, and direct matching is performed by calculating whether the numerical value of the bounding_box_overlap of the 2D truth box and the 3D projection box is greater than the set threshold. If the IoU of the 2D truth box and the 3D projection box of the same category is greater than the set threshold, the 2D truth box and the 3D projection box will be associated. Because the information of the original 3D truth box of the 3D projection box is retained, the 2D truth box can be given 3D information, such as horizontal and vertical depth and horizontal and vertical speed information, and because it is a continuous frame truth, the target tracking ID of the 2D truth box and the 3D projection frame can also be associated. In this round of matching, targets that are close to the vehicle and not too coupled with other targets will be jointly matched. However, because this round of matching is completely based on the IoU value, only when the IoU result is greater than the set threshold will it be associated, which will cause many missed matches. Therefore, the first round of matching is a coarse match.

[0128] Optionally, because the 3D true value projection frame of the target farther away not only expands but also offsets, there will be no IoU information and it cannot be matched. In order to solve the problem of more true value targets in complex scenes, or targets occluding and overlapping each other, a two-round matching is proposed. The second round of matching is based on the matching process after eliminating the results of the first round of matching. This round of matching will only associate the targets whose IoU values ​​of the 2D true value frame and the 3D projection frame are not 0. The IoU values ​​that are not 0 with the IoU value of the 2D true value frame can be arranged from large to small, and the Euclidean distance of the pixel coordinates of the center point of the 2D true value frame and the 3D projection frame is calculated and sorted, and the IoU sorting result and the center point Euclidean distance sorting result are combined for matching. Double judgment can be performed on the two types of sorting results. For example, the 2D truth box with index 1 has an index of 3 with the largest IoU value of the 3D truth projection box, and the 2D truth box with the closest Euclidean distance to the center pixel coordinates of the 3D truth projection box with index 3 has an index of 1. That is, after double judgment, the 2D truth box with index 1 and the 3D truth projection box with index 3 are associated and matched.

[0129] Optionally, the second round of matching can also solve the matching of targets whose IoU values ​​of 2D true value frames and 3D projection frames are not 0, and the third round of matching mainly solves the matching of targets whose IoU values ​​are 0. First, based on the elimination of the first and second round matching results, the 2D true value frames and 3D target frames with IoU values ​​of 0 are screened out, and the matching results of each 2D true value frame in the historical 10 consecutive frames (if less than 10 frames, from the starting frame to the current frame) are calculated and analyzed, and the joint annotation information of the key frames is combined to assist in optimization.

[0130] Figure 7 is a schematic diagram of another optional perceptual model evaluation method according to an embodiment of the present application; Figure 7As shown in the figure, the specific matching ideas are as follows: First, assume that the Car target with 2D true value ID 5 in the nth frame has no 3D projection frame target matching it after the first and second rounds of matching. That is, the IoU between the Car target with 2D true value ID 5 and all 3D projection frames is 0. At the beginning of the third round of matching, we can first count whether the Car target with 2D true value ID 5 has always existed in the past 10 consecutive frames. If it has always existed, it is the most ideal situation. If it has not always existed, then count the continuous scenes in which the Car target with 2D true value ID 5 in the history has lasted to the current frame, such as the n-5th frame to the nth frame. If the Car target with 2D true value ID 5 has always appeared in the past 10 consecutive frames, count the matching of the Car target with 2D true value ID 5 in the past 10 consecutive frames. The Car target with 2D truth value ID 5 has matched with the Car target with 3D projection ID 3 5 times, 3D projection ID 6 4 times, and 3D projection ID 7 1 time in 10 consecutive historical frames. The historical matching results are arranged in descending order. From the descending results, it can be seen that the Car target with 2D truth value ID 5 and the Car target with 3D projection ID 3 have matched the most times in the historical matching stage. This is the process of counting the matching information of the 3D projection frame based on the 2D truth value target frame, and it is also necessary to count the matching information of the 2D truth value target frame based on the 3D projection frame.

[0131] For example, based on the result that the Car target with 2D true value ID 5 matches the Car target with 3D projection ID 3 the most times in the above historical matching stage, the matching information of the Car target with 3D projection ID 3 in the historical matching stage can be counted. Assuming that the Car target with 3D projection ID 3 also matches the Car target with 2D true value ID 5 the most times in the historical matching information, then the Car target with 2D true value ID 5 will be matched with the Car target with 3D projection ID 3 in the nth frame; if the Car target with 3D projection ID 3 does not match the Car target with 2D true value ID 5 the most times in the historical matching information, then the historical matching information of the Car target with 3D projection ID 6, which has the second most matches with the Car target with 2D true value ID 5, can be counted based on the statistical results of the above step, and so on.

[0132] After the above two steps, assuming that the Car target with 2D true value ID 5 has been matched with the Car target with 3D projection ID 3, the matching sequence is corrected and optimized based on the manual joint annotation results of the key frames. The specific ideas are as follows:

[0133] First, find the nearest key frame based on the current nth frame. The key frame is manually extracted and sent for manual labeling. The number of key frames in a continuous frame segment is fixed, and the number of key frames and the key frame extraction rules are set in advance. If the nearest key frame cannot be found, no key frame correction is performed. Assume that based on the nth frame, the nearest key frame m is found. Based on the joint annotation information of the key frame, the joint annotation matching information of the 2D continuous frame truth value and the 3D continuous frame truth value of the key frame can be known. Based on the joint annotation information of the key frame, the Car target with the 2D truth value ID of 5 in the current nth frame is calculated based on the 2D truth box marked by the joint matching of the key frame. The purpose is: first, to confirm whether the Car target with the 2D truth value ID of 5 in the nth frame of the current continuous frame segment also exists in the key frame m; second, if so, the Car target with the 2D truth value ID of 5 in the nth frame is marked with the 2D truth value ID of 14 by the manual joint matching of the key frame. Correspondingly, the IoU between the Car target with 3D true value ID 15 in key frame m that matches the Car target with 2D true value ID 14 and the Car target with 3D projection ID 3 in the nth frame is calculated to be greater than the threshold. If it is greater than the threshold condition, it is considered that the Car target with 2D true value ID 14 and the Car target with 3D true value ID 15 that are matched by the joint annotation of key frame m are completely corresponding to the Car target with 2D true value ID 5 and the Car target with 3D projection ID 3 that are matched in the nth frame, thereby confirming that the result based on the third round of matching for the nth frame is correct. Then, in the n-10th frame, the Car target with 2D true value ID 5 matches the 3D projection true value ID 7, and this match only appears in one frame, which can be optimized with the help of key frames. Assume that the key frame closest to the n-10th frame is also key frame m. This assumption is valid in most cases because the number of key frames is small and the span is large. Based on the previous step, we know that the Car target with 2D truth value ID 5 in the nth frame corresponds to the Car target with 2D truth value ID 14 marked by manual joint matching in the key frame. The Car target with 3D truth value ID 15 in the key frame m matches the Car target with 2D truth value ID 14. Based on the Car target with 3D truth value ID 15, the IoU of all 3D projection frames in the n-10th frame is calculated, and then they are sorted from large to small for matching. It is concluded that in all 3D projection frames of the n-10th frame, the Car target with ID 6 corresponds to the Car target with 3D truth value ID 15 in the key frame m. Based on this, the 3D projection truth value ID of the Car target matching the 2D truth value ID 5 in the n-10th frame is modified to 6.

[0134] Optionally, by joint matching and annotation of the true values ​​of 2D and 3D continuous frames and the true values ​​of the continuous frame fragments, the 2D true value box and the 3D true value box are associated. Each 2D continuous frame true value target box will include target category, 2D target ID, 3D target ID, 2D target box xmin, ymin, xmax, ymax, target head or tail longitudinal distance, target head or tail lateral distance, target side longitudinal distance, target side lateral distance, target longitudinal speed, target lateral speed, 3D true value box x, y, z, rz, l, w, h and 8 corner point coordinate information of the three-dimensional rectangle of the target on the 2D image. The specific joint matching annotation result examples are as follows: Car 16 2187 1198 2333 1314 28.89 -4.35 31.08 -3.44 21.13 -0.77 31.06 -4.41 0.74 -0.025 4.34 1.93 1.63 2177 1202 2297 1203 2342 1197 2204 1196 2176 1304 22961304 2341 1315 2203 1315Based on the joint labeled true value, the detection results of the vehicle-side algorithm model to be evaluated under the continuous frame fragments of the joint labeled true value are evaluated.

[0135] Optionally, first, the joint matching annotations and the feedback results of the algorithm to be evaluated are correlated and output in the format required for cloud evaluation. When the true value and algorithm results are correlated in the 3D space in a pure 3D evaluation, there is no correlation information and no correlation can be performed due to the large depth error of the vehicle-side perception algorithm for distant targets. However, this method is not performed in 3D space, but the 2D frame of the joint matching annotation is correlated, which can effectively solve the problem of no correlation information in pure 3D space. After correlation, single frame evaluation, single data segment evaluation, and multiple data segment evaluation can be performed:

[0136] Multi-data fragment evaluation: The evaluation set of this evaluation system is generated by combining multiple data fragments. Multi-fragment evaluation can intuitively reflect the comprehensive performance of the vehicle-side perception algorithm. The fragments that make up the evaluation set, based on the scene recognition labels of the data slicing module, can better balance the diversity and richness of the fragment labels contained in the evaluation set, and show the performance indicators of the vehicle-side perception algorithm under a variety of scene labels as much as possible, so as to make the evaluation results more convincing and representative. Through the evaluation index results, targeted performance analysis can also be carried out. For example, the evaluation indicators show that the overall indicators of the pedestrian category are poor. Based on this result, further single-fragment evaluation can be carried out.

[0137] Single data segment evaluation: Single segment evaluation can further analyze the performance of the algorithm for specific categories under different scene labels, and can also further analyze the performance of the algorithm for specific scene labels under different target categories. For example, based on the multi-data segment evaluation results, the overall evaluation indicators of the pedestrian category are poor. Through the single segment evaluation results, the evaluation results of the pedestrian category in each segment are filtered and grouped, and the possible reasons for the poor pedestrian category evaluation results can be analyzed, such as poor pedestrian detection effect on a rainy night.

[0138] Single-frame evaluation: Single-frame evaluation can customize the evaluation set. For example, to evaluate the detection ability of the vehicle-side perception algorithm for small targets, all single frames containing small targets can be selected for evaluation through object storage. And because single frames can also be mapped back to the fragment information of the single frame and detailed fragment scene label information through object storage, single-frame evaluation can be combined with single-fragment evaluation and multi-fragment evaluation to form a complete evaluation system.

[0139] Optionally, first, based on the data screening slicing and joint matching annotation modules, assume that the joint matching annotation truth values ​​of 20 continuous frame segments are generated. In the data screening slicing module, scene labels will be generated for the 20 continuous frame segments, and the 20 continuous frame segments will be divided based on the scene labels, for example, based on weather labels, light labels, road scene labels, etc. Evaluation can be performed in each divided label, because the joint annotation truth value contains depth information, and the target can be divided into different distance segments according to the longitudinal distance of the target from the vehicle. In each distance segment, each type of target is evaluated, and the evaluation indicators include the absolute error, relative error, and quantile of the depth information, the absolute error and quantile of the speed information, and the total number of each type of target in the distance segment. For the evaluation under each scene label segment, the evaluation indicators include recall rate, precision rate, MOTA, MOTP, FN (missed detection), FP (false detection), etc. Assuming that based on the 2D and 3D joint annotation truth results of the above 20 consecutive frame segments, the longitudinal and lateral distance information of each target from the vehicle can be obtained. Based on the truth information, each type of target is first divided into distance segments. For example, for the Car category targets, based on the joint annotation truth results, they are divided into five intervals of 0~10m, 10~30m, 30~60m, 60~100m, and 100~150m from the vehicle, and the target set of the Car category based on the truth distance division of these 20 consecutive frame segments is obtained. Based on the feedback results of the vehicle-side algorithm model in these 20 consecutive frame segments, the recall rate, precision rate, MOTA, MOTP and other general target tracking indicators of the Car category can be calculated.

[0140] Optionally, it is necessary to calculate the statistical indicators of the Car category within the range of 0~10m from the vehicle. First, it is necessary to obtain the above-mentioned target set with a distance of 0~10m from the vehicle based on the joint true value annotation. The absolute horizontal and longitudinal distance error set of each return target matching the true value target can be calculated. The matching of the return target and the joint annotation target is mainly based on the Hungarian algorithm. Then, based on the error set, the absolute error standard deviation, mean, and variance can be calculated. At the same time, the relative horizontal and longitudinal distance error set of each return target matching the true value target can also be calculated. Then, based on the error set, the relative error standard deviation, mean, and variance can be calculated. For speed, the absolute horizontal and longitudinal speed error set of each return target matching the true value target can be calculated. Then, based on the error set, the absolute error standard deviation, mean, and variance can be calculated. In order to calculate the quantiles of the distance error and speed error, the distance error set and the speed error set can be arranged from large to small, and then the 0.95 quantile (i.e., 2 times sigma) is calculated.

[0141] Through the above implementation of the present application, true value data of two-dimensional and three-dimensional information can be obtained, which can make the perception model evaluation avoid the problem that the three-dimensional space cannot be associated with the true value due to the large error in the distance measurement of long-distance targets, and can also provide more information on the basis of two-dimensional data. And it can be divided into distance measurement and speed measurement evaluation, and classification based on scene labels can be performed to obtain more detailed evaluation results and provide clearer guidance for subsequent model updates. The update of the perception model can achieve significant results based on less data.

[0142] It should be noted that, for the aforementioned method embodiments, for the sake of simplicity, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the present application is not limited by the described order of actions, because according to the present application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the present application.

[0143] According to another aspect of the embodiment of the present application, a perceptual model evaluation device for implementing the above-mentioned perceptual model evaluation method is also provided. Figure 8 As shown, the device comprises:

[0144] A prediction unit 802, configured to input the road data collected by the ego vehicle in a plurality of consecutive frame segments into the perception model to be evaluated, and obtain a plurality of corresponding model prediction results, wherein the road data includes a scene label for indicating a driving scene of the ego vehicle in a continuous frame segment;

[0145] The model evaluation unit 804 is used to match the model prediction result and the true value data corresponding to the road data, and evaluate the perception model to be evaluated to obtain the evaluation result;

[0146] The data acquisition unit 806 is used to determine the scene label carried by the road data corresponding to the evaluation result that meets the preset conditions as the target scene label, obtain the training data containing the target scene label and train the perception model to be evaluated.

[0147] Optionally, the above-mentioned model evaluation unit 804 includes: a category evaluation module, which is used to obtain model prediction results and true value data corresponding to road data collected by multiple continuous frame segments to evaluate the perception model to be evaluated, and obtain category evaluation results, wherein the category evaluation results are used to indicate evaluation results corresponding to multiple categories of perception targets; a model evaluation module, which is used to determine the category of the perception target to be evaluated based on the category evaluation results, and use the road data collected by multiple continuous frame segments to evaluate the perception model to be evaluated on the dimension of the category of the perception target to be evaluated to obtain an evaluation result.

[0148] Optionally, the above-mentioned data acquisition unit 806 includes: a single-segment evaluation module, which is used to evaluate the perception model to be evaluated with the model prediction results and true value data corresponding to each of the road data collected from multiple continuous frame segments, and obtain the evaluation results corresponding to each of the multiple continuous frame segments; a result comparison module, which is used to determine the evaluation results that meet the preset conditions in the evaluation results corresponding to each of the multiple continuous frame segments as the evaluation results to be determined; and a label determination module, which is used to determine the repeated scene labels in the scene labels carried by the road data corresponding to the evaluation results to be determined as the target scene labels.

[0149] Optionally, the above-mentioned data acquisition unit 806 also includes: a data selection module, used to select the target road data collected in a single frame from the road data containing the target scene label; a single frame evaluation module, used to determine the model prediction results and true value data corresponding to the target road data, and match the model prediction results and true value data corresponding to the target road data to obtain a single frame evaluation result.

[0150] Optionally, the above-mentioned model evaluation unit 804 also includes: a distance determination module, which is used to determine the distance between at least one reference obstacle in the road data and the vehicle according to the true value data corresponding to the road data, wherein the reference obstacle is used to indicate the static obstacles and dynamic obstacles around the vehicle; a distance interval division module, which is used to divide at least one reference obstacle into at least one reference obstacle set according to the distance between at least one reference obstacle and the vehicle and a preset distance threshold; an interval evaluation module, which is used to evaluate the perception model to be evaluated in turn according to the road data corresponding to at least one reference obstacle set to obtain an evaluation result.

[0151] Optionally, the above-mentioned model evaluation unit 804 also includes: a matching module, used to match the model prediction result corresponding to the reference obstacle with the true value data corresponding to the reference obstacle; a first error calculation module, used to calculate the difference between the model prediction result and the distance in the true value data to obtain an absolute distance error; a second error calculation module, used to calculate the ratio between the absolute distance error and the distance in the true value data to obtain a relative distance error; a third error calculation module, used to calculate the difference between the speed in the model prediction result and the true value data to obtain an absolute speed error.

[0152] Optionally, the above-mentioned data acquisition unit 806 also includes: a data determination module, used to determine the target category and the target distance threshold interval; a data acquisition module, used to obtain target road data containing a target scene label and a target category within the target distance threshold interval, and determine the target road data as training data; a model training module, used to use the training data to train the perception model to be evaluated.

[0153] Optionally, the prediction unit 802 includes: a data screening module, which is used to screen the reference road data to determine the road data that meets the preset data conditions, wherein the reference road data is the road data continuously collected by the vehicle; a data cutting module, which is used to cut the road data that meets the preset data conditions in the reference road data to obtain the road data in multiple continuous frame segments that are consistent in time; a data labeling module, which is used to perform scene recognition on the road data to obtain the corresponding scene label, and add the scene label to the road data to obtain the updated road data.

[0154] Optionally, the above-mentioned model evaluation unit 804 also includes: a truth value acquisition module, which is used to input the road data collected by the vehicle in multiple continuous frame segments into the truth value model to obtain two-dimensional truth value data and three-dimensional truth value data corresponding to the road data; a truth value association module, which is used to associate the two-dimensional truth value data with the three-dimensional truth value data to obtain the truth value data corresponding to the road data.

[0155] Optionally, in this embodiment, the embodiments to be implemented by the above-mentioned various unit modules can refer to the above-mentioned various method embodiments, which will not be repeated here.

[0156] According to another aspect of the embodiment of the present application, an electronic device for implementing the above-mentioned perception model evaluation method is also provided. The electronic device may be Fig. 9 The terminal device or server shown in the figure. This embodiment is described by taking the electronic device as a terminal device as an example. Fig. 9 As shown, the electronic device includes a memory 902 and a processor 904. The memory 902 stores a computer program, and the processor 904 is configured to execute the steps in any of the above method embodiments through the computer program.

[0157] Optionally, in this embodiment, the electronic device may be located in at least one network device among a plurality of network devices of a computer network.

[0158] Optionally, in this embodiment, the processor may be configured to perform the following steps through a computer program:

[0159] S1, inputting road data collected by the ego vehicle in a plurality of continuous frame segments into the perception model to be evaluated, and obtaining a plurality of corresponding model prediction results, wherein the road data includes a scene label for indicating a driving scene of the ego vehicle in a continuous frame segment;

[0160] S2, matching the model prediction results and the true value data corresponding to the road data, and evaluating the perception model to be evaluated to obtain the evaluation results;

[0161] S3, determining the scene label carried by the road data corresponding to the evaluation result that meets the preset conditions as the target scene label, obtaining the training data containing the target scene label and training the perception model to be evaluated.

[0162] Alternatively, a person skilled in the art may understand that: Fig. 9 The structure shown is for illustration only, and the electronic device may also be a smart phone (such as an Android phone, an iOS phone, etc.), a tablet computer, a PDA, a mobile Internet device (Mobile Internet Devices, MID), a PAD, and other terminal devices. Fig. 9 The electronic device and the electronic equipment described above are not limited in structure. Fig. 9 More or fewer components (such as network interfaces, etc.) as shown in, or with Fig. 9 Different configurations are shown.

[0163] Among them, the memory 902 can be used to store software programs and modules, such as the program instructions / modules corresponding to the evaluation method and device of the perception model in the embodiment of the present application. The processor 904 executes various functional applications and data processing by running the software programs and modules stored in the memory 902, that is, realizing the above-mentioned evaluation method of the perception model. The memory 902 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 902 may further include a memory remotely located relative to the processor 904, and these remote memories may be connected to the terminal via a network. Examples of the above-mentioned networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof. Among them, the memory 902 can be specifically, but not limited to, used to store information such as road images. As an example, such as Fig. 9 As shown, the memory 902 may include, but is not limited to, the prediction unit 802, the model evaluation unit 804, and the data acquisition unit 806 in the evaluation device of the perception model. In addition, it may also include, but is not limited to, other module units in the evaluation device of the perception model, which will not be described in detail in this example.

[0164] Optionally, the transmission device 906 is used to receive or send data via a network. Specific examples of the network may include a wired network and a wireless network. In one example, the transmission device 906 includes a network adapter (Network Interface Controller, NIC), which can be connected to other network devices and routers via a network cable so as to communicate with the Internet or a local area network. In one example, the transmission device 906 is a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0165] In addition, the electronic device mentioned above further includes: a display 908 for displaying a simulated road scene; and a connection bus 910 for connecting various module components in the electronic device mentioned above.

[0166] In other embodiments, the terminal device or server may be a node in a distributed system, wherein the distributed system may be a blockchain system, and the blockchain system may be a distributed system formed by connecting the multiple nodes in the form of network communication. Among them, the nodes may form a peer-to-peer (P2P) network, and any form of computing device, such as a server, terminal or other electronic device, may become a node in the blockchain system by joining the peer-to-peer network.

[0167] According to one aspect of the present application, a computer program product is provided, the computer program product comprising a computer program / instruction, the computer program / instruction comprising a program code for executing the method shown in the flow chart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication part, and / or installed from a removable medium. When the computer program is executed by a central processing unit, various functions provided by the embodiments of the present application are executed.

[0168] The serial numbers of the above-mentioned embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.

[0169] According to one aspect of the present application, a computer-readable storage medium is provided, and a processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the above-mentioned perceptual model evaluation method.

[0170] Optionally, in this embodiment, the computer-readable storage medium may be configured to store a computer program for performing the following steps:

[0171] S1, inputting road data collected by the ego vehicle in a plurality of continuous frame segments into the perception model to be evaluated, and obtaining a plurality of corresponding model prediction results, wherein the road data includes a scene label for indicating a driving scene of the ego vehicle in a continuous frame segment;

[0172] S2, matching the model prediction results and the true value data corresponding to the road data, and evaluating the perception model to be evaluated to obtain the evaluation results;

[0173] S3, determining the scene label carried by the road data corresponding to the evaluation result that meets the preset conditions as the target scene label, obtaining the training data containing the target scene label and training the perception model to be evaluated.

[0174] Optionally, in this embodiment, a person of ordinary skill in the art may understand that all or part of the steps in the various methods of the above embodiments may be completed by instructing hardware related to the terminal device through a program, and the program may be stored in a computer-readable storage medium, and the storage medium may include: a flash drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0175] If the integrated units in the above embodiments are implemented in the form of software functional units and sold or used as independent products, they can be stored in the above computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for enabling one or more computer devices (which may be personal computers, servers or network devices, etc.) to execute all or part of the steps of the above methods in each embodiment of the present application.

[0176] In the above embodiments of the present application, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, please refer to the relevant description of other embodiments.

[0177] In the several embodiments provided in the present application, it should be understood that the disclosed client can be implemented in other ways. Among them, the device embodiments described above are only schematic. For example, the division of the above units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.

[0178] The units described above as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0179] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0180] The above are only preferred implementations of the present application. It should be pointed out that ordinary technicians in this technical field can make several improvements and modifications without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.

Claims

1. A method for evaluating a perception model, characterized in that: include: Inputting road data collected by the ego vehicle in a plurality of continuous frame segments into the perception model to be evaluated to obtain a plurality of corresponding model prediction results, wherein the road data includes a scene label for indicating a driving scene of the ego vehicle in one of the continuous frame segments; Associating the two-dimensional true value data with the three-dimensional true value data to obtain the true value data corresponding to the road data, matching the model prediction result corresponding to the road data with the true value data, and evaluating the perception model to be evaluated to obtain an evaluation result; Determining the scene label carried by the road data corresponding to the evaluation result that meets the preset conditions as the target scene label, acquiring training data containing the target scene label and training the perception model to be evaluated; Wherein, determining the scene label carried by the road data corresponding to the evaluation result that meets the preset condition as the target scene label includes: Obtaining a category evaluation result using the road data collected by the plurality of continuous frame segments, and determining the perception target category corresponding to the result less than a preset threshold in the category evaluation result as the perception target category to be evaluated, wherein the category evaluation result is used to indicate the evaluation results of all objects and participants around the self-vehicle of a specific category in different scenarios; Determine an evaluation result of the perceived target category to be evaluated on a single data segment, and determine the scene label carried by the road data corresponding to the evaluation result that meets a preset condition as a target scene label; Among them, determining the evaluation results of the continuous frame segments includes: determining the distance between at least one reference obstacle in the road data and the vehicle itself according to the true value data corresponding to the road data collected by the continuous frame segments, and dividing the reference obstacles into reference obstacle sets according to the distance between the reference obstacles and the vehicle itself, and evaluating the perception model to be evaluated in turn according to the road data corresponding to at least one reference obstacle set to obtain the evaluation results.

2. The method according to claim 1, characterized in that The step of evaluating the perception model to be evaluated to obtain an evaluation result includes: Acquire the model prediction results and the true value data corresponding to the road data collected by the multiple continuous frame segments to evaluate the perception model to be evaluated, and obtain category evaluation results, wherein the category evaluation results are used to indicate evaluation results corresponding to multiple categories of perception targets; The category of the perception target to be evaluated is determined based on the category evaluation result, and the perception model to be evaluated is evaluated on the dimension of the perception target category to be evaluated using the road data collected by the multiple continuous frame segments to obtain the evaluation result.

3. The method according to claim 2, characterized in that The step of determining the scene label carried by the road data corresponding to the evaluation result that meets the preset condition as the target scene label includes: Evaluate the perception model to be evaluated based on the model prediction results and the true value data corresponding to each of the road data collected by the multiple continuous frame segments, and obtain evaluation results corresponding to each of the multiple continuous frame segments; Determining, among the evaluation results corresponding to each of the plurality of continuous frame segments, the evaluation result satisfying the preset condition as the evaluation result to be determined; The repeated scene label in the scene labels carried by the road data corresponding to the to-be-determined evaluation result is determined as the target scene label.

4. The method according to claim 2, characterized in that: After determining the scene label carried by the road data corresponding to the evaluation result that meets the preset condition as the target scene label, the method includes: Selecting target road data collected in a single frame from the road data containing the target scene label; Determine the model prediction result and the true value data corresponding to the target road data, and match the model prediction result and the true value data corresponding to the target road data to obtain a single-frame evaluation result.

5. The method according to claim 2, characterized in that: The matching of the model prediction result corresponding to the road data with the true value data and evaluating the perception model to be evaluated to obtain the evaluation result includes: Determining a distance between at least one reference obstacle in the road data and the vehicle according to true value data corresponding to the road data, wherein the reference obstacle is used to indicate static obstacles and dynamic obstacles around the vehicle; Differentiate the at least one reference obstacle into at least one reference obstacle set according to the distance between the at least one reference obstacle and the vehicle and a preset distance threshold; The perception model to be evaluated is evaluated in sequence according to the road data corresponding to the at least one reference obstacle set to obtain the evaluation result.

6. The method according to claim 5, characterized in that The matching of the model prediction result and the true value data corresponding to the road data includes: Matching the model prediction result corresponding to the reference obstacle with the true value data corresponding to the reference obstacle; The step of evaluating the perception model to be evaluated to obtain an evaluation result includes at least one of the following: Calculate the difference between the model prediction result and the distance in the true value data to obtain the absolute distance error; Calculate the ratio between the absolute distance error and the distance in the true value data to obtain a relative distance error; The absolute speed error is obtained by calculating the difference between the speed in the model prediction result and the speed in the true value data.

7. The method according to any one of claims 1 to 6, characterized in that The obtaining of training data including the target scene label and training the perception model to be evaluated includes: Determine target category and target distance threshold interval; Acquire target road data including the target scene label and the target category within the target distance threshold interval, and determine the target road data as the training data; The training data is used to train the perception model to be evaluated.

8. The method according to any one of claims 1 to 6, characterized in that Before inputting the road data collected by the vehicle in the plurality of continuous frame segments into the perception model to be evaluated, the method includes: Screening the reference road data to determine road data that meets preset data conditions, wherein the reference road data is road data continuously collected by the vehicle; Cutting the road data that meets the preset data condition in the reference road data to obtain the road data in the multiple continuous frame segments that are consistent in time; Scene recognition is performed on the road data to obtain the corresponding scene label, and the scene label is added to the road data to obtain updated road data.

9. The method according to any one of claims 1 to 6, characterized in that: Before matching the model prediction result corresponding to the road data with the true value data, the method includes: Inputting the road data collected by the vehicle in the plurality of continuous frame segments into a truth model to obtain two-dimensional truth data and three-dimensional truth data corresponding to the road data; The two-dimensional true value data and the three-dimensional true value data are associated to obtain the true value data corresponding to the road data.

10. A perception model evaluation device, characterized in that: include: A prediction unit, used to input road data collected by the ego vehicle in a plurality of continuous frame segments into the perception model to be evaluated, and obtain a plurality of corresponding model prediction results, wherein the road data includes a scene label for indicating a driving scene of the ego vehicle in one of the continuous frame segments; A model evaluation unit, used to associate the two-dimensional true value data with the three-dimensional true value data to obtain the true value data corresponding to the road data, match the model prediction result corresponding to the road data with the true value data, and evaluate the perception model to be evaluated to obtain an evaluation result; A data acquisition unit, configured to determine the scene label carried by the road data corresponding to the evaluation result that meets the preset conditions as a target scene label, acquire training data containing the target scene label and train the perception model to be evaluated; The data acquisition unit is further used to: obtain a category evaluation result using the road data collected by the multiple continuous frame segments, and determine the perception target category corresponding to the result less than a preset threshold in the category evaluation result as the perception target category to be evaluated, wherein the category evaluation result is used to indicate the evaluation results of all objects and participants around the self-vehicle of a specific category in different scenarios; Determine an evaluation result of the perceived target category to be evaluated on a single data segment, and determine the scene label carried by the road data corresponding to the evaluation result that meets a preset condition as a target scene label; Among them, determining the evaluation results of the continuous frame segments includes: determining the distance between at least one reference obstacle in the road data and the vehicle itself according to the true value data corresponding to the road data collected by the continuous frame segments, and dividing the reference obstacles into reference obstacle sets according to the distance between the reference obstacles and the vehicle itself, and evaluating the perception model to be evaluated in turn according to the road data corresponding to at least one reference obstacle set to obtain the evaluation results.

11. A computer-readable storage medium, characterized in that: The computer-readable storage medium includes a stored program, wherein the program executes the method described in any one of claims 1 to 9 when executed.

12. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instructions are executed by a processor, the method according to any one of claims 1 to 9 is implemented.

13. An electronic device comprising a memory and a processor, characterized in that: A computer program is stored in the memory, and the processor is configured to execute the method according to any one of claims 1 to 9 through the computer program.

Images