Sensor evaluation method, device and system and server

Abstract

The invention discloses a sensor evaluation method, device and system and a server. The method comprises the following steps: determining a test environment and test time; if the test environment and the test time are met, synchronously collecting sensing data in the test environment through the truth value sensing equipment group and the sensor group to be tested; performing data processing on the first original data collected by the truth value sensing equipment group to obtain a truth value data group; performing data processing on second original data acquired by the sensor group to be detected to obtain a data group to be detected; associating the truth value data set with the to-be-tested data set through a data association algorithm to obtain an associated evaluation index; and determining an evaluation result of the to-be-detected sensor group at least based on a comparison result of the associated evaluation index and a preset value.

Description

Technical Field

[0001] This application relates to the field of sensor evaluation, and particularly to a sensor evaluation method, device, system and server. Background Art

[0002] The sensing module is one of the basic modules of autonomous driving, and accurate sensing ability is crucial for the autonomous driving of vehicles. Different types of sensors have different sensing performances. In order to meet the requirements of sensing accuracy and cost, it is necessary to select appropriate sensors.

[0003] Currently, in order to meet the requirements of sensing performance, multiple different sensors are usually selected for sensing fusion, which requires evaluating the detection accuracy of the sensors for sensing fusion. Summary of the Invention

[0004] In view of this, this application provides a sensor evaluation method, device, system and server, and the specific solutions are as follows:

[0005] A sensor evaluation method includes:

[0006] Determine the test environment and test time;

[0007] If the test environment and test time are satisfied, synchronously collect the sensing data in the test environment through the ground truth sensing device group and the to-be-tested sensor group;

[0008] Process the first raw data collected by the ground truth sensing device group to obtain a ground truth data group; process the second raw data collected by the to-be-tested sensor group to obtain a to-be-tested data group;

[0009] Associate the ground truth data group with the to-be-tested data group through a data association algorithm to obtain an associated evaluation index;

[0010] Determine the evaluation result of the to-be-tested sensor group at least based on the comparison result between the associated evaluation index and a preset value.

[0011] Further, the step of associating the ground truth data group with the to-be-tested data group through a data association algorithm to obtain an associated evaluation index includes:

[0012] Compare whether the actual target object in the ground truth data group and the measured target object in the to-be-tested data group are the same target object through a data association algorithm, and determine the comparison result as the associated evaluation result.

[0013] Further, the step of determining the evaluation result of the to-be-tested sensor group at least based on the comparison result between the associated evaluation index and a preset value includes:

[0014] Determine the ratio of the number of measurement target objects in the to-be-tested data group to the number of actual target objects in the true value data group as the detection rate;

[0015] Compare the detection rate with the detection rate threshold to determine the evaluation result of the detection rate of the to-be-tested sensor group.

[0016] Further, determining the evaluation result of the to-be-tested sensor group based at least on the comparison result between the associated evaluation index and the preset value includes:

[0017] Determine the false alarm rate as the ratio of the number of measurement target objects in the to-be-tested data group that do not match the actual target objects in the true value data group to the number of all measurement target objects in the to-be-tested data group;

[0018] Compare the false alarm rate with the false alarm rate threshold to determine the evaluation result of the false alarm rate of the to-be-tested sensor group.

[0019] Further, determining the evaluation result of the to-be-tested sensor group based at least on the comparison result between the associated evaluation index and the preset value includes:

[0020] Determine the motion information comparison value as the ratio or difference of the target motion information of the measurement target objects in the to-be-tested data group to the target motion information of the actual target objects in the true value data group;

[0021] Compare the motion information comparison value with the motion information comparison threshold to determine the evaluation result of the motion information of the to-be-tested sensor group;

[0022] Wherein, the motion information at least includes: the position, speed, acceleration or angular velocity information of the target object.

[0023] Further, associating the true value data group with the to-be-tested data group through a data association algorithm includes:

[0024] Align the true value data group obtained at the test time with the to-be-tested data group in terms of time;

[0025] Associate the true value data in the true value data group corresponding to each frame timestamp with the measurement data in the to-be-tested data group through a data association algorithm.

[0026] Further, processing the first raw data collected by the true value perception device group to obtain a true value data group includes:

[0027] Obtain the point cloud data, image data and millimeter wave data collected by the true value perception devices in the true value perception device group;

[0028] Identifying the type of the target object and tracking the trajectory of the target object from the point cloud data through deep learning;

[0029] Correcting the type of the identified target object from the image data;

[0030] Correcting the trajectory of the target object from the millimeter wave data to obtain a true value data set with the type and trajectory correction of the target object completed.

[0031] A sensor evaluation system, comprising:

[0032] A true value perception device group, used to collect perception data in the test environment at the test time to obtain first raw data;

[0033] A sensor under test group, used to collect perception data in the test environment at the test time to obtain second raw data;

[0034] A server, used to determine the test environment and test time; if the test environment and test time are satisfied, collect the perception data in the test environment synchronously through the true value perception device group and the sensor under test group; process the first raw data collected by the true value perception device group to obtain a true value data set; process the second raw data collected by the sensor under test group to obtain a data set to be tested; associate the true value data set with the data set to be tested through a data association algorithm to obtain an associated evaluation index; determine the evaluation result of the sensor under test group at least based on the comparison result between the associated evaluation index and a preset value.

[0035] A sensor evaluation device, comprising:

[0036] A first determination unit, used to determine the test environment and test time;

[0037] An acquisition unit, used to collect the perception data in the test environment synchronously through the true value perception device group and the sensor under test group when it is determined that the current test environment and test time are satisfied;

[0038] A processing unit, used to process the first raw data collected by the true value perception device group to obtain a true value data set; process the second raw data collected by the sensor under test group to obtain a data set to be tested;

[0039] An association unit, used to associate the true value data set with the data set to be tested through a data association algorithm to obtain an associated evaluation index;

[0040] A second determination unit, used to determine the evaluation result of the sensor under test group at least based on the comparison result between the associated evaluation index and a preset value.

[0041] A server, comprising:

[0042] A processor, configured to determine a test environment and a test time; if the test environment and the test time are satisfied, collect perception data in the test environment synchronously through a group of ground truth perception devices and a group of sensors to be tested; perform data processing on first raw data collected by the group of ground truth perception devices to obtain a group of ground truth data; perform data processing on second raw data collected by the group of sensors to be tested to obtain a group of data to be tested; associate the group of ground truth data with the group of data to be tested through a data association algorithm to obtain an associated evaluation index; determine an evaluation result of the group of sensors to be tested at least based on a comparison result between the associated evaluation index and a preset value;

[0043] A memory, configured to store a program for the processor to execute the above processing procedure.

[0044] As can be seen from the above technical solutions, the sensor evaluation method, device, system and server disclosed in this application determine a test environment and a test time; if the test environment and the test time are satisfied, collect perception data in the test environment synchronously through a group of ground truth perception devices and a group of sensors to be tested; perform data processing on first raw data collected by the group of ground truth perception devices to obtain a group of ground truth data; perform data processing on second raw data collected by the group of sensors to be tested to obtain a group of data to be tested; associate the group of ground truth data with the group of data to be tested through a data association algorithm to obtain an associated evaluation index; determine an evaluation result of the group of sensors to be tested at least based on a comparison result between the associated evaluation index and a preset value. This solution obtains the data collected by the group of ground truth perception devices and the group of sensors to be tested respectively under the same test environment and test time, associates the obtained data, obtains an associated evaluation instruction, and determines the evaluation result of the group of sensors to be tested based on the associated evaluation index, so as to realize the evaluation of the detection accuracy of the sensor group for perception fusion. Description of the Drawings

[0045] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings required for use in the description of the embodiments or the prior art. Obviously, the drawings in the following description are only some embodiments of the present application. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0046] Figure 1 It is a flowchart of a sensor evaluation method disclosed in an embodiment of the present application;

[0047] Figure 2 It is a flowchart of a sensor evaluation method disclosed in an embodiment of the present application;

[0048] Figure 3A schematic diagram of the motion information of a certain target object in a real data group and a to-be-tested data group at discrete timestamps disclosed in an embodiment of the present application;

[0049] Figure 4 A schematic structural diagram of a sensor evaluation system disclosed in an embodiment of the present application;

[0050] Figure 5 A schematic structural diagram of a sensor evaluation device disclosed in an embodiment of the present application;

[0051] Figure 6 A schematic structural diagram of a server disclosed in an embodiment of the present application. Detailed implementation manners

[0052] Next, the technical solutions in the embodiments of the present application will be clearly and completely described with reference to the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. All other embodiments obtained by those of ordinary skill in the art based on the embodiments in the present application without creative efforts shall fall within the protection scope of the present application.

[0053] The present application discloses a sensor evaluation method, and its flowchart is as Figure 1 shown, including:

[0054] Step S11, determining a test environment and a test time;

[0055] Step S12, if the test environment and the test time are satisfied, synchronously collecting perception data in the test environment through a ground truth perception device group and a to-be-tested sensor group;

[0056] Step S13, performing data processing on the first raw data collected by the ground truth perception device group to obtain a ground truth data group; performing data processing on the second raw data collected by the to-be-tested sensor group to obtain a to-be-tested data group;

[0057] Step S14, associating the ground truth data group with the to-be-tested data group through a data association algorithm to obtain an associated evaluation index;

[0058] Step S15, determining the evaluation result of the to-be-tested sensor group at least based on the comparison result between the associated evaluation index and a preset value.

[0059] For an intelligent driving vehicle, the perception module is one of the important modules. The perception module usually consists of multiple sensors, and the perception of the surrounding road conditions is achieved through the data collected by the multiple sensors respectively, so as to achieve the purpose of safe driving of the intelligent driving vehicle.

[0060] In this embodiment, the overall perception accuracy of the perception module is evaluated by evaluating the data collected by the sensor group.

[0061] Set the test environment and test time. When the test environment is met and the test time is reached, test the sensor group to be tested in the perception module. The test can be specifically as follows: Collect the perception data in the test environment through the true value perception device group and the sensor group to be tested respectively, and perform correlation comparison on the collected data respectively, so as to obtain the evaluation result of the sensor group to be tested and determine the perception accuracy of the perception module.

[0062] Among them, the test environment and test time come from the scenario requirements of the business side or the requirement side, and targeted scenario data collection and evaluation are carried out.

[0063] Among them, the perception data at least includes: point cloud data, image data and millimeter wave data. Correspondingly, the sensor group to be tested at least includes: lidar sensors, image acquisition sensors, millimeter wave radar sensors, etc. In addition, the sensor group to be tested can also include: carrier phase differential sensor RTK, etc. Among them, the lidar sensor can be a high-line beam lidar sensor, and the image acquisition sensor can be a high-definition image acquisition sensor.

[0064] Correspondingly, the true value perception device group also at least includes the above true value sensors, such as: true value lidar sensors, true value image acquisition sensors, true value millimeter wave radar sensors, etc.

[0065] Under the same test environment and test time, collect the perception data through the true value perception device group and the sensor group to be tested respectively, so that the scenario and time corresponding to the data collected by the sensor group to be tested are exactly the same as the scenario and time corresponding to the data collected by the true value perception device group. Only when the scenario and time are exactly the same can there be a basis for comparison and evaluation.

[0066] After obtaining the first raw data collected by the true value perception device group and the second raw data collected by the sensor group to be tested, process the first raw data and the second raw data respectively to obtain the true value data group and the data group to be tested.

[0067] Among them, the true value data group can include: the data obtained by identifying and correcting the target type, tracking and correcting the trajectory of all the collected real point cloud data, real image data and real millimeter wave data.

[0068] Specifically, obtain the point cloud data, image data and millimeter wave data collected by the true value perception devices in the true value perception device group. Identify the type of the target object and track the trajectory of the target object through the point cloud data by means of deep learning. Correct the type of the identified target object through the image data, and correct the trajectory of the target object through the millimeter wave data to obtain the true value data group with the type and trajectory of the target object corrected.

[0069] Among them, lidar sensors, image acquisition sensors, millimeter-wave radar sensors, carrier-phase differential sensors RTK, the to-be-tested sensor group, and industrial control computers, 4G modules, switches and other hardware are deployed on the vehicle side. The carrier-phase differential sensor RTK obtains satellite time and provides timing for the industrial control computer. The lidar sensors, millimeter-wave radar sensors, and the to-be-tested sensor group are connected to the industrial control computer to obtain system time. The lidar sensor provides timing for the image acquisition sensor through a radar trigger device, so that all sensors meet the requirements of time synchronization.

[0070] That is, the data included in the true value data group are the data obtained after identifying and correcting the data collected by the true value sensing device. At this time, the data in the obtained true value data group are the sensing data after multi-sensor fusion;

[0071] Similarly, the to-be-tested data group includes: the data obtained after identifying and correcting the target type, tracking and correcting the trajectory of all collected measurement point cloud data, measurement image data, and measurement millimeter-wave data.

[0072] That is: Obtain the measurement point cloud data, measurement image data, and measurement millimeter-wave data collected by the to-be-tested sensors in the to-be-tested sensor group. Identify the type of target object and track the trajectory of the target object in the measurement point cloud data through deep learning. Correct the type of the identified target object through the measurement image data, and correct the trajectory of the target object through the strategic millimeter-wave data to obtain the to-be-tested data group with the type and trajectory of the target object corrected.

[0073] After obtaining the true value data group and the to-be-tested data group, associate the true value data group with the to-be-tested data group through a data association algorithm to obtain an association evaluation index, and determine the evaluation result of the to-be-tested sensor group based on the association evaluation index.

[0074] Associating the true value data group with the to-be-tested data group through a data association algorithm can be specifically: Compare whether the actual target object in the true value data group and the measurement target object in the to-be-tested data group are the same target object through the data association algorithm, and determine the comparison result as the association evaluation result.

[0075] The true value data group includes the target object, and the data group to be measured also includes the target object. Compare whether the target object in the true value data group and the target object in the data group to be measured are the same target object, so as to determine the associated evaluation index; compare the associated evaluation index with the preset value. If the associated evaluation index is greater than the preset value, it can be determined that the evaluation result of the sensor group to be measured is passed and the accuracy is relatively high. If the associated evaluation index is not greater than the preset value, it can be determined that the evaluation result of the sensor group to be measured is not passed and the accuracy is relatively low, so as to adjust the sensors in the sensor group to be measured based on this evaluation result, thereby improving the detection accuracy of the sensor group to be measured.

[0076] The sensor evaluation method disclosed in this embodiment determines the test environment and test time; if the test environment and test time are satisfied, the true value perception device group and the sensor group to be measured are used to synchronously collect the perception data in the test environment; the first raw data collected by the true value perception device group is processed to obtain the true value data group; the second raw data collected by the sensor group to be measured is processed to obtain the data group to be measured; the true value data group and the data group to be measured are associated through a data association algorithm to obtain an associated evaluation index; at least based on the comparison result of the associated evaluation index and the preset value, the evaluation result of the sensor group to be measured is determined. This solution obtains the data collected by the true value perception device group and the sensor group to be measured respectively under the same test environment and test time, associates the obtained data to obtain an associated evaluation instruction, and determines the evaluation result of the sensor group to be measured based on the associated evaluation index, so as to realize the evaluation of the detection accuracy of the sensor group for perception fusion.

[0077] This embodiment discloses a sensor evaluation method, and its flowchart is as Figure 2 shown, including:

[0078] Step S21: Determine the test environment and test time;

[0079] Step S22: If the test environment and test time are satisfied, the true value perception device group and the sensor group to be measured are used to synchronously collect the perception data in the test environment;

[0080] Step S23: Process the first raw data collected by the true value perception device group to obtain the true value data group, and process the second raw data collected by the sensor group to be measured to obtain the data group to be measured;

[0081] Step S24: Associate the true value data group and the data group to be measured through a data association algorithm to obtain an associated evaluation index;

[0082] Step S25: Determine the detection rate as the ratio of the number of measurement target objects in the data group to be measured to the number of actual target objects in the true value data group;

[0083] Step S26: Compare the detection rate with the detection rate threshold to determine the evaluation result of the detection rate of the sensor group to be tested.

[0084] There can be multiple associated evaluation metrics, such as: detection rate, missed detection rate, false alarm rate, motion information comparison value, etc.

[0085] Two or more of the above multiple metrics can be used as the associated evaluation metrics for the current sensor group to be tested simultaneously, or only one of the above multiple metrics can be selected as the associated evaluation metric for the current sensor group to be tested. The selection of the associated evaluation metric is related to the application scenario or function of the sensor to be tested.

[0086] When determining the associated evaluation metric, first align the coordinates and time of the data group to be tested and the true value data group. Aligning the coordinates can make the positions corresponding to the data collected by the data group to be tested exactly correspond to the positions corresponding to the data collected by the true value data group. Aligning the time can make each frame of data collected by the data group to be tested correspond to each frame of data collected by the true value data group in time.

[0087] Both the data group to be tested and the true value data group include data corresponding to multiple different frames. Making each frame of data in the data group to be tested correspond exactly to each frame of data in the true value data group in time can make the actions of the target object collected by the data group to be tested theoretically exactly the same as those of the target object collected by the true value data group. Whether they are actually the same or not requires comparing the data corresponding to each frame timestamp, so as to obtain a comparison result for each frame timestamp.

[0088] If the associated evaluation metric is the detection rate, or at least includes the detection rate, the detection rate can be determined first. The detection rate is the ratio of the number of measured target objects in the data group to be tested to the number of actual target objects in the true value data group. Compare this ratio with the detection rate threshold to determine the evaluation result of the detection rate of the sensor to be tested.

[0089] The detection rate is the ratio of the target objects detected by the sensor group to be tested that exactly match the real target objects.

[0090] Determining the detection rate needs to be based on the comparison results obtained for each frame timestamp. Determine whether the measured target objects detected by the sensor group to be tested in each frame timestamp are the same as the actual target objects. If they are the same, it is determined that the sensor to be tested detects the target object in this frame timestamp. If they are not the same, it is determined that the sensor to be tested does not detect the target object in this frame timestamp.

[0091] Afterwards, the number of all detected target objects is determined as the number of measured target objects in the measured data set collected by the sensor under test at the test time, which is used as the dividend for determining the ratio of the detection rate. The number of actual target objects collected by the ground truth sensor at the test time is used as the divisor for determining the ratio of the detection rate. The ratio obtained by dividing the dividend by the divisor is determined as the detection rate.

[0092] If only the detection rate is used as the associated evaluation index, when the detection rate is greater than the detection rate threshold, it can be determined that the detection accuracy of the sensor group under test is relatively high; if the detection rate is not greater than the detection rate threshold, it can be determined that the detection accuracy of the sensor group under test is relatively low. If there are multiple associated evaluation indexes, when the detection rate is not greater than the detection rate threshold, it can also be determined that the detection rate of the sensor group under test is relatively low.

[0093] In addition, the false negative rate can also be used as at least one of the associated evaluation indexes, that is: determining the false negative rate also needs to be based on the comparison results obtained for each frame timestamp. For each frame timestamp, it is determined whether the measured target objects detected by the sensor group under test are the same as the actual target objects. If they are not the same, it is determined that the measured target objects detected by the sensor group under test in this frame timestamp do not match the actual target objects. Determine the ratio of the number of measured target objects detected by the sensor group under test that do not match the actual target objects to the number of actual target objects during the test time. Or, directly after determining the number of measured target objects collected by the sensor under test that are the same as the actual target objects during the test time, subtract the number of measured target objects collected by the sensor under test that are the same as the actual target objects from the number of actual target objects detected by the ground truth sensor during the test time. The result obtained is the number of measured target objects collected by the sensor group under test that do not match the actual target objects, and then divide it by the number of actual target objects to obtain the false negative rate. Or, the false negative rate can also be directly obtained by subtracting the detection rate from 1.

[0094] Compare the false negative rate with the false negative rate threshold. If the false negative rate is less than the false negative rate threshold, it can be determined that the false negative rate of the sensor group under test is relatively low and meets the detection accuracy standard; if the false negative rate is not less than the false negative rate threshold, it can be determined that the false negative rate of the sensor group under test is relatively high and does not meet the detection accuracy standard.

[0095] In addition, the false alarm rate can also be used as at least one of the associated evaluation indexes, that is: determine the false alarm rate as the ratio of the number of measured target objects in the measured data set that do not match the actual target objects in the ground truth data set to the number of all measured target objects in the measured data set; compare the false alarm rate with the false alarm rate threshold to determine the evaluation result of the false alarm rate of the sensor group under test.

[0096] Among them, the difference between the false alarm rate and the missed alarm rate is as follows: the divisor of the false alarm rate is the number of all measurement target objects in the data group to be measured, while the divisor of the missed alarm rate is the number of all actual target objects in the true value data group; and for both the false alarm rate and the missed alarm rate, the dividend is the number of measurement target objects that match the actual target objects in the data group to be measured.

[0097] The false alarm rate can be used to determine the probability of the target objects detected incorrectly by the sensor to be measured itself, rather than the gap between it and the true value sensing device group.

[0098] Furthermore, the evaluation result of the sensor group to be measured can also be determined by the motion information comparison value.

[0099] That is: the ratio or difference between the target motion information of the measurement target objects in the data group to be measured and the target motion information of the actual target objects in the true value data group is determined as the motion information comparison value; the motion information comparison value is compared with the motion information comparison threshold to determine the evaluation result of the motion information of the sensor group to be measured; among them, the motion information at least includes: the position, speed, acceleration or angular velocity information of the target object.

[0100] The motion information can be: the size, position, speed, heading, yaw, acceleration, angular velocity, trajectory and their respective covariances of the target object.

[0101] By analyzing and processing the values detected by each sensor to be measured in the sensor group to be measured, the above-mentioned motion information of the measurement target object is obtained. Similarly, by analyzing and processing the values collected by each true value sensing device in the true value sensing device group, the above-mentioned motion information of the actual target object is obtained. Then, the motion information of the measurement target object is compared with the motion information of the actual target object to determine the evaluation result of the sensor group to be measured.

[0102] Comparing the motion information of the measurement target object with the motion information of the actual target object can be: comparing the above-mentioned various motion information one by one. When the ratio or difference between the measured motion information and the true motion information corresponding to each type of motion information is greater than its comparison threshold, it can be determined that the detection accuracy of this type of motion information is relatively low. When the number of types of motion information with relatively low detection accuracy reaches a certain specific threshold, it can be determined that the sensor group to be measured fails the evaluation.

[0103] Specifically, as Figure 3 shown, it is a schematic diagram of the motion information of a certain target object in the true data group and the data group to be measured at discrete time stamps. The distance between the points of the two curves at the same time stamp represents the detection accuracy of the sensor to be measured for this motion information.

[0104] Among them, GT represents the true value data group, DUT represents the data group to be measured, t is the discrete time stamp, and s is the motion information of the target object. The motion information can be information such as size, position, speed, heading, yaw, acceleration, angular velocity, etc.

[0105] Taking the motion information as acceleration as an example for illustration, when the time stamp is 100, the ordinate of the point corresponding to the GT curve at this time stamp is greater than the ordinate of the point corresponding to the DUT curve at this time stamp. That is, at this moment, the acceleration of the data group to be measured at this moment is less than the acceleration of the true value data group at this moment. The difference between the two can be determined according to the specific values; within the time period from 0 to 300 of the time stamp, the difference between the two curves is relatively small, while within the time period from 300 to 400 of the time stamp, the difference between the two curves is relatively large. The difference in acceleration between the two curves within this time period can be determined, so as to determine the detection accuracy of the sensor to be measured.

[0106] In addition, based on the analysis of Figure 3 the shown curve graph, it can be determined that the detection accuracy of the sensor to be measured is different in different time periods, and the reason for the different detection accuracies of the sensor to be measured in different time periods can be further determined based on its specific motion values.

[0107] The sensor evaluation method disclosed in this embodiment determines the test environment and test time; if the test environment and test time are met, the true value perception device group and the sensor group to be measured are used to synchronously collect the perception data in the test environment; the first raw data collected by the true value perception device group is processed to obtain the true value data group; the second raw data collected by the sensor group to be measured is processed to obtain the data group to be measured; the true value data group and the data group to be measured are associated through a data association algorithm to obtain an associated evaluation index; at least based on the comparison result between the associated evaluation index and the preset value, the evaluation result of the sensor group to be measured is determined. This solution obtains the data collected by the true value perception device group and the sensor group to be measured respectively under the same test environment and test time, associates the obtained data to obtain an associated evaluation instruction, and determines the evaluation result of the sensor group to be measured based on the associated evaluation index, so as to realize the evaluation of the detection accuracy of the perception fusion sensor group.

[0108] This embodiment discloses a sensor evaluation system, and its structural schematic diagram is as Figure 4 shown, including:

[0109] The true value perception device group 41, the sensor group 42 to be measured, and the server 43.

[0110] Among them, the true value perception device group 41 is used to collect the perception data in the test environment at the test time to obtain the first raw data;

[0111] The sensor group 42 to be tested is used to collect perception data in a test environment during a test time to obtain second raw data;

[0112] The server 43 is used to determine the test environment and the test time; if the test environment and the test time are met, the perception data in the test environment is collected synchronously through the ground truth perception device group and the sensor group to be tested; the first raw data collected by the ground truth perception device group is processed to obtain a ground truth data group; the second raw data collected by the sensor group to be tested is processed to obtain a to-be-tested data group; the ground truth data group and the to-be-tested data group are associated through a data association algorithm to obtain an associated evaluation index; at least based on the comparison result between the associated evaluation index and a preset value, the evaluation result of the sensor group to be tested is determined.

[0113] The sensor evaluation system disclosed in this embodiment is implemented based on the sensor evaluation method disclosed in the above embodiment, and will not be elaborated here.

[0114] The sensor evaluation system disclosed in this embodiment determines the test environment and the test time; if the test environment and the test time are met, the perception data in the test environment is collected synchronously through the ground truth perception device group and the sensor group to be tested; the first raw data collected by the ground truth perception device group is processed to obtain a ground truth data group; the second raw data collected by the sensor group to be tested is processed to obtain a to-be-tested data group; the ground truth data group and the to-be-tested data group are associated through a data association algorithm to obtain an associated evaluation index; at least based on the comparison result between the associated evaluation index and a preset value, the evaluation result of the sensor group to be tested is determined. This solution obtains the data collected by the ground truth perception device group and the sensor group to be tested respectively under the same test environment and test time, associates the obtained data to obtain an associated evaluation instruction, and determines the evaluation result of the sensor group to be tested based on the associated evaluation index, so as to realize the evaluation of the detection accuracy of the perception fusion sensor group.

[0115] This embodiment discloses a sensor evaluation device, and its structural schematic diagram is as Figure 5 shown, including:

[0116] A first determination unit 51, a collection unit 52, a processing unit 53, an association unit 54 and a second determination unit 55.

[0117] Among them, the first determination unit 51 is used to determine the test environment and the test time;

[0118] The collection unit 52 is used to synchronously collect the perception data in the test environment through the ground truth perception device group and the sensor group to be tested when it is determined that the current test environment and test time are met;

[0119] The processing unit 53 is used to process the first raw data collected by the ground truth perception device group to obtain a ground truth data group; and process the second raw data collected by the sensor under test group to obtain a data group to be measured.

[0120] The association unit 54 is used to associate the ground truth data group with the data group to be measured through a data association algorithm to obtain an associated evaluation index.

[0121] The second determination unit 55 is used to determine the evaluation result of the sensor under test group at least based on the comparison result between the associated evaluation index and a preset value.

[0122] The sensor evaluation device disclosed in this embodiment is implemented based on the sensor evaluation method disclosed in the above embodiment, and will not be elaborated here.

[0123] The sensor evaluation device disclosed in this embodiment determines the test environment and test time; if the test environment and test time are satisfied, the ground truth perception device group and the sensor under test group are used to synchronously collect the perception data in the test environment; the first raw data collected by the ground truth perception device group is processed to obtain a ground truth data group; the second raw data collected by the sensor under test group is processed to obtain a data group to be measured; the ground truth data group and the data group to be measured are associated through a data association algorithm to obtain an associated evaluation index; and the evaluation result of the sensor under test group is determined at least based on the comparison result between the associated evaluation index and a preset value. This solution obtains the data collected by the ground truth perception device group and the sensor under test group respectively in the same test environment and test time, associates the obtained data to obtain an associated evaluation instruction, and determines the evaluation result of the sensor under test group based on the associated evaluation index, so as to realize the evaluation of the detection accuracy of the sensor group for perception fusion.

[0124] This embodiment discloses a server, and its structural schematic diagram is as Figure 6 shown, including:

[0125] A processor 61 and a memory 62.

[0126] Among them, the processor 61 is used to determine the test environment and test time; if the test environment and test time are satisfied, the ground truth perception device group and the sensor under test group are used to synchronously collect the perception data in the test environment; the first raw data collected by the ground truth perception device group is processed to obtain a ground truth data group; the second raw data collected by the sensor under test group is processed to obtain a data group to be measured; the ground truth data group and the data group to be measured are associated through a data association algorithm to obtain an associated evaluation index; and the evaluation result of the sensor under test group is determined at least based on the comparison result between the associated evaluation index and a preset value;

[0127] The memory 62 is used to store the program for the processor to execute the above processing procedure.

[0128] The server disclosed in this embodiment is implemented based on the sensor evaluation method disclosed in the above embodiment, which will not be elaborated here.

[0129] The server disclosed in this embodiment determines the test environment and test time; if the test environment and test time are met, the true value perception device group and the to-be-tested sensor group are used to synchronously collect the perception data in the test environment; the first raw data collected by the true value perception device group is processed to obtain a true value data group; the second raw data collected by the to-be-tested sensor group is processed to obtain a to-be-tested data group; the true value data group and the to-be-tested data group are associated through a data association algorithm to obtain an associated evaluation index; at least based on the comparison result between the associated evaluation index and a preset value, the evaluation result of the to-be-tested sensor group is determined. This solution obtains the data collected by the true value perception device group and the to-be-tested sensor group respectively under the same test environment and test time, associates the obtained data to obtain an associated evaluation instruction, and determines the evaluation result of the to-be-tested sensor group based on the associated evaluation index, so as to realize the evaluation of the detection accuracy of the sensor group for perception fusion.

[0130] The embodiment of the present application also provides a readable storage medium, on which a computer program is stored. The computer program is loaded and executed by a processor to implement the steps of the above sensor evaluation method. The specific implementation process can refer to the description of the corresponding part of the above embodiment, and will not be elaborated in this embodiment.

[0131] The present application also proposes a computer program product or a computer program. The computer program product or the computer program includes computer instructions, and the computer instructions are stored in a computer-readable storage medium. The processor of the electronic device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the electronic device executes the methods provided in various optional implementation manners in the above aspects of the sensor evaluation method or the sensor evaluation system. The specific implementation process can refer to the description of the corresponding embodiment above and will not be elaborated.

[0132] The various embodiments in this specification are described in a progressive manner. Each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the description of the method part.

[0133] Those skilled in the art may further realize that the units and algorithm steps of each example described in combination with the embodiments disclosed herein can be implemented by electronic hardware, computer software, or a combination of the two. To clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described according to functions in the above description. Whether these functions are executed in a hardware or software manner depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods for each specific application to implement the described functions, but such implementation should not be considered to exceed the scope of this application.

[0134] The steps of the methods or algorithms described in combination with the embodiments disclosed herein can be directly implemented by hardware, software modules executed by a processor, or a combination of the two. The software modules can be placed in a random access memory (RAM), internal memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, removable disk, CD-ROM, or any other form of storage medium well-known in the technical field.

[0135] The above description of the disclosed embodiments enables those skilled in the art to implement or use this application. Various modifications to these embodiments will be obvious to those skilled in the art, and the general principles defined herein can be implemented in other embodiments without departing from the spirit or scope of this application. Therefore, this application will not be limited to these embodiments shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for evaluating sensors, characterized in that, it includes: Determine the test environment and test time; If the test environment and test time are met, synchronously collect the sensed data in the test environment through the ground truth sensing device group and the sensor group to be tested; Perform data processing on the first raw data collected by the ground truth sensing device group to obtain a ground truth data group; perform data processing on the second raw data collected by the sensor group to be tested to obtain a data group to be tested; Associate the ground truth data group with the data group to be tested through a data association algorithm to obtain an associated evaluation index; Determine the evaluation result of the sensor group to be tested based at least on the comparison result between the associated evaluation index and a preset value.

2. The method according to claim 1, characterized in that, the step of associating the ground truth data group with the data group to be tested through a data association algorithm to obtain an associated evaluation index includes: Compare whether the actual target object in the ground truth data group and the measured target object in the data group to be tested are the same target object through a data association algorithm, and determine the comparison result as the associated evaluation result.

3. The method according to claim 2, characterized in that, the step of determining the evaluation result of the sensor group to be tested based at least on the comparison result between the associated evaluation index and a preset value includes: Determine the ratio of the number of measured target objects in the data group to be tested to the number of actual target objects in the ground truth data group as the detection rate; Compare the detection rate with a detection rate threshold to determine the evaluation result of the detection rate of the sensor group to be tested.

4. The method according to claim 2, characterized in that, the step of determining the evaluation result of the sensor group to be tested based at least on the comparison result between the associated evaluation index and a preset value includes: Determine the ratio of the number of measured target objects in the data group to be tested that do not match the actual target objects in the ground truth data group to the number of all measured target objects in the data group to be tested as the false alarm rate; Compare the false alarm rate with a false alarm rate threshold to determine the evaluation result of the false alarm rate of the sensor group to be tested.

5. The method according to claim 1, characterized in that, the step of determining the evaluation result of the sensor group to be tested based at least on the comparison result between the associated evaluation index and a preset value includes: Determine the ratio or difference between the target motion information of the measured target objects in the data group to be tested and the target motion information of the actual target objects in the ground truth data group as the motion information comparison value; Compare the motion information comparison value with a motion information comparison threshold to determine the evaluation result of the motion information of the sensor group to be tested; wherein, the motion information at least includes: the position, speed, acceleration or angular velocity information of the target object.

6. The method according to claim 1, characterized in that, the step of associating the ground truth data group with the data group to be tested through a data association algorithm includes: Align the ground truth data group obtained at the test time with the data group to be tested in terms of time; Associate the true value data in the true value data group corresponding to each frame of timestamp with the measurement data in the data to be measured through a data association algorithm.

7. The method according to claim 1, wherein, the data processing of the first raw data collected by the true value perception device group to obtain a true value data group includes: obtain the point cloud data, image data and millimeter wave data collected by the true value perception devices in the true value perception device group; identify the type of the target object and track the trajectory of the target object through the point cloud data in a deep learning manner; correct the type of the identified target object through the image data; correct the trajectory of the target object through the millimeter wave data to obtain a true value data group with the type and trajectory of the target object corrected.

8. A sensor evaluation system, wherein, comprising: a true value perception device group for collecting perception data in a test environment at a test time to obtain first raw data; a data to be measured sensor group for collecting perception data in a test environment at a test time to obtain second raw data; a server for determining a test environment and a test time; if the test environment and test time are satisfied, synchronously collect the perception data in the test environment through the true value perception device group and the data to be measured sensor group; perform data processing on the first raw data collected by the true value perception device group to obtain a true value data group; perform data processing on the second raw data collected by the data to be measured sensor group to obtain a data to be measured group; associate the true value data group with the data to be measured group through a data association algorithm to obtain an associated evaluation index; determine the evaluation result of the data to be measured sensor group at least based on the comparison result between the associated evaluation index and a preset value.

9. A sensor evaluation device, wherein, comprising: a first determination unit for determining a test environment and a test time; a collection unit for synchronously collecting the perception data in the test environment through the true value perception device group and the data to be measured sensor group when it is determined that the current test environment and test time are satisfied; a processing unit for performing data processing on the first raw data collected by the true value perception device group to obtain a true value data group; perform data processing on the second raw data collected by the data to be measured sensor group to obtain a data to be measured group; an association unit for associating the true value data group with the data to be measured group through a data association algorithm to obtain an associated evaluation index; a second determination unit for determining the evaluation result of the data to be measured sensor group at least based on the comparison result between the associated evaluation index and a preset value.

10. A server, wherein, comprising: a processor for determining a test environment and a test time; if the test environment and test time are satisfied, synchronously collect the perception data in the test environment through the true value perception device group and the data to be measured sensor group; perform data processing on the first raw data collected by the true value perception device group to obtain a true value data group; Perform data processing on the second original data collected by the to-be-tested sensor group to obtain a to-be-tested data group; associate the true value data group with the to-be-tested data group through a data association algorithm to obtain an associated evaluation index; determine the evaluation result of the to-be-tested sensor group based at least on the comparison result between the associated evaluation index and a preset value; A memory for storing a program for the processor to execute the above processing procedure.

Images