Evaluation method and evaluation device for perception fusion algorithm

Abstract

Embodiments of the present disclosure provide an evaluation method and device for a perception fusion algorithm, a storage medium and an electronic device. The evaluation method comprises: synchronizing a true value sensing module and a measured sensing module arranged on a vehicle; obtaining first collection result data of the true value sensing module and second collection result data of the measured sensing module; and obtaining an evaluation result of a perception fusion algorithm to be tested based on the first collection result data and the second collection result data. Embodiments of the present disclosure can give an evaluation for the perception result of the measured perception fusion system, thereby being applied to the development and testing of a measured perception system on a vehicle to evaluate or accept the perception and fusion algorithm performance of the developed measured perception system on the vehicle.

Description

TECHNICAL FIELD

[0001] The embodiment of the present disclosure relates to the technical field of automatic driving information perception, in particular to a method and device for evaluating a perception fusion algorithm, a storage medium and an electronic device. BACKGROUND

[0002] With the rapid development of intelligent vehicles, the development of perception and fusion functions of vehicle-mounted sensing systems is becoming increasingly important. A set of automatic evaluation schemes for vehicle-mounted perception and fusion algorithm development with high detection accuracy and strict time alignment is particularly important. However, the existing technology cannot strictly synchronize the time and space between the vehicle-mounted sensing system and the true value system, which leads to the inability to efficiently and accurately evaluate the vehicle-mounted sensing module during the development of vehicle-mounted perception and fusion algorithms. SUMMARY

[0003] In view of the above problems in the prior art, the embodiment of the present disclosure provides a method and device for evaluating a perception fusion algorithm, a storage medium and an electronic device.

[0004] To solve the above technical problems, the embodiment of the present disclosure adopts the following technical solutions:

[0005] A method for evaluating a perception fusion algorithm, comprising: synchronizing a true value sensing module and a measured sensing module arranged on a vehicle; obtaining first acquisition result data of the true value sensing module and second acquisition result data of the measured sensing module; and obtaining an evaluation result of the perception fusion algorithm to be tested based on the first acquisition result data and the second acquisition result data.

[0006] In some embodiments, the true value sensing module at least includes a mechanical laser radar, a millimeter wave radar and a camera device; and the measured sensing module at least includes a solid-state laser radar, a millimeter wave radar and a camera device.

[0007] In some embodiments, the spatial synchronization of the true value sensing module and the measured sensing module arranged on the vehicle comprises: obtaining a position parameter of a sensor in the measured sensing module; and determining whether the position synchronization is successful based on the position parameter and the true value sensing module.

[0008] In some embodiments, the time synchronization of the true value sensing module and the measured sensing module arranged on the vehicle comprises: receiving a signal reference time standard through a clock source; achieving time information synchronization between the mechanical laser radar or the solid-state laser radar through a PTP network time synchronization protocol; synchronously exposing the camera device through an external trigger mode; and achieving time information synchronization with the millimeter wave radar through CANTSyn.

[0009] In some embodiments, the obtaining the first collection result data of the true value sensor module and the second collection result data of the measured sensor module comprises: respectively obtaining first collection raw data from the true value sensor module and second collection raw data from the measured sensor module; obtaining the first collection result data by a true value processing algorithm from the first collection raw data; and obtaining the second collection result data by the perception fusion algorithm from the second collection raw data.

[0010] In some embodiments, the first collection result data or the second collection result data comprises any one or a combination of multiple of target type, target ID, color, orientation, three-dimensional equation coefficients of lane line or curb, two-dimensional bounding box, three-dimensional bounding box, semantic annotation, light source state, traffic signal indication type, traffic signal duration, distance, speed, acceleration.

[0011] In some embodiments, the evaluation result is represented by a test index, and the test index comprises at least one of accuracy, precision, recall, multi-target tracking accuracy, multi-target tracking precision, average precision, average intersection over union value, distance precision, speed precision, acceleration precision, average precision, average recall, average average precision, and average average recall.

[0012] The present disclosure also provides an evaluation device for a perception fusion algorithm, comprising: a synchronization module configured to synchronize a true value sensor module and a measured sensor module arranged on a vehicle; a first obtaining module configured to obtain first collection result data of the true value sensor module and second collection result data of the measured sensor module; and a second obtaining module configured to obtain an evaluation result of a perception fusion algorithm to be tested based on the first collection result data and the second collection result data.

[0013] The present disclosure also provides a storage medium storing a computer program, wherein the computer program is executed by a processor to implement the steps of any of the above methods.

[0014] The present disclosure also provides an electronic device comprising at least a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of any of the above methods when executing the computer program stored in the memory.

[0015] The beneficial effect of the embodiments of the present disclosure is that the embodiments of the present disclosure obtain the collection data of the true value sensing module and the collection data of the measured sensing module based on the same space-time standard by time synchronization and space synchronization of the measured sensing module and the true value sensing module, further obtain the true value perception result by running the true value algorithm processing of the true value sensing module, obtain the measured perception result by running the to-be-measured perception or fusion algorithm processing of the measured sensing module, compare the measured perception result data and the true value perception result data, and output the evaluation result of the measured perception system result, so as to give the evaluation for the perception result of the measured perception fusion system, thereby being applied to the development and test of the measured perception system on vehicle to evaluate or accept the performance of the perception and fusion algorithm of the developed measured perception system on vehicle. BRIEF DESCRIPTION OF DRAWINGS

[0016] In order to more clearly illustrate the technical solutions in the embodiments of the present disclosure or the prior art, the following will briefly introduce the drawings needed to be used in the embodiments or the prior art description. Obviously, the drawings in the following description are only some embodiments described in the present disclosure, and other drawings can be obtained by those skilled in the art without any creative labor.

[0017] Figure 1 The structure schematic diagram of the vehicle-mounted perception fusion evaluation system of the embodiments of the present disclosure;

[0018] Figure 2 The step schematic diagram of the evaluation method for the perception fusion algorithm of the embodiments of the present disclosure;

[0019] Figure 3 The step schematic diagram of the evaluation method for the perception fusion algorithm of the embodiments of the present disclosure. DETAILED DESCRIPTION

[0020] The various schemes and features of the present disclosure are described herein with reference to the accompanying drawings.

[0021] It should be understood that various modifications can be made to the embodiments described herein. Therefore, the above description should not be regarded as limiting, but only as an example of the embodiments. Those skilled in the art will think of other modifications within the scope and spirit of the present disclosure.

[0022] The accompanying drawings, which are included in the specification and form a part of the specification, illustrate embodiments of the present disclosure and serve to explain the principles of the present disclosure together with the above general description of the present disclosure and the following detailed description of the embodiments.

[0023] These and other characteristics of the present disclosure will become apparent from the following description of the preferred forms given, by way of non-limiting example, with reference to the attached drawings.

[0024] It should also be understood that, although the present disclosure has been described in relation to the particular examples, many other modifications and / or alternative arrangements can be utilized by one skilled in the art to practice the present disclosure as well, which are intended to be within the scope of the claims as expressed in the following claims.

[0025] The above and other aspects, features, and advantages of the present disclosure will become more apparent from the following detailed description, taken in conjunction with the accompanying drawings, which illustrate one embodiment of the present disclosure by way of example.

[0026] Specific embodiments of the present disclosure will be described hereinafter with reference to the accompanying drawings; however, it will be understood that the disclosed embodiments are merely examples of the present disclosure, which can be implemented in various ways. Well-known and / or repetitive functions and structures are not described in detail to avoid unnecessary or redundant details that can obscure the present disclosure. Therefore, specific structural and functional details disclosed herein are not intended to be limiting, but are merely representative of the present disclosure as a basis for the claims and for teaching one skilled in the art to variously employ the present disclosure in virtually any appropriate detailed structure.

[0027] The specification can use phrases such as "in one embodiment", "in another embodiment", "in yet another embodiment", or "in other embodiments", which can refer to one or more of the same or different embodiments under the present disclosure.

[0028] The first embodiment of the present disclosure is used to evaluate the accuracy of the perception system under test based on the comparison between the target real data acquired by the sensor and the target estimated data acquired by the perception fusion algorithm in the automatic driving scenario, where the core of the perception system under test is the perception fusion algorithm. Here, the target can refer to a person, object, etc. that needs to be perceived on the road during the automatic driving of the vehicle.

[0029] Specifically, in the automatic driving scenario of the vehicle, the target for perception test here includes different data such as target type, target ID, color, orientation, lane line or curb cubic equation coefficients, two-dimensional bounding box, three-dimensional bounding box, semantic labeling, light source state, traffic signal indication type, traffic signal duration, distance, speed, acceleration, etc. Of course, the data of the target for perception test is not limited to the above examples. For the above-mentioned data, specific examples are as follows,

[0030] Among them, the target type for perception test here includes, for example, the following types:

[0031] (a) Vehicle: The vehicle types here include passenger cars, trucks, buses, semi-trailer tractors, rail trains, engineering vehicles, police cars, ambulances, fire engines, school buses, water trucks, tricycles, bicycles, electric bicycles, motorcycles, etc.

[0032] (b) Pedestrian: Pedestrian here should cover various postures including standing, walking, riding, sitting, lying, holding, carrying, being held, being carried, pushing, squatting, bending, and mannequins (referring to poster figures, sculptures, model people, and other non-real people) and the like;

[0033] (c) Traffic marking: The types of traffic marking here should include all types of marking, prohibition, and warning marking specified in GB 5768.3-2009;

[0034] (d) Traffic signal light: The types of traffic signal light here should include motor vehicle signal light, non-motor vehicle signal light, left-turn non-motor vehicle signal light, pedestrian crossing signal light, lane signal light, direction indication signal light, flashing warning signal light, level crossing signal light, and U-turn signal light and the like;

[0035] (e) Temporary cover: Temporary cover here includes, for example, cone barrels, water horses, construction fences, and the like;

[0036] (f) Traffic sign: The types of traffic sign here include at least all types of warning signs, prohibition signs, and indication signs specified in GB 5768.2-2009 and the like;

[0037] (g) Road edge;

[0038] (h) Drivable area: The drivable area here refers to the identification area of the road area that the ego vehicle can pass through in the field of view.

[0039] Further, the target ID here refers to a mark for marking and tracking traffic participants such as pedestrians and vehicles.

[0040] Further, the color here includes the color of the target such as traffic signal light, traffic marking, and the like of the road, wherein the color of the traffic signal light includes red, green, and yellow, and the color of the traffic marking is generally white and yellow.

[0041] Further, the orientation here refers to the direction of travel or indication direction of the target, wherein the target includes vehicles, pedestrians, animals, traffic signal lights, traffic signs, and the like, and the orientation is specified as follows for different targets:

[0042] a) Vehicle: The orientation is given in the angle relative to the X-axis of the vehicle coordinate system (defined);

[0043] b) Pedestrian, animal: The orientation is given in the north, northeast, east, southeast, south, southwest, west, and northwest directions;

[0044] c) The orientation of the traffic signal light and the traffic sign is given in the horizontal or vertical direction.

[0045] Furthermore, the coefficients of the cubic equation for lane lines or curbs mentioned here refer to the description of lane lines or curbs using cubic equations, which are described in detail below:

[0046] a) C0: Lateral distance from the near endpoint to the origin of the vehicle coordinate system;

[0047] b) C1: Slope, indicating whether the line deviates to the left or right;

[0048] c)C2: Curvature, the radius of a circle, is used to indicate whether a curve is large or small;

[0049] d)C3: Curvature change rate, which will show a more obvious change when a large bend suddenly becomes a small bend.

[0050] Furthermore, the two-dimensional bounding box mentioned here is used to frame the position of the target object boundary in the image using a two-dimensional rectangular box.

[0051] Furthermore, the three-dimensional bounding box mentioned here is a three-dimensional rectangular box that outlines the position of the target object's boundary in the image.

[0052] Furthermore, the semantic annotation mentioned here describes the contour boundaries (driving areas, traffic markings, etc.) and center lines (lane lines, road boundaries, etc.) of the target object in the image using multi-point coordinates, and the multi-point coordinates constitute a semantic annotation point set.

[0053] Furthermore, the light source state mentioned here refers to the identification of the on and off states of artificial light sources.

[0054] Furthermore, the traffic signal indicator type mentioned here refers to the identification status of the traffic signal indicator type, which is divided into seven categories: no type, left turn, straight ahead, right turn, U-turn, no entry, and others.

[0055] Furthermore, the duration of the traffic signal light mentioned here refers to the countdown timer displayed on the traffic signal light.

[0056] Furthermore, the distances mentioned here refer to the lateral and longitudinal distances from the target to the origin of the vehicle coordinate system.

[0057] Furthermore, the speed mentioned here refers to the absolute speed of the target, including the vector speed of vehicles, pedestrians, cyclists, animals, etc. within the field of vision.

[0058] Furthermore, the acceleration mentioned here refers to the absolute acceleration of the target, including the vector acceleration of vehicles, pedestrians, cyclists, animals, etc. within the field of vision.

[0059] like Figure 1 As shown, Figure 1The structure of an evaluation system for a perception fusion algorithm in a vehicle-mounted system is shown, which includes a true value sensor module 10, a measured sensor module 20, a positioning module 30, a time synchronization module 40, and a data collection module 50; the above-mentioned modules interact to provide data for the perception fusion algorithm and the measured perception system in an autonomous driving scenario.

[0060] The true value sensor module 10 includes at least a high-line mechanical laser radar, a blind-filling radar, front and rear millimeter wave radars, an intelligent forward-looking camera, a combined inertial navigation system, a CAN card, a switch, etc.; it is used to directly obtain relevant data of the specified target, such as point cloud data, CAN target data, bus CAN data, etc.

[0061] The measured sensor module 20 includes at least a solid-state laser radar, a forward-looking camera, a rear-view camera, a side-view camera, and a millimeter wave radar, etc.; it is used to provide basic data as the input of the perception fusion algorithm, such as video data, point cloud data, CAN target data, etc.

[0062] The positioning module 30 is used to obtain the positioning information of different sensors in the vehicle, which includes at least GNSS (Global Navigation Satellite System), PPS, high-precision IMU, high-precision RTK, etc.

[0063] The time synchronization module 40 includes at least an FPGA video board card, a network card, a CAN card, and a clock source, etc.; it is used to synchronize the time information of the true value sensor module 10 and the measured sensor module 20, wherein, in the time synchronization process, the signal reference time standard of the GNSS and the PPS is received through the clock source, then the time information synchronization between the mechanical laser radar or the solid-state laser radar and other sensors is realized through the PTP network time synchronization protocol, the synchronization exposure of various cameras can also be controlled through the external trigger mode, and the time information synchronization with the millimeter wave radar can be realized through CANTSyn.

[0064] The data collection module 50 is used to collect and arrange the processed sensor data, which can include the point cloud data, CAN target data, bus CAN data, etc. collected by the true value sensor module 10, and the video data, point cloud data, CAN target data, etc. collected by the measured sensor module 20.

[0065] The evaluation system for the measured perception system of the vehicle comprises a true value post-processing module, a to-be-measured algorithm module and a comparative analysis module. The true value post-processing module is used to analyze the data collected by the true value sensing module 10 to obtain result data. The to-be-measured algorithm module is used to calculate the data collected by the measured sensing module 20 through a perception fusion algorithm to obtain result data. The comparative analysis module is used to compare the two result data to evaluate the accuracy of the perception fusion algorithm.

[0066] In the embodiments of the present disclosure, before the accuracy of the perception fusion algorithm is evaluated, the evaluation system needs to be installed on the vehicle for data collection, and specifically, the true value sensing module 10, the measured sensing module 20, the time synchronization module 40 and the like need to be installed.

[0067] The evaluation system can objectively evaluate the accuracy of the measured perception system. Specifically, as shown in Figure 2 The embodiments of the present disclosure specifically relate to an evaluation method for a perception fusion algorithm of a vehicle-mounted system, which comprises the following steps:

[0068] S101, synchronizing the true value sensing module and the measured sensing module arranged on the vehicle.

[0069] In this step, the true value sensing module and the measured sensing module arranged on the vehicle are synchronized. Specifically, in the process of evaluating the perception fusion algorithm, in order to ensure that the result data output by the perception fusion algorithm is as close to the true value as possible to improve the accuracy of the perception fusion algorithm, it is necessary to ensure that the data collected by the measured sensing module 20 is as close to the true value as possible. Therefore, it is necessary to ensure that the data collected by the measured sensing module 20 and the data collected by the true value sensing module 10 are kept in the same time and space dimensions. Therefore, the true value sensing module 10 and the measured sensing module 20 arranged on the vehicle need to be synchronized first. The synchronization here includes spatial synchronization and temporal synchronization.

[0070] The spatial synchronization of the true value sensing module 10 and the measured sensing module 20 needs to be completed under the same reference for spatial synchronization, Figure 3This document illustrates the specific process of calibrating the truth sensor module 10 and the measured sensor module 20 under the same reference, thereby achieving spatial synchronization to realize a reference coordinate system. First, the vehicle needs to be driven into the control console of the multi-sensor system calibration site. The vehicle's parking position is adjusted via the control console, for example, by centering the vehicle. Then, the OBD interface of the vehicle is connected to the control console to collect the vehicle's VIN code, sensor configuration information, etc.

[0071] like Figure 3 As shown, the specific spatial synchronization method includes the following steps:

[0072] S201, Obtain the position parameters of the sensor in the sensor module under test.

[0073] In this step, the position parameters of the sensors in the sensor module under test are acquired. These sensors may include, for example, solid-state LiDAR, a front-view camera, a rear-view camera, a side-view camera, and millimeter-wave radar. Data is collected from multiple selected sensors in the sensor module 20 on the vehicle. Simultaneously, the collected data is calibrated under a unified spatial reference based on a pre-established truth space reference to obtain the positional relationship parameters between the sensors in the sensor module under test and the extrinsic parameter matrix of each sensor in the vehicle body coordinate system. This truth space reference can be pre-determined, for example, by the positioning module 30.

[0074] S202, determine whether the position synchronization is successful based on the position parameters and the truth sensing module.

[0075] In this step, the success of the position calibration is determined based on the position parameters and the true value sensing module. Specifically, after obtaining the position parameters of the sensor in the tested sensing module 20 through step S201, the position parameters can be compared with the direct position information of the sensor in the true value tested module 10 to determine whether the position calibration was successful. If the positions are inconsistent, the position of the corresponding sensor in the tested sensing module 20 is adjusted for comparison again.

[0076] If the position calibration is successful, disconnect the OBD interface and release the vehicle's centering control. If the position calibration fails, handle the exception according to the abnormal handling procedure. For example, a calibration attempt threshold can be set; if the calibration fails after three attempts, save the sensor data and calibration failure information for later analysis of the cause of the failure. Finally, complete the vehicle multi-sensor calibration process and drive out of the vehicle.

[0077] In addition, the time synchronization module 40 can be used to synchronize the true value sensing module 10 and the measured sensing module 20 in time, so as to realize the unification in the time dimension.

[0078] The data collected by the true value sensing module 10 and the data collected by the measured sensing module 20 will be subjected to time synchronization processing of all sensors such as laser radar and all collected data such as video data by the time synchronization module 40.

[0079] In the time synchronization process, the clock source receives the signal reference time standard of the GNSS and the PPS, and then realizes the time information synchronization between the sensors such as mechanical laser radar or solid-state laser radar through the PTP network time synchronization protocol. In addition, the external trigger mode can be used to control the synchronization exposure of various cameras, and the CANTSyn can be used to realize the time information synchronization with the millimeter wave radar.

[0080] Further, the clock source data can be corrected by generating a second pulse through a time card, and then the time stamp of the collected data such as the laser radar and the video data can be corrected according to the RTK time stamp, and the time synchronization of the millimeter wave radar can be realized by the time card. For example, the time synchronization mechanism between the video and the radar is to trigger the camera and the solid-state laser radar data by rotating the high-line mechanical laser radar to a certain angle, so as to ensure that the data of each sensor can be recorded with a unified time stamp mechanism, and the synchronization accuracy is less than 1 ms, which is in the level of hundreds of microseconds. Finally, the data collected by the true value sensing module 10 and the data collected by the measured sensing module 20 have the same time stamp.

[0081] In this way, by realizing the unification in the time dimension and the space dimension between the measured sensing module and the true value sensing module, the problem that the evaluation reference value of the measured perception system cannot be strictly aligned can be solved, the automatic evaluation of the measured perception system can be realized, and the time alignment is accurate, the automation degree is high, and the test efficiency is high.

[0082] S102, acquiring the first collection result data of the true value sensing module and the second collection result data of the measured sensing module.

[0083] After the true value sensing module and the measured sensing module arranged on the vehicle are synchronized in step S101, the first collection result data of the true value sensing module and the second collection result data of the measured sensing module are acquired in this step.

[0084] Firstly, after the true value sensing module 10 and the measured sensing module 20 perform data acquisition after time and space synchronization is realized, first acquisition raw data and second acquisition raw data are obtained respectively. Further, the first acquisition raw data is input to the true value post-processing module, and first acquisition result data is obtained through a true value processing algorithm, and at the same time, the second acquisition raw data is input to the algorithm to be measured, and the perception or fusion result data to be measured, i.e. second acquisition result data, is obtained through the perception fusion algorithm to be measured.

[0085] The first acquisition raw data here refers to relevant data of a specified target, such as point cloud data, CAN target data, bus CAN data, etc. The second acquisition raw data here refers to basic data provided as an input of the perception fusion algorithm, such as video data, point cloud data, CAN target data, etc.

[0086] The first acquisition result data and the second acquisition result data here are arbitrary data based on a target, such as an arbitrary possible combination of different data of a target type, a target ID, a color, an orientation, a lane line or a curb cubic equation coefficient, a two-dimensional bounding box, a three-dimensional bounding box, semantic labeling, a light source state, a traffic signal lamp indication type, a traffic signal lamp duration, a distance, a speed, an acceleration, etc. such as the speed of a pedestrian in front, the duration of a traffic signal lamp, the distance and curvature of a lane line, etc.

[0087] S103, obtaining an evaluation result of the perception fusion algorithm to be measured based on the first acquisition result data and the second acquisition result data.

[0088] After obtaining the first acquisition result data of the true value sensing module and the second acquisition result data of the measured sensing module through the above step S102, in this step, an evaluation result of the perception fusion algorithm to be measured is obtained based on the first acquisition result data and the second acquisition result data. The evaluation result here is represented by a test index.

[0089] Specifically, the test index here can select different types of indexes according to the evaluation needs, such as accuracy, precision, recall, multi-target tracking accuracy, multi-target tracking precision, average precision, average intersection over union value, distance precision, speed precision, acceleration precision, average precision, average recall, average average precision, and average average recall, etc.

[0090] Among them, the above different test indexes are introduced respectively as follows:

[0091] 1. Accuracy (accuracy): a classification performance index, which refers to the sum of the number of targets correctly inferred by the model divided by the total number of test targets, and the formula is as follows:

[0092]

[0093] Wherein, TP refers to the prediction is positive (P), actually predicted correctly (T), that is, the correct rate of positive judgment; TN refers to the prediction is negative (N), actually predicted correctly (T), that is, the correct rate of negative judgment; FP refers to the prediction is positive (P), actually predicted incorrectly (F), false positive rate, that is, the positive judgment into negative; FN refers to the prediction is negative (N), actually predicted incorrectly (F), false negative rate, that is, the positive judgment into negative.

[0094] 2. Precision: the number of correctly inferred positive examples / the total number of inferred positive examples, also known as precision rate, the formula is as follows:

[0095]

[0096] 3. Recall: the number of correctly inferred positive examples / the total number of actual positive examples, also known as recall rate, the formula is as follows:

[0097]

[0098] 4. Accuracy of multi-target tracking (MOTA), which reflects the accuracy of determining the number of targets and related attributes of the targets, is used to count the error accumulation in tracking, including FP, FN, IDS w , the formula is as follows:

[0099]

[0100] Wherein, m t : is FN, the number of missed detections, that is, in the t frame, the target o j There is no hypothesis position matching it; fp t : is FP, the number of false detections, that is, in the t frame, the hypothesis position h j There is no tracking target matching it; mme t : is IDS w , the number of misassignments, that is, in the t frame, the number of ID switches of the tracking target, which often occurs in the case of occlusion.

[0101] 5. Accuracy of multi-target tracking (MOTP), which reflects the accuracy of determining the target position, is used to measure the accuracy of determining the target position, the formula is as follows:

[0102]

[0103] Wherein, c t : represents the number of matches between the t frame target o i And hypothesis hj. represents the target o of the t-th frame i The distance between its paired hypothesis position, i.e., the matching error.

[0104] 6、mAccuracy, the average of each target class accuracy.

[0105]

[0106] 7、mIoU: a standard measure for semantic segmentation, which calculates the intersection over union of the inferred contour area and the real area.

[0107]

[0108] 8、Distance accuracy: the percentage of distance error of the measured target in the test scene, with the denominator being the real distance between the target and the ego vehicle.

[0109]

[0110] wherein DUT x represents the measured object X direction distance; DUT y represents the measured object Y direction distance; GT x represents the measured object X direction distance true value; GT y represents the measured object X direction distance true value.

[0111] 9、Speed accuracy: the percentage of speed error of the measured target in the test scene, with the denominator being the real speed of the target.

[0112]

[0113] wherein DUT vx represents the measured object X direction speed; DUT vy represents the measured object Y direction speed GT xv represents the measured object X direction speed true value; GT vy represents the measured object X direction speed true value.

[0114] 10、Acceleration accuracy: the percentage of acceleration error of the measured target in the test scene, with the denominator being the real acceleration of the target.

[0115]

[0116] wherein DUT ax represents the measured object X direction acceleration; GT ay represents the measured object Y direction acceleration; GT ax represents the measured object X direction acceleration true value; GT ay represents the measured object X direction acceleration true value.

[0117] 11. Average precision (AP) is used to evaluate the performance of a detection algorithm for target recognition.

[0118]

[0119] 12. Average recall (AR) is used to evaluate the performance of a detection algorithm for target recognition.

[0120]

[0121] 13. Average average precision (mAP) is used to evaluate the performance of a detection algorithm for target recognition of multiple categories.

[0122] 14. Average average recall (mAR) is used to evaluate the performance of a detection algorithm for target recognition of multiple categories.

[0123] The embodiment of the present disclosure synchronizes the measured sensor module and the true value sensor module in time and space, so as to obtain the collection data of the true value sensor module and the collection data of the measured sensor module based on the same space-time standard, further processes the true value sensor module to obtain the true value perception result, processes the measured sensor module to obtain the measured perception result, compares the measured perception result data and the true value perception result data, and outputs the evaluation result of the measured perception system result, so as to evaluate the perception result of the measured perception fusion system, and thus the performance of the perception and fusion algorithm of the developed vehicle-mounted measured perception system is evaluated or accepted in the development and test of the vehicle-mounted measured perception system.

[0124] The second embodiment of the present disclosure relates to an evaluation device for a perception fusion algorithm, which comprises a synchronization module, a first acquisition module and a second acquisition module, which cooperate with each other, wherein:

[0125] The synchronization module is used to synchronize the true value sensor module and the measured sensor module arranged on the vehicle.

[0126] The first acquisition module is used to acquire the first collection result data of the true value sensor module and the second collection result data of the measured sensor module.

[0127] The second acquisition module is used to acquire the evaluation result of the measured perception fusion algorithm based on the first collection result data and the second collection result data.

[0128] Further, the true value sensor module at least comprises a mechanical laser radar, a millimeter wave radar and a camera device; and the measured sensor module at least comprises a solid-state laser radar, a millimeter wave radar and a camera device.

[0129] The synchronization module comprises:

[0130] A position acquisition unit is configured to acquire a position parameter of a sensor in the measured sensor module.

[0131] A synchronization judgment unit is configured to judge whether the position synchronization is successful based on the position parameter and the true value sensor module.

[0132] The synchronization module further comprises:

[0133] A receiving unit is configured to receive a signal reference time standard through a clock source.

[0134] A first synchronization unit is configured to realize time information synchronization between the mechanical laser radar or the solid-state laser radar through a PTP network time synchronization protocol.

[0135] A second synchronization unit is configured to control the camera to perform synchronous exposure through an external trigger mode.

[0136] A third synchronization unit is configured to realize time information synchronization between the millimeter wave radar through CANTSyn.

[0137] The first acquisition module comprises:

[0138] A collected data acquisition unit is configured to acquire first collected raw data from the true value sensor module and second collected raw data from the measured sensor module.

[0139] A first calculation unit is configured to acquire first collected result data from the first collected raw data through a true value processing algorithm.

[0140] A second calculation unit is configured to acquire second collected result data from the second collected raw data through the perception fusion algorithm.

[0141] Further, the first collected result data or the second collected result data comprises any one or a combination of multiple of target type, target ID, color, orientation, three times equation coefficient of lane line or curb, two-dimensional bounding box, three-dimensional bounding box, semantic annotation, light source state, traffic signal lamp indication type, traffic signal lamp duration, distance, speed, acceleration.

[0142] Further, the evaluation result is represented by a test index, and the test index comprises at least one of accuracy, precision, recall, multi-target tracking accuracy, multi-target tracking precision, average precision, average intersection over union value, distance precision, speed precision, acceleration precision, average precision, average recall, average average precision, and average average recall.

[0143] The embodiment of the present disclosure obtains the collection data of the true value sensing module and the collection data of the measured sensing module based on the same space-time standard by time synchronization and space synchronization of the measured sensing module and the true value sensing module, further obtains the true value perception result by running the true value algorithm processing on the true value sensing module, obtains the measured perception result by running the to-be-measured perception or fusion algorithm processing on the measured sensing module, compares the measured perception result data and the true value perception result data, and outputs the evaluation result of the measured perception system result, so as to give the evaluation for the perception result of the measured perception fusion system, thereby being applied to the development and test of the measured perception system on the vehicle to evaluate or accept the performance of the perception and fusion algorithm of the developed measured perception system on the vehicle.

[0144] The third embodiment of the present disclosure provides a storage medium, which is a computer readable medium and stores a computer program. The computer program is executed by a processor to implement the method provided by the first and third embodiments of the present disclosure, including the following steps S11 to S13:

[0145] S11, synchronizing the true value sensing module and the measured sensing module arranged on the vehicle;

[0146] S12, obtaining the first collection result data of the true value sensing module and the second collection result data of the measured sensing module;

[0147] S13, obtaining the evaluation result of the to-be-measured perception fusion algorithm based on the first collection result data and the second collection result data.

[0148] Further, the computer program is executed by the processor to implement other methods provided by the first embodiment of the present disclosure

[0149] The embodiment of the present disclosure obtains the collection data of the true value sensing module and the collection data of the measured sensing module based on the same space-time standard by time synchronization and space synchronization of the measured sensing module and the true value sensing module, further obtains the true value perception result by running the true value algorithm processing on the true value sensing module, obtains the measured perception result by running the to-be-measured perception or fusion algorithm processing on the measured sensing module, compares the measured perception result data and the true value perception result data, and outputs the evaluation result of the measured perception system result, so as to give the evaluation for the perception result of the measured perception fusion system, thereby being applied to the development and test of the measured perception system on the vehicle to evaluate or accept the performance of the perception and fusion algorithm of the developed measured perception system on the vehicle.

[0150] The fourth embodiment of the present disclosure provides an electronic device including at least a memory and a processor, the memory storing a computer program, and the processor implementing the method provided by any of the embodiments of the present disclosure when executing the computer program stored on the memory. For example, the electronic device computer program steps are as follows S21 to S23:

[0151] S21, synchronizing the true value sensor module and the measured sensor module arranged on the vehicle;

[0152] S22, obtaining the first acquisition result data of the true value sensor module and the second acquisition result data of the measured sensor module;

[0153] S23, obtaining the evaluation result of the to-be-tested perception fusion algorithm based on the first acquisition result data and the second acquisition result data.

[0154] Further, the processor also executes the computer program in the third embodiment

[0155] The embodiments of the present disclosure synchronize the measured sensor module and the true value sensor module in time and space, so as to obtain the acquisition data of the true value sensor module and the acquisition data of the measured sensor module based on the same space-time standard. Further, the true value perception result is obtained by running the true value algorithm on the true value sensor module, the measured perception result is obtained by running the to-be-tested perception or fusion algorithm on the measured sensor module, the measured perception result data and the true value perception result data are compared, and the evaluation result of the measured perception system result is output, so as to evaluate the perception result of the measured perception fusion system, thereby being applied to the development and test of the vehicle-mounted measured perception system to evaluate or accept the performance of the perception and fusion algorithm of the developed vehicle-mounted measured perception system.

[0156] The storage medium can be included in the electronic device, or can exist separately and not be assembled into the electronic device.

[0157] The storage medium carries one or more programs, and when the one or more programs are executed by the electronic device, the electronic device is caused to: obtain at least two internet protocol addresses; send a node evaluation request including the at least two internet protocol addresses to a node evaluation device, wherein the node evaluation device selects an internet protocol address from the at least two internet protocol addresses and returns; receive the internet protocol address returned by the node evaluation device; and wherein the obtained internet protocol address indicates an edge node in a content distribution network.

[0158] Alternatively, the storage medium described above carries one or more programs, when the one or more programs are executed by the electronic device, the electronic device is caused to: receive a node evaluation request comprising at least two internet protocol addresses; select an internet protocol address from the at least two internet protocol addresses; return the selected internet protocol address; wherein the received internet protocol address indicates an edge node in a content distribution network.

[0159] Computer program code for carrying out operations of the present disclosure can be written in any of one or more programming languages or combinations of languages including object or visual programming languages such as Java, Smalltalk, C++ or conventional procedural programming languages such as the "C" programming language or similar programming languages. The program code can execute entirely on the passenger computer, partly on the passenger computer, as a stand-alone software package, partly on the passenger computer and partly on a remote computer or entirely on the remote computer or server. In the latter scenario, the remote computer can be connected to the passenger computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection can be made to an external computer (for example, through the Internet using an Internet Service Provider).

[0160] It should be noted that the storage medium in the present disclosure can be a computer readable signal medium or a computer readable storage medium or any combination of the above two. The computer readable storage medium may, for example, but is not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or apparatus, or any combination of the above. More specific examples of the computer readable storage medium can include, but are not limited to, an electrical connection having one or more wires, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present disclosure, the computer readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, device or apparatus. In the present disclosure, the computer readable signal medium can include a data signal carried in a baseband or as a part of a carrier wave, which carries computer readable program code. Such a propagated data signal can take many forms, including but not limited to an electromagnetic signal, an optical signal or any suitable combination of the above. The computer readable signal medium can also be any storage medium other than the computer readable storage medium that can send, propagate or transmit a program for use by or in conjunction with an instruction execution system, device or apparatus. The program code contained in the storage medium can be transmitted by any suitable medium, including but not limited to a wire, a cable, a RF (radio frequency) or the like, or any suitable combination of the above.

[0161] The flow diagrams and block diagrams in the drawings are schematic illustrations of possible architectures, functions and operations of systems, methods and computer program products in accordance with various embodiments of the present disclosure. In this regard, each block in the flow diagrams and block diagrams can represent a module, a segment, or a portion of code, which comprises one or more executable instructions for implementing the specified logical functions. It should also be noted that in some alternative implementations, the functions noted in the blocks can occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently or the blocks may, in fact, be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams and / or flow diagrams, and combinations of blocks in the block diagrams and / or flow diagrams, can be implemented by special purpose hardware-based systems that perform the specified functions or operations, or combinations of special purpose hardware and computer instructions.

[0162] The units described in the embodiments of the present disclosure can be implemented by means of software, or by hardware. In some cases, the name of the unit does not constitute a limitation on the unit itself.

[0163] The functions described above in this document can be performed, at least in part, by one or more hardware logic components. For example, and without limitation, illustrative types of hardware logic components that can be used include Field-programmable Gate Arrays (FPGAs), Application-specific Integrated Circuits (ASICs), Application-specific Standard Products (ASSPs), System-on-a-chip systems (SOCs), Complex Programmable Logic Devices (CPLDs), etc.

[0164] In the context of the present disclosure, a machine-readable medium can be a tangible medium that contains or stores a program for use by or in connection with an instruction execution system, apparatus, or device. The machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples of the machine-readable storage medium will include one or more lines of a program of a processor, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0165] The above description is merely exemplary of the present disclosure and the application of the principles thereof. It is not intended to limit the disclosure to the precise forms disclosed. Rather, it is intended to cover such alternatives, modifications, and equivalents as can be suggested by the principles of the disclosure and other printed publications that are prior art to the present disclosure.

[0166] Further, although operations are depicted in a particular, sequential order, this should not be understood as requiring or implying that the operations are performed in the order illustrated, or that they are performed sequentially, as in some cases, the operations can be performed in parallel, or the various steps in an operation can be performed at different times. Similarly, while the above discussion has included several specific examples, these should not be construed as limiting the scope of the disclosure, as other specific examples falling within the scope of the claims are possible. Some of the features of the disclosure discussed in the context of separate embodiments can also be combined with each other in single embodiments. Conversely, various features of the disclosure, which are described in the context of a single embodiment, can also be implemented in multiple embodiments. The scope of the disclosure is defined by the appended claims, rather than the specific embodiments that are described above in order to provide what is understood to be a practical disclosure.

[0167] Although the subject matter has been described in language specific to structural features and / or methodological acts, it is to be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or acts described above. Rather, the specific features and acts described above are disclosed as example forms of implementing the claims.

[0168] The above detailed description sets forth numerous specific details of the present disclosure. However, it is contemplated that not all embodiments of the present disclosure will include all of the specific details described above. In other words, the present disclosure is intended to be illustrative, rather than restrictive. Many other embodiments will become apparent to those skilled in the art upon reading and understanding the above description and the appended claims.

Claims

1. A method for evaluating a perceptual fusion algorithm, characterized in that, The method comprises: time synchronization and space synchronization of a true value sensor module and a measured sensor module arranged on a vehicle; obtaining first acquisition result data of the true value sensor module and second acquisition result data of the measured sensor module; obtaining an evaluation result of a perception fusion algorithm to be tested based on the first acquisition result data and the second acquisition result data, wherein the evaluation result is represented by a test index, and the test index comprises multi-target tracking accuracy and / or multi-target tracking precision; wherein the space synchronization of the true value sensor module and the measured sensor module arranged on the vehicle comprises: setting a centering position of the vehicle on a console of a multi-sensor system calibration site, connecting a vehicle-mounted diagnostic system interface of the vehicle through the console, and collecting sensor configuration information of the vehicle; obtaining position parameters of multiple sensors in the measured sensor module from the sensor configuration information, wherein the position parameters comprise position relationship parameters between multiple sensors in the measured sensor module and an external parameter matrix of each sensor in the measured sensor module in a vehicle body coordinate system; determining whether the position synchronization is successful based on the position parameters and direct position information of sensors in the true value sensor module; in response to the number of calibration failures of the position synchronization exceeding a threshold value, saving acquisition data of the sensors in the measured sensor module, acquisition data of the sensors in the true value sensor module, and calibration failure information; and in response to the position synchronization being successful, disconnecting the connection between the console and the vehicle-mounted diagnostic system interface to release the centering position control.

2. The evaluation method according to claim 1, characterized by The true value sensor module at least comprises a mechanical laser radar, a millimeter wave radar and a camera device; and the measured sensor module at least comprises a solid-state laser radar, a millimeter wave radar and a camera device.

3. The evaluation method according to claim 2, characterized by, The time synchronization of the true value sensor module and the measured sensor module arranged on the vehicle comprises: receiving a signal reference time standard through a clock source; synchronizing time information between the mechanical laser radar or the solid-state laser radar through a PTP network time synchronization protocol; controlling the camera device to perform synchronous exposure through an external trigger mode; synchronizing time information between the millimeter wave radar through CANTSyn.

4. The evaluation method according to claim 1, characterized by The method comprises: respectively obtaining first acquisition raw data from the true value sensor module and second acquisition raw data from the measured sensor module; obtaining first acquisition result data from the first acquisition raw data through a true value processing algorithm; obtaining second acquisition result data from the second acquisition raw data through the perception fusion algorithm.

5. The evaluation method according to claim 1, characterized by The first acquisition result data or the second acquisition result data comprises any one or a combination of multiple of target types, target IDs, colors, orientations, three-order equation coefficients of lane lines or kerbs, two-dimensional bounding boxes, three-dimensional bounding boxes, semantic annotations, light source states, traffic signal lamp indication types, traffic signal lamp durations, distances, speeds and accelerations.

6. The evaluation method according to claim 1, characterized by The test indicators further include at least one of accuracy, precision, recall, average precision, average intersection over union, distance accuracy, speed accuracy, acceleration accuracy, average precision, average recall, average average precision, and average average recall.

7. An evaluation device for a perceptual fusion algorithm, characterized in that The method comprises the following steps: a synchronization module, configured to synchronize time and space of a true value sensor module and a measured sensor module arranged on a vehicle; a first acquisition module, configured to acquire first acquisition result data of the true value sensor module and second acquisition result data of the measured sensor module; a second acquisition module, configured to acquire an evaluation result of a perception fusion algorithm to be tested based on the first acquisition result data and the second acquisition result data, wherein the evaluation result is represented by a test indicator, and the test indicator comprises multi-target tracking accuracy and / or multi-target tracking precision; wherein the space synchronization of the true value sensor module and the measured sensor module arranged on the vehicle comprises: setting a centering position of the vehicle on a console of a multi-sensor system calibration site, and connecting a vehicle onboard diagnostic system interface through the console to acquire sensor configuration information of the vehicle; acquiring position parameters of multiple sensors in the measured sensor module from the sensor configuration information, wherein the position parameters comprise position relationship parameters between the multiple sensors in the measured sensor module and an external parameter matrix of each sensor in the multiple sensors in the measured sensor module in a vehicle body coordinate system; determining whether the position synchronization is successful based on the position parameters and direct position information of sensors in the true value sensor module; in response to a calibration number of times that the position synchronization is unsuccessful exceeding a calibration number of times threshold, saving acquisition data of the sensors in the measured sensor module, acquisition data of the sensors in the true value sensor module, and calibration failure information; and in response to the position synchronization being successful, disconnecting the console from the vehicle onboard diagnostic system interface to release the centering position control.

8. A storage medium storing a computer program, characterized by The computer program, when executed by a processor, implements the steps of the method of any one of claims 1 to 6.

9. An electronic device comprising at least a memory, a processor, said memory having stored thereon a computer program, characterized in that, The processor, when executing the computer program on the memory, implements the steps of the method of any one of claims 1 to 6.

Images