Algorithm evaluation method and device, electronic equipment, storage medium and program product

Abstract

The invention relates to an algorithm evaluation method and device, electronic equipment, a storage medium and a program product, and relates to the field of automatic driving or intelligent driving. The method comprises the steps of obtaining at least one piece of test data and at least one piece of truth value data corresponding to target point cloud data; wherein the test data comprises two-dimensional coordinate information of at least one detection point obtained by carrying out obstacle detection on the target point cloud data through a to-be-evaluated obstacle detection algorithm, and the truth value data is used for reflecting the two-dimensional coordinate information of a marked area where an obstacle is located in the target point cloud data; according to the timestamp information, matching the at least one piece of test data with the at least one piece of truth value data to obtain at least one pair of first test data and first truth value data with the same timestamp; and determining error information of the obstacle detection algorithm to be evaluated according to the first test data and the first truth value data. According to the invention, the accuracy of obstacle detection algorithm evaluation can be improved.

Description

Technical Field

[0001] The present application relates to the field of autonomous driving or intelligent driving, and in particular to the field of algorithm evaluation technology, and specifically to an algorithm evaluation method, device, electronic device, storage medium and program product. Background Art

[0002] As cars enter the intelligent era, LiDAR has become one of the core sensors for realizing environmental perception in the field of autonomous driving or intelligent driving. LiDAR has more accurate three-dimensional detection capabilities. It can collect point cloud data of obstacles by scanning the surrounding environment, and then process these point cloud data through obstacle detection algorithms to determine obstacles or drivable areas, so that the intelligent driving system can avoid obstacles, slow down or brake according to the determined obstacles or drivable areas. In this process, the accuracy of the obstacle detection algorithm is crucial.

[0003] In the relevant technical solutions, when testing the obstacle detection algorithm, it is necessary to synchronize the data collected by different sensors including lidar in time and space to obtain the true data and the test data output by the obstacle detection algorithm; however, different sensors have different data collection cycles due to the constraints of the physical characteristics of their own hardware or their design limitations, so there may be data that cannot be completely synchronized in time, which ultimately leads to low accuracy of the comparison results between the true data and the test data, resulting in poor accuracy of the test results of the obstacle detection algorithm. Summary of the invention

[0004] The present application provides an algorithm evaluation method, device, electronic device, storage medium and program product to at least solve the technical problem of poor accuracy of the test results of the obstacle detection algorithm in the prior art. The technical solution of the present application is as follows:

[0005] According to the first aspect provided by the present application, an algorithm evaluation method is provided, the method comprising: obtaining at least one test data and at least one true value data corresponding to target point cloud data; wherein the test data comprises two-dimensional coordinate information of at least one detection point obtained by performing obstacle detection on the target point cloud data through the obstacle detection algorithm to be evaluated, and the true value data is used to reflect the two-dimensional coordinate information of the marked area where the obstacle in the target point cloud data is located; according to timestamp information, matching at least one of the test data and at least one of the true value data to obtain at least one pair of first test data and first true value data with the same timestamp; determining error information of the obstacle detection algorithm to be evaluated based on the first test data and the first true value data.

[0006] According to the above technical means, the true value data and the test data are obtained based on the same target point cloud data processing, so it can be ensured that the obstacles represented by the true value data and the test data that are successfully matched according to the timestamp information are completely consistent in time and space, avoiding the inconsistency in time and space between the obstacles marked by the true value data and the test data due to the different periods of sensor data collection, thereby improving the accuracy of the obstacle detection algorithm evaluation.

[0007] In one possible implementation, the coordinate systems of the test data and the true value data are the same, and the error information of the obstacle detection algorithm to be evaluated includes the shortest distance between a first target detection point in the first test data and a boundary of a marked area of ​​the first true value data, and the first target detection point is a detection point among at least one detection point of the first test data that is closest to the boundary of the marked area of ​​the first true value data.

[0008] According to the above technical means, the shortest distance between the detection point in the test data and the boundary of the marking area is determined as the error information. When the deviation between the detection point and the marking area representing the same obstacle is small, it can prevent the detection point on one side of the detected obstacle from being calculated with the other side boundary of the marking area representing the real obstacle to obtain erroneous error information.

[0009] In one possible implementation, error information of the obstacle detection algorithm to be evaluated is determined based on first test data and first true value data, including: expanding the boundary of a marked area of ​​the first true value data by a preset threshold to obtain an enlarged area; taking a detection point located in the enlarged area among at least one detection point of the first test data as a candidate detection point; and determining the error information of the obstacle detection algorithm to be evaluated based on the boundary of at least one candidate detection point and the marked area of ​​the first true value data.

[0010] According to the above technical means, by enlarging the marked area and selecting the detection points in the enlarged area, the detection points located near the marked area can be selected, and only the distance between these detection points located near the marked area and the marked area is calculated, thereby avoiding meaningless traversal calculations of a large number of detection points far away from the marked area, and effectively reducing the amount of calculation when evaluating the obstacle detection algorithm.

[0011] In one possible implementation, error information of the obstacle detection algorithm to be evaluated is determined based on the first test data and the first true value data, including: determining a target edge of the marked area based on a relative position relationship between a test vehicle and the marked area of ​​the first true value data; and determining error information of the obstacle detection algorithm to be evaluated based on at least one detection point of the first test data and the target edge.

[0012] According to the above technical means, when calculating the deviation between the detection point and the marked area of ​​the true value data, only the detection point and the edge of the marked area used to represent the position where the obstacle may collide with the test vehicle are calculated. This can reduce the amount of calculation without affecting the accuracy of the test of the obstacle detection algorithm.

[0013] In one possible implementation, error information of an obstacle detection algorithm to be evaluated is determined based on at least one detection point of first test data and a target edge, including: determining a detection area based on at least one detection point of the first test data; wherein the detection area is a minimum area covering at least one detection point of the first test data; determining at least one second target detection point of the first test data; wherein the second target detection point is a detection point whose distance from a boundary of the detection area is less than or equal to a second preset distance; and determining error information of the obstacle detection algorithm to be evaluated based on at least one second target detection point of the first test data and a target edge of the marked area.

[0014] According to the above technical means, the minimum area of ​​the detection area covering the detection points can approximately represent the maximum possible contact area of ​​the obstacles represented by these detection points. Further selecting detection points that are closer to the detection area can prevent errors from being generated by comparing the detection points that represent the non-possible contact positions of the obstacles with the marked areas used to represent the actual possible contact areas of the obstacles, thereby improving the accuracy of obstacle algorithm evaluation. For example, for a series of detection points used to represent vehicle obstacles, the detection points close to the boundaries among these detection points reflect the positions where the obstacles may collide with the test vehicle, while the detection points close to the center among these detection points reflect the positions where the rear windows of the vehicles will not cause collisions or other obstacles to the test vehicle, while the marked areas in the true value data mark the positions where the obstacle vehicles may actually collide with the test vehicle; therefore, selecting the distance between the detection points close to the boundaries and the marked areas among a series of detection points used to represent the three-dimensional contour of an obstacle can accurately reflect the deviation between the detection points and the true value.

[0015] In one possible implementation, the error information of the obstacle detection algorithm to be evaluated includes the shortest distance between a first target detection point in the first test data and a boundary of a marked area of ​​the first true value data; the method further includes: determining the mean and variance of the shortest distances in different intervals based on at least one shortest distance between each pair of first test data and first true value data with the same timestamp.

[0016] According to the above technical means, the error information obtained by comparing each pair of test data and true value data is divided into different intervals according to the data size, and the mean and variance in each interval are calculated, which is conducive to reflecting the error degree of the obstacle evaluation algorithm as a whole, and reflecting to a certain extent whether there are abnormalities in the evaluation process through the numerical values ​​of the variances in different intervals and the comparison of the variances in different intervals.

[0017] According to the second aspect provided by the present application, an algorithm evaluation device is provided, which includes an acquisition unit, a matching unit and a determination unit. The acquisition unit is used to acquire at least one test data and at least one true value data corresponding to the target point cloud data; wherein the test data includes the two-dimensional coordinate information of at least one detection point obtained by performing obstacle detection on the target point cloud data by the obstacle detection algorithm to be evaluated, and the true value data is used to reflect the two-dimensional coordinate information of the marked area where the obstacle in the target point cloud data is located; the matching unit is used to match at least one test data and at least one true value data according to the timestamp information, and obtain at least one pair of first test data and first true value data with the same timestamp; the determination unit is used to determine the error information of the obstacle detection algorithm to be evaluated according to the first test data and the first true value data.

[0018] In one possible implementation, the coordinate systems of the test data and the true value data are the same, and the error information of the obstacle detection algorithm to be evaluated includes the shortest distance between a first target detection point in the first test data and a boundary of a marked area of ​​the first true value data, and the first target detection point is a detection point among at least one detection point of the first test data that is closest to the boundary of the marked area of ​​the first true value data.

[0019] In one possible implementation, the determination unit is specifically used to expand the boundary of the marked area of ​​the first true value data by a preset threshold to obtain an enlarged area; use a detection point located in the enlarged area among at least one detection point of the first test data as a candidate detection point; and determine the error information of the obstacle detection algorithm to be evaluated based on the boundary of the at least one candidate detection point and the marked area of ​​the first true value data.

[0020] In a possible implementation, the determination unit is specifically used to determine the target edge of the marked area based on the relative position relationship between the test vehicle and the marked area of ​​the first true value data; and determine the error information of the obstacle detection algorithm to be evaluated based on at least one detection point of the first test data and the target edge.

[0021] In one possible implementation, the determination unit is specifically used to determine a detection area based on at least one detection point of the first test data; wherein the detection area is the minimum area covering at least one detection point of the first test data; determine at least one second target detection point of the first test data; wherein the second target detection point is a detection point whose distance from the boundary of the detection area is less than or equal to a second preset distance; and determine error information of the obstacle detection algorithm to be evaluated based on at least one second target detection point of the first test data and the target edge of the marked area.

[0022] In one possible implementation, the error information of the obstacle detection algorithm to be evaluated includes the shortest distance between a first target detection point in the first test data and a boundary of a marked area of ​​the first true value data; the determination unit is further used to determine the mean and variance of the shortest distances in different intervals based on at least one shortest distance of each pair of first test data and first true value data with the same timestamp.

[0023] According to a third aspect provided by the present application, an electronic device is provided, comprising: a processor and a memory for storing instructions executable by the processor, wherein the processor is configured to execute the instructions to implement the method of the first aspect and any possible implementation method thereof.

[0024] According to the fourth aspect provided by the present application, a computer-readable storage medium is provided. When the instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device is enabled to execute the method of the above-mentioned first aspect and any possible implementation method thereof.

[0025] According to the fifth aspect provided by the present application, a computer program product is provided, the computer program product comprising computer instructions, and when the computer instructions are executed on an electronic device, the electronic device executes the method of the above-mentioned first aspect and any possible implementation manner thereof.

[0026] Therefore, the above technical features of the present application have the following beneficial effects:

[0027] (1) The true value data and the test data are obtained based on the same target point cloud data processing. Therefore, it can be ensured that the true value data and the test data that are successfully matched according to the timestamp information are completely consistent in time and space. This avoids the inconsistency in time and space between the obstacles marked by the true value data and the test data due to the different periods of sensor data collection, thereby improving the accuracy of the obstacle detection algorithm evaluation.

[0028] (2) The shortest distance between the detection point in the test data and the boundary of the marking area is determined as the error information. When the deviation between the detection point and the marking area representing the same obstacle is small, it can prevent the detection point used to represent one side of the detected obstacle from being calculated with the other side boundary of the marking area representing the real obstacle to obtain erroneous error information.

[0029] (3) By enlarging the marked area and selecting the detection points in the enlarged area, the detection points located near the marked area can be selected, and the distance between the detection points and the marked area is calculated only for these detection points located near the marked area, thereby avoiding meaningless traversal calculations for a large number of detection points far away from the marked area, and effectively reducing the amount of calculation when evaluating the obstacle detection algorithm.

[0030] (4) When calculating the deviation between the detection point and the marked area of ​​the true value data, only the detection point and the edge of the marked area representing the position where the obstacle may collide with the test vehicle are calculated. This can reduce the amount of calculation without affecting the accuracy of the test of the obstacle detection algorithm.

[0031] (5) The minimum area of ​​the detection point covered by the detection area can approximately represent the maximum possible contact area of ​​the obstacles represented by these detection points. Further selecting detection points that are close to the detection area can prevent errors caused by comparing the detection points that represent the non-possible contact positions of the obstacles with the marked areas used to represent the actual possible contact areas of the obstacles, thereby improving the accuracy of obstacle algorithm evaluation. For example, for a series of detection points used to represent vehicle obstacles, the detection points close to the boundaries of these detection points reflect the positions where the obstacles may collide with the test vehicle, while the detection points close to the center of these detection points reflect the positions where the rear windows of the vehicles will not cause collisions and other obstacles to the test vehicle, while the marked areas in the true value data mark the positions where the obstacle vehicles may actually collide with the test vehicle; therefore, selecting the distance between the detection points close to the boundaries of a series of detection points used to represent the three-dimensional contour of an obstacle and the marked areas can accurately reflect the deviation between the detection points and the true value.

[0032] (6) The error information obtained by comparing each pair of test data and true value data is divided into different intervals according to the data size, and the mean and variance in each interval are calculated. This is conducive to reflecting the error degree of the obstacle evaluation algorithm as a whole, and through the numerical values ​​of the variances in different intervals and the comparison of the variances in different intervals, it can reflect to a certain extent whether there are abnormalities in the evaluation process.

[0033] It should be noted that the technical effects brought about by any implementation method in the second to fifth aspects can refer to the technical effects brought about by the corresponding implementation method in the first aspect, and will not be repeated here.

[0034] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] The drawings herein are incorporated into the specification and constitute a part of the specification, illustrate embodiments consistent with the present application, and together with the specification are used to explain the principles of the present application, and do not constitute an improper limitation on the present application.

[0036] Figure 1 is a structural schematic diagram of an algorithm evaluation system according to an exemplary embodiment;

[0037] Figure 2 is a flow chart of an algorithm evaluation method according to an exemplary embodiment;

[0038] Figure 3 is a flow chart of another algorithm evaluation method according to an exemplary embodiment;

[0039] Figure 4 is an example diagram of a two-dimensional coordinate system according to an exemplary embodiment;

[0040] Figure 5 is a flow chart of another algorithm evaluation method according to an exemplary embodiment;

[0041] Figure 6 is an example diagram showing a test vehicle and a marking area according to an exemplary embodiment;

[0042] Figure 7 is a flow chart of another algorithm evaluation method according to an exemplary embodiment;

[0043] Figure 8 is an example diagram showing a marking area and a detection point according to an exemplary embodiment;

[0044] Fig. 9 is a flow chart of another algorithm evaluation method according to an exemplary embodiment;

[0045] Fig.10 is an example diagram of matching data on a two-dimensional coordinate system according to an exemplary embodiment;

[0046] Fig.11 is a block diagram of an algorithm evaluation device according to an exemplary embodiment;

[0047] Fig.12 It is a block diagram of an electronic device according to an exemplary embodiment. DETAILED DESCRIPTION

[0048] In order to enable ordinary persons in the art to better understand the technical solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the accompanying drawings.

[0049] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present application. On the contrary, they are merely examples of devices and methods consistent with some aspects of the present application as detailed in the attached claims.

[0050] Before introducing the algorithm evaluation method provided by this application in detail, a brief introduction to the prior art related to this application is first given.

[0051] As cars enter the intelligent era, LiDAR has become one of the core sensors for realizing environmental perception in the field of autonomous driving or intelligent driving. LiDAR has more accurate three-dimensional detection capabilities. It can collect point cloud data of obstacles by scanning the surrounding environment, and then process these point cloud data through obstacle detection algorithms to determine obstacles or drivable areas, so that the intelligent driving system can avoid obstacles, slow down or brake according to the determined obstacles or drivable areas. In this process, the accuracy of the obstacle detection algorithm is crucial.

[0052] In the relevant technical solutions, when testing the obstacle detection algorithm, it is necessary to synchronize the data collected by different sensors including lidar in time and space to obtain the true data and the test data output by the obstacle detection algorithm; however, different sensors have different data collection cycles due to the constraints of the physical characteristics of their own hardware or their design limitations, so there may be data that cannot be completely synchronized in time, which ultimately leads to low accuracy of the comparison results between the true data and the test data, resulting in poor accuracy of the test results of the obstacle detection algorithm.

[0053] In response to the above technical problems, the present application provides an algorithm evaluation method, the idea of ​​which is to compare the test data and the true data obtained by different processing based on the same target point cloud data, so as to ensure that the obstacles represented by the test data and the true data are completely consistent in time and space, thereby avoiding the inconsistency in time and space between the obstacles marked by the true data and the test data due to the different periods of sensor data collection, thereby improving the accuracy of the obstacle detection algorithm evaluation.

[0054] Before introducing the algorithm evaluation method provided by this application in detail, a brief introduction to the implementation environment (implementation architecture) involved in this application is first given.

[0055] Figure 1 Figure 2 shows a schematic diagram of the structure of an algorithm evaluation system. Figure 1 As shown, the algorithm evaluation system includes a data source device 10 , a detection server 20 , a labeling server 30 and an evaluation server 40 .

[0056] The data source device 10 is connected to the detection server 20 and the annotation server 30 respectively, and the detection server 20 and the annotation server 30 are also connected to the evaluation server 40. Exemplarily, the connection between the data source device 10 and the detection server 20 and the annotation server 30, and the connection between the evaluation server 40 and the detection server 20 and the annotation server 30 include wired connection or wireless connection. For example, the wired connection can be a wired connection realized by optical fiber or other physical transmission media; the wireless connection can be a Bluetooth connection, a wireless fidelity (Wi-Fi) connection, etc.

[0057] Exemplarily, the data source device 10, the detection server 20, the annotation server 30 and the evaluation server 40 may be a single electronic device such as a blade server and a rack-mounted server, or may be a (distributed) server cluster composed of multiple servers.

[0058] It should be noted that the data source device 10 , the detection server 20 , the annotation server 30 or the evaluation server 40 may be different physical devices or the same physical device.

[0059] In some embodiments, the data source device 10 is used to send target point cloud data to the detection server 20 and the annotation server 30 .

[0060] In some embodiments, the detection server 20 is installed with an obstacle detection algorithm to be evaluated. The detection server 20 is used to process the target point cloud data received from the data source device 10 through the obstacle detection algorithm to be evaluated, obtain at least one test data, and send at least one test data to the evaluation server 40.

[0061] In some embodiments, the annotation server 30 is used to determine at least one true value data based on the target point cloud data received from the data source device 10, and send at least one true value data to the evaluation server 40; the true value data is used to reflect obstacles in the target point cloud data.

[0062] In some embodiments, the evaluation server 40 is used to determine the error information of the obstacle detection algorithm to be evaluated by matching and comparing at least one test data received from the detection server 20 and at least one true value data received from the annotation server 30.

[0063] For ease of understanding, the algorithm evaluation method provided in this application is specifically introduced below with reference to the accompanying drawings.

[0064] Figure 2 FIG. 1 is a flow chart of an algorithm evaluation method according to an exemplary embodiment. Figure 2 As shown, the algorithm evaluation method includes the following steps:

[0065] S101. Obtain at least one test data and at least one true value data corresponding to the target point cloud data.

[0066] The test data includes the two-dimensional coordinate information of at least one detection point obtained by performing obstacle detection on the target point cloud data through the obstacle detection algorithm to be evaluated, and the true value data is used to reflect the two-dimensional coordinate information of the marked area where the obstacle in the target point cloud data is located. The target point cloud data includes at least one point cloud data collected at different times, and at least one test data is determined by performing obstacle detection on at least one point cloud data through the obstacle detection algorithm to be evaluated; the true value data is determined by performing artificial intelligence pre-labeling and manual inspection on at least one point cloud data. The at least one point cloud data collected at different times includes at least one (frame) point cloud data collected at each different time point in a continuous period of time, and can also be at least one point cloud data collected at different discontinuous time points.

[0067] As a possible implementation method, it is implemented in Python language, and the information of at least one detection point generated by the obstacle detection algorithm to be evaluated is packaged into one data as test data; the two-dimensional coordinate information of the marked area is packaged into one data as true value data.

[0068] S102: Match at least one test data and at least one true value data according to the timestamp information to obtain at least one pair of first test data and first true value data with the same timestamp.

[0069] The timestamp information of the test data or the true value data is determined according to the timestamp of the point cloud data from which the test data or the true value data is obtained, and the timestamp information obtained according to the timestamp of the same point cloud data is the same.

[0070] As a possible implementation method, test data with the same timestamp information as the true value data is searched according to the timestamp information of the true value data, test data with the same timestamp information as the true value data is recorded, test data without the same timestamp information as the true value data is discarded, and the discarded test data is recorded.

[0071] S103: Determine error information of the obstacle detection algorithm to be evaluated according to the first test data and the first true value data.

[0072] The error information of the obstacle detection algorithm to be evaluated includes the distance between the detection point and the marked area.

[0073] It can be understood that the true value data and the test data are obtained based on the same target point cloud data processing, so it can be guaranteed that the true value data and the test data that are successfully matched according to the timestamp information are completely consistent in time and space, avoiding the inconsistency in time and space between the obstacles marked by the true value data and the test data due to the different periods of sensor data collection, thereby improving the accuracy of the obstacle detection algorithm evaluation.

[0074] In some embodiments, the coordinate systems of the test data and the true value data are the same, and the error information of the obstacle detection algorithm to be evaluated includes the shortest distance between a first target detection point in the first test data and a boundary of a marked area of ​​the first true value data, and the first target detection point is a detection point among at least one detection point of the first test data that is closest to the boundary of the marked area of ​​the first true value data.

[0075] It can be understood that by determining the shortest distance between the detection point in the test data and the boundary of the marking area as the error information, when the deviation between the detection point representing the same obstacle and the marking area is small, it can prevent the detection point used to represent one side of the detected obstacle from being calculated with the other side boundary of the marking area representing the real obstacle to obtain erroneous error information.

[0076] In some embodiments, Figure 3 As shown, S103 may include;

[0077] S201 . Enlarge the boundary of a marked area of ​​first true value data by a preset threshold to obtain an enlarged area.

[0078] As a possible implementation manner, the two-dimensional coordinate information for representing the enlarged area is determined according to the two-dimensional coordinate information of the marked area where the obstacle is located in the target point cloud data reflected by the true value data and a preset threshold.

[0079] S202: Take a detection point located in the enlarged area among at least one detection point of the first test data as a candidate detection point.

[0080] Wherein, whether the detection point is located in the enlarged area is determined according to the two-dimensional coordinate information of each detection point in the at least one detection point and the two-dimensional coordinate information of the enlarged area.

[0081] As a possible implementation, the two-dimensional coordinate information of the marked area where the obstacle in the target point cloud data reflected by the true value data is located includes the size of the marked area, the two-dimensional coordinates of the center point of the marked area, and the deflection angle of the marked area relative to the two-dimensional coordinate system (in most scenarios, the obstacle marked by the marked area is a vehicle, and the deflection angle should be the heading angle of the vehicle). The two-dimensional coordinate information of the magnified area includes the size of the magnified area, the two-dimensional coordinates of the center point of the magnified area, and the deflection angle of the magnified area relative to the two-dimensional coordinate system. S202 may include:

[0082] S2021. Correct the angle of the two-dimensional coordinate system according to the deflection angle of the magnified area relative to the two-dimensional coordinate system to obtain a corrected two-dimensional coordinate system; wherein any two parallel sides of the magnified area are parallel to the horizontal axis or the vertical axis of the corrected two-dimensional coordinate system.

[0083] S2022. Determine the coordinates of the center point of the enlarged area and the detection point in the corrected two-dimensional coordinate system according to the deflection angle of the enlarged area relative to the two-dimensional coordinates, the two-dimensional coordinates of the center point of the enlarged area, and the two-dimensional coordinates of the detection point.

[0084] S2023. Determine the coordinates of four vertices of the enlarged area in the corrected two-dimensional coordinate system according to the size of the enlarged area and the coordinates of the center point of the enlarged area in the corrected two-dimensional coordinate system.

[0085] S2024: Determine whether the detection point is located in the enlarged area according to the coordinates of the four vertices of the enlarged area in the corrected two-dimensional coordinate system and the coordinates of the detection point in the corrected two-dimensional coordinate system.

[0086] For example, Figure 4 is an example diagram of a two-dimensional coordinate system according to an exemplary embodiment. Figure 4 As shown in the figure, the two-dimensional coordinate system represented by the solid line is the original two-dimensional coordinate system, and the two-dimensional coordinate system represented by the dotted line is the corrected two-dimensional coordinate system. 1 (x 1 ,y 1 ), A 2 (x 2 ,y 2 ), A 3 (x 3 ,y 3 ) and A 4 (x 4 ,y 4 ) are the four vertices of the magnified area, point B(x b,y b ) and point C(x c ,y c ) are two detection points, and the deflection angle of the magnified area is α. It is known that for any point X(x x ,y x ), the coordinates of the point in the modified two-dimensional coordinate system obtained by rotating θ clockwise around the origin of the original two-dimensional coordinate system are:

[0087] X(xcosα-ysinα,xsinα+ycosα)

[0088] So we can get point A 1 , A 2 , A 3 , A 4 , B and C in the corrected two-dimensional coordinate system (in order to achieve the purpose of determining whether the detection point in the corrected two-dimensional coordinate system is located in the magnified area in this embodiment, at least only the coordinates of the two vertices constituting the diagonal line of the magnified area need to be determined). For the sake of simplicity in the following description, the coordinates of these points in the corrected two-dimensional coordinate system are assumed to be A 1 (x ` 1 ,y ` 1 ), A 2 (x ` 2 ,y ` 2 ), A 3 (x ` 3 ,y ` 3 ), A 4 (x ` 4 ,y 4 ` )、B(x ` b ,y ` b ) and C(x ` c ,y c ` ).Depend on Figure 4 From the spatial relationship shown in the figure, it can be seen that for any point Y(x y ,y y ), the criteria for determining whether the point is in the magnified area include:

[0089]

[0090] Therefore, we can pass point A2 , A 3 Comparison of the abscissa and ordinate of point B, point C determines whether point B and point C are located in the enlarged area.

[0091] S203: Determine error information of the obstacle detection algorithm to be evaluated according to a boundary between at least one candidate detection point and a marked area of ​​the first true value data.

[0092] It can be understood that by enlarging the marked area and selecting the detection points in the enlarged area, the detection points located near the marked area can be selected, and the distance between the detection points and the marked area is calculated only for these detection points located near the marked area, thereby avoiding meaningless traversal calculations of a large number of detection points far away from the marked area, and effectively reducing the amount of calculation when evaluating the obstacle detection algorithm.

[0093] In some embodiments, Figure 5 As shown, S103 may include:

[0094] S301 . Determine a target edge of the marked area according to a relative position relationship between a test vehicle and the marked area of ​​first true value data.

[0095] Among them, the test vehicle faces forward and is equipped with a laser radar, and the target point cloud data is obtained through the detection of the laser radar. The marking area is a rectangular area located in front of the test vehicle. The target edge of the marking area is the edge of the four edges of the marking area that is directly opposite or opposite to the test vehicle. Specifically, for the marking area with the same direction as the test vehicle, if the marking area is located in front of the test vehicle, its target edge is the edge at the bottom of the marking area; if the marking area is located in front of the left of the test vehicle, its target edge is the edge at the bottom and the right side of the marking area; if the marking area is located in front of the right of the test vehicle, its target edge is the edge at the bottom and the left side of the marking area. For the marking area with a different direction from the test vehicle, if the direction of the marking area is between the front direction and the left direction of the test vehicle, its target edge is the edge at the bottom and the left side of the marking area; if the direction of the marking area is between the front direction and the right direction of the test vehicle, its target edge is the edge at the bottom and the right side of the marking area.

[0096] For example, Figure 6 is an example diagram showing a test vehicle and a marking area according to an exemplary embodiment. Figure 6 As shown, there are marked areas A, B, C, D and E in front of the test vehicle, and the target edges of each marked area are Figure 6 Indicated in bold.

[0097] It should be noted that the method in this embodiment can be applied to the enlarged area described in S201-S203, and the method for determining the target edge of the enlarged area can refer to the above description.

[0098] S302: Determine error information of the obstacle detection algorithm to be evaluated according to at least one detection point and a target edge of the first test data.

[0099] It can be understood that when calculating the deviation between the detection point and the marked area of ​​the true value data, only calculating the detection point and the edges of the marked area used to represent the location where the obstacle may collide with the test vehicle can reduce the amount of calculation without affecting the accuracy of the test of the obstacle detection algorithm.

[0100] In some embodiments, Figure 7 As shown, S302 can be implemented as follows:

[0101] S401. Determine a detection area according to at least one detection point of first test data.

[0102] The detection area is the minimum area covering at least one detection point of the first test data.

[0103] As a possible implementation manner, at least one detection point is clustered to generate a cluster, and then the cluster generated by clustering is fitted into a minimum area covering at least one detection point of the first test data.

[0104] S402: Determine at least one second target detection point of the first test data.

[0105] The second target detection point is a detection point whose distance from the boundary of the detection area is less than or equal to a second preset distance.

[0106] S403: Determine error information of the obstacle detection algorithm to be evaluated according to at least one second target detection point of the first test data and the target edge of the marked area.

[0107] Wherein, referring to the description in S301, the target edge may include one or more edges of the marked area, and the error information of the obstacle detection algorithm to be evaluated includes the shortest distance between each detection point and the one or more edges in at least one detection point.

[0108] For example, Figure 8 is an example diagram showing a marking area and a detection point according to an exemplary embodiment. Figure 8 As shown, the target edges of the marked area are the bottom edge and the left edge, and the distance between the detection point A and the target edge at the bottom of the marked area is L 1 , the distance between the detection point A and the target edge on the left side of the marked area is L 2 , Easy to know L1 >L 2 , so the error information of the obstacle detection algorithm to be evaluated based on the detection point A includes L 2 .

[0109] It is understandable that the minimum area of ​​the detection area covering the detection points can approximately represent the maximum possible contact area of ​​the obstacles represented by these detection points. Further selecting detection points that are closer to the detection area can prevent errors from being generated by comparing the detection points that represent the non-possible contact positions of the obstacles with the marked areas used to represent the actual possible contact areas of the obstacles, thereby improving the accuracy of the obstacle algorithm evaluation. For example, for a series of detection points used to represent vehicle obstacles, the detection points close to the boundaries of these detection points reflect the positions where the obstacles may collide with the test vehicle, while the detection points close to the center of these detection points reflect the positions where the rear windows of the vehicle will not cause collisions or other obstacles to the test vehicle, while the marked areas in the true value data mark the positions where the obstacle vehicle may actually collide with the test vehicle; therefore, selecting the distance between the detection points close to the boundaries of a series of detection points used to represent the three-dimensional contour of an obstacle and the marked areas can accurately reflect the deviation between the detection points and the true value.

[0110] In some embodiments, the error information of the obstacle detection algorithm to be evaluated includes the shortest distance between a first target detection point in the first test data and a boundary of a marked area of ​​the first true value data; the method may also include: determining the mean and variance of the shortest distances in different intervals based on at least one shortest distance of at least one pair of test data and true value data with the same timestamp.

[0111] It can be understood that the error information obtained by comparing each pair of test data and true value data is divided into different intervals according to the data size, and the mean and variance in each interval are calculated, which is conducive to reflecting the error degree of the obstacle evaluation algorithm as a whole, and through the comparison of the variance values ​​in different intervals and the variances in different intervals, it can reflect to a certain extent whether there are abnormalities in the evaluation process.

[0112] The algorithm evaluation method in this application is introduced below in conjunction with specific embodiments. Fig. 9 As shown, the algorithm evaluation method provided in the embodiment of the present application can be implemented in the following manner:

[0113] S601. Obtain at least one test data and at least one true value data.

[0114] Among them, at least one test data and at least one true value data are determined based on the same target point cloud data. Please refer to the content described in S101, which will not be repeated here.

[0115] S602: Match at least one test data and at least one true value data to obtain matching data.

[0116] The matching data includes at least one pair of first test data and first true value data with the same timestamp. The position information of each pair of first test data and first true value data with the same timestamp is described by the same two-dimensional coordinate system; or, each pair of first test data and first true value data with the same timestamp is in the same two-dimensional coordinate system.

[0117] S603: Enlarge the marked area in the matching data to obtain an enlarged area.

[0118] S604: Determine the target edge of the enlarged area.

[0119] The target edge of the magnified area is determined according to the relative position relationship between the magnified area and the test vehicle; the target point cloud data in S601 is collected by the laser radar installed on the test vehicle.

[0120] S605: Determine candidate detection points.

[0121] Among them, the candidate detection point is located in the enlarged area.

[0122] S606: Determine the detection area.

[0123] The detection area is the smallest area covering all candidate detection points.

[0124] S607: Determine a second target detection point.

[0125] The distance between the second target detection point and the boundary of the detection point is less than or equal to the first preset distance.

[0126] S608: Determine the error information of the algorithm to be evaluated and the mean and variance of the error information in different intervals.

[0127] For example, Fig.10 FIG. 1 is an example diagram of matching data on a two-dimensional coordinate system according to an exemplary embodiment. Fig.10 As shown in the figure, rectangle A is the marked area, rectangle B is the magnified area, and rectangle C is the detection area. The marked area, magnified area, and detection area are represented by dotted lines, and the target edges of the magnified area and the detection area are represented by solid lines. In addition to the detection points covered by the magnified area, Fig.10There are other detection points in the two-dimensional coordinate system shown. These detection points are not in the magnified area and will not be determined as candidate detection points, so they will not participate in the calculation after S605. The detection points in the detection area close to the left and bottom solid lines (i.e., the target edge of the detection area) will be determined as second target detection points, and finally the shortest distance between each second target detection point and the target edge of the magnified area is determined as the error information of the obstacle detection algorithm to be evaluated; further, the mean and variance of these shortest distances in different intervals can also be included.

[0128] The above mainly introduces the solution provided by the embodiment of the present application from the perspective of the method. In order to achieve the above functions, the algorithm evaluation device or electronic device includes a hardware structure and / or software module corresponding to the execution of each function. It should be easily appreciated by those skilled in the art that, in combination with the units and algorithm steps of each example described in the embodiments disclosed herein, the present application can be implemented in the form of hardware or a combination of hardware and computer software. Whether a function is executed in the form of hardware or computer software driving hardware depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0129] The embodiment of the present application can exemplarily divide the functional modules of the algorithm evaluation device or electronic device according to the above method. For example, the algorithm evaluation device or electronic device may include various functional modules corresponding to the various functional divisions, or two or more functions may be integrated into one processing module. The above-mentioned integrated modules can be implemented in the form of hardware or in the form of software functional modules. It should be noted that the division of modules in the embodiment of the present application is schematic and is only a logical functional division. There may be other division methods in actual implementation.

[0130] Fig.11 is a block diagram of an algorithm evaluation device according to an exemplary embodiment. Fig.11 The algorithm evaluation device 700 includes an acquisition unit 701, a matching unit 702 and a determination unit 703.

[0131] The acquisition unit 701 is used to acquire at least one test data and at least one true value data corresponding to the target point cloud data; wherein the test data includes two-dimensional coordinate information of at least one detection point obtained by performing obstacle detection on the target point cloud data through the obstacle detection algorithm to be evaluated, and the true value data is used to reflect the two-dimensional coordinate information of the marked area where the obstacle in the target point cloud data is located.

[0132] The matching unit 702 is used to match at least one test data and at least one true value data according to the timestamp information to obtain at least one pair of first test data and first true value data with the same timestamp.

[0133] The determination unit 703 is used to determine error information of the obstacle detection algorithm to be evaluated according to the first test data and the first true value data.

[0134] Optionally, the coordinate systems of the test data and the true value data are the same, and the error information of the obstacle detection algorithm to be evaluated includes the shortest distance between a first target detection point in the first test data and a boundary of a marked area of ​​the first true value data, where the first target detection point is the detection point among at least one detection point of the first test data that is closest to the boundary of the marked area of ​​the first true value data.

[0135] Optionally, the determination unit 703 is specifically used to expand the boundary of the marked area of ​​the first true value data by a preset threshold to obtain an enlarged area; take a detection point located in the enlarged area among at least one detection point of the first test data as a candidate detection point; and determine the error information of the obstacle detection algorithm to be evaluated based on the boundary of at least one candidate detection point and the marked area of ​​the first true value data.

[0136] Optionally, the determination unit 703 is specifically used to determine the target edge of the marked area based on the relative position relationship between the test vehicle and the marked area of ​​the first true value data; and determine the error information of the obstacle detection algorithm to be evaluated based on at least one detection point of the first test data and the target edge.

[0137] Optionally, the determination unit 703 is specifically used to determine a detection area based on at least one detection point of the first test data; wherein the detection area is the minimum area covering at least one detection point of the first test data; determine at least one second target detection point of the first test data; wherein the second target detection point is a detection point whose distance from the boundary of the detection area is less than or equal to a second preset distance; determine the error information of the obstacle detection algorithm to be evaluated based on at least one second target detection point of the first test data and the target edge of the marked area.

[0138] Optionally, the error information of the obstacle detection algorithm to be evaluated includes the shortest distance between the first target detection point in the first test data and the boundary of the marked area of ​​the first true value data; the determination unit 703 is also used to: determine the mean and variance of the shortest distances in different intervals based on at least one shortest distance between each pair of first test data and first true value data with the same timestamp.

[0139] Regarding the device in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.

[0140] Fig.12 FIG. 1 is a block diagram of an electronic device according to an exemplary embodiment. Fig.12 As shown, the electronic device 800 includes but is not limited to: a processor 801 and a memory 802 .

[0141] The memory 802 is used to store executable instructions of the processor 801. It can be understood that the processor 801 is configured to execute instructions to implement the algorithm evaluation method in the above embodiment.

[0142] It should be noted that those skilled in the art can understand that Fig.12 The electronic device structure shown in the figure does not constitute a limitation on the electronic device, and the electronic device may include Fig.12 More or fewer components may be shown, or certain components may be combined, or the components may be arranged differently.

[0143] The processor 801 is the control center of the electronic device. It uses various interfaces and lines to connect various parts of the entire electronic device. By running or executing software programs and / or modules stored in the memory 802, and calling data stored in the memory 802, it performs various functions of the electronic device and processes data, thereby monitoring the electronic device as a whole. The processor 801 may include one or more processing units. Optionally, the processor 801 may integrate an application processor and a modem processor, wherein the application processor mainly processes the operating system, user interface, and application programs, and the modem processor mainly processes wireless communications. It is understandable that the above-mentioned modem processor may not be integrated into the processor 801.

[0144] The memory 802 can be used to store software programs and various data. The memory 802 can mainly include a program storage area and a data storage area, wherein the program storage area can store an operating system, an application program required by at least one functional module (acquisition unit 701, matching unit 702, and determination unit 703), etc. In addition, the memory 802 can include a high-speed random access memory, and can also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other volatile solid-state storage devices.

[0145] In an exemplary embodiment, a computer-readable storage medium including instructions is also provided, such as a memory 802 including instructions. The above instructions can be executed by the processor 801 of the electronic device 800 to implement the algorithm evaluation method in the above embodiment.

[0146] In actual implementation, Fig.11 The functions of the acquisition unit 701, the matching unit 702 and the determination unit 703 in Fig.12The processor 801 in the embodiment calls the computer program stored in the memory 802. The specific execution process can refer to the description of the algorithm evaluation method in the above embodiment, which will not be repeated here.

[0147] Optionally, the computer-readable storage medium may be a non-temporary computer-readable storage medium, for example, the non-temporary computer-readable storage medium may be a read-only memory (ROM), a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.

[0148] In an exemplary embodiment, the present application also provides a computer program product including one or more instructions, and the one or more instructions can be executed by a processor of an electronic device to complete the algorithm evaluation method in the above embodiment.

[0149] It should be noted that when the instructions in the above-mentioned computer-readable storage medium or one or more instructions in the computer program product are executed by the processor of the electronic device, the various processes of the above-mentioned algorithm evaluation method embodiment are implemented, and the same technical effect as the above-mentioned algorithm evaluation method can be achieved. To avoid repetition, they will not be repeated here.

[0150] Through the description of the above implementation methods, technical personnel in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional modules is used as an example. In actual applications, the above-mentioned functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above.

[0151] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic, for example, the division of modules or units is only a logical function division, and there may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0152] The units described as separate components may or may not be physically separated, and the components shown as units may be one physical unit or multiple physical units, that is, they may be located in one place or distributed in multiple different places. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0153] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0154] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a readable storage medium. Based on this understanding, the technical solution of the embodiment of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions to enable a device (which can be a single-chip microcomputer, chip, etc.) or a processor (processor) to perform all or part of the steps of the various embodiments of the present application. The aforementioned storage medium includes: various media that can store program codes, such as USB flash drives, mobile hard drives, ROM, RAM, magnetic disks or optical disks.

[0155] The above are only specific implementations of the present application, but the protection scope of the present application is not limited thereto, and any changes or substitutions within the technical scope disclosed in the present application should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. An algorithm evaluation method, characterized in that: The method comprises: Acquire at least one test data and at least one true value data corresponding to the target point cloud data; wherein the test data includes two-dimensional coordinate information of at least one detection point obtained by performing obstacle detection on the target point cloud data through the obstacle detection algorithm to be evaluated, and the true value data is used to reflect the two-dimensional coordinate information of the marked area where the obstacle in the target point cloud data is located; Matching at least one of the test data and at least one of the true value data according to the timestamp information to obtain at least one pair of first test data and first true value data with the same timestamp; Error information of the obstacle detection algorithm to be evaluated is determined according to the first test data and the first true value data.

2. The algorithm evaluation method according to claim 1, characterized in that: The test data and the true value data have the same coordinate system, and the error information of the obstacle detection algorithm to be evaluated includes the shortest distance between a first target detection point in the first test data and a boundary of a marked area of ​​the first true value data, and the first target detection point is a detection point among at least one detection point of the first test data that is closest to the boundary of the marked area of ​​the first true value data.

3. The algorithm evaluation method according to claim 1, characterized in that: The step of determining error information of the obstacle detection algorithm to be evaluated according to the first test data and the first true value data includes: Expanding the boundary of the marked area of ​​the first true value data by a preset threshold to obtain an enlarged area; taking a detection point located in the enlarged area among at least one detection point of the first test data as a candidate detection point; Error information of the obstacle detection algorithm to be evaluated is determined according to a boundary between the at least one candidate detection point and a marked area of ​​the first true value data.

4. The algorithm evaluation method according to claim 1, characterized in that: The step of determining error information of the obstacle detection algorithm to be evaluated according to the first test data and the first true value data includes: Determining a target edge of the marked area according to a relative position relationship between the test vehicle and the marked area of ​​the first true value data; Error information of the obstacle detection algorithm to be evaluated is determined according to at least one detection point of the first test data and the target edge.

5. The algorithm evaluation method according to claim 4, characterized in that: The determining error information of the obstacle detection algorithm to be evaluated according to at least one detection point of the first test data and the target edge includes: Determine a detection area according to at least one detection point of the first test data; wherein the detection area is a minimum area covering at least one detection point of the first test data; Determine at least one second target detection point of the first test data; wherein the second target detection point is the detection point whose distance from the boundary of the detection area is less than or equal to a second preset distance; Error information of the obstacle detection algorithm to be evaluated is determined according to at least one second target detection point of the first test data and a target edge of the marked area.

6. The algorithm evaluation method according to any one of claims 1 to 5, characterized in that: The error information of the obstacle detection algorithm to be evaluated includes the shortest distance between the first target detection point in the first test data and the boundary of the marked area of ​​the first true value data; the method further includes: According to at least one shortest distance between each pair of first test data and first true value data with the same timestamp, means and variances of the shortest distances in different intervals are determined.

7. An algorithm evaluation device, characterized in that: The device comprises: an acquisition unit, a matching unit and a determination unit; The acquisition unit is used to acquire at least one test data and at least one true value data corresponding to the target point cloud data; wherein the test data includes two-dimensional coordinate information of at least one detection point obtained by performing obstacle detection on the target point cloud data through the obstacle detection algorithm to be evaluated, and the true value data is used to reflect the two-dimensional coordinate information of the marked area where the obstacle in the target point cloud data is located; The matching unit is used to match at least one of the test data and at least one of the true value data according to the timestamp information to obtain at least one pair of first test data and first true value data with the same timestamp; The determining unit is used to determine error information of the obstacle detection algorithm to be evaluated according to the first test data and the first true value data.

8. An electronic device, characterized in that: include: processor; a memory for storing instructions executable by the processor; The processor is configured to execute the instructions to implement the method according to any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that: When the computer-executable instructions stored in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device is capable of performing the method as claimed in any one of claims 1 to 6.

10. A computer program product, characterized in that The computer program product comprises computer instructions, and when the computer instructions are executed on an electronic device, the electronic device performs the method according to any one of claims 1 to 6.

Images