Evaluation methods, devices, equipment, and media for autonomous driving perception algorithms

Abstract

This disclosure provides a method, apparatus, electronic device, computer-readable storage medium, and computer program product for evaluating autonomous driving perception algorithms, relating to the field of artificial intelligence, and particularly to the field of autonomous driving technology. The implementation scheme includes: acquiring scene data and test results, wherein the test results indicate whether a first traffic participant in a target scenario will pose a driving risk to the main vehicle; determining a target area based on the scene data, wherein the target area indicates the area that the first traffic participant may pass through within a first time period; processing the scene data using an autonomous driving perception algorithm to obtain simulation test data; determining at least one second traffic participant falling into the target area within the first time period and at least one perception test result corresponding to the at least one second traffic participant based on the simulation test data; and assessing the perception capability of the autonomous driving perception algorithm based on the road test results and at least one perception test result.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the field of artificial intelligence, in particular to the field of automatic driving, and more particularly to a method and apparatus for evaluating an automatic driving perception algorithm, an electronic device, a computer readable storage medium, and a computer program product. BACKGROUND

[0002] In the related art, when evaluating an automatic driving perception algorithm, it is usually necessary to determine whether the perception result of the algorithm is consistent with the actual road test result by manually reviewing simulation test video data, which is costly and inefficient. SUMMARY

[0003] The present disclosure provides a method and apparatus for evaluating an automatic driving perception algorithm, an electronic device, a computer readable storage medium, and a computer program product.

[0004] According to an aspect of the present disclosure, a method for evaluating an automatic driving perception algorithm is provided, comprising: obtaining scene data and a measured result, wherein the scene data is collected when a host vehicle drives in a target scene including a first traffic participant, and the measured result indicates whether the first traffic participant in the target scene will cause driving risk to the host vehicle; determining a target region based on the scene data, wherein the target region indicates a region that the first traffic participant is likely to pass through in a first time period; processing the scene data using the automatic driving perception algorithm to obtain simulation test data output by the automatic driving perception algorithm; determining at least one second traffic participant falling into the target region in the first time period and at least one perception test result corresponding to the at least one second traffic participant based on the simulation test data, wherein each perception test result indicates whether the corresponding second traffic participant is perceived to cause driving risk to the host vehicle; and evaluating the perception ability of the automatic driving perception algorithm based on the road measured result and the at least one perception test result.

[0005] According to another aspect of the present disclosure, there is provided an evaluation device for an automatic driving perception algorithm, comprising: an acquisition unit configured to acquire scene data and a measured result, wherein the scene data is collected when a host vehicle drives in a target scene including a first traffic participant, and the measured result indicates whether the first traffic participant in the target scene causes driving risk to the host vehicle; a first determination unit configured to determine a target area based on the scene data, wherein the target area indicates an area that the first traffic participant is likely to pass through in a first time period; a processing unit configured to process the scene data using the automatic driving perception algorithm to obtain simulation test data output by the automatic driving perception algorithm; a second determination unit configured to determine, based on the simulation test data, at least one second traffic participant in the target area in the first time period and at least one perception test result corresponding to the at least one second traffic participant, wherein each perception test result indicates whether the corresponding second traffic participant is perceived to cause driving risk to the host vehicle; and an evaluation unit configured to evaluate the perception capability of the automatic driving perception algorithm based on the road measured result and the at least one perception test result.

[0006] According to another aspect of the present disclosure, there is provided an electronic device, comprising: at least one processor; and a memory communicatively connected to the at least one processor; the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the above-mentioned evaluation method for the automatic driving perception algorithm.

[0007] According to another aspect of the present disclosure, there is provided a non-transitory computer-readable storage medium storing computer instructions for causing a computer to perform the above-mentioned evaluation method for the automatic driving perception algorithm.

[0008] According to another aspect of the present disclosure, there is provided a computer program product comprising a computer program which, when executed by a processor, implements the above-mentioned evaluation method for the automatic driving perception algorithm.

[0009] According to one or more embodiments of the present disclosure, the target area that a traffic participant is likely to pass through is determined based on the scene data generated by actual road testing, and the perception capability of the automatic driving perception algorithm for the target area is scored in simulation testing, so that the target area associated with the traffic participant being evaluated can be used to replace the traffic participant to implement perception capability evaluation, overcoming the defect that the same traffic participant cannot be locked in multiple tests due to ID inconsistency of the same traffic participant in the scene data of road testing and the test data of simulation testing, effectively improving the evaluation efficiency for the automatic driving perception algorithm, and greatly reducing the evaluation cost.

[0010] It is to be understood that the details set forth herein do not limit the scope of the embodiments of the present disclosure but merely contemplate it to a preferred embodiment. Other features, aspects, and advantages of the present disclosure will become apparent from the following description, from the appended claims, and from the numerous drawings that are presented. BRIEF DESCRIPTION OF DRAWINGS

[0011] The accompanying drawings, which are included to provide a further understanding of the application and are incorporated in

[0012] Figure 1 shows a schematic diagram of an exemplary system in which various methods described herein can be implemented, in accordance with an embodiment of the present disclosure;

[0013] Figure 2 shows an exemplary flowchart of a method for evaluating an autonomous driving perception algorithm, in accordance with an embodiment of the present disclosure;

[0014] Figure 3 shows a partial exemplary flowchart of a method for evaluating an autonomous driving perception algorithm, in accordance with an embodiment of the present disclosure;

[0015] Figure 4 shows another partial exemplary flowchart of a method for evaluating an autonomous driving perception algorithm, in accordance with an embodiment of the present disclosure;

[0016] Figure 5 shows a schematic diagram of determining a target region from a first position and a second position, in accordance with an embodiment of the present disclosure;

[0017] Figure 6 shows yet another partial exemplary flowchart of a method for evaluating an autonomous driving perception algorithm, in accordance with an embodiment of the present disclosure;

[0018] Figure 7 shows still another partial exemplary flowchart of a method for evaluating an autonomous driving perception algorithm, in accordance with an embodiment of the present disclosure;

[0019] Figure 8 shows a structural block diagram of an evaluation apparatus for an autonomous driving perception algorithm, in accordance with an embodiment of the present disclosure; and

[0020] Figure 9 shows a structural block diagram of an exemplary electronic device that can be used to implement an embodiment of the present disclosure. DETAILED DESCRIPTION

[0021] Exemplary embodiments of the present disclosure are described herein below with reference to the accompanying drawings, in which various specific details are set forth to assist in understanding the present disclosure. It will be apparent, however, to one of ordinary skill in the art that various changes and modifications can be made to the embodiments described herein without departing from the scope of the present disclosure. Also, descriptions of well-known functions and constructions are omitted for clarity and conciseness.

[0022] In the present disclosure, the terms "first", "second", and the like are used to describe various elements only for the purpose of distinguishing one element from another, and do not intend to limit the positions, sequence, or importance of the elements. In some examples, a first element and a second element can refer to the same instance of the element, and in some cases, they can refer to different instances of the element based on the context of the description.

[0023] The terms used in the description of various described examples in the present disclosure are only for the purpose of describing particular examples and are not intended to be limiting. Unless specifically defined otherwise, an element that is a singular can be plural and vice versa. Also, the term "and / or" used in the present disclosure encompasses any and all possible combinations of the listed items.

[0024] Embodiments of the present disclosure will be described in detail below with reference to the accompanying drawings.

[0025] Figure 1 A schematic diagram of an example system 100 in which various methods and apparatus described herein can be implemented according to embodiments of the present disclosure is shown. Referring to Figure 1 The system 100 includes a motor vehicle 110, a server 120, and one or more communication networks 130 coupling the motor vehicle 110 to the server 120.

[0026] In embodiments of the present disclosure, the motor vehicle 110 can include a computing device according to embodiments of the present disclosure and / or be configured to perform a method according to embodiments of the present disclosure.

[0027] The server 120 can run one or more services or software applications that enable the evaluation method for automated driving perception algorithms. In certain embodiments, the server 120 can also provide other services or software applications, which can include non-virtual environments and virtual environments. In Figure 1In the configuration shown, server 120 may include one or more components that implement the functions performed by server 120. These components may include software components, hardware components, or combinations thereof that can be executed by one or more processors. A user of motor vehicle 110 may sequentially interact with server 120 using one or more client applications to utilize the services provided by these components. It should be understood that various different system configurations are possible and may differ from system 100. Therefore, Figure 1 This is an example of a system used to implement the various methods described herein, and is not intended to be limiting.

[0028] Server 120 may include one or more general-purpose computers, special-purpose server computers (e.g., PC (personal computer) servers, UNIX servers, mid-range servers), blade servers, mainframe computers, server clusters, or any other suitable arrangement and / or combination. Server 120 may include one or more virtual machines running a virtual operating system, or other computing architectures involving virtualization (e.g., one or more flexible pools of logical storage devices that can be virtualized to maintain virtual storage devices for servers). In various embodiments, server 120 may run one or more services or software applications that provide the functionality described below.

[0029] The computing unit in server 120 can run one or more operating systems, including any of the aforementioned operating systems and any commercially available server operating system. Server 120 can also run any of a variety of additional server applications and / or middleware applications, including HTTP servers, FTP servers, CGI servers, JAVA servers, database servers, etc.

[0030] In some implementations, server 120 may include one or more applications to analyze and merge data feeds and / or event updates received from vehicle 110. Server 120 may also include one or more applications to display data feeds and / or real-time events via one or more display devices of vehicle 110.

[0031] Network 130 can be any type of network well known to those skilled in the art, and can support data communication using any of a variety of available protocols (including, but not limited to, TCP / IP, SNA, IPX, etc.). By way of example only, one or more networks 130 can be satellite communication networks, local area networks (LANs), Ethernet-based networks, token ring networks, wide area networks (WANs), the Internet, virtual networks, virtual private networks (VPNs), intranets, extranets, blockchain networks, public switched telephone networks (PSTNs), infrared networks, wireless networks (including, for example, Bluetooth, WiFi), and / or any combination of these with other networks.

[0032] System 100 may also include one or more databases 150. In some embodiments, these databases may be used to store data and other information. For example, one or more of the databases 150 may be used to store information such as audio files and video files. The data repository 150 may reside in various locations. For example, a data repository used by server 120 may be local to server 120, or it may be located away from server 120 and may communicate with server 120 via a network-based or dedicated connection. The data repository 150 may be of different types. In some embodiments, the data repository used by server 120 may be a database, such as a relational database. One or more of these databases may store, update, and retrieve data from and from the database in response to commands.

[0033] In some embodiments, one or more of the databases 150 may also be used by an application to store application data. The databases used by the application may be of different types, such as key-value stores, object stores, or regular stores supported by a file system.

[0034] Motor vehicle 110 may include sensors 111 for sensing the surrounding environment. Sensors 111 may include one or more of the following sensors: a visual camera, an infrared camera, an ultrasonic sensor, a millimeter-wave radar, and a lidar (LiDAR). Different sensors can provide different detection accuracy and range. Cameras may be mounted in front of, behind, or at other locations on the vehicle. Visual cameras can capture the situation inside and outside the vehicle in real time and present it to the driver and / or passengers. In addition, by analyzing the images captured by the visual cameras, information such as traffic light indications, intersection conditions, and the operating status of other vehicles can be obtained. Infrared cameras can capture objects in night vision conditions. Ultrasonic sensors may be mounted around the vehicle to measure the distance of objects outside the vehicle using the strong directionality of ultrasound. Millimeter-wave radar may be mounted in front of, behind, or at other locations on the vehicle to measure the distance of objects outside the vehicle using the characteristics of electromagnetic waves. LiDAR may be mounted in front of, behind, or at other locations on the vehicle to detect the edges and shape information of objects, thereby performing object recognition and tracking. Due to the Doppler effect, the radar device can also measure the speed changes of the vehicle and moving objects.

[0035] The motor vehicle 110 may also include a communication device 112. The communication device 112 may include a satellite positioning module capable of receiving satellite positioning signals (e.g., BeiDou, GPS, GLONASS, and GALILEO) from satellite 141 and generating coordinates based on these signals. The communication device 112 may also include a module for communicating with a mobile communication base station 142. The mobile communication network can implement any suitable communication technology, such as current or emerging wireless communication technologies (e.g., 5G technology) like GSM / GPRS, CDMA, and LTE. The communication device 112 may also have a vehicle-to-everything (V2X) module, configured to enable vehicle-to-the-world communication, for example, vehicle-to-vehicle (V2V) communication with other vehicles 143 and vehicle-to-infrastructure (V2I) communication with infrastructure 144. Furthermore, the communication device 112 may also have a module configured to communicate with a user terminal 145 (including but not limited to smartphones, tablets, or wearable devices such as watches) via, for example, a wireless local area network conforming to the IEEE 802.11 standard or Bluetooth. Using the communication device 112, the motor vehicle 110 can also access the server 120 via the network 130.

[0036] The motor vehicle 110 may also include a control unit 113. The control unit 113 may include a processor, such as a central processing unit (CPU) or a graphics processing unit (GPU), or other dedicated processors, that communicates with various types of computer-readable storage devices or media. The control unit 113 may include an autonomous driving system for automatically controlling various actuators in the vehicle. The autonomous driving system is configured to control the powertrain, steering system, and braking system of the motor vehicle 110 (not shown) via multiple actuators in response to inputs from multiple sensors 111 or other input devices to control acceleration, steering, and braking respectively, without human intervention or with limited human intervention. Some processing functions of the control unit 113 can be implemented via cloud computing. For example, some processing can be performed using an onboard processor while other processing can be performed using cloud computing resources. The control unit 113 may be configured to perform methods according to this disclosure. Furthermore, the control unit 113 may be implemented as an example of a computing device on the motor vehicle side (client) according to this disclosure.

[0037] Figure 1 The system 100 can be configured and operated in various ways to enable the application of the various methods and apparatus described in this disclosure.

[0038] In related technologies, when evaluating autonomous driving perception algorithms, it is usually necessary to manually review the video data of simulation tests to determine whether the perception results of the algorithm are consistent with the actual road test results, which is costly and inefficient.

[0039] to this end, Figure 2 An exemplary flowchart of an evaluation method for an autonomous driving perception algorithm according to an embodiment of the present disclosure is shown.

[0040] like Figure 2 As shown, according to an embodiment of this disclosure, an evaluation method 200 for an autonomous driving perception algorithm is provided, comprising: acquiring scene data and test results, wherein the scene data is collected when the main vehicle is driving in a target scene including a first traffic participant, and the test results indicate whether the first traffic participant in the target scene will pose a driving risk to the main vehicle (step 210); determining a target area based on the scene data, wherein the target area indicates the area that the first traffic participant may pass through in a first time period (step 220); processing the scene data using an autonomous driving perception algorithm to obtain simulation test data output by the autonomous driving perception algorithm (step 230); determining at least one second traffic participant that falls into the target area in the first time period and at least one perception test result corresponding to the at least one second traffic participant based on the simulation test data, wherein each perception test result indicates whether the corresponding second traffic participant is perceived to pose a driving risk to the main vehicle (step 240); and evaluating the perception capability of the autonomous driving perception algorithm based on the road test results and at least one perception test result (step 250).

[0041] According to the evaluation method for autonomous driving perception algorithms in this disclosure, the target area that the traffic participant may pass through is determined based on the scene data generated from actual road tests. The perception ability of the autonomous driving perception algorithm for the target area is scored in the simulation test. This allows the target area associated with the traffic participant to be used to replace the traffic participant in the evaluation to achieve perception ability evaluation. This overcomes the defect that the same traffic participant cannot be locked in multiple tests due to the inconsistency of ID between the scene data of the actual road test and the test data of the simulation test. This effectively improves the evaluation efficiency of autonomous driving perception algorithms and significantly reduces the evaluation cost.

[0042] In step 210, scenario data and test results are acquired. The scenario data is collected when the main vehicle is driving in the target scenario including the first traffic participant. The test results indicate whether the first traffic participant in the target scenario will pose a driving risk to the main vehicle.

[0043] In some embodiments, the scene data described above may be video data and / or point cloud data, and there are no limitations on this.

[0044] In some embodiments, the aforementioned scenario data and test results may be collected directly from actual road tests of the main vehicle in the target scenario, or they may be obtained from historical actual road test scenario data and test results from a database, without limitation.

[0045] In some embodiments, the first traffic participant includes, but is not limited to, motor vehicles, non-motor vehicles, pedestrians, and other dynamic obstacles that may appear in the target scene.

[0046] In one example, the first traffic participant is a motor vehicle, and the target scenario is a situation where the main vehicle and the motor vehicle are traveling in opposite directions along the lane lines; in another example, the first traffic participant is a fallen leaf, and the target scenario is a fallen leaf being blown by the wind towards the main vehicle. It should be understood that the above examples are for illustrative purposes only and are not intended to limit the scope of the traffic activity.

[0047] In step 220, a target area is determined based on scene data, wherein the target area indicates the area that the first traffic participant may pass through during the first time period.

[0048] Figure 3 A partial exemplary flowchart of an evaluation method for autonomous driving perception algorithms according to embodiments of the present disclosure is shown.

[0049] According to some embodiments, such as Figure 3 As shown, step 220 includes: determining the first position of the first traffic participant at the start time of the first time period and the second position at the end time of the first time period based on scene data (step 321); and determining the target area based on the first position and the second position (step 322).

[0050] By observing the positions of the first traffic participant at the start and end times, we can obtain the positional changes of the first traffic participant within the first time period. Based on these positional changes, we can at least partially determine the area traversed by the first traffic participant and use it as the target area associated with the first traffic participant. This effectively improves the association between the target area and the first traffic participant, thereby enhancing the evaluation accuracy of the algorithm's perception capability based on the positions of the first traffic participant and the target.

[0051] In step 321, the first position of the main vehicle at the start time of the first time period and the second position at the end time of the first time period are determined based on the scene data.

[0052] In some embodiments, a start frame corresponding to the start time of the first time period and an end frame corresponding to the end time of the first time period can be obtained from the scene data to determine the first position of the main vehicle from the start frame and the second position of the main vehicle from the end frame.

[0053] In some embodiments, the first and second locations mentioned above can be actual geographical coordinates based on a high-precision map, or location coordinates based on other specific reference systems, and there are no restrictions on this.

[0054] In step 322, the target area is determined based on the first position and the second position.

[0055] In some embodiments, the first position and the second position may fall within the scope of the target area, or they may fall outside the scope of the target area, or one may fall within the scope of the target area and the other may fall outside the scope of the target area, without limitation.

[0056] In some embodiments, the shape of the target region can be regular or irregular, without limitation.

[0057] For scenarios where the primary traffic participant is a motor vehicle or non-motor vehicle traveling along the lane lines, etc. Figure 4 Another exemplary flowchart of an evaluation method for autonomous driving perception algorithms according to embodiments of the present disclosure is shown.

[0058] According to some embodiments, such as Figure 4 As shown, step 322 includes: using the line connecting the first position and the second position as a reference, moving the first position and the second position to the left by a first distance in a direction perpendicular to the line to obtain the first corner point and the second corner point (step 4221); using the line connecting the first position and the second position as a reference, moving the first position and the second position to the right by a second distance in a direction perpendicular to the line to obtain the third corner point and the fourth corner point (step 4222); and taking the rectangular area enclosed by the first corner point, the second corner point, the third corner point and the fourth corner point as the target area (step 4223).

[0059] By using this method to obtain the four corner points to form a rectangle, the pattern of the first traffic participant driving along the lane line is effectively adapted. This allows for a more accurate inclusion of the area that the first traffic participant may pass through in the target area, effectively enhancing the correlation between the target area and the first traffic participant. This, in turn, improves the evaluation efficiency while ensuring the accuracy of the evaluation of the autonomous driving perception algorithm.

[0060] Figure 5 A schematic diagram illustrating the determination of a target region based on a first position and a second position according to an embodiment of the present disclosure is shown. Figure 5 As shown, the coordinates of the first position A relative to the current reference frame are (x... a y a The coordinates of the second position B relative to the current reference frame are (x...). b y b ).

[0061] In step 4221, taking the line AB connecting the first position A and the second position B as a reference, the first position A and the second position B are moved to the left by a first distance d1 in a direction perpendicular to the line AB to obtain the first corner point C and the second corner point D.

[0062] In step 4222, the first position A and the second position B are moved to the right by a second distance d2 along a direction perpendicular to the line AB to obtain the third corner point E and the fourth corner point F.

[0063] In step 4223, the rectangular area CDEF enclosed by the first corner point C, the second corner point D, the third corner point E, and the fourth corner point F is taken as the target area.

[0064] In some embodiments, the first distance d1 and the second distance d2 can be the same or different. In one example, the lane width of the first traffic participant is 5 meters, so the first distance d1 and the second distance d2 can both be set to 5 meters. In this case, the target area CDEF is twice the lane width, and regardless of whether the first traffic participant changes lanes during the first time period, its travel route can be included in the target area CDEF.

[0065] It should be noted that the above examples are for illustrative purposes only and are not intended to limit the scope of the example. For example, the first distance d1 can be 0.5 meters, 3 meters, or 10 meters, and there are no restrictions on this.

[0066] In step 230, the scene data is processed using an autonomous driving perception algorithm to obtain the simulation test data output by the autonomous driving perception algorithm.

[0067] In some embodiments, after obtaining the simulation test data, the data frames in the simulation test data can be filtered based on the timestamp to retain the data frames corresponding to the first time period and delete other data frames, thereby reducing the storage space occupancy rate.

[0068] In step 240, based on simulation test data, at least one second traffic participant that falls into the target area within the first time period and at least one perception test result corresponding to the at least one second traffic participant are determined, wherein each perception test result indicates whether the corresponding second traffic participant is perceived to pose a driving risk to the main vehicle.

[0069] By obtaining the perception test results of the autonomous driving algorithm for the target area in the first time period based on the perception test results of the second traffic participant that intersects with the target area in the first time period, it is possible to avoid matching traffic participants in the actual road test and the simulation test one by one, so as to achieve the above evaluation.

[0070] However, when the target area is large, some traffic participants located far from the main vehicle may appear in the target area during the initial time period, but they may not pose a driving risk to the main vehicle. Based on this, Figure 6 Further exemplary flowcharts of an evaluation method for autonomous driving perception algorithms according to embodiments of the present disclosure are shown.

[0071] According to some embodiments, such as Figure 6 As shown, step 240 includes: obtaining multiple data frames corresponding to the first time period from the simulation test data (step 641); for each data frame, determining at least one third traffic participant whose distance from the main vehicle is less than the collision threshold in the data frame (step 6421); and designating the third traffic participant that falls into the target area among the at least one third traffic participant as the second traffic participant (step 6422); and traversing multiple data frames to obtain at least one second traffic participant (step 643).

[0072] By determining the distance between the third traffic participant and the main vehicle, it is possible to rule out the possibility that some third traffic participants who are far away from the main vehicle and will not pose a collision risk but will pass through the target area are the second traffic participants, thereby further improving the efficiency and accuracy of evaluating autonomous driving perception algorithms.

[0073] Figure 7 A further exemplary flowchart of an evaluation method for autonomous driving perception algorithms according to embodiments of the present disclosure is shown.

[0074] According to some embodiments, such as Figure 7 As shown, step 240 includes: for each second traffic participant, obtaining the first size perceived by the second traffic participant in the first data frame, wherein the first data frame is the data frame with the smallest perceived size of the second traffic participant among multiple data frames, and the first size includes at least one of the first length, first width, and first height of the second traffic participant (step 7411); obtaining the second size of the second traffic participant perceived in the second data frame corresponding to the first size, wherein the second data frame is the data frame with the largest perceived size of the second traffic participant among multiple data frames, and the second size includes at least one of the second length, second width, and second height of the second traffic participant (step 7412); determining the size change value of the second traffic participant based on the first size and the second size (step 7413); determining the perception test result corresponding to the second traffic participant based on the temporal relationship between the first data frame and the second data frame and the size change value (step 7414); and traversing at least one second traffic participant to obtain at least one perception test result (step 742).

[0075] For steps 7411 and 7412, in order to obtain the size change value of the second traffic participant, the second size needs to correspond to the first size. For example, when the first size includes the first height, the second size must include at least the second height; similarly, when the first size includes the first width and the first length, the second size must include at least the second width and the second length, and so on.

[0076] In some embodiments, the area or volume of the second traffic participant can be calculated based on the obtained length, width and height, so as to use the area or volume of the second traffic participant as the first dimension and the second dimension mentioned above.

[0077] For steps 7413 and 7414, the dimensional change values ​​include, but are not limited to, the difference between the second dimension and the first dimension, the absolute value of the difference between the second dimension and the first dimension, and the dimensional change rate of the second dimension relative to the first dimension, etc.

[0078] In one example, the size change value is the difference between the second size and the first size. When the first time corresponding to the first data frame is earlier than the second time corresponding to the second data frame, the size change value is positive. The larger the size change value, the greater the distance the second traffic participant moves in the direction closer to the main vehicle, and the greater the perceived driving risk posed by the second traffic participant to the main vehicle. When the first time corresponding to the first data frame is later than the second time corresponding to the second data frame, the size change value is negative. The smaller the size change value, the greater the distance the second traffic participant moves in the direction away from the main vehicle, and the lower the perceived driving risk posed by the second traffic participant to the main vehicle. Based on this, the size change value will always be positively correlated with the perceived driving risk posed by the second traffic participant to the main vehicle.

[0079] For dynamic traffic participants, perceived changes in their size can effectively characterize the likelihood of them posing a driving risk to the primary vehicle. For example, in a scenario where a vehicle is traveling in the opposite direction to a primary vehicle in an adjacent lane, the greater the perceived size change of the vehicle, the higher the likelihood that it is encroaching on the primary vehicle's current lane, and consequently, the higher the potential driving risk to the primary vehicle. Therefore, based on the aforementioned perceived size changes, the perception test results for that traffic participant can be determined simply and efficiently, thus improving evaluation efficiency.

[0080] In some embodiments, the size change value may be the length change value, width change value, and / or height change value of the second traffic participant; in other embodiments, its area or volume may also be calculated based on the obtained length, width, and height of the second traffic participant, and the area change value or volume change value of the second traffic participant may be used as the aforementioned size change value, without limitation.

[0081] According to some embodiments, step 7414 includes: assigning a risk score to the second traffic participant based on the size change value, and using the risk score as the perception test result corresponding to the second traffic participant, wherein the size change value is positively correlated with the risk score. Furthermore, step 240 includes: determining a risk perception score for the autonomous driving perception algorithm based on at least one perception test result, wherein the risk perception score is positively correlated with the score of at least one risk score corresponding to at least one perception test result.

[0082] Based on the size changes of the second traffic participant, the autonomous driving perception algorithm can be evaluated in a quantitative way, thereby reducing the difficulty of evaluation and improving evaluation efficiency.

[0083] In some embodiments, the risk perception score may be the average, weighted average, maximum or minimum value of at least one risk score, without limitation.

[0084] According to some embodiments, the above method 200 further includes: returning the above-mentioned scene data, actual test results, simulation test data and evaluation results of the perception capability of the autonomous driving perception algorithm as training data to the autonomous driving perception algorithm, so as to update the network model parameters of the autonomous driving perception algorithm.

[0085] By returning the above data and evaluation results to the autonomous driving perception algorithm, the direction of the next generation update and iteration of the autonomous driving perception algorithm model can be effectively determined to adjust the network parameters, thereby improving the iteration efficiency and iteration effect of the algorithm model.

[0086] Figure 8 A structural block diagram of an evaluation apparatus for autonomous driving perception algorithms according to an embodiment of the present disclosure is shown.

[0087] like Figure 8As shown, according to an embodiment of this disclosure, an evaluation device 800 for an autonomous driving perception algorithm is provided, comprising: an acquisition unit 810 configured to acquire scene data and test results, wherein the scene data is collected when a main vehicle is driving in a target scene including a first traffic participant, and the test results indicate whether the first traffic participant in the target scene will pose a driving risk to the main vehicle; a first determination unit 820 configured to determine a target area based on the scene data, wherein the target area indicates an area that the first traffic participant may pass through in a first time period; a processing unit 830 configured to process the scene data using an autonomous driving perception algorithm to obtain simulation test data output by the autonomous driving perception algorithm; a second determination unit 840 configured to determine at least one second traffic participant falling into the target area in the first time period and at least one perception test result corresponding to the at least one second traffic participant based on the simulation test data, wherein each perception test result indicates whether the corresponding second traffic participant is perceived to pose a driving risk to the main vehicle; and an evaluation unit 850 configured to evaluate the perception capability of the autonomous driving perception algorithm based on the road test results and at least one perception test result.

[0088] Here, the operation of each of the above units 810 to 850 of the evaluation device 800 for autonomous driving perception algorithms is similar to the operation of steps 210 to 250 described above, and will not be repeated here.

[0089] According to embodiments of this disclosure, an electronic device, a readable storage medium, and a computer program product are also provided.

[0090] refer to Figure 9 The present invention describes a structural block diagram of an electronic device 900 that can serve as a server or client of the present disclosure, which is an example of a hardware device that can be applied to various aspects of the present disclosure. The electronic device is intended to represent various forms of digital electronic computer devices, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0091] like Figure 9As shown, the electronic device 900 includes a computing unit 901, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 902 or a computer program loaded into a random access memory (RAM) 903 from a storage unit 908. The RAM 903 may also store various programs and data required for the operation of the electronic device 900. The computing unit 901, ROM 902, and RAM 903 are interconnected via a bus 904. An input / output (I / O) interface 905 is also connected to the bus 904.

[0092] Multiple components in electronic device 900 are connected to I / O interface 905, including: input unit 906, output unit 907, storage unit 908, and communication unit 909. Input unit 906 can be any type of device capable of inputting information to electronic device 900. Input unit 906 can receive input digital or character information and generate key signal inputs related to user settings and / or function control of electronic device, and can include, but is not limited to, a mouse, keyboard, touchscreen, trackpad, trackball, joystick, microphone, and / or remote control. Output unit 907 can be any type of device capable of presenting information, and can include, but is not limited to, a monitor, speaker, video / audio output terminal, vibrator, and / or printer. Storage unit 908 can include, but is not limited to, disk and optical disk. Communication unit 909 allows electronic device 900 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks, and can include, but is not limited to, modems, network cards, infrared communication devices, wireless communication transceivers and / or chipsets, such as Bluetooth devices, 802.11 devices, WiFi devices, WiMax devices, cellular communication devices, and / or the like.

[0093] The computing unit 901 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 901 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning network algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 901 performs the various methods and processes described above, such as method 200. In some embodiments, method 200 is now a computer software program tangibly contained in a machine-readable medium, such as storage unit 908. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 900 via ROM 902 and / or communication unit 909. When the computer program is loaded into RAM 903 and executed by the computing unit 901, method 200 or more steps described above may be performed. Alternatively, in other embodiments, the computing unit 901 may be configured to perform method 200 by any other suitable means (e.g., by means of firmware).

[0094] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0095] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0096] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0097] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0098] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as data servers), or middleware components (e.g., application servers), or frontend components (e.g., user computers with graphical user interfaces or web browsers through which users can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., communication networks). Examples of communication networks include local area networks (LANs), wide area networks (WANs), the Internet, and blockchain networks.

[0099] Computer systems can include clients and servers. Clients and servers are generally located far apart and typically interact via communication networks. Client-server relationships are created by computer programs running on the respective computers and having a client-server relationship with each other. Servers can be cloud servers, servers in distributed systems, or servers incorporating blockchain technology.

[0100] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this disclosure can be performed in parallel, sequentially, or in a different order, as long as the desired result of the technical solution disclosed in this disclosure can be achieved, and this is not limited herein.

[0101] While embodiments or examples of this disclosure have been described with reference to the accompanying drawings, it should be understood that the methods, systems, and devices described above are merely exemplary embodiments or examples, and the scope of the invention is not limited by these embodiments or examples, but only by the granted claims and their equivalents. Various elements in the embodiments or examples may be omitted or replaced by their equivalents. Furthermore, the steps may be performed in a different order than that described in this disclosure. Further, various elements in the embodiments or examples may be combined in various ways. Importantly, as the technology evolves, many elements described herein can be replaced by equivalents that appear after this disclosure.

Claims

1. A method for evaluating an automatic driving perception algorithm, comprising: obtaining scene data and a measured result, wherein the scene data is collected when a host vehicle drives in a target scene including a first traffic participant, and the measured result indicates whether the first traffic participant in the target scene causes driving risk to the host vehicle; determining a target region based on the scene data, wherein the target region indicates a region that the first traffic participant is likely to pass through in a first time period; processing the scene data using the automatic driving perception algorithm to obtain simulation test data output by the automatic driving perception algorithm; determining at least one second traffic participant in the target region in the first time period and at least one perception test result corresponding to the at least one second traffic participant based on the simulation test data, wherein each perception test result indicates whether a corresponding second traffic participant is perceived to cause driving risk to the host vehicle, and wherein the at least one perception test result is determined based on a change in size of the at least one second traffic participant perceived in the first time period; and evaluating a perception capability of the automatic driving perception algorithm based on the measured result and the at least one perception test result.

2. The method of claim 1, wherein, The determining the target region based on the scene data comprises: determining a first position of the first traffic participant at a start time of the first time period and a second position of the first traffic participant at an end time of the first time period based on the scene data; and determining the target region based on the first position and the second position.

3. The method of claim 2, wherein, The determining the target region based on the first position and the second position comprises: moving the first position and the second position to the left by a first distance in a direction perpendicular to a line connecting the first position and the second position to obtain a first corner point and a second corner point; moving the first position and the second position to the right by a second distance in the direction perpendicular to the line to obtain a third corner point and a fourth corner point; and taking a rectangular region surrounded by the first corner point, the second corner point, the third corner point, and the fourth corner point as the target region.

4. The method of any one of claims 1-3, wherein, The determining the at least one second traffic participant in the target region in the first time period comprises: obtaining a plurality of data frames corresponding to the first time period from the simulation test data; for each data frame, determining at least one third traffic participant in the data frame that is less than a collision threshold from the host vehicle; and taking a third traffic participant in the at least one third traffic participant that falls into the target region as the second traffic participant; and traversing the plurality of data frames to obtain the at least one second traffic participant.

5. The method of claim 4, wherein, The determining the at least one perception test result corresponding to the at least one second traffic participant comprises: for each second traffic participant, obtaining a first size of the second traffic participant perceived in a first data frame, wherein the first data frame is a data frame in which the second traffic participant is perceived to have a smallest size in the plurality of data frames, and the first size comprises at least one of a first length, a first width, and a first height of the second traffic participant; obtaining a second size of the second traffic participant corresponding to the first size and perceived in a second data frame, wherein the second data frame is a data frame in which the second traffic participant is perceived to have a largest size in the plurality of data frames, and the second size comprises at least one of a second length, a second width, and a second height of the second traffic participant; determining a size change value of the second traffic participant based on the first size and the second size; and determining a perception test result corresponding to the second traffic participant based on a timing relationship between the first data frame and the second data frame and the size change value; and traversing the at least one second traffic participant to obtain the at least one perception test result.

6. The method of claim 5, wherein, The determining of the perception test result corresponding to the second traffic participant based on the size change value comprises: performing a risk score on the second traffic participant based on the size change value, and taking a score of the risk score as the perception test result corresponding to the second traffic participant, wherein the size change value is positively correlated with the score of the risk score. And wherein the evaluating of the perception capability of the automatic driving perception algorithm based on the actual measurement result and the at least one perception test result comprises: determining a risk perception score of the automatic driving perception algorithm according to the at least one perception test result, wherein the risk perception score is positively correlated with at least one risk score corresponding to the at least one perception test result.

7. The method of any one of claims 1-3, further comprising: returning the scene data, the actual measurement result, the simulation test data, and the evaluation result of the perception capability of the automatic driving perception algorithm as training data to the automatic driving perception algorithm to update network model parameters of the automatic driving perception algorithm.

8. An evaluation device for an automatic driving perception algorithm, comprising: an obtaining unit configured to obtain scene data and an actual measurement result, wherein the scene data is collected when a host vehicle drives in a target scene including a first traffic participant, and the actual measurement result indicates whether the first traffic participant in the target scene causes a driving risk to the host vehicle; a first determining unit configured to determine a target region based on the scene data, wherein the target region indicates a region that the first traffic participant is likely to pass through in a first time period; a processing unit configured to process the scene data by using the automatic driving perception algorithm to obtain simulation test data output by the automatic driving perception algorithm; and a second determining unit configured to determine a perception test result corresponding to the first traffic participant based on a timing relationship between a first data frame and a second data frame and a size change value of the first traffic participant, wherein the first data frame is a data frame in which the first traffic participant is perceived to have a smallest size in the plurality of data frames, the second data frame is a data frame in which the first traffic participant is perceived to have a largest size in the plurality of data frames, the size change value is determined based on a first size and a second size of the first traffic participant, the first size comprises at least one of a first length, a first width, and a first height of the first traffic participant, the second size comprises at least one of a second length, a second width, and a second height of the first traffic participant, and the perception test result corresponding to the first traffic participant is determined based on a score of a risk score of the first traffic participant. a second determining unit, configured to determine, based on the simulation test data, at least one second traffic participant falling into the target region in the first time period and at least one perception test result corresponding to the at least one second traffic participant, wherein each perception test result indicates whether the corresponding second traffic participant is perceived to cause driving risk to the host vehicle, and wherein the at least one perception test result is determined based on a variation of a size of the at least one second traffic participant perceived in the first time period; and an evaluation unit, configured to evaluate the perception capability of the automatic driving perception algorithm based on the actual measurement result and the at least one perception test result.

9. The apparatus of claim 8, wherein, The first determining unit comprises: a first determining sub-unit, configured to determine, based on the scene data, a first position of the first traffic participant at a start time of the first time period and a second position of the first traffic participant at an end time of the first time period; and a second determining sub-unit, configured to determine the target region based on the first position and the second position.

10. The apparatus of claim 9, wherein, The second determining sub-unit comprises: a first corner point sub-unit, configured to move the first position and the second position by a first distance to the left along a direction perpendicular to a line connecting the first position and the second position, to obtain a first corner point and a second corner point; a second corner point sub-unit, configured to move the first position and the second position by a second distance to the right along the direction perpendicular to the line, to obtain a third corner point and a fourth corner point; and a third determining sub-unit, configured to take a rectangular region surrounded by the first corner point, the second corner point, the third corner point and the fourth corner point as the target region.

11. The apparatus of any of claims 8-10, wherein, The second determining unit comprises: an obtaining sub-unit, configured to obtain, from the simulation test data, a plurality of data frames corresponding to the first time period; a first processing sub-unit, configured to, for each data frame, determine at least one third traffic participant in the data frame having a distance to the host vehicle less than a collision threshold; and take a third traffic participant falling into the target region among the at least one third traffic participant as a second traffic participant; and a first traversing sub-unit, configured to traverse the plurality of data frames to obtain the at least one second traffic participant.

12. The apparatus of claim 11, wherein, The second determining unit comprises: a second processing sub-unit, configured to, for each second traffic participant, obtain a first size of the second traffic participant perceived in a first data frame, wherein the first data frame is a data frame in which the second traffic participant has a smallest perceived size among the plurality of data frames, and the first size comprises at least one of a first length, a first width and a first height of the second traffic participant; and a second traversing sub-unit, configured to traverse the at least one perception test result corresponding to the second traffic participant to obtain a perception test result of the second traffic participant. obtaining a second size corresponding to the first size that the second traffic participant is perceived in a second data frame, wherein the second data frame is a data frame in which the second traffic participant is perceived to have the largest size in the plurality of data frames, and the second size includes at least one of a second length, a second width, and a second height of the second traffic participant; determining a size change value of the second traffic participant based on the first size and the second size; and determining a perception test result corresponding to the second traffic participant based on a timing relationship between the first data frame and the second data frame and the size change value; and traversing the at least one second traffic participant to obtain the at least one perception test result.

13. The apparatus of claim 12, wherein, The second processing subunit includes: a scoring subunit configured to score a risk of the second traffic participant based on the size change value, and take a score of the risk score as the perception test result corresponding to the second traffic participant, wherein the size change value is positively correlated with the score of the risk score; and wherein the evaluation unit includes: a third determining subunit configured to determine a risk perception score of the autonomous driving perception algorithm according to the at least one perception test result, wherein the risk perception score is positively correlated with at least one risk score corresponding to the at least one perception test result.

14. The apparatus of any one of claims 8-10, further comprising: an updating unit configured to return the scene data, the actual measurement result, the simulation test data, and the evaluation result of the perception capability of the autonomous driving perception algorithm to the autonomous driving perception algorithm as training data to update network model parameters of the autonomous driving perception algorithm.

15. An electronic device, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-7.

16. A non-transitory computer readable storage medium having stored thereon computer instructions, wherein, The computer instructions are used to enable the computer to perform the method of any one of claims 1-7.

17. A computer program product comprising a computer program, wherein, The computer program, when executed by a processor, implements the method of any one of claims 1-7. The computer program, when executed by a processor, implements the method of any one of claims 1-7.

Images