Autonomous driving evaluation method and apparatus, and autonomous driving device

Abstract

The present application provides an autonomous driving evaluation method and apparatus, and an autonomous driving device, capable of simultaneously evaluating the sensing capability and the planning control capability of an autonomous driving algorithm, and improving the reliability of autonomous driving algorithm evaluation. The method comprises: acquiring label information corresponding to environment information of an environment where an autonomous driving device is located, wherein the label information is used for indicating an actual state of a target object in the environment; acquiring a processing result of an autonomous driving algorithm for the environment information, wherein the autonomous driving algorithm is used for sensing the environment information and performing planning control on a driving behavior on the basis of an environment sensing result; and on the basis of the label information and the processing result, obtaining the environment sensing capability and the planning control capability of the autonomous driving algorithm.

Description

Automatic driving evaluation method, device and automatic driving equipment Technical Field

[0001] The present application relates to the field of autonomous driving, and in particular to an autonomous driving evaluation method, device, and autonomous driving equipment. Background Art

[0002] Autonomous driving technology relies on the collaboration of computer vision, radar, monitoring devices, and global positioning systems to enable motor vehicles to achieve autonomous driving without the need for active human operation. Autonomous driving vehicles use various computing systems to help transport passengers from one location to another. Some autonomous driving vehicles may require some initial or continuous input from an operator (such as a navigator, driver, or passenger). Autonomous driving vehicles allow operators to switch from manual mode to autonomous driving mode or a mode in between. Since autonomous driving technology does not require humans to drive motor vehicles, it can theoretically effectively avoid human driving errors, reduce the occurrence of traffic accidents, and improve road transportation efficiency. Therefore, autonomous driving technology is receiving more and more attention.

[0003] Currently, to ensure the effectiveness of autonomous driving, it is often necessary to evaluate the autonomous driving system (also known as the autonomous driving algorithm) installed on the vehicle to ensure that the autonomous driving system can plan and control reasonable driving behavior in various environments. However, current evaluation schemes generally only evaluate the autonomous driving planning and control algorithm, which is not comprehensive enough.

[0004] Summary of the Invention

[0005] The present application provides an autonomous driving evaluation method, apparatus, and autonomous driving equipment, which can simultaneously evaluate the perception and planning control capabilities of the autonomous driving algorithm based on label information corresponding to environmental information, thereby improving the reliability of the autonomous driving algorithm evaluation.

[0006] To achieve the above objectives, this application adopts the following technical solutions:

[0007] In a first aspect, the present application provides an evaluation method for autonomous driving, the method comprising: obtaining label information corresponding to environmental information of the environment in which the autonomous driving device is located, the label information being used to indicate the actual state of the target object in the environment; obtaining a processing result of the autonomous driving algorithm based on the environmental information, the autonomous driving algorithm being used to perceive the environmental information and plan and control the driving behavior based on the environmental perception result; and determining the environmental perception capability and planning and control capability of the autonomous driving algorithm based on the label information and the processing result.

[0008] The solution of the first aspect described above can obtain corresponding label information indicating the actual state of objects in the environment based on environmental information in the autonomous driving device's environment. This allows for a system-level assessment of the autonomous driving algorithm's perception and planning and control capabilities based on the actual environmental characteristics, thereby achieving system-level protection for the entire process from acquiring environmental information to perceiving it and making driving behavior decisions. This system-level assessment can identify not only perception anomalies (such as missed detections and false detections) in the autonomous driving algorithm, but also anomalies in its planning and control capabilities, enabling targeted optimization of the algorithm's perception and planning and control capabilities.

[0009] In some possible implementations, obtaining label information corresponding to the environmental information includes: obtaining label information corresponding to the environmental information sent by an external device; or labeling the environmental information to obtain label information corresponding to the environmental information. In this way, the device performing the autonomous driving assessment method provided in this solution may have its own environmental feature labeling capabilities, or may obtain actual environmental features from other devices.

[0010] In some possible implementations, environmental information includes static object information and dynamic object information. Static object information includes at least one of traffic light information, lane information, and static obstacle information. Dynamic object information includes at least one of people, vehicles, and animals. This allows for system-level evaluation of various capabilities of autonomous driving algorithms, including their ability to navigate traffic lights, maintain speed limits, avoid other vehicles, yield to pedestrians, and confirm right-of-way.

[0011] Optionally, the device that executes the autonomous driving evaluation method provided by this solution can itself mark the static targets and dynamic targets contained in the environmental information, and can also mark the status information of the static targets and dynamic targets, such as the color and size of the static targets, and the moving speed and direction of the dynamic targets.

[0012] Optionally, the environmental information may be manually annotated, so that the electronic device can obtain label information corresponding to the manually annotated environmental information.

[0013] In some possible implementations, determining the evaluation result of the autonomous driving algorithm based on the label information and processing results includes: evaluating whether the processing result is reasonable based on the label information; if the processing result is not reasonable, obtaining an environmental perception result obtained by the autonomous driving algorithm from the processing result; and determining the environmental perception and planning and control capabilities of the autonomous driving algorithm based on a comparison between the label information and the environmental perception result. In this way, the rationality of the driving behavior decisions made by the autonomous driving algorithm based on environmental perception can be evaluated based on actual environmental characteristics, thereby locating unreasonable problem events. Furthermore, by comparing the actual environmental characteristics with the environmental characteristics perceived by the autonomous driving algorithm, it is possible to automatically analyze whether the cause of the problem event is due to an issue with the autonomous driving algorithm's perception capabilities or its regulatory control capabilities, thereby achieving the purpose of dividing the problem into issues with the perception algorithm and those with the regulatory control algorithm. This allows for targeted optimization of the autonomous driving algorithm's perception and planning and control capabilities.

[0014] In some possible implementations, the above-mentioned determination of the autonomous driving algorithm's environmental perception capability and planning and control capability based on the comparison between the tag information and the environmental perception results includes: determining that the autonomous driving algorithm's environmental perception capability is abnormal when the tag information is inconsistent with the environmental perception results; and determining that the autonomous driving algorithm's planning and control capability is abnormal when the tag information is consistent with the environmental perception results. Thus, when the environmental characteristics perceived by the autonomous driving algorithm are inconsistent with the actual environmental characteristics, it can be assumed that the autonomous driving algorithm did not perceive accurate environmental characteristics, and therefore it can be determined that the cause of the automatic analysis problem event stems from a perception problem in the autonomous driving algorithm. When the environmental characteristics perceived by the autonomous driving algorithm are consistent with the actual environmental characteristics, it can be assumed that the autonomous driving algorithm perceived accurate environmental characteristics, and therefore it can be determined that the cause of the automatic analysis problem event stems from a planning and control problem in the autonomous driving algorithm.

[0015] In some possible implementations, the device itself that executes the autonomous driving evaluation method provided by this solution can optimize the autonomous driving algorithm based on the label information and the above-mentioned processing results.

[0016] Optionally, if the label information indicates that the autonomous driving algorithm's processing results based on environmental information are unreasonable, the autonomous driving algorithm can be optimized based on a comparison between the label information and the autonomous driving algorithm's environmental perception results to improve the autonomous driving algorithm's environmental perception and planning and control capabilities. This achieves system-level optimization of the autonomous driving algorithm's perception and planning and control capabilities.

[0017] In some possible implementations, the autonomous driving algorithm includes an environmental perception algorithm and a planning and control algorithm. When label information is inconsistent with the environmental perception results, the autonomous driving algorithm's environmental perception capabilities may be considered abnormal. In this case, the environmental perception algorithm may be optimized to improve the autonomous driving algorithm's environmental perception capabilities. When label information is consistent with the environmental perception results, the autonomous driving algorithm's planning and control capabilities may be considered abnormal. In this case, the planning and control algorithm may be optimized to optimize the autonomous driving algorithm's planning and control capabilities. In this way, based on real-world environmental characteristics, problems with the perception and planning and control algorithms can be automatically analyzed and optimized accordingly.

[0018] In some possible implementations, the autonomous driving evaluation method further includes: displaying a first prompt message, when the tag information is inconsistent with the environmental perception result, to indicate an abnormality in the autonomous driving algorithm's environmental perception capability; and displaying a second prompt message, when the tag information is consistent with the environmental perception result, to indicate an abnormality in the autonomous driving algorithm's planning and control capability. In this way, relevant personnel are promptly alerted when an abnormality in the autonomous driving algorithm occurs.

[0019] In some possible implementations, the autonomous driving evaluation method further includes: determining a first value of the autonomous driving algorithm, the first value indicating the probability that the autonomous driving algorithm's processing results for the environment are reasonable; determining a second value of the autonomous driving algorithm, the second value indicating the probability that the processing results for the environment are unreasonable due to an abnormality in the autonomous driving algorithm's environmental perception capabilities; or determining a third value of the autonomous driving algorithm, the third value indicating the probability that the processing results for the environment are unreasonable due to an abnormality in the autonomous driving algorithm's planning and control capabilities. This allows for a system-level pass rate assessment of the autonomous driving algorithm, and allows analysis of whether failed scenarios are due to perception anomalies or planning and control anomalies.

[0020] In some possible implementations, evaluating the rationality of processing results based on label information includes determining whether the processing results are consistent with the driving behavior evaluation criteria based on a predetermined correspondence between the label information and the driving behavior evaluation criteria; and evaluating the processing results as unreasonable if the processing results are consistent with the driving behavior evaluation criteria. This allows for the design of driving behavior evaluation criteria based on real-world environmental characteristics, rather than the perception results of autonomous driving algorithms, which can improve the efficiency of problem discovery and analysis.

[0021] In some possible implementations, obtaining the processing results of the autonomous driving algorithm on the environmental information includes: invoking the autonomous driving algorithm to perform a simulation based on the environmental information, and obtaining the simulation results as the processing results. In this way, the solution of this application can also be applied to a simulation test platform to obtain a safe and reliable autonomous driving algorithm before it is installed on a real vehicle.

[0022] In a second aspect, an autonomous driving evaluation device is provided, which has the function of implementing the above-mentioned first aspect or any possible design method thereof. The function can be implemented by hardware or by hardware executing corresponding software. The hardware or software includes one or more modules or units corresponding to the above-mentioned functions. For example, the autonomous driving evaluation device may include a label acquisition module, an algorithm processing module, and an algorithm evaluation module. Among them:

[0023] A label acquisition module is used to obtain label information corresponding to the environmental information of the environment in which the autonomous driving device is located. The label information is used to indicate the actual status of the target object in the environment;

[0024] An algorithm processing module is used to obtain the processing results of the autonomous driving algorithm based on environmental information. The autonomous driving algorithm is used to perceive environmental information and plan and control driving behavior based on the environmental perception results;

[0025] The algorithm evaluation module is used to obtain the environmental perception and planning control capabilities of the autonomous driving algorithm based on label information and processing results.

[0026] In some possible implementations, the autonomous driving evaluation device may further include an environment acquisition module for acquiring environmental information of the environment in which the autonomous driving device is located.

[0027] In some possible implementations, the tag acquisition module may be configured to acquire tag information corresponding to the aforementioned environmental information sent by an external device.

[0028] In some possible implementations, the label acquisition module may be used to label the above-mentioned environmental information to obtain label information corresponding to the above-mentioned environmental information.

[0029] In some possible implementations, the algorithm evaluation module may include a system evaluation unit, a perception acquisition unit, and a label comparison unit. The system evaluation unit is configured to evaluate the rationality of a processing result based on label information; the perception acquisition unit is configured to, if the processing result is assessed as unreasonable, obtain an environmental perception result obtained by the autonomous driving algorithm from the processing result; and the label comparison unit is configured to determine the environmental perception capability and planning and control capability of the autonomous driving algorithm based on a comparison between the label information and the environmental perception result.

[0030] In some possible implementations, the system evaluation unit may be used to: determine whether a processing result is consistent with the driving behavior evaluation standard based on a correspondence between preset label information and the driving behavior evaluation standard; and when the processing result is consistent with the driving behavior evaluation standard, evaluate the processing result as unreasonable.

[0031] In some possible implementations, the label comparison unit may be used to: when the label information is inconsistent with the environmental perception result, determine that the evaluation result of the autonomous driving algorithm is abnormal in perception capability; when the label information is consistent with the environmental perception result, determine that the evaluation result of the autonomous driving algorithm is abnormal in planning and control capability.

[0032] In some possible implementations, the autonomous driving evaluation device further includes a system optimization module configured to optimize the autonomous driving algorithm based on the label information and the processing results.

[0033] Optionally, the above-mentioned system optimization module can be used for: when it is evaluated that the processing results of the autonomous driving algorithm based on environmental information are unreasonable based on the label information, the autonomous driving algorithm can be optimized based on the comparison results between the label information and the environmental perception results of the autonomous driving algorithm to optimize the environmental perception ability and planning and control ability of the autonomous driving algorithm.

[0034] In some possible implementations, the autonomous driving algorithm includes an environment perception algorithm and a planning and control algorithm. The system optimization module may be configured to: when the tag information is inconsistent with the environment perception results, optimize the environment perception algorithm to optimize the environment perception capability of the autonomous driving algorithm; and when the tag information is consistent with the environment perception results, optimize the planning and control algorithm to optimize the planning and control capability of the autonomous driving algorithm.

[0035] In some possible embodiments, the autonomous driving evaluation device also includes a display module for displaying a first prompt message when the label information is inconsistent with the environmental perception result, to indicate that the environmental perception capability of the autonomous driving algorithm is abnormal; and for displaying a second prompt message when the label information is consistent with the environmental perception result, to indicate that the planning and control capability of the autonomous driving algorithm is abnormal.

[0036] In some possible implementations, the autonomous driving evaluation device further includes a system statistics module. The system statistics module is configured to: determine a first value of the autonomous driving algorithm, the first value indicating the probability that a processing result of the autonomous driving algorithm in an environment is reasonable; determine a second value of the autonomous driving algorithm, the second value indicating the probability that an unreasonable processing result is caused by an abnormality in the autonomous driving algorithm's perception capability in the environment; or determine a third value of the autonomous driving algorithm, the third value indicating the probability that an unreasonable processing result is caused by an abnormality in the autonomous driving algorithm's planning and control capability in the environment.

[0037] In some possible implementations, the environmental information includes static target information and dynamic target information.

[0038] In some possible implementations, the algorithm processing module may be an algorithm simulation module, which can be used to call the autonomous driving algorithm for simulation based on environmental information to obtain simulation results as processing results.

[0039] In a third aspect, an autonomous driving evaluation device is provided, comprising a memory and one or more processors; the memory and the processor are coupled; the memory is used to store program code, the program code comprising instructions, and when the processor executes the instructions, the device executes the autonomous driving evaluation method in any possible implementation of the first aspect above.

[0040] In a fourth aspect, a controller is provided, which includes the autonomous driving evaluation device according to the second aspect or the third aspect.

[0041] In a fifth aspect, an autonomous driving device is provided, comprising the controller according to the fourth aspect, or the autonomous driving evaluation device according to the third aspect. The autonomous driving device may be an autonomous vehicle, a robot, or the like.

[0042] In a sixth aspect, a chip system is provided. The chip system includes one or more interface circuits and one or more processors. The interface circuits and processors are interconnected via a circuit. The interface circuits are configured to receive a signal and send the signal to the processor, the signal including an instruction. The processor is configured to execute the instruction and perform the autonomous driving assessment method according to any possible implementation of the first aspect.

[0043] It can be understood that the beneficial effects that can be achieved by the above-mentioned device of the second aspect, the device of the third aspect, the controller of the fourth aspect, the automatic driving equipment of the fifth aspect and the chip system of the sixth aspect can refer to the beneficial effects of the first aspect and any possible implementation thereof, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] FIG1 is a structural schematic diagram of a vehicle provided in an embodiment of the present application;

[0045] FIG2 is a second structural diagram of a vehicle provided in an embodiment of the present application;

[0046] FIG3 is a schematic diagram of the structure of a computer system provided in an embodiment of the present application;

[0047] FIG4 is a first schematic diagram of an application of a cloud-side commanded autonomous driving vehicle according to an embodiment of the present application;

[0048] FIG5 is a second schematic diagram of an application of a cloud-side commanded autonomous driving vehicle provided by an embodiment of the present application;

[0049] FIG6 is a schematic structural diagram of a simulation test system provided in an embodiment of the present application;

[0050] FIG7 is a schematic diagram of a flow chart of an autonomous driving evaluation method provided in an embodiment of the present application;

[0051] FIG8 is a schematic diagram of a flow chart of another autonomous driving evaluation method provided in an embodiment of the present application;

[0052] FIG9 is a schematic diagram of a perception anomaly of an autonomous driving algorithm provided in an embodiment of the present application. DETAILED DESCRIPTION

[0053] The technical solutions in the embodiments of the present application will be described below in conjunction with the drawings in the embodiments of the present application. In the following, the terms "first" and "second" are used for descriptive purposes only and cannot be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Thus, the features defined as "first" and "second" may explicitly or implicitly include one or more of the features. It should be understood that in the present application, "at least one" refers to one or more, and "more" refers to two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships can exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural.

[0054] The solution of the embodiment of the present application can be applied to electronic devices. Among them, the electronic device can be an autonomous driving vehicle, a robot, or other autonomous driving equipment with autonomous driving capabilities, and the embodiment of the present application does not limit this. In some embodiments, the solution of the embodiment of the present application can also be applied to other devices (such as cloud servers, algorithm testing platforms, algorithm simulation platforms, etc.) that have the ability to test and evaluate autonomous driving algorithms. The electronic device can implement the autonomous driving evaluation method provided by the embodiment of the present application through the components it contains (including hardware and software), that is, according to the environmental information in the surrounding environment of the autonomous driving device, obtain corresponding label information for indicating the actual state of the target object in the environment, and obtain the processing result of the autonomous driving algorithm on the environmental information, the autonomous driving algorithm is used to perceive the environmental information and plan and control the driving behavior based on the perception result, and then determine the evaluation result of the autonomous driving algorithm based on the label information and the processing result, and the evaluation result is used to indicate whether the perception ability and planning and control ability of the autonomous driving algorithm are abnormal.

[0055] It can be understood that the autonomous driving evaluation method provided in the embodiment of the present application can evaluate the perception ability and planning and control ability of the autonomous driving algorithm based on the label information corresponding to the environmental information, thereby not only being able to discover the perception anomaly problems of the autonomous driving algorithm, but also being able to discover the planning and control anomaly problems of the autonomous driving algorithm, and then being able to optimize the perception ability and planning and control ability of the autonomous driving algorithm in a targeted manner.

[0056] The following will take an electronic device as an autonomous driving vehicle (hereinafter referred to as the vehicle) as an example to schematically illustrate the solution provided in the embodiments of the present application.

[0057] Please refer to Figure 1, which shows a functional block diagram of a vehicle 100 provided in an embodiment of the present application. The vehicle 100 may include various devices, components, etc. disposed in the vehicle 100 and / or on the body of the vehicle 100. In one embodiment, the devices and components disposed in the vehicle 100 may include, but are not limited to, an autonomous driving system (also known as an autonomous driving algorithm) and autonomous driving functional applications. It is understood that vehicles with certain autonomous driving capabilities are typically equipped with an autonomous driving system.

[0058] The vehicle 100 may include various subsystems, such as a travel system 110, a sensor system 120, a control system 130, one or more peripheral devices 140, a power supply 150, a computer system 160, and a user interface 170. Alternatively, the vehicle 100 may include more or fewer subsystems, and each subsystem may include multiple components. In addition, each subsystem and component of the vehicle 100 may be interconnected via wired or wireless connections.

[0059] The propulsion system 110 may include components that provide powered motion for the vehicle 100. In one embodiment, the propulsion system 110 may include an engine 111, a transmission 112, an energy source 113, and wheels 114. The engine 111 may be an internal combustion engine, an electric motor, an air compression engine, or a combination of other types of engines. The engine 111 converts the energy source 113 into mechanical energy.

[0060] Examples of energy source 113 include gasoline, solar panels, batteries, and other sources of electricity. Energy source 113 can also provide energy to other systems of vehicle 100. Transmission 112 can transmit mechanical power from engine 111 to wheels 114.

[0061] The sensor system 120 may include several sensors that sense information about the environment surrounding the vehicle 100. For example, the sensor system 120 may include a positioning system 121 (the positioning system may be a global positioning system (GPS), a BeiDou system, or other positioning systems), an inertial measurement unit (IMU) 122, a radar 123, a lidar 124, and a camera 125. The sensor system 120 may also include sensors that monitor the internal systems of the vehicle 100 (for example, an in-vehicle air quality monitor, a fuel gauge, an oil temperature gauge, etc.). Sensor data from one or more of these sensors can be used to detect objects and their corresponding characteristics (position, shape, direction, speed, etc.). Such detection and recognition are key functions for the safe operation of the autonomous driving of the vehicle 100.

[0062] Positioning system 121 may be used to estimate the geographic location of vehicle 100. IMU 122 is used to sense changes in position and orientation of vehicle 100 based on inertial acceleration. In one embodiment, IMU 122 may be a combination of an accelerometer and a gyroscope.

[0063] Radar 123 may utilize radio signals to sense objects within the surrounding environment of vehicle 100. In some embodiments, in addition to sensing objects, radar 123 may also be used to sense the speed and / or heading of the objects.

[0064] LiDAR 124 may utilize laser light to sense objects in the environment in which vehicle 100 is located. In some embodiments, LiDAR 124 may include one or more laser sources, a laser scanner, and one or more detectors, among other system components.

[0065] The camera 125 may be used to capture multiple images of the surrounding environment of the vehicle 100, as well as multiple images of the interior of the vehicle cabin. The camera 125 may be a still camera or a video camera.

[0066] The control system 130 controls the operation of the vehicle 100 and its components. The control system 130 may include various components, including a steering system 131, a throttle 132, a brake unit 133, a computer vision system 134, a path control system 135, and an obstacle avoidance system 136. The steering system 131, which may be a steering wheel system, is operable to adjust the direction of the vehicle 100. The throttle 132 is used to control the operating speed of the engine 111 and, in turn, the speed of the vehicle 100. The brake unit 133 is used to control the deceleration of the vehicle 100.

[0067] The computer vision system 134 can be operated to process and analyze images captured by the camera 125 to identify objects and / or features in the environment surrounding the vehicle 100. The objects and / or features may include traffic signs, road conditions, and obstacles. The computer vision system 134 can use object recognition algorithms, structure from motion (SFM) algorithms, video tracking, and other computer vision techniques. In some embodiments, the computer vision system 134 can be used to map the environment, track objects, estimate the speed of objects, and so on.

[0068] The route control system 135 is used to determine the driving route of the vehicle 100. In some embodiments, the route control system 135 can combine data from sensors, the positioning system 121, and one or more predetermined maps to determine the driving route for the vehicle 100.

[0069] The obstacle avoidance system 136 is used to identify, assess, and avoid or otherwise negotiate potential obstacles in the environment of the vehicle 100 .

[0070] In this embodiment of the present application, the computer system 160 may invoke the computer vision system 134 to sense and detect images captured by the sensor system 120, such as the camera 125, to sense and detect the status information of static and dynamic environmental objects in the surrounding environment of the vehicle 100. The computer system 160 transmits the sensed and detected status information of the static and dynamic environmental objects to the route control system 135 and the obstacle avoidance system 136, so that the route control system 135 and the obstacle avoidance system 136 can plan and control the driving behavior of the vehicle 100, such as deceleration, yielding, lane changing, etc.

[0071] Of course, in one example, the control system 130 may include additional or alternative components other than those shown and described, or may also include a portion of the components shown above.

[0072] The vehicle 100 interacts with external sensors, other vehicles, other computer systems, or users via peripheral devices 140. The peripheral devices 140 may include a wireless communication system 141, an onboard computer 142, a microphone 143, and / or a speaker 144.

[0073] In some embodiments, peripheral devices 140 provide a means for a user of vehicle 100 to interact with user interface 170. For example, onboard computer 142 can provide information to the user of vehicle 100. User interface 170 can also operate onboard computer 142 to receive user input. Onboard computer 142 can be operated via a touch screen. In other cases, peripheral devices 140 can provide a means for vehicle 100 to communicate with other devices located within the vehicle.

[0074] The wireless communication system 141 can communicate wirelessly with one or more devices directly or via a communication network. The device can be a cloud server of the vehicle or another server. For example, the wireless communication system 141 can use 3G cellular communication, 4G cellular communication, or 5G cellular communication. The wireless communication system 141 can communicate with a wireless local area network (WLAN) using a wireless fidelity (Wi-Fi) network. In some embodiments, the wireless communication system 141 can communicate directly with the device using an infrared link or Bluetooth.

[0075] Power source 150 may provide power to various components of vehicle 100 .

[0076] Some or all functions of the vehicle 100 are controlled by a computer system 160. The computer system 160 may include at least one processor 161 that executes instructions 1621 stored in a non-transitory computer-readable medium such as a memory 162. The computer system 160 may also be a plurality of computing devices that control individual components or subsystems of the vehicle 100 in a distributed manner.

[0077] Processor 161 can be any conventional processor, such as a central processing unit (CPU), a microprocessor (MCU), or other conventional chips. Alternatively, the processor can be a dedicated device such as an application-specific integrated circuit (ASIC) or other hardware-based processor. Although FIG1 functionally illustrates a processor, memory, and other components in the same physical housing, one skilled in the art will appreciate that the processor, computer system, or memory can actually include multiple processors, computer systems, or memories that can be stored in the same physical housing, or multiple processors, computer systems, or memories that can not be stored in the same physical housing. For example, the memory can be a hard drive or other storage medium located in a different physical housing. Therefore, references to a processor or computer system will be understood to include references to a collection of processors, computer systems, or memories that can operate in parallel, or a collection of processors, computer systems, or memories that can not operate in parallel. Rather than using a single processor to perform the steps described herein, some components, such as the steering assembly and the deceleration assembly, can each have their own processor that performs only calculations related to the functions specific to the component.

[0078] In some embodiments, the processor 161 may be a processor of an onboard processing device such as a vehicle computer, a domain controller, a mobile data center (MDC), or an onboard computer.

[0079] In various aspects described herein, the processor can be located remotely from the vehicle and in wireless communication with the vehicle. In other aspects, some of the processes described herein are performed on a processor disposed within the vehicle while others are performed by a remote processor, including taking the necessary steps to perform a single maneuver.

[0080] In some embodiments, memory 162 may include instructions 1621 (e.g., program logic) that are executable by processor 161 to perform various functions of vehicle 100, including those described above. Memory 162 may also include additional instructions, including instructions for sending data to, receiving data from, interacting with, and / or controlling one or more of travel system 110, sensor system 120, control system 130, and peripherals 140.

[0081] In addition to the instructions 1621, the memory 162 may also store data such as road maps, route information, the vehicle's location, direction, speed, and other such vehicle data, as well as other information. This information may be used by the vehicle 100 and the computer system 160 during operation of the vehicle 100 in autonomous, semi-autonomous, and / or manual modes.

[0082] For example, in one possible embodiment, the memory 162 may obtain environmental information about the vehicle's surrounding environment, such as images of traffic lights, other vehicles, road edges, and obstacles such as green belts, obtained by sensors in the sensor system 120. Thus, the processor 161 may obtain this environmental information from the memory 162 and invoke the computer vision system 134 or other components of the vehicle 100 to perceive this environmental information and thereby detect the status of objects in the vehicle's current environment.

[0083] The memory 162 can also obtain perception results of environmental information from the computer vision system 134 or other components of the vehicle 100. The perception results are status information of the target objects in the environment in which the vehicle is currently located. For example, it can be the perceived information of whether there are other vehicles, pedestrians, green belts, lanes and other targets near the vehicle's current environment. It can also be the perceived driving direction, driving speed, and distance from the vehicle itself of other vehicles, pedestrians and other targets. It can also be the color information of the perceived target object - traffic lights (traffic lights).

[0084] In addition to the above, the memory 162 may also store the vehicle's own state information and tag information corresponding to environmental information. The vehicle's state information includes, but is not limited to, the vehicle's position, speed, acceleration, and heading angle. The tag information corresponding to the environmental information may include actual state information of objects in the vehicle's current environment, such as the actual direction, speed, and distance of other vehicles, pedestrians, green belts, lanes, and other objects, as well as the actual color information of traffic lights.

[0085] In this way, the processor 161 can obtain these perception results and the vehicle's own state information from the memory 162, and based on these perception results and the vehicle's own state information, plan and control the driving behavior of the vehicle 100 to control the automatic driving of the vehicle 100. The memory 162 can also obtain the planning and control results of the processor 161.

[0086] In an embodiment of the present application, the processor 161 can also obtain label information corresponding to the environmental information from the memory 162 to evaluate the planning control results and perception results based on the label information to automatically discover whether there are problems with perception and planning control.

[0087] User interface 170 is used to provide information to or receive information from a user of vehicle 100. Optionally, user interface 170 may interact with one or more input / output devices within the set of peripheral devices 140, such as one or more of wireless communication system 141, onboard computer 142, microphone 143, and speaker 144.

[0088] Computer system 160 can control vehicle 100 based on information obtained from various subsystems (e.g., travel system 110, sensor system 120, and control system 130) and information received from user interface 170. For example, computer system 160 can control steering system 131 to change the vehicle's forward direction based on information from sensor system 120 and obstacle avoidance system 136, thereby avoiding obstacles detected by sensor system 120 and obstacle avoidance system 136. In some embodiments, computer system 160 can control many aspects of vehicle 100 and its subsystems.

[0089] In one possible implementation, the computer system 160 may be an autonomous driving system (also known as an autonomous driving algorithm), an MDC platform, or other domain controller responsible for autonomous driving or intelligent driving assistance computing. This application does not impose any restrictions on this.

[0090] Alternatively, one or more of the above components may be installed or associated separately from the vehicle 100. For example, the memory 162 may be partially or completely separate from the vehicle 100. The above components may be coupled together for communication via wired and / or wireless means.

[0091] Optionally, the above components are only an example. In actual applications, the components in the above modules may be added or deleted according to actual needs. Figure 1 should not be understood as a limitation to the embodiments of the present application.

[0092] Vehicle 100 traveling on a road can determine its driving strategy, such as adjusting the current speed of vehicle 100, based on the state information of objects in its surrounding environment that it senses and detects. Objects in the surrounding environment of vehicle 100 can include other vehicles, pedestrians, traffic lights, green belts, or other types of objects. In some examples, each object in the surrounding environment can be considered independently, and speed adjustment instructions for vehicle 100 can be determined based on the object's respective characteristics, such as its current speed, acceleration, and distance from the vehicle.

[0093] The vehicle 100 may be a car, truck, motorcycle, bus, ship, airplane, helicopter, lawn mower, recreational vehicle, amusement park vehicle, construction equipment, tram, golf cart, train, and cart, etc., and the embodiments of the present application do not impose any particular limitation.

[0094] In other embodiments of the present application, the vehicle 100 may further include hardware structures and / or software modules to implement the aforementioned functions in the form of hardware structures, software modules, or a combination of hardware structures and software modules. Whether a particular function is implemented in the form of hardware structures, software modules, or a combination of hardware structures and software modules depends on the specific application and design constraints of the technical solution.

[0095] 2 , illustratively, a vehicle 200 may include the following modules:

[0096] Environment Acquisition Module 201: This module is used to acquire environmental information from the surrounding environment of vehicle 200 using onboard sensors and / or roadside sensors. Optionally, vehicle 200 may be vehicle 100 in Figure 1 . Roadside and onboard sensors may include lidar, millimeter-wave radar, or visual sensors. Environmental information may include video data or image data originally collected by the sensors. Environment Acquisition Module 201 is also used to transmit the environmental information acquired by the sensors to Algorithm Processing Module 203.

[0097] It should be noted that the solution provided in this application can obtain environmental information in the surrounding environment of the vehicle 200 in a variety of ways. The methods of obtaining environmental information in the surrounding environment of the vehicle 200 in the relevant technologies can all be adopted in the embodiments of this application.

[0098] The label acquisition module 202 is configured to acquire label information corresponding to the environmental information. The label information indicates the actual state of an object in the environment. Optionally, the label acquisition module 202 may acquire the label information corresponding to the environmental information from an external device via a wireless communication module. In one embodiment, the label acquisition module 202 may also acquire the environmental information from the environmental acquisition module 201 and annotate the environmental information using a ground truth labeling tool on the vehicle 200 to obtain the label information corresponding to the environmental information. The environmental acquisition module 201 is further configured to transmit the acquired label information to the algorithm evaluation module 204.

[0099] It should be noted that the solution provided in this application can obtain tag information in a variety of ways, and the methods for obtaining tag information in the relevant technologies can all be adopted in the embodiments of this application.

[0100] Algorithm processing module 203 is configured to receive environmental information transmitted by environment acquisition module 201 and invoke the autonomous driving algorithm to process the environmental information to obtain a processing result. The autonomous driving algorithm is configured to perceive the environmental information to obtain a perception result, and based on the perception result, plan and control the driving behavior of vehicle 100 to obtain a planning and control result. Algorithm processing module 203 is further configured to transmit the obtained algorithm processing result to algorithm evaluation module 204.

[0101] The perception results may include information such as the position, speed, and direction of travel of other vehicles, pedestrians, and other targets in the surrounding environment of the vehicle 100, and may also include information such as the color of the detected target - a traffic light. The planning and control results may include lateral decisions and / or longitudinal decisions. The lateral decision is mainly made by means such as left and right steering to try to bypass the target and continue driving. The longitudinal decision is mainly made by means such as front and rear acceleration, deceleration, and braking to try to avoid collisions with targets directly in front of or behind the vehicle.

[0102] Algorithm evaluation module 204 is configured to receive the label information sent by label acquisition module 202 and the processing results sent by algorithm processing module 203, and determine an evaluation result of the autonomous driving algorithm based on the label information and processing results. The evaluation result indicates whether there are any problems with the autonomous driving algorithm's perception and planning and control capabilities.

[0103] In some embodiments, the vehicle 200 may also include a display module (not shown in FIG. 2 ) for receiving the evaluation results sent by the algorithm evaluation module 204 and displaying different evaluation results using different colors to facilitate intuitive viewing of different evaluation results.

[0104] In some embodiments, the vehicle 200 may further include a storage component (not shown in FIG. 2 ) for storing executable codes of the above-mentioned modules. Running these executable codes may implement part or all of the method flow of the embodiments of the present application.

[0105] In one possible implementation, as shown in FIG3 , the computer system 160 shown in FIG1 may include a processor 301 coupled to a system bus 302. The processor 301 may be one or more processors, each of which may include one or more processor cores. A display adapter 303 may drive a display 324, which is coupled to the system bus 302. The system bus 302 is coupled to an input / output (I / O) bus 305 via a bus bridge 304. An I / O interface 306 is coupled to the I / O bus 305. The I / O interface 306 communicates with various I / O devices, such as an input device 307 (e.g., a keyboard, mouse, touch screen, etc.), a media tray 308 (e.g., a multimedia interface), a transceiver 309 (capable of sending and / or receiving radio communication signals), a camera 310 (capable of capturing still and moving digital video images), and an external universal serial bus (USB) port 311. Optionally, the interface connected to the I / O interface 306 may be a USB interface.

[0106] Processor 301 may be any conventional processor, including a reduced instruction set computer (RISC) processor, a complex instruction set computer (CISC) processor, or a combination thereof. Alternatively, processor 301 may be a dedicated device such as an application specific integrated circuit (ASIC). Alternatively, processor 301 may be a neural network processor or a combination of a neural network processor and the conventional processors described above.

[0107] Alternatively, in various embodiments described herein, computer system 160 may be located remotely from the autonomous vehicle and in wireless communication with the autonomous vehicle. In other aspects, some of the processes described herein may be executed on a processor within the autonomous vehicle, while other processes may be performed by a remote processor, including taking the actions required to execute a single maneuver.

[0108] Computer system 160 can communicate with a software deployment server 313 via a network interface 312. Optionally, network interface 312 can be a hardware network interface, such as a network card. Network 314 can be an external network, such as the Internet, or an internal network, such as Ethernet or a virtual private network (VPN). Optionally, network 314 can also be a wireless network, such as a wireless fidelity (Wi-Fi) network or a cellular network.

[0109] The hard drive interface 315 is coupled to the system bus 302. The hard drive interface 315 is connected to a hard drive 316. The system memory 317 is coupled to the system bus 302. The data running in the system memory 317 may include the operating system (OS) 318 and application programs 319 of the computer system 160.

[0110] Operating system (OS) 318 includes, but is not limited to, a shell 320 and a kernel 321. Shell 320 is an interface between the user and kernel 321 of OS 318. Shell 320 is the outermost layer of OS 318. The shell manages the interaction between the user and OS 318: it waits for user input, interprets user input to OS 318, and processes various outputs from OS 318.

[0111] The kernel 321 consists of the portion of the operating system 318 that manages memory, files, peripherals, and system resources, and directly interacts with the hardware. The kernel 321 of the operating system 318 typically runs processes and provides inter-process communication, as well as CPU time slice management, interrupt management, memory management, and I / O management.

[0112] Applications 319 include programs 323 related to autonomous driving algorithms, such as those that manage the autonomous vehicle's interactions with obstacles on the road, control the autonomous vehicle's route and speed, and control the autonomous vehicle's interactions with other vehicles / autonomous vehicles on the road. Applications 319 also reside on the system of deploying server 313. In one embodiment, when application 319 is needed, computer system 160 can download application 319 from deploying server 313.

[0113] Sensor 322 is associated with computer system 160. Sensor 322 is used to detect the environment surrounding computer system 160. For example, sensor 322 can detect surrounding objects such as cars, people, and traffic lights. Alternatively, if computer system 160 is located in a self-driving car, sensor 322 can be at least one of a camera, an infrared sensor, and other devices.

[0114] In other embodiments of the present application, computer system 160 can also receive information from or transfer information to other computer systems. Alternatively, sensor data collected from sensor system 120 of vehicle 100 can be transferred to another computer for processing. As shown in FIG4 , data from computer system 160 can be transmitted via a network to a cloud-side computer system 410 for further processing.

[0115] In one example, computer system 160 may include a server having multiple computers, such as a load balancing server cluster. To receive, process, and transmit data from computer system 160, server 420 exchanges information with different nodes of the network. Computer system 410 may have a configuration similar to computer system 160 and include a processor 430, memory 440, instructions 450, and data 460.

[0116] In one example, data 460 from server 420 may include weather-related information. For example, server 420 may receive, monitor, store, update, and transmit various information related to objects in the surrounding environment. This information may include target classification, target shape information, and target tracking information, for example, in the form of reports, radar information, or forecasts.

[0117] FIG5 illustrates an example of interaction between an autonomous vehicle and a cloud service center (cloud server). The cloud service center can receive information (such as data collected by vehicle sensors or other information) from vehicles 513 and 512 within its operating environment 500 via a network 511, such as a wireless communication network. Vehicles 513 and 512 can be autonomous vehicles.

[0118] Based on the received data, cloud service center 520 runs its stored programs related to autonomous driving algorithms to control vehicles 513 and 512. These programs can include managing the interaction between the autonomous vehicle and obstacles on the road, controlling the route or speed of the autonomous vehicle, or controlling the interaction between the autonomous vehicle and other autonomous vehicles on the road.

[0119] For example, cloud service center 520 can provide portions of a map to vehicles 513 and 512 via network 511. In other examples, operations can be divided among different locations. For example, multiple cloud service centers can receive, verify, combine, and / or transmit information reports. In some examples, information reports and / or sensor data can also be transmitted between vehicles. Other configurations are also possible.

[0120] In some examples, the cloud service center 520 sends to the autonomous vehicle suggested solutions for possible driving situations within the environment (e.g., notifying the vehicle of an obstacle ahead and how to get around it). For example, the cloud service center 520 can assist the vehicle in determining how to proceed when faced with a specific obstacle within the environment. The cloud service center 520 sends to the autonomous vehicle a response indicating how the vehicle should proceed in a given scenario. For example, based on the collected sensor data, the cloud service center 520 can sense the presence of a temporary stop sign ahead on the road. Accordingly, the cloud service center 520 sends a driving behavior planning control for the vehicle to pass the obstacle (e.g., instructing the vehicle to change lanes to another road).

[0121] In some embodiments, the cloud service center 520 may also obtain actual state information of objects in the autonomous vehicle's environment from other servers and evaluate its own perception detection results and driving behavior planning and control results based on the actual state information. In some embodiments, the cloud service center 520 may also obtain actual state information of objects in the autonomous vehicle's environment from other servers and transmit it to the autonomous vehicle.

[0122] In some embodiments, when the above-mentioned solution is implemented through other devices capable of testing and evaluating autonomous driving algorithms, such as cloud servers, algorithm testing platforms, algorithm simulation platforms, etc., that is, based on the label information corresponding to the environmental information, the perception ability and planning and control ability of the autonomous driving algorithm are evaluated at the same time. The other devices capable of testing and evaluating autonomous driving algorithms may also include an environment acquisition module, a label acquisition module, an algorithm processing module, and an algorithm evaluation module.

[0123] As an example, Figure 6 shows a schematic diagram of the architecture of a simulation test system provided by this application. As shown in Figure 6, the simulation test system may include a simulation test device 610, an autonomous driving algorithm module 620, and an onboard hardware platform 630.

[0124] The simulation test device 610 includes an environment acquisition module 611 , a label acquisition module 612 , an algorithm processing module 613 and an algorithm evaluation module 614 .

[0125] The environment acquisition module 611 is used to obtain environmental information of the vehicle's surrounding environment through a scene database.

[0126] Among them, the scene database includes environmental data of multiple scenes. The scenes in the scene database can be scenes obtained through road testing, or scenes obtained through other means. For example, environmental data in and around the vehicle can be collected through actual vehicle road testing, and the road test data can be extracted to form a road test scene, and the scene obtained through the road test can be added to the scene database. In addition, the scene database can also include other available scenes, such as scenes provided by a sharing platform, or scenes obtained through other commercially available channels. The scene database can be stored locally, such as a simulation test system, workstation, etc., or it can be stored in a cloud server. The scene database can be updated in real time and can be shared by multiple simulation test platforms.

[0127] It should be noted that the scenarios described in this application can be understood as a collection of various environmental information and constraints surrounding the vehicle, including the road, weather, traffic signals, and traffic participants. For example, scenarios can be categorized by road type, such as highway scenarios, urban road scenarios, off-road road scenarios, and desert road scenarios. Another example is that scenarios can be categorized by function, such as traffic light intersection environments, speed limit environments, environments with other vehicles present, and environments with pedestrians present.

[0128] In some embodiments, various environmental data in the scene database may include corresponding label information, and the label information may be content that is manually annotated on the environmental data, or content that is annotated on the environmental data using an annotation tool.

[0129] The label acquisition module 612 is used to obtain label information corresponding to the environmental information. The label information is used to indicate the actual state of the target object in the environment. The label acquisition module 612 can obtain label information obtained by manually annotating the environmental information or by annotating the environmental information using a ground truth annotation tool.

[0130] The algorithm processing module 613 is used to simulate and run the autonomous driving algorithm based on the environmental information obtained above to obtain simulation processing results. The simulation results include simulation perception results and simulation planning and control results.

[0131] Optionally, the algorithm processing module 613 may construct a simulation scenario based on the environmental information obtained above, and execute the autonomous driving algorithm based on the simulation scenario. The simulation scenario may be a computer-generated scenario, which may be generated by collecting various environmental data in the real world and recreating the real-world scenario using a simulation tool. This application does not impose any restrictions on this.

[0132] Optionally, when there are multiple autonomous driving algorithms that need to be tested, the algorithm processing module 613 is also used to provide a visual interface, allowing relevant personnel to select the autonomous driving algorithm that needs to be tested, and to view the scene effects after the test.

[0133] Algorithm evaluation module 614 is configured to determine an evaluation result of the autonomous driving algorithm based on the label information and processing results. The evaluation result indicates whether there are any issues with the autonomous driving algorithm's perception and planning and control capabilities in the test simulation scenario.

[0134] Optionally, the algorithm processing module 613 can design an evaluation algorithm based on the aforementioned label information. The evaluation dimensions typically include traffic regulations (such as speeding, running a red light, crossing the line, and collisions), and the riding experience (such as sudden braking, lane keeping, and safe spacing). For example, if the color label of the traffic light at the intersection ahead is red, the vehicle's driving behavior through the intersection is unreasonable.

[0135] Optionally, when the algorithm processing module 613 simulates and runs the autonomous driving algorithm based on environmental information in multiple scenarios to obtain processing results of multiple scenario simulations, the algorithm evaluation module 614 can also calculate the pass rate of the test based on the above-mentioned label information and the processing results of multiple scenario simulations after the multiple scenario simulations are completed, thereby determining the evaluation results of the autonomous driving algorithm.

[0136] As shown in Figure 6, the autonomous driving algorithm module 620 includes core autonomous driving algorithms including perception algorithms and planning and control algorithms. Among them, the perception algorithm is used to perceive and detect the environmental information obtained above, so as to perceive and detect whether there is a target object from the environmental information, and perceive the state information of the detected target object. The planning and control algorithm is also referred to as the regulation and control algorithm in the embodiment of this application. The regulation and control algorithm is used for planning and controlling the vehicle, so that the vehicle can drive along the planned and controlled path, or perform acceleration, braking and other actions according to the results of the planning and control. This application does not limit the specific driving tasks or the actions performed by the vehicle.

[0137] The autonomous driving algorithm module 620 can implement autonomous driving in real environments through the vehicle hardware platform 630, and in simulated scenarios through the algorithm processing module 613. Functions such as perception and planning control can be implemented through the vehicle hardware platform 630, which includes a sensor system 631 and a computer system 632. The sensor system 631 collects environmental information, while the computer system 632 processes and evaluates the autonomous driving algorithm.

[0138] In some embodiments, the autonomous driving algorithm module 620, when deployed on the vehicle hardware platform 630, can also execute the autonomous driving evaluation method provided in the embodiments of the present application. For example, the computer system 632 can also include an environment acquisition module, a label acquisition module, an algorithm processing module, and an algorithm evaluation module.

[0139] It should be noted that the autonomous driving evaluation method provided in the embodiments of this application can be applied not only to the testing of autonomous driving systems, but also to the testing of other systems such as advanced driving assistance systems (ADAS). This application does not limit the type of autonomous driving system to be tested.

[0140] The methods in the following embodiments can all be implemented in a vehicle having the aforementioned hardware structure or other devices capable of testing and evaluating autonomous driving algorithms. For example, autonomous driving vehicles can also be servers, test platforms, or simulation platforms capable of testing and evaluating autonomous driving algorithms. For example, processors 301 and 430, as well as simulation test device 610, in computer system 160 mentioned above.

[0141] Please refer to Figure 7, which shows an autonomous driving evaluation method proposed in this application. The method is applied to electronic devices, which can be autonomous vehicles, robots, or other autonomous driving devices with autonomous driving capabilities, or other devices capable of testing and evaluating autonomous driving algorithms, such as cloud servers, algorithm testing platforms, and algorithm simulation platforms. As shown in Figure 7, the method includes steps S701-S704:

[0142] S701. Obtain environmental information of the environment where the autonomous driving device is located.

[0143] In an embodiment of the present application, the autonomous driving device may collect environmental information of its own environment through a variety of sensors during driving, and may also integrate the various environmental information currently collected to obtain integrated environmental information.

[0144] Alternatively, the environmental information may be traffic flow information and road conditions collected by various sensors installed on the autonomous driving device during driving. This information may include raw data from static objects such as traffic signs, traffic lights (e.g., red and green lights), and lane markings, as well as raw data from dynamic objects such as surrounding vehicles, pedestrians, and obstacles. These sensors may include lidar, millimeter-wave radar, ultrasonic radar, and monocular or binocular cameras.

[0145] Road surface information can reflect various road conditions. Traffic signs can indicate speed limits, upper and lower speed limits, and stop signs ahead. Traffic light colors can indicate whether to stop, turn left, or turn right. Lane markings can indicate vehicle direction, turning radius, lane directions, and lane change permissions. Vehicles, pedestrians, and obstacles can indicate the obstacle's relative position to the vehicle, lane information, and relative speed.

[0146] Optionally, obtaining environmental information of the environment in which the autonomous driving device is located may be obtaining all raw data collected by the autonomous driving device through sensors. The raw data may be an environmental image, and the environmental image may include the above-mentioned environmental information.

[0147] In some embodiments, when the electronic device is an autonomous driving device, it can directly access all raw data collected by sensors in real time. In some embodiments, when the electronic device is a test platform capable of testing and evaluating autonomous driving algorithms, the autonomous driving device can upload all raw data collected by sensors to the test platform, which can then add the data reported by the autonomous driving device to a scenario database. This allows the test platform to obtain environmental information from the scenario database.

[0148] S702: Acquire tag information corresponding to the environmental information, where the tag information is used to indicate the actual state of the target object in the environment.

[0149] In the embodiments of this application, when evaluating the autonomous driving algorithm's ability to process environmental information, it is necessary to obtain label information corresponding to the environmental information. Label information indicates the actual state of an object in the environment, which can also be referred to as the true value of the state of the object in the environment. For example, the actual color of a traffic light is referred to as the true value of the traffic light color.

[0150] For example, if the environmental information includes a traffic light image captured by an autonomous driving device, the electronic device can obtain the actual color state of the traffic light, such as red, as the traffic light color label in the traffic light image, i.e., the true value of the traffic light color. For example, if the environmental information includes a traffic sign image captured by an autonomous driving device, the electronic device can obtain the actual sign information, such as a speed limit of 30 km / h, as the speed limit label in the traffic sign image, i.e., the true value of the speed limit.

[0151] In some embodiments, when the electronic device is an autonomous driving device, the autonomous driving device can obtain label information corresponding to environmental information from an external device. For example, with authorization from a traffic management department, the autonomous driving device can obtain the actual color status information of traffic lights at various intersections from the relevant server of the traffic management department while driving. For another example, the autonomous driving device can be equipped with a true value acquisition device that can capture each target object in the environment surrounding the autonomous driving device and the actual status of each target object. Optionally, the true value acquisition device includes a radar that can capture the actual position, actual driving speed, and actual distance between each target object and the autonomous driving device.

[0152] Optionally, the autonomous driving device can be equipped with a true value annotation tool. After acquiring environmental information, the autonomous driving device can use the annotation tool to annotate the environmental information. The annotated content is referred to as a label, thereby obtaining label information corresponding to the environmental information. For example, using the annotation tool, an image of a traffic light captured by the autonomous driving device can be annotated as follows: "The color of the traffic light is green."

[0153] In some embodiments, when the electronic device is a test platform capable of testing and evaluating autonomous driving algorithms, after acquiring environmental information, the test platform can manually annotate the environmental information. For example, if a human eye observes that a traffic light image captured by the autonomous driving device is red, the image can be manually annotated as follows: "The color of the traffic light is red." Of course, annotation tools can also be used to annotate the environmental information to obtain the label information (ground truth) corresponding to the environmental information.

[0154] S703. Obtain the processing results of the autonomous driving algorithm on the environmental information. The autonomous driving algorithm is used to perceive the environmental information and plan and control the driving behavior based on the environmental perception results.

[0155] In an embodiment of the present application, the autonomous driving algorithm can be used to perceive environmental information, and after obtaining the perception results, the driving behavior of the autonomous driving device can be planned and controlled based on the environmental perception results to obtain a planning and control result.

[0156] Optionally, the autonomous driving algorithm may include an environmental perception algorithm and a planning and control algorithm (also known as a planning and control algorithm). The environmental perception algorithm is used to perceive and detect environmental information to obtain environmental perception results. The environmental perception results are used to characterize the objects and / or state information of the objects perceived and detected by the autonomous driving algorithm from the environmental information. The environmental perception results can be the perception and detection of whether there are other vehicles, pedestrians, traffic lights (such as traffic lights), green belts, lanes and other targets in the environment, or the perception and detection of the position, speed and other state information of other vehicles, pedestrians and other targets, or the perception and detection of the color information of traffic lights (such as traffic lights).

[0157] The control algorithm is used to plan and control the driving behavior of the autonomous driving device based on the perception results detected by the environmental perception algorithm, thereby obtaining a planning and control result. The planning and control result may include lateral and / or longitudinal decisions. Lateral decisions primarily use left and right steering to avoid objects as much as possible. Longitudinal decisions primarily use forward and backward acceleration, deceleration, and braking to minimize collisions with objects directly in front of or behind the vehicle.

[0158] In an embodiment of the present application, the electronic device may invoke an autonomous driving algorithm to process environmental information and obtain a processing result, which may include an environmental perception result and a planning and control result.

[0159] In some embodiments, when the electronic device is an autonomous driving device, the autonomous driving device may run the perception algorithm within its onboard autonomous driving algorithm to sense and detect environmental information collected in real time by sensors, thereby obtaining an environmental perception result. The device may then run the planning and control algorithm within its onboard autonomous driving algorithm to make decisions, i.e., plan and control, regarding its driving behavior based on the environmental perception result. This enables the autonomous driving device to travel along a planned and controlled driving path, or to perform actions such as acceleration and braking based on the planned and controlled result.

[0160] In some embodiments, when the electronic device is a test platform capable of testing and evaluating autonomous driving algorithms, after obtaining environmental information from a scenario database, the test platform can invoke the autonomous driving algorithm to be evaluated. Based on this environmental information, the autonomous driving algorithm to be evaluated is then simulated to obtain a processing result of the simulation. This processing result includes a perception result of the simulated environmental information and a simulation result obtained by simulated decision-making, i.e., simulated planning and control, of the autonomous driving device's driving behavior based on this simulated perception result. The autonomous driving algorithm to be evaluated can be one or more. If there are multiple autonomous driving algorithms to be evaluated, the test platform can simulate multiple autonomous driving algorithms simultaneously based on the environmental information.

[0161] Optionally, when the test platform obtains environmental information of multiple scenarios from the scenario database, after calling the autonomous driving algorithm to be evaluated, the test platform can repeatedly simulate the autonomous driving algorithm to be evaluated based on the environmental information of different scenarios, that is, replay the environmental perception algorithm and regulation and control algorithm in the autonomous driving algorithm to be evaluated in different scenarios, so as to achieve the purpose of testing the same autonomous driving algorithm in different scenarios.

[0162] S704. Obtain the environmental perception capability and planning control capability of the autonomous driving algorithm based on the label information and processing results.

[0163] In an embodiment of the present application, when it is necessary to evaluate the autonomous driving algorithm, the environmental perception ability and decision-making ability (i.e., planning and control ability) of the autonomous driving algorithm can be simultaneously evaluated based on the label information to evaluate whether the perception ability and planning and control ability of the autonomous driving algorithm are abnormal or unreasonable, thereby being able to promptly discover the perception problems, planning and control problems, and system-level protection problems of the autonomous driving system, thereby improving the reliability of the autonomous driving algorithm evaluation and realizing system-level protection of the autonomous driving equipment, that is, protection of the entire process from the sensor receiving traffic flow information and road conditions to the autonomous driving equipment making corresponding driving behavior decisions.

[0164] In some embodiments, an evaluation algorithm, also known as a checker, can be designed to evaluate the autonomous driving algorithm based on various label information. This allows the user to select a corresponding evaluation algorithm when evaluating the autonomous driving algorithm.

[0165] As a method, an evaluation algorithm can be used to assess the planning and control results generated by the autonomous driving algorithm based on perception results. The evaluation algorithm may include at least one evaluation dimension. This evaluation dimension may include traffic regulations, such as speeding, running red lights, crossing the line, and collisions, as well as passenger experience, such as sudden braking, lane keeping, and safe spacing. This is done to assess whether the planning and control results generated by the autonomous driving algorithm based on perception results comply with traffic regulations and / or conventional passenger experience standards.

[0166] As an example, taking the label information as the color label of a traffic light, the designed evaluation algorithm may include that when the color label of the traffic light at the intersection ahead is red, it is unreasonable for the autonomous driving algorithm to make a driving behavior decision to pass through the intersection; or when the color label of the traffic light at the intersection ahead is green, it is unreasonable for the autonomous driving algorithm to make a driving behavior decision to brake.

[0167] It can be understood that label information indicates the actual state of objects in the environment, such as the true color of a traffic light. Therefore, an evaluation algorithm designed based on accurate label information can accurately assess whether the planning and control results currently generated by the autonomous driving algorithm based on perception results meet the driving behavior evaluation criteria, that is, whether they are reasonable. Meeting the driving behavior evaluation criteria can refer to whether the actions of the autonomous driving device meet traffic regulations, kinematics, and the riding experience when the test platform invokes the autonomous driving control algorithm for simulation in the test scenario. It can also refer to whether the actions of the real vehicle meet traffic regulations, kinematics, and the riding experience when the driving control algorithm is running in the real vehicle.

[0168] For example, taking the environmental information including the image of a traffic light at the intersection ahead as an example, when the electronic device obtains the traffic light color label of red in the traffic light image through the aforementioned method, if the driving behavior decision made in the processing result of the automatic driving algorithm is to pass the intersection ahead, the electronic device can evaluate that the driving behavior decision made is unreasonable based on the evaluation algorithm corresponding to the color label of the traffic light, and the driving behavior decision has the problem of running a red light.

[0169] As another approach, the evaluation algorithm can also be used to evaluate the perception results of the autonomous driving algorithm. The evaluation algorithm may include two evaluation dimensions: false detection and missed detection. False detection can be understood as the incorrect perception of the type of target object when perceiving and detecting environmental information, such as perceiving a bicycle as a motorcycle, or the incorrect perception of the state information of the target object, such as perceiving the red light of a traffic light as a green light or yellow light. Missed detection can be understood as the failure to perceive the target object or the state information of the target object when perceiving and detecting environmental information, such as there are three traffic lights in the environmental image of the intersection ahead, and the autonomous driving algorithm only perceives and detects two traffic lights from the environmental image, or perceives and detects three traffic lights but only perceives the colors of two traffic lights.

[0170] As an example, taking the label information as the color label of a traffic light, the designed evaluation algorithm may include, for example, when the color label of the traffic light at the intersection ahead is red, it is unreasonable for the autonomous driving algorithm to perceive and recognize that green or yellow is unreasonable; or when the color label of the traffic light at the intersection ahead is green, it is unreasonable for the autonomous driving algorithm to perceive and recognize that red or yellow is unreasonable.

[0171] It can be understood that label information indicates the actual status of the target object in the environment, such as the actual color status of the traffic light. Therefore, the evaluation algorithm designed based on accurate label information can accurately evaluate whether the current perception results of the autonomous driving algorithm are accurate.

[0172] As a method, the evaluation algorithm can also be used to simultaneously assess the perception results and planning and control results of the autonomous driving algorithm. For example, if the environmental information includes an image of a traffic light at a forward intersection, and the electronic device obtains the traffic light image through the aforementioned method and the traffic light color label is red, it is unreasonable for the autonomous driving algorithm to perceive and recognize the traffic light color in the traffic light image as green or yellow. If the autonomous driving algorithm perceives and recognizes the traffic light color as red, it is unreasonable for the autonomous driving algorithm to make a behavioral decision to pass the forward intersection.

[0173] In some embodiments, the tag information corresponding to the environmental information, i.e., the annotated environmental truth value, can be stored as a JSON file in an object storage service (OBS) bucket. A truth value parsing framework is designed to call the OBS bucket results for the truth value and perform field parsing on the JSON file annotated with the environmental truth value (e.g., a JSON file annotated with a traffic light). This encapsulates the truth value call algorithm to provide an interface for the evaluation algorithm checker to call.

[0174] In some embodiments, after evaluating the autonomous driving algorithm's environmental perception and planning and control capabilities based on tag information and processing results, the electronic device can automatically capture problematic events that do not meet the evaluation criteria, i.e., are unreasonable. These problematic events are used to indicate an abnormality in the autonomous driving algorithm. Optionally, the event information carried by the problematic event can include processing results that do not meet the evaluation criteria, such as running a red light or accidentally braking at a green light. Optionally, the event information carried by the problematic event can also include timestamp information of the time the problem occurred, so that the electronic device can retrieve corresponding data based on the corresponding timestamp information to analyze the cause of the problematic event.

[0175] In some embodiments, after the electronic device determines the evaluation result of the autonomous driving algorithm, it can also report the problem event to the cloud server, and the cloud server will retrieve the corresponding data to analyze the cause of the problem event.

[0176] In some embodiments, after evaluating the environmental perception and planning control capabilities of the autonomous driving algorithm based on label information and processing results, the environmental perception and planning control capabilities of the autonomous driving algorithm can also be optimized based on the evaluated abnormal problem events to obtain an optimized autonomous driving algorithm and re-run it on the autonomous driving device.

[0177] The embodiment of the present application provides an evaluation method for autonomous driving. The electronic device can obtain corresponding label information indicating the actual status of the target object in the environment based on the environmental information in the environment where the autonomous driving device is located, and obtain the processing results of the autonomous driving algorithm on the environmental information. Among them, the autonomous driving algorithm is used to perceive the environmental information and plan and control the driving behavior based on the environmental perception results. The electronic device can then determine the environmental perception capability and planning and control capability of the autonomous driving algorithm based on the label information and processing results. While realizing the evaluation of the system-level capability of the autonomous driving algorithm, it can also promptly discover perception anomalies and planning and control anomalies in the autonomous driving algorithm. The reliability and safety of the autonomous driving algorithm evaluation are improved.

[0178] Please refer to Figure 8, which shows another autonomous driving evaluation method proposed in this application. Based on the label information corresponding to the environmental information, the perception problem and planning and control problem of the autonomous driving algorithm can be automatically divided, thereby enabling targeted optimization of the perception algorithm and planning and control algorithm. As shown in Figure 8, the method includes steps S801-S808:

[0179] S801. Obtain environmental information of the environment where the autonomous driving device is located.

[0180] In some embodiments, electronic devices can evaluate the autonomous driving algorithm's perception and planning and control capabilities in different driving environments. These driving environments can include traffic lights, speed limits, the presence of other vehicles, the presence of pedestrians, and lane markings. This allows for a system-level assessment of the autonomous driving algorithm's ability to navigate traffic lights, speed limits, avoid other vehicles, avoid pedestrians, and confirm right-of-way, from perception to decision-making.

[0181] In some embodiments, the electronic device can obtain environmental information about the environment surrounding the autonomous driving device when the autonomous driving device is in a target driving environment, so as to specifically evaluate the autonomous driving algorithm in the target driving environment. The target driving environment can be one or more of the aforementioned traffic light environment, speed limit environment, other vehicle presence environment, pedestrian presence environment, lane marking environment, etc., or other driving environments. This is not limited in the present embodiment.

[0182] Optionally, environmental information can be information of interest in the target driving environment. For example, in a traffic light environment, the information of interest is the color status of the traffic light; in a speed limit environment, the information of interest is the speed limit status indicated by the traffic sign; in an environment with other vehicles, the information of interest is the location, speed, and direction of other vehicles; and in a lane line environment, the information of interest is the number of lanes, the location of lane lines, and other information.

[0183] In some embodiments, when the electronic device is a test platform capable of testing and evaluating autonomous driving algorithms, the autonomous driving device can be driven in various driving environments and collect environmental information about the device's surroundings during driving, which can be uploaded to the test platform's scenario database. When the test platform needs to evaluate the autonomous driving algorithm's perception and planning and control capabilities in a target driving environment, the test platform can obtain environmental information related to the target driving environment from the scenario database to conduct a targeted evaluation of the autonomous driving algorithm in that target driving environment.

[0184] Taking the target driving environment as a traffic light intersection environment as an example, the test platform can obtain traffic light images in the surrounding environment of the autonomous driving device when the autonomous driving device is at different traffic light intersections from the scene database, so as to conduct a targeted evaluation of the autonomous driving algorithm's ability to pass traffic light intersections.

[0185] In some embodiments, personnel can create assessment tasks on an electronic device to conduct targeted evaluations of certain capabilities of an autonomous driving algorithm. For example, personnel can create a task to assess the autonomous driving algorithm's ability to navigate traffic lights. This allows the electronic device to automatically acquire environmental information including traffic light scenarios.

[0186] S802: Acquire tag information corresponding to the environmental information, where the tag information is used to indicate the actual state of the target object in the environment.

[0187] In some embodiments, after creating an evaluation task on an electronic device, a person can select a corresponding evaluation algorithm. For example, after creating an evaluation task for an autonomous driving algorithm's ability to navigate a traffic light intersection, a person can select a corresponding algorithm for evaluating the rationality of the traffic light intersection.

[0188] In some embodiments, relevant personnel can create multiple evaluation tasks on the electronic device and, correspondingly, select multiple corresponding evaluation algorithms to simultaneously evaluate the multiple capabilities of the autonomous driving algorithm.

[0189] S803: Obtain the processing result of the autonomous driving algorithm on the environmental information.

[0190] S804: Evaluate whether the processing result is reasonable based on the tag information.

[0191] In an embodiment of the present application, the electronic device can perform a rationality evaluation of the processing results of the autonomous driving algorithm based on the label information to evaluate whether the processing results are reasonable.

[0192] Since the processing results of the autonomous driving algorithm on environmental information may include the perception results obtained by the autonomous driving algorithm of the environmental information, and the planning and control results obtained by planning and controlling the driving behavior of the autonomous driving device based on the perception results, the above-mentioned evaluation of whether the processing results are reasonable based on the label information may be an evaluation of whether the planning and control results obtained by the autonomous driving algorithm based on the perception results of the environmental information are reasonable.

[0193] For example, if environmental information includes an image of a traffic light at the intersection ahead, and the color label of the traffic light at the intersection ahead is red, and the autonomous driving algorithm makes a driving behavior decision to pass through the intersection based on the perceived color of the traffic light, then the traffic light color label can be used to assess that the autonomous driving algorithm's driving behavior decision to pass through the intersection is unreasonable. However, it is not certain whether the driving behavior decision is due to an abnormality in the autonomous driving algorithm's perception or an abnormality in the autonomous driving algorithm's planning and control. Therefore, further comparison of the traffic light color label with the autonomous driving algorithm's perception output is necessary to accurately locate the source of the problem.

[0194] In some embodiments, the environmental information acquired by the electronic device may include corresponding timestamp information, and accordingly, the tag information may also include corresponding timestamp information, so that the electronic device can determine the tag information corresponding to the environmental information based on the timestamp information.

[0195] In some embodiments, the electronic device may use each timestamp as an observation point for evaluating whether the processing results of the autonomous driving algorithm are reasonable, so as to evaluate the processing results of the corresponding timestamps based on the label information of different timestamps.

[0196] S805. When the evaluation processing result is unreasonable, obtain the environmental perception result obtained by the autonomous driving algorithm perceiving the environmental information from the processing result.

[0197] In an embodiment of the present application, when evaluating the rationality of the processing results, the electronic device can automatically capture problem events in which the processing results do not meet the evaluation criteria, that is, are unreasonable, and compare the label information of the problem event with the perception results to define the problem of the problem event, thereby achieving the purpose of diverting the problem into perception algorithm problems and control algorithm problems.

[0198] Because the processing results of the autonomous driving algorithm on environmental information may include the perception results obtained by the autonomous driving algorithm perceiving the environmental information, as well as the planning and control results obtained by planning and controlling the driving behavior of the autonomous driving device based on the perception results, if it is assessed that the driving behavior planned and controlled by the autonomous driving algorithm based on the perception results of the environmental information is unreasonable, the electronic device may obtain the perception results obtained by the autonomous driving algorithm perceiving the environmental information and compare the perception results with the label information.

[0199] S806: Compare the tag information with the environment perception result.

[0200] In some embodiments, the comparison between the label information and the perception result can be to determine whether the target object detected from the environmental information is consistent with the target object marked in the label information, and whether there are omissions or false detections. It can also be to determine whether the status information of the target object detected from the environmental information is consistent with the status information of the target object marked in the label information, and whether there are omissions or false detections.

[0201] Taking the example of an image of a traffic light at the intersection ahead where environmental information includes the image, as shown in (a) in FIG9 , the label information corresponding to the traffic light image can be as shown in (b) in FIG9 , which marks the traffic lights 901 and 902 existing in the environment of the intersection ahead, as well as the corresponding true colors of the traffic lights.

[0202] When the evaluation results indicate that a red light running problem event has occurred, if the automatic perception algorithm detects the traffic light image and the perception result is as shown in FIG9(c), the electronic device can compare the label information corresponding to the traffic light image shown in FIG9(b) to determine that the label information is inconsistent with the perception result. Furthermore, it can be determined that the automatic perception algorithm has missed the detection of the traffic light 901.

[0203] If the automatic perception algorithm senses and detects the traffic light image, and the perception result is as shown in FIG9(d), the electronic device can compare the label information corresponding to the traffic light image shown in FIG9(b) to determine that the label information is inconsistent with the perception result. Furthermore, the automatic perception algorithm can determine that the color of traffic light 901 was misdetected by the automatic perception algorithm, misidentifying the red light as a yellow light.

[0204] It can be understood that if the automatic perception algorithm perceives and detects the traffic light image and the perception result is consistent with the label information corresponding to the traffic light image shown in (b) in Figure 9, it can be determined that the perception ability of the autonomous driving algorithm is normal.

[0205] In some embodiments, when evaluating the rationality of the processing results, if the evaluation processing results are reasonable, the electronic device may also compare the label information with the perception results to evaluate the system-level capabilities of the autonomous driving algorithm.

[0206] Understandably, if the autonomous driving algorithm's processing results are assessed as reasonable, but the label information is inconsistent with the perception results, the algorithm can be considered to have good handling capabilities for certain edge cases. For example, in scenarios such as traffic light intersections at night or in foggy and rainy weather, autonomous driving algorithms are prone to missed or false detections. Based on current weather information, the autonomous driving algorithm can intelligently downgrade to manual driving mode, such as switching to manual driving mode or driving at a safe and slow speed to obtain a close-up image of the traffic light before planning and controlling. This approach, rather than evaluating the perception algorithm and the control algorithm separately, involves a system-level assessment of the autonomous driving equipment, encompassing the entire process from the moment the sensors receive traffic flow information and road conditions to the moment the autonomous driving equipment makes the appropriate driving decisions. This ensures that the autonomous driving algorithm has sufficient handling capabilities for certain edge cases, improving the reliability and safety of the autonomous driving algorithm evaluation.

[0207] S807: When the label information is inconsistent with the environmental perception result, it is determined that the environmental perception capability of the autonomous driving algorithm is abnormal.

[0208] In an embodiment of the present application, when the label information is inconsistent with the perception result, it can be determined that the perception ability of the autonomous driving algorithm is abnormal, and it can be further determined whether it is a missed detection or a false detection.

[0209] For example, when the evaluation and processing results determine that a red light running problem event occurred, if it is determined that the automatic perception algorithm missed a red light detection, the problem event can be characterized as a red light running problem caused by a missed red light detection, and the problem with the autonomous driving algorithm's perception algorithm can be located, and the need for optimization of the autonomous driving algorithm's perception algorithm can be located. If it is determined that the automatic perception algorithm falsely detected a red light, the problem event can be characterized as a red light running problem caused by a false red light detection, and the problem with the autonomous driving algorithm's control algorithm can be located, and the need for optimization of the control algorithm can be used to optimize the autonomous driving algorithm's environmental perception capabilities.

[0210] In some embodiments, when the electronic device is an autonomous driving device, if the autonomous driving algorithm determines that its perception capability is abnormal, a first prompt message may be displayed on the display screen (display module) of the autonomous driving device to alert the driver of the abnormality in the autonomous driving system's perception. Optionally, the autonomous driving device may also automatically report relevant data corresponding to the abnormality to a cloud server to facilitate subsequent optimization and update of the autonomous driving algorithm.

[0211] S808: When the label information is consistent with the environmental perception result, it is determined that the planning and control capability of the autonomous driving algorithm is abnormal.

[0212] In an embodiment of the present application, when the label information is consistent with the perception result, it can be determined that the perception result of the autonomous driving algorithm is correct, that is, the perception ability is normal, and then it can be determined that the unreasonable processing result is because there is a problem with the planning and control algorithm in the autonomous driving algorithm, that is, the evaluation result of the autonomous driving algorithm is that the planning and control ability is abnormal.

[0213] For example, if the evaluation results indicate a red light running incident, and the automatic perception algorithm determines that the traffic light is red, consistent with the label information, then the automatic perception algorithm's perception results can be confirmed to be correct. However, if the autonomous driving algorithm's control algorithm is still able to make driving decisions through the intersection when the light is red, indicating a problem with the control algorithm. At this point, the problematic incident can be characterized as a red light running incident caused by a control anomaly, and the problem with the autonomous driving algorithm's control algorithm has been identified. Optimization of the control algorithm is therefore necessary to optimize the autonomous driving algorithm's planning and control capabilities.

[0214] In some embodiments, when the electronic device is an autonomous driving device, if the autonomous driving algorithm's evaluation result indicates an abnormality in planning and control capabilities, a second prompt message may be displayed on the autonomous driving device's display screen (display module) to alert the driver of the abnormality in the autonomous driving system's planning and control. Optionally, the autonomous driving device may also automatically report relevant data corresponding to the abnormality to a cloud server to facilitate subsequent optimization and update of the autonomous driving algorithm.

[0215] In some embodiments, the electronic device may also calculate the pass rate of the autonomous driving algorithm in different driving environments based on the evaluation results, or may calculate the pass rate of the autonomous driving algorithm in a target driving environment. The pass rate may be the probability that the autonomous driving algorithm's processing of the environment is reasonable, such as a first value; the probability that the autonomous driving algorithm's processing of the environment is unreasonable due to an abnormality in the autonomous driving algorithm's perception capability, such as a second value; or the probability that the autonomous driving algorithm's processing of the environment is unreasonable due to an abnormality in the autonomous driving algorithm's planning and control capability, such as a third value.

[0216] Taking the target driving environment as a traffic light intersection environment as an example, the electronic device can automatically capture problem events in which the processing results of the autonomous driving algorithm in multiple traffic light intersection environments are unreasonable by replaying the environmental information of the autonomous driving algorithm and evaluating the replayed processing results based on the corresponding label information. Based on the problem events, the traffic light intersection pass rate of the autonomous driving algorithm can be counted, and the abnormal causes of the autonomous driving algorithms that failed, that is, the autonomous driving algorithms that generated problem events, can be counted. For example, the probability of running a red light due to abnormal perception of the autonomous driving algorithm, the probability of running a red light due to abnormal planning and control of the autonomous driving algorithm, and the probability of sudden braking at a green light due to abnormal perception of the autonomous driving algorithm and the probability of sudden braking at a green light due to abnormal planning and control of the autonomous driving algorithm can be counted.

[0217] In the autonomous driving evaluation method provided by the embodiments of the present application, an electronic device can evaluate the autonomous driving algorithm based on label information to automatically identify problem events. By comparing the label information with the autonomous driving algorithm's perception output, the problem event can be demarcated as a problem module, thereby separating the problem into perception algorithm issues and regulatory control algorithm issues. Specifically, for red light running problems caused by missed red light detection, the evaluation algorithm automatically identifies the problem event and, by comparing the label information with the autonomous driving algorithm's perception output topic, automatically locates the problem module as a perception algorithm issue. For red light running problems caused by incorrect regulatory control decisions, the evaluation algorithm automatically identifies the problem event and, by comparing the label information with the autonomous driving algorithm's perception output topic, automatically locates the problem module as a regulatory control algorithm issue. This method not only enables simultaneous evaluation of the autonomous driving algorithm's perception and planning and control capabilities, but also promptly detects perception and planning and control anomalies within the autonomous driving algorithm, thereby improving the reliability and safety of autonomous driving algorithm evaluation.

[0218] It is understandable that, in order to realize the above functions, the above-mentioned autonomous driving equipment and the like include hardware structures and / or software modules corresponding to the execution of each function. It should be easily appreciated by those skilled in the art that, in combination with the units and algorithm steps of the various examples described in the embodiments disclosed herein, the embodiments of the present application can be implemented in the form of hardware or a combination of hardware and computer software. Whether a function is executed in the form of hardware or computer software driving hardware depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the embodiments of the present application.

[0219] It should be noted that the module division in the embodiments of this application is illustrative and represents only one logical functional division. In actual implementation, other divisions may be used. For example, an autonomous driving device may include memory, a processor, a communication interface, and a bus. The memory, processor, and communication interface communicate with each other via a bus.

[0220] An embodiment of the present application provides an evaluation device for consistent autonomous driving, which can implement the methods of each step in the above embodiment.

[0221] In the embodiment of the present application, the autonomous driving evaluation device can be divided into functional modules according to the above-mentioned method example. When each functional module is divided according to each function, Figures 2 and 6 show a possible structural diagram of the autonomous driving evaluation device involved in the above-mentioned embodiment. The device includes an environment acquisition module, a label acquisition module, an algorithm processing module, and an algorithm evaluation module. Of course, the autonomous driving evaluation device may also include other modules, or the autonomous driving evaluation device may include fewer modules.

[0222] An embodiment of the present application provides a computer-readable storage medium storing one or more programs, wherein the one or more programs include instructions that, when executed by a computer, enable the computer to perform the methods of each step in the above embodiment.

[0223] The embodiments of the present application also provide a computer program product comprising instructions, which, when executed on a computer, enables the computer to execute the methods of each step in the above embodiments.

[0224] An embodiment of the present application provides an autonomous driving evaluation device, comprising a processor and a memory; wherein the memory is used to store computer program instructions, and the processor is used to run the computer program instructions to enable the autonomous driving evaluation device to execute the autonomous driving evaluation method performed in the steps of the above embodiment.

[0225] Those skilled in the art can clearly understand that, for the convenience and simplicity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments, and will not be repeated here. If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or the part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in each embodiment of the present application. The aforementioned storage medium includes various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0226] The above description is only a specific implementation method of the present application, but the protection scope of the present application is not limited thereto. Any changes or replacements within the technical scope disclosed in the present application should be included in the protection scope of the present application.

Claims

1. An evaluation method for autonomous driving, characterized in that, Including: Obtain label information corresponding to the environmental information of the environment where the autonomous driving device is located, and the label information is used to indicate the actual state of the target object in the environment; Obtain the processing result of the autonomous driving algorithm based on the environmental information, and the autonomous driving algorithm is used to sense the environmental information and plan and control the driving behavior based on the environmental perception result; Based on the label information and the processing result, obtain the environmental perception ability and the planning and control ability of the autonomous driving algorithm.

2. The method according to claim 1, characterized in that, The obtaining the label information corresponding to the environmental information of the environment where the autonomous driving device is located includes: Obtain the label information corresponding to the environmental information of the environment where the autonomous driving device is located sent by an external device; or Annotate the environmental information of the environment where the autonomous driving device is located to obtain the label information corresponding to the environmental information.

3. The method according to claim 1 or 2, characterized in that, The obtaining the environmental perception ability and the planning and control ability of the autonomous driving algorithm based on the label information and the processing result includes: Based on the label information, evaluate whether the processing result is reasonable; When it is evaluated that the processing result is unreasonable, obtain, from the processing result, the environmental perception result obtained by the autonomous driving algorithm for sensing the environmental information; Based on the comparison result between the label information and the environmental perception result, obtain the environmental perception ability and the planning and control ability of the autonomous driving algorithm.

4. The method according to claim 3, wherein The obtaining the environmental perception ability and the planning and control ability of the autonomous driving algorithm based on the comparison result between the label information and the environmental perception result includes: When the label information is inconsistent with the environmental perception result, determine that the environmental perception ability of the autonomous driving algorithm is abnormal; When the label information is consistent with the environmental perception result, determine that the planning and control ability of the autonomous driving algorithm is abnormal.

5. The method according to claim 3, wherein The obtaining the environmental perception ability and the planning and control ability of the autonomous driving algorithm based on the comparison result between the label information and the environmental perception result includes: Based on the comparison result between the label information and the environmental perception result, optimize the autonomous driving algorithm to optimize the environmental perception ability and the planning and control ability of the autonomous driving algorithm.

6. The method according to claim 5, wherein The autonomous driving algorithm includes an environmental perception algorithm and a planning and control algorithm. The optimizing the autonomous driving algorithm based on the comparison result between the label information and the environmental perception result includes: When the label information is inconsistent with the environmental perception result, optimize the environmental perception algorithm to optimize the environmental perception ability of the autonomous driving algorithm; When the label information is consistent with the environmental perception result, optimize the planning and control algorithm to optimize the planning and control ability of the autonomous driving algorithm.

7. The method according to any one of claims 3 to 6, characterized in that The method further includes: When the label information is inconsistent with the environmental perception result, display a first prompt message; When the label information is consistent with the environmental perception result, display a second prompt message.

8. The method according to any one of claims 1-7, characterized in that, The method further includes: Determine the probability that the processing result of the autonomous driving algorithm in the environment is reasonable; or Determine the probability that the processing result of the autonomous driving algorithm is unreasonable due to the abnormal environmental perception ability of the autonomous driving algorithm in the environment; or Determine the probability that the processing result of the autonomous driving algorithm is unreasonable due to the abnormal planning and control ability of the autonomous driving algorithm in the environment.

9. The method according to any one of claims 1-8, characterized in that, The environmental information includes static target information and dynamic target information.

10. The method according to any one of claims 3-9, characterized in that, Evaluating whether the processing result is reasonable according to the label information includes: Judging whether the processing result is consistent with the driving behavior evaluation standard according to the corresponding relationship between the preset label information and the driving behavior evaluation standard; When the processing result is consistent with the driving behavior evaluation standard, evaluate that the processing result is unreasonable.

11. The method according to any one of claims 1-10, characterized in that, Obtaining the processing result of the autonomous driving algorithm based on the environmental information includes: Based on the environmental information, call the autonomous driving algorithm for simulation to obtain the simulation result as the processing result.

12. An evaluation device for autonomous driving, characterized in that, The device includes a label acquisition module, an algorithm processing module, and an algorithm evaluation module, where: The label acquisition module is used to acquire label information corresponding to the environmental information of the environment where the autonomous driving device is located, and the label information is used to indicate the actual state of the target in the environment; The algorithm processing module is used to obtain the processing result of the autonomous driving algorithm based on the environmental information, and the autonomous driving algorithm is used to perceive the environmental information and plan and control the driving behavior based on the environmental perception result; The algorithm evaluation module is used to obtain the environmental perception ability and planning and control ability of the autonomous driving algorithm according to the label information and the processing result.

13. An evaluation device for autonomous driving, characterized in that, The device includes a memory and one or more processors; the memory and the processor are coupled; the memory is used to store program code, and the program code includes instructions. When the processor executes the instructions, the device executes the method according to any one of claims 1-11.

14. An autonomous driving device, characterized in that, The autonomous driving device includes the autonomous driving evaluation device according to claim 13.

15. A vehicle, characterized in that, The vehicle includes the autonomous driving evaluation device according to claim 13.

16. A readable storage medium, characterized in that, Including instructions, when the instructions run on the autonomous driving evaluation device, the autonomous driving evaluation device is caused to execute the method according to any one of claims 1-11.

Images