Method, apparatus, device and medium for generating autonomous driving training samples

Abstract

The present application discloses a method, apparatus, device and medium for generating an autonomous driving training sample, which is applied to the field of driverless driving. The method includes: in the autonomous driving mode, the autonomous driving algorithm controls the autonomous driving device to automatically drive based on the autonomous driving logic; during the automatic driving process, collect the operation instructions input through the human-machine interface on the autonomous driving device, and the operation instructions are used to label the control intention of the human on the autonomous driving device during the automatic driving process; use the driving data of the automatic driving process as a sample and the control intention corresponding to the operation instructions as a label to generate a training sample, and the training sample is used to optimize the autonomous driving logic of the autonomous driving algorithm according to the control intention. This method can improve the annotation accuracy of the control intention.

Description

Technical Field

[0001] This application relates to the field of driverless, and particularly to a method, apparatus, device and medium for generating an automatic driving training sample. Background Art

[0002] With the development of network technology and the Internet, intelligent driverless vehicles have gradually become well-known to people. The behavior decision-making of automatic driving is the core module reflecting the intelligence of driverless vehicles. Its purpose is to enable driverless vehicles to have human-like decision-making capabilities and be able to issue control instructions that take into account safety, comfort, and traffic efficiency in various complex scenarios.

[0003] In related technologies, an automatically driving model is trained using manually marked control intentions. When the vehicle is in the automatic driving mode, driving data in different driving scenarios is collected through an in-vehicle system. Then, the marked personnel play back the driving data collected in different scenarios offline. For abnormal behaviors that may cause danger, combined with data from multiple sensors and predetermined marking rules, the control instructions that the driverless vehicle should take in the relevant scenarios are marked.

[0004] In the methods of related technologies, the control intentions are marked manually offline based on the driving data. Since the marked personnel cannot actually perceive the driving scenarios, the marking results are inconsistent with the decisions of actual drivers, and the accuracy rate is relatively low. Summary of the Invention

[0005] Embodiments of this application provide a method, apparatus, device and medium for generating an automatic driving training sample, which can improve the marking accuracy of control intentions. The technical solutions are as follows:

[0006] According to one aspect of this application, a method for generating an automatic driving training sample is provided. The method is executed by an automatic driving device, and the method includes:

[0007] In the automatic driving mode, the automatic driving algorithm controls the automatic driving device to automatically drive based on the automatic driving logic;

[0008] During the automatic driving process, operation instructions input through the human-machine interface on the automatic driving device are collected. The operation instructions are used to mark the control intention of the human for the automatic driving device during the automatic driving process;

[0009] Taking the driving data of the automatic driving process as a sample and the control intention corresponding to the operation instructions as a label, a training sample is generated. The training sample is used to optimize the automatic driving logic of the automatic driving algorithm according to the control intention.

[0010] According to another aspect of this application, a device for generating an automatic driving training sample is provided. The device includes:

[0011] A driving module, configured to, in an autonomous driving mode, control the autonomous driving device to automatically drive based on an autonomous driving algorithm and an autonomous driving logic.

[0012] An interaction module, configured to, during the automatic driving process, collect operation instructions input through a human-machine interface on the autonomous driving device, where the operation instructions are used to label the control intention of a human for the autonomous driving device during the automatic driving process.

[0013] A generation module, configured to use the driving data of the automatic driving process as a sample and the control intention corresponding to the operation instructions as a label to generate a training sample, where the training sample is used to optimize the autonomous driving logic of the autonomous driving algorithm according to the control intention.

[0014] According to another aspect of the present application, a computer device is provided, including: a processor and a memory, where at least one instruction, at least one program, a code set, or an instruction set is stored in the memory, and the at least one instruction, the at least one program, the code set, or the instruction set is loaded and executed by the processor to implement the method for generating an autonomous driving training sample as described in the above aspect.

[0015] According to another aspect of the present application, a computer-readable storage medium is provided, where at least one instruction, at least one program, a code set, or an instruction set is stored in the storage medium, and the at least one instruction, the at least one program, the code set, or the instruction set is loaded and executed by a processor to implement the method for generating an autonomous driving training sample as described in the above aspect.

[0016] On the other hand, an embodiment of the present application provides a computer program product or a computer program, including computer instructions, where the computer instructions are stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the method for generating an autonomous driving training sample provided in the above optional implementation manner.

[0017] The beneficial effects brought by the technical solution provided by the embodiment of the present application at least include:

[0018] When the autonomous driving device is in the autonomous driving mode, the human operator annotates the control intention through the man-machine interface in the autonomous driving device. Compared with the method in the related art where the annotator annotates the control intention based on the offline video, the technical solution provided by the embodiments of the present application can perform the annotation more accurately. Since the offline video is the picture captured by the camera on the autonomous driving device, the picture perspective is fixed and single, and there is a deviation between the perspective provided by the camera and the observation perspective of the driver in the real driving environment, resulting in the offline annotation method being unable to truly restore the driving scene, and the annotator cannot annotate based on the real driving scene. In the technical solution provided by the embodiments of the present application, the annotator is inside the autonomous driving device, and the autonomous driving device is automatically driven by the autonomous driving algorithm, and the annotator can directly observe the driving situation of the autonomous driving device, the road surface condition, and the positional relationship with the surrounding obstacles from the first perspective of the cab, so as to accurately annotate the control intention during the driving process of the autonomous driving device. Using the method provided by the embodiments of the present application to annotate the control intention can accurately annotate based on the real driving condition of the autonomous driving device, making the annotation result closer to the decision-making of the driver during actual driving and improving the accuracy of the annotation. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following will briefly introduce the drawings required for the description of the embodiments. Obviously, the following-described drawings are only some embodiments of the present application. For those of ordinary skill in the art, without creative efforts, other drawings can be obtained based on these drawings.

[0020] Figure 1 is a block diagram of the implementation environment provided by an exemplary embodiment of the present application;

[0021] Figure 2 is a flowchart of a method for generating an autonomous driving training sample provided by another exemplary embodiment of the present application;

[0022] Figure 3 is a schematic diagram of a method for generating an autonomous driving training sample provided by another exemplary embodiment of the present application;

[0023] Figure 4 is a schematic diagram of a method for generating an autonomous driving training sample provided by another exemplary embodiment of the present application;

[0024] Figure 5 is a schematic diagram of a device for generating an autonomous driving training sample provided by another exemplary embodiment of the present application;

[0025] Figure 6 is a schematic diagram of the structure of an autonomous driving device provided by an exemplary embodiment of the present application. Detailed implementation manners

[0026] To make the objectives, technical solutions and advantages of the present application clearer, the following will further describe the implementation manners of the present application in detail with reference to the accompanying drawings.

[0027] First, the nouns involved in the embodiments of the present application will be introduced:

[0028] User Interface (UI) controls: refer to any visible controls or elements that can be seen on the user interface of an application, such as controls like pictures, input boxes, text boxes, buttons, labels, etc. Some of the UI controls respond to user operations. For example, when a user clicks on the switch account control, a switch account request can be sent to the server.

[0029] Please refer to Figure 1 , which shows a schematic diagram of an implementation environment involved in an embodiment of the present invention. The implementation environment includes: an autonomous driving device 100, and the autonomous driving device 100 can be a driverless vehicle (driverless car), a drone, etc.

[0030] In an optional implementation manner, the autonomous driving device 100 includes a decision-making module 101, a control module 102, a Controller Area Network (CAN) bus 103, an execution module 104, and a Human-Machine Interface (HMI) 105.

[0031] Exemplarily, the decision-making module 101 operates independently, converts decision-making instructions into control instructions for controlling the movement of the autonomous driving device 100 through the control module 102, and issues the control instructions to the execution module 104 through the CAN bus 103. At the same time, the HMI 105 receives input instructions, which are input manually through the HMI. The input instructions are collected and recorded by the CAN bus 103 and do not participate in the movement control of the autonomous driving device.

[0032] Optionally, the HMI includes at least one of a turn signal lever and a headlight lever.

[0033] Figure 2 shows a flowchart of a method for generating an autonomous driving training sample provided by an exemplary embodiment of the present application. The method can be applied to an autonomous driving device as shown in Figure 1 . The method includes the following steps:

[0034] Step 201, in the autonomous driving mode, the autonomous driving algorithm controls the autonomous driving device to automatically drive based on the autonomous driving logic.

[0035] The automatic driving mode is a mode in which a computer controls an automatic driving device to travel. The computer can be embedded in the automatic driving device, installed on the automatic driving device, or remotely control the automatic driving device through any communication method.

[0036] Exemplarily, the automatic driving mode in the embodiments of the present application does not support manual driving. That is, in the automatic driving mode, the automatic driving device does not respond to operation instructions input manually to travel. The operation instructions input manually are only used as annotation information and are collected and used to generate a control intention.

[0037] Exemplarily, the automatic driving device includes an automatic driving processor, a general processor, and a CAN (Controller Area Network) bus. The automatic driving processor and the general processor communicate through the CAN bus.

[0038] The automatic driving processor is used to execute the processing and operation of relevant data in the automatic driving mode and is used to collect data for the control intention. The general processor is used to execute the processing and operation of the basic data of the automatic driving device and is responsible for coordinating the work among various processors.

[0039] Exemplarily, the automatic driving device can be a driverless vehicle, a driverless delivery vehicle, a driverless transport vehicle, etc. Exemplarily, the automatic driving device can be applied to the unmanned transportation scenario, and all driving devices operating in the unmanned transportation scenario are automatic driving devices. The automatic driving algorithm can be an algorithm trained for the unmanned transportation scenario. Exemplarily, since the application scenario of the automatic driving device is the unmanned transportation scenario and there are few human factors involved in the scenario, the collection of training samples is based on the automatic driving mode. The vehicle always travels in the automatic driving mode, and the operation instructions input manually do not have an actual control effect, thereby ensuring that the collected training samples are appropriate for the unmanned transportation scenario.

[0040] In the automatic driving mode, the automatic driving processor controls the automatic driving device to travel automatically through the automatic driving algorithm.

[0041] Exemplarily, the automatic driving algorithm has an automatic driving logic. The automatic driving logic refers to the logical method by which the automatic driving algorithm outputs control instructions based on the input data. Exemplarily, the control instructions output by the automatic driving algorithm control the automatic driving device to travel automatically.

[0042] Step 202, during the automatic driving process, collect the operation instructions input through the human-machine interface on the automatic driving device. The operation instructions are used to annotate the human control intention for the automatic driving device during the automatic driving process.

[0043] Exemplarily, the annotator rides in the autonomous driving device. During the automatic driving process of the autonomous driving device, operation instructions are input through the HMI in the autonomous driving device to achieve real-time annotation of control intentions during the autonomous driving process.

[0044] Exemplarily, the human-machine interface (HMI) in the autonomous driving device can be any interface on the autonomous driving device that can receive human-machine interaction operations. For example, the human-machine interface can include at least one of a turn signal lever and a headlight lever. Additionally, the human-machine interface can also include at least one of a steering wheel, a brake, an accelerator, a clutch, a gear shift lever, a windshield wiper lever, and various buttons.

[0045] The operation instructions are instructions generated by the annotator's manual operation on the human-machine interface, and the human-machine interface responds to the manual operation. For example, when the annotator toggles the turn signal lever, a corresponding steering instruction can be generated.

[0046] Exemplarily, the operation instructions are invalid control instructions in the autonomous driving mode. That is, the autonomous driving device does not respond to the operation instructions input manually, and these operation instructions are not used for the driving control of the autonomous driving device but only for information collection.

[0047] Exemplarily, the main processor receives the operation instructions input through the human-machine interface on the autonomous driving device. The autonomous driving processor obtains the operation instructions from the main processor through the CAN bus. That is, the operation instructions input through the human-machine interface on the autonomous driving device are received by the main processor; the autonomous driving processor obtains the operation instructions from the main processor.

[0048] Step 203: Use the driving data during the automatic driving process as samples and the control intention corresponding to the operation instructions as labels to generate training samples, which are used to optimize the autonomous driving logic of the autonomous driving algorithm according to the control intention.

[0049] The autonomous driving device determines the control intention based on the driving state of the autonomous driving device when the operation instructions are received and the operation instructions; collects the driving data during the automatic driving process; and generates training samples, which include the driving data and the control intention.

[0050] The driving state is controlled by the autonomous driving device itself. The driving state can include at least one of a constant speed state, an acceleration state, a lane change state, a lane borrowing state, a lane keeping state, a reverse state, a parking state, a climbing state, a downhill state, a high-speed driving state, and a slow driving state.

[0051] Among them, the lane keeping state means that the autonomous driving device maintains driving in one lane. Or, the lane keeping state means that the autonomous driving device maintains a state of driving straight ahead.

[0052] The lane - changing state refers to: the autonomous driving device is changing from one lane to another. For example, the lane - changing state refers to the process in which the autonomous driving device changes from the first lane to the second lane. The first lane is parallel to the second lane, or the first lane intersects with the second lane. Exemplarily, the lane - changing state starts from when the tires of the autonomous driving device start to turn and ends when the tires of the autonomous driving device return to the straight - ahead position. Exemplarily, if the first lane is parallel to the second lane, after the lane - changing state ends, the position of the autonomous driving device changes laterally (perpendicular to the driving direction), and the driving direction remains unchanged. If the first lane intersects with the second lane, after the lane - changing state ends, the driving direction of the autonomous driving device changes.

[0053] The lane - borrowing state refers to: the autonomous driving device makes a lateral avoidance. For example, the lane - borrowing state refers to the process in which when there is an obstacle in front of the autonomous driving device, the autonomous driving device avoids the obstacle to the left or right and returns to its original position after passing the obstacle. Exemplarily, the lane - borrowing state includes: the process in which the autonomous driving device turns in one direction and then turns in the opposite direction. The lane on which the autonomous driving device travels remains unchanged before and after the lane - borrowing state, and the driving direction remains unchanged. Taking the example that the autonomous driving device was originally at the x position laterally (perpendicular to the driving direction), after the lane - borrowing state ends, the autonomous driving device returns to the x position laterally, or near the x position.

[0054] The control intention is the decision that the autonomous driving device should make under the current driving state marked by the annotator, or the control intention is the guiding opinion marked by the annotator for the driving instruction executed by the autonomous driving device. For example, the control intention marked by the annotator can be: annotation information related to the steering start time, steering end time, braking start time, lane - changing start time, lane - changing end time, lane - changing duration, lane - borrowing start time, lane - borrowing end time, lane - borrowing lateral distance, driving speed, driving direction, etc.

[0055] Exemplarily, the autonomous driving device collects the driving data within a period of time before receiving the operation instruction.

[0056] Exemplarily, the control intention is determined based on the driving state and operation instruction of the autonomous driving device; the driving data of the control intention and driving state and the operation data of the operation instruction are collected to generate a training sample. The control intention, the driving data of the driving state, and the operation data of the operation instruction correspond one by one. Among them, the control intention is the driving intention of the annotator determined based on the driving state and operation instruction.

[0057] The driving data of the driving state includes at least one of: the driving state of the autonomous driving device when receiving the operation instruction, the duration of being in this driving state, the start time of this driving state, the end time of this driving state, the driving distance of this driving state, the driving speed of this driving state, the vehicle route record in this driving state, and the driving video in this driving state.

[0058] The operation data of the operation instruction includes at least one of the reception time of the operation instruction, the type of the operation instruction, and the HMI that receives / generates the operation instruction.

[0059] Based on the training samples, it can be learned in what driving scenarios the annotator marked what driving decisions, so that the autonomous driving algorithm can learn the driving decisions of the annotator based on the training samples, thereby enabling the autonomous driving algorithm to better control the driving of the autonomous driving device.

[0060] Exemplarily, the autonomous driving device invokes the autonomous driving algorithm to control the driving of the autonomous driving device. The autonomous driving algorithm is trained using the collected training samples.

[0061] Exemplarily, the autonomous driving algorithm can be an algorithm of a machine learning model or an algorithm of a neural network model. The training sample includes the input data and the labeled data of the autonomous driving algorithm. The input data is input into the autonomous driving algorithm to obtain the output data. According to the gap between the output data and the labeled data, the autonomous driving algorithm is trained so that after iterative training, the autonomous driving algorithm can obtain the labeled data more accurately according to the input data.

[0062] Exemplarily, the autonomous driving algorithm can output a control instruction based on the driving data of the autonomous driving device in the past period of time. That is, the input of the autonomous driving algorithm is the driving data of the autonomous driving device in the past first time period, and the output is the control instruction, which is used to control the autonomous driving device to perform autonomous driving.

[0063] The manually annotated control intention is used to correct the control instruction.

[0064] Exemplarily, the autonomous driving algorithm can also output a control instruction based on the driving data of the autonomous driving device at the current moment. That is, the input of the autonomous driving algorithm is the driving data of the autonomous driving device at the current moment, and the output is the control instruction, which is used to control the autonomous driving device to perform autonomous driving.

[0065] Exemplarily, the autonomous driving device can also send the training samples to a computer, and the computer is used to train the autonomous driving algorithm based on the training samples. Exemplarily, the computer can be a server or a terminal.

[0066] Exemplarily, the autonomous driving processor obtains the driving state of the autonomous driving device from the main processor through the CAN bus, and annotates the control intention based on the driving state and the operation instruction of the autonomous driving device.

[0067] In summary, in the method provided in this embodiment, when the autonomous driving device is in the autonomous driving mode, a human operator annotates the control intention through the human-machine interface in the autonomous driving device. Compared with the method in the related art where annotators annotate control intentions based on offline videos, the technical solution provided in this application embodiment can perform annotation more accurately. Since the offline video is the picture captured by the camera on the autonomous driving device, the picture perspective is fixed and single, and there is a deviation between the perspective provided by the camera and the observation perspective of the driver in the real driving environment, resulting in the offline annotation method being unable to truly restore the driving scenario, and the annotator being unable to annotate based on the real driving scenario. In the technical solution provided in this application embodiment, the annotator is inside the autonomous driving device, and the autonomous driving device is automatically driven by the autonomous driving algorithm. The annotator can directly observe the driving situation of the autonomous driving device, the road surface condition, and the positional relationship with surrounding obstacles from the first perspective of the driver's cab, so as to accurately annotate the control intention during the driving process of the autonomous driving device. Using the method provided in this application embodiment to annotate the control intention can perform accurate annotation based on the real driving condition of the autonomous driving device, making the annotation result closer to the driver's decision-making during actual driving and improving the accuracy of annotation.

[0068] Exemplarily, as shown in Table 1, taking the HMI including a turn signal lever and a headlight lever, and the driving states including a lane change state and a lane keeping state as an example, the annotation of the control intention based on the driving state and the operation instruction is described.

[0069] Exemplarily, the turn signal lever includes: a left gear position, a neutral position, and a right gear position; the headlight lever includes: an up gear position, a neutral position, and a down gear position.

[0070] Table 1

[0071]

[0072] 1. In response to receiving a lane change operation in the lane keeping state, determine that the control intention is a lane change. Mark the moment when the lane change operation is received as the lane change initiation time.

[0073] The lane change operation is an operation used to express the lane change intention.

[0074] For example, in response to receiving a left turn operation in the lane keeping state, annotate the control intention as a left lane change; in response to receiving a right turn operation in the lane keeping state, annotate the control intention as a right lane change.

[0075] The left turn operation can be to move the turn signal lever from the neutral position to the left gear position, and the right turn operation can be to move the turn signal lever from the neutral position to the right gear position.

[0076] When factors such as navigation, overtaking, and risks generate a need to change lanes, the autonomous driving device needs to start changing lanes from the lane-keeping state. At this time, the annotator observes the traffic flow situation in the target lane and the relative distance and relative speed between the adjacent vehicles and the host vehicle. When the annotator believes that the lane-changing initiation timing is met, the annotator expresses the intention to initiate a lane change through the turn signal lever signal.

[0077] 2. In response to receiving a reverse lane-changing operation in the lane-changing state, determine that the control intention is to cancel the lane change; where the lane-changing state is used to change the driving lane of the autonomous driving device. Mark the moment when the reverse lane-changing operation is received as the lane-change cancellation timing.

[0078] The reverse lane-changing operation is a lane-changing operation in the direction opposite to the lane-changing direction of the current lane-changing state.

[0079] For example, in response to receiving a right-turning operation in the left-lane-changing state, mark the control intention as canceling the lane change; in response to receiving a left-turning operation in the right-lane-changing state, mark the control intention as canceling the lane change.

[0080] In the lane-changing state, the annotator always pays attention to the state of the social vehicle closer to the host vehicle (autonomous driving device) in the target lane. When it is found that the vehicle in front or behind is too close to the host vehicle and has no intention of giving way, the annotator can express the intention to cancel the lane change through the reverse turn signal lever, that is, if currently changing lanes to the left, cancel through the right lever; if currently changing lanes to the right, cancel the lane change through the left lever.

[0081] 3. In response to receiving a duration control operation in the lane-changing state, determine that the control intention is to change the lane-changing duration; where the lane-changing state is used to change the driving lane of the autonomous driving device.

[0082] The duration control operation is an operation used to express the intention of adjusting the lane-changing duration. Exemplarily, the duration control operation can be the operation of toggling the headlight lever in the lane-changing state.

[0083] For example, in response to receiving an upward operation of the headlight lever in the left-lane-changing state or the right-lane-changing state, mark the control intention as reducing the lane-changing duration; in response to receiving a downward operation of the headlight lever in the left-lane-changing state or the right-lane-changing state, mark the control intention as increasing the lane-changing duration.

[0084] The lane-changing duration refers to the total duration from the start of the lane-changing state to the end of the lane-changing state.

[0085] After the lane change is initiated, the autonomous driving device is in the lane change state. Considering factors such as comfort and efficiency, the annotator can annotate the lane change completion time of the autonomous driving device. Flipping the headlight lever upwards indicates the desire for the autonomous driving device to complete the lane change faster, i.e., the lane change time decreases; flipping the headlight lever downwards indicates the desire for the autonomous driving device to complete the lane change more smoothly, i.e., the lane change time increases.

[0086] Figure 3 The figure shows a schematic diagram of a method for generating an autonomous driving training sample provided by an exemplary embodiment of the present application. In the lane keeping state 301, the annotator can mark the lane change intention and the lane change initiation timing by operating the turn signal lever. After the autonomous driving algorithm controls the autonomous driving device to enter the lane change state 302, the annotator can mark the lane change duration by operating the headlight lever, or mark the lane change cancellation intention and the lane change cancellation timing by operating the turn signal lever. The autonomous driving algorithm controls the autonomous driving device to cancel the lane change, or the autonomous driving algorithm controls the autonomous driving device to complete the lane change and return to the lane keeping state 301.

[0087] Exemplarily, as shown in Table 2, taking the HMI including a turn signal lever and a headlight lever, and the driving states including the lane borrowing state and the lane keeping state as an example, the control intention is annotated based on the driving state and the operation instruction.

[0088] Exemplarily, the turn signal lever includes: a left gear position, a neutral position, and a right gear position; the headlight lever includes: an up gear position, a neutral position, and a down gear position.

[0089] Table 2

[0090]

[0091] 1. In response to receiving a lane borrowing operation in the lane keeping state, determine that the control intention is to borrow a lane. Mark the moment when the lane borrowing operation is received as the lane borrowing initiation timing.

[0092] The lane borrowing operation is an operation used to express the lane borrowing intention.

[0093] For example, in response to receiving an upward operation of the headlight lever in the lane keeping state, mark the control intention as borrowing a lane to the left; in response to receiving a downward operation of the headlight lever in the lane keeping state, mark the control intention as borrowing a lane to the right.

[0094] The upward operation of the headlight lever can be to move the headlight lever from the neutral position to the up gear position, and the downward operation of the headlight lever can be to move the headlight lever from the neutral position to the down gear position.

[0095] When the distance between the autonomous driving device and a low-speed obstacle ahead in the same lane or an obstacle in the oncoming lateral direction that is relatively close is less than a certain threshold, the autonomous driving device needs to make a lane-changing avoidance maneuver and switch from the lane-keeping state to the lane-changing state. At this time, the annotator needs to comprehensively judge the distance to the target obstacle and the distance to other lateral obstacles, decide whether to initiate a lane change, and use the headlight lever to express the decision-making intention. Pushing the headlight lever upward indicates a left lane change, and pushing the headlight lever downward indicates a right lane change.

[0096] 2. In response to receiving a distance adjustment operation in the lane-changing state, determine that the control intention is a lane-changing distance adjustment; wherein, the lane-changing state is used to control the autonomous driving device to make a lateral avoidance maneuver.

[0097] The distance adjustment operation is an operation used to express the intention of adjusting the lateral moving distance of the lane change. For example, the distance adjustment operation can be an operation of toggling the turn signal lever in the lane-changing state.

[0098] The lane-changing distance refers to the maximum distance that the autonomous driving device moves laterally in the lane-changing state, with the lateral direction perpendicular to the driving direction.

[0099] For example, in response to receiving a right-turn operation in the left lane-changing state, determine that the control intention is a decrease in the lateral distance; in response to receiving a left-turn operation in the left lane-changing state, mark the control intention as an increase in the lateral distance; in response to receiving a left-turn operation in the right lane-changing state, mark the control intention as a decrease in the lateral distance; in response to receiving a right-turn operation in the right lane-changing state, mark the control intention as an increase in the lateral distance.

[0100] After the lane change is initiated, the annotator observes the lateral distances between the host vehicle and the target obstacle, the lateral obstacles, and the road edge to ensure that the host vehicle does not have a collision risk with the obstacles due to being too close. If the distance is inappropriate, the annotator can use the turn signal lever to express the intention of adjusting the lateral distance. When making a left lane change, the left turn signal lever indicates that the lateral distance needs to be further increased, and the reverse turn signal lever indicates that the lateral distance needs to be decreased; when making a right lane change, the right turn signal lever indicates that the lateral distance needs to be further increased, and the left turn signal lever expresses that the lateral distance needs to be decreased.

[0101] 3. In response to receiving a lane-change completion operation in the lane-changing state, determine that the control intention is a lane-change completion; wherein, the lane-changing state is used to control the autonomous driving device to make a lateral avoidance maneuver. Mark the moment when the lane-change completion operation is received as the lane-change completion timing.

[0102] The lane-change completion operation is an operation used to express the intention of lane-change completion. Exemplarily, the lane-change completion operation can be an operation of toggling the headlight lever in the lane-changing state.

[0103] For example, in response to receiving a downward operation of the headlight lever in the left lane borrowing state, the control intention is marked as the lane borrowing is completed; in response to receiving an upward operation of the headlight lever in the right lane borrowing state, the control intention is marked as the lane borrowing is completed.

[0104] When the ego vehicle (autonomous driving equipment) and the target obstacle have a certain safe distance and the lateral position of the original lane is safe enough, the lane borrowing is completed and the ego vehicle returns to the lateral position of the original lane. At this time, the annotator can express the intention of completing the lane borrowing by using the reverse headlight lever, that is, push the headlight lever downward when borrowing the left lane, and push the headlight lever upward when borrowing the right lane, to indicate that the ego vehicle can complete the lane borrowing and return to the lane keeping state.

[0105] Figure 4 A schematic diagram of a method for generating autonomous driving training samples provided by an exemplary embodiment of the present application is shown. In the lane keeping state 301, the annotator can annotate the intention of borrowing lanes and the timing of initiating the borrowing lanes by operating the headlight lever. After the autonomous driving algorithm controls the autonomous driving device to enter the borrowing lane state 303, the annotator can annotate the distance of the borrowing lanes by operating the turn signal lever, or annotate the intention of completing the borrowing lanes and the timing of completing the borrowing lanes by operating the headlight lever. The autonomous driving algorithm controls the autonomous driving device to return to the lane keeping state 301 after the borrowing lanes are completed.

[0106] In summary, the method provided in the embodiment of the present application is based on the existing HMI input device on the autonomous driving device, and designs an input operation-decision mapping relationship based on a state machine, so that labeling personnel can mark lane changing and borrowing decision intentions online, efficiently and accurately based on the first-person perspective, providing an effective reference for the autonomous driving algorithm to learn human driving capabilities.

[0107] The method provided in the embodiment of the present application is completed by the annotator from the driver's perspective based on the actual scene observation input, using the existing input control equipment on the vehicle, and realizing intention labeling in real time, accurately and efficiently without adding additional burden to the annotator. In addition, based on the preset finite state machine, the same input can express different intentions, which can further realize more refined parameter labeling and provide more references for decision algorithm design.

[0108] The method provided in the embodiment of the present application is an online annotation method for designing lane change and lane borrowing decisions of vehicles. In order to reduce the annotation burden of the annotators, the universal turn signal lever and headlight lever on the vehicle are used as HMI input devices to express the decision intention of the annotators, and the mapping between input and intention conforms to the driving habits of human drivers.

[0109] The following is an embodiment of the device of the present application. For details not described in detail in the embodiment of the device, reference can be made to the corresponding records in the above method embodiment, and they will not be repeated herein.

[0110] Figure 5 The figure shows a schematic structural diagram of a generation device for autonomous driving training samples provided by an exemplary embodiment of the present application. This device can be implemented as all or part of an autonomous driving device through software, hardware, or a combination of both. The device includes:

[0111] A driving module 401, configured to, in the autonomous driving mode, automatically drive the autonomous driving device based on an autonomous driving algorithm and an autonomous driving logic;

[0112] An interaction module 404, configured to, during the automatic driving process, collect operation instructions input through a human-machine interface on the autonomous driving device, where the operation instructions are used to label the control intention of a human for the autonomous driving device during the automatic driving process;

[0113] A generation module 405, configured to use the driving data of the automatic driving process as a sample and the control intention corresponding to the operation instructions as a label to generate a training sample, where the training sample is used to optimize the autonomous driving logic of the autonomous driving algorithm according to the control intention.

[0114] In an optional embodiment, a labeling module 402 is configured to determine the control intention based on the driving state of the autonomous driving device and the operation instructions when the operation instructions are received;

[0115] The generation module 405 is configured to collect the driving data of the automatic driving process;

[0116] The generation module 405 is configured to generate a training sample, where the training sample includes the driving data and the control intention.

[0117] In an optional embodiment, the labeling module 402 is configured to, in response to receiving a lane change operation in the lane keeping state, determine the control intention as a lane change.

[0118] In an optional embodiment, the labeling module 402 is configured to, in response to receiving a left turn operation in the lane keeping state, determine the control intention as a left lane change;

[0119] The labeling module 402 is configured to, in response to receiving a right turn operation in the lane keeping state, determine the control intention as a right lane change.

[0120] In an optional embodiment, the labeling module 402 is configured to determine the moment when the lane change operation is received as the lane change initiation time.

[0121] In an alternative embodiment, the annotation module 402 is configured to determine that the control intention is to cancel the lane change in response to receiving a reverse lane change operation in the lane change state;

[0122] Wherein, the lane change state is used to change the driving lane of the autonomous driving device.

[0123] In an alternative embodiment, the annotation module 402 is configured to determine that the control intention is to cancel the lane change in response to receiving a right turn operation in the left lane change state;

[0124] The annotation module 402 is configured to determine that the control intention is to cancel the lane change in response to receiving a left turn operation in the right lane change state.

[0125] In an alternative embodiment, the annotation module 402 is configured to determine the moment when the reverse lane change operation is received as the lane change cancellation timing.

[0126] In an alternative embodiment, the annotation module 402 is configured to determine that the control intention is to change the lane change duration in response to receiving a duration control operation in the lane change state;

[0127] Wherein, the lane change state is used to change the driving lane of the autonomous driving device.

[0128] In an alternative embodiment, the annotation module 402 is configured to determine that the control intention is to reduce the lane change duration in response to receiving an upward operation of the headlight lever in the left lane change state or the right lane change state;

[0129] The annotation module 402 is configured to determine that the control intention is to increase the lane change duration in response to receiving a downward operation of the headlight lever in the left lane change state or the right lane change state.

[0130] In an alternative embodiment, the annotation module 402 is configured to determine that the control intention is to borrow a lane in response to receiving a lane borrowing operation in the lane keeping state.

[0131] In an alternative embodiment, the annotation module 402 is configured to determine that the control intention is to borrow a lane to the left in response to receiving an upward operation of the headlight lever in the lane keeping state;

[0132] The annotation module 402 is configured to determine that the control intention is to borrow a lane to the right in response to receiving a downward operation of the headlight lever in the lane keeping state.

[0133] In an alternative embodiment, the annotation module 402 is configured to determine the moment when the lane borrowing operation is received as the lane borrowing initiation timing.

[0134] In an alternative embodiment, the annotation module 402 is configured to determine that the control intention is a lane-changing distance adjustment in response to receiving a distance adjustment operation in the lane-changing state;

[0135] wherein the lane-changing state is used to control the autonomous driving device to perform lateral avoidance.

[0136] In an alternative embodiment, the annotation module 402 is configured to determine that the control intention is a decrease in the lateral distance in response to receiving a right-turning operation in the left-lane-changing state;

[0137] The annotation module 402 is configured to determine that the control intention is an increase in the lateral distance in response to receiving a left-turning operation in the left-lane-changing state;

[0138] The annotation module 402 is configured to determine that the control intention is a decrease in the lateral distance in response to receiving the left-turning operation in the right-lane-changing state;

[0139] The annotation module 402 is configured to determine that the control intention is an increase in the lateral distance in response to receiving the right-turning operation in the right-lane-changing state.

[0140] In an alternative embodiment, the annotation module 402 is configured to determine that the control intention is the completion of lane-changing in response to receiving a lane-changing completion operation in the lane-changing state;

[0141] wherein the lane-changing state is used to control the autonomous driving device to perform lateral avoidance.

[0142] In an alternative embodiment, the annotation module 402 is configured to determine that the control intention is the completion of lane-changing in response to receiving a downward operation of the headlight lever in the left-lane-changing state;

[0143] The annotation module 402 is configured to determine that the control intention is the completion of lane-changing in response to receiving an upward operation of the headlight lever in the right-lane-changing state.

[0144] In an alternative embodiment, the annotation module 402 is configured to determine the moment when the lane-changing completion operation is received as the lane-changing completion timing.

[0145] In an alternative embodiment, the human-machine interface includes at least one of a turn signal lever and a headlight lever.

[0146] In an alternative embodiment, it is characterized in that

[0147] The annotation module 402 is configured to determine the control intention based on the driving state and the operation instruction of the autonomous driving device;

[0148] The annotation module 402 is configured to collect the control intention, the driving data of the driving state, and the operation data of the operation instruction, and generate a training sample.

[0149] In an alternative embodiment, the device further includes:

[0150] A training module 403 is configured to train the autonomous driving algorithm using the collected training samples.

[0151] In an alternative embodiment, the autonomous driving device includes an autonomous driving processor and a general processor;

[0152] The annotation module 402 is configured to obtain the driving state of the autonomous driving device from the general processor through the autonomous driving processor.

[0153] In an alternative embodiment, the autonomous driving device includes an autonomous driving processor and a general processor;

[0154] An interaction module 404 is configured to receive an operation instruction input through a human-machine interface on the autonomous driving device through the general processor;

[0155] The interaction module 404 is configured to obtain the operation instruction from the general processor through the autonomous driving processor.

[0156] In an alternative embodiment, the autonomous driving device includes an autonomous driving processor; the step of annotating the control intention based on the driving state and the operation instruction of the autonomous driving device is executed by the autonomous driving processor.

[0157] In an alternative embodiment, the device further includes:

[0158] A sending module is configured to send the training sample to a computer, and the computer is configured to train the autonomous driving algorithm based on the training sample.

[0159] Please refer to Figure 6 , which shows a structural block diagram of an autonomous driving device 1300 provided by an exemplary embodiment of the present application.

[0160] Generally, the autonomous driving device 1300 includes a processor 1301 and a memory 1302.

[0161] The processor 1301 may include one or more processing cores, such as a quad-core processor, an octa-core processor, etc. The processor 1301 may be implemented in at least one hardware form of DSP (Digital Signal Processing), FPGA (Field-Programmable Gate Array), or PLA (Programmable Logic Array). The processor 1301 may also include a main processor and a coprocessor. The main processor is a processor for processing data in the wake state, also known as the CPU (Central Processing Unit); the coprocessor is a low-power processor for processing data in the standby state. In some embodiments, the processor 1301 may be integrated with a GPU (Graphics Processing Unit), and the GPU is responsible for rendering and drawing the content to be displayed on the display screen. In some embodiments, the processor 1301 may further include an AI (Artificial Intelligence) processor, and the AI processor is used to process computational operations related to machine learning.

[0162] The memory 1302 may include one or more computer-readable storage media, and the computer-readable storage media may be tangible and non-transitory. The memory 1302 may further include high-speed random access memory and non-volatile memory, such as one or more disk storage devices and flash storage devices. In some embodiments, the non-transitory computer-readable storage media in the memory 1302 is used to store at least one instruction, and the at least one instruction is to be executed by the processor 1301 to implement the method for generating an autonomous driving training sample provided in this application.

[0163] Those skilled in the art can understand that Figure 6 the structure shown in

[0164] does not constitute a limitation on the autonomous driving device 1300, and may include more or fewer components than shown in the figure, or combine certain components, or adopt a different component layout.

[0165] The present application also provides a computer device, which includes a processor and a memory. At least one instruction, at least one program, a code set or an instruction set is stored in the storage medium, and the at least one instruction, at least one program, the code set or the instruction set is loaded and executed by the processor to implement the method for generating an autonomous driving training sample provided in each of the above method embodiments.

[0166] The present application also provides a computer-readable storage medium, in which at least one instruction, at least one program, a code set or an instruction set is stored, and the at least one instruction, at least one program, the code set or the instruction set is loaded and executed by the processor to implement the method for generating an autonomous driving training sample provided in each of the above method embodiments.

[0167] The present application also provides a computer program product or a computer program, which includes computer instructions, and the computer instructions are stored in a computer-readable storage medium. The processor of the computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the method for generating an autonomous driving training sample provided in the above optional implementation manners.

[0168] It should be understood that "a plurality of" mentioned herein refers to two or more. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B may represent: A exists alone, A and B exist simultaneously, and B exists alone. The character " / " generally represents an "or" relationship between the associated objects before and after.

[0169] Those of ordinary skill in the art can understand that all or part of the steps for implementing the above embodiments can be completed by hardware, or can be completed by a program instructing relevant hardware. The program mentioned above can be stored in a computer-readable storage medium, and the storage medium mentioned above can be a read-only memory, a magnetic disk or an optical disc, etc.

[0170] The above are only optional embodiments of the present application, and are not intended to limit the present application. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A method for generating an autonomous driving training sample, characterized in that, The method is executed by an autonomous driving device, and the method includes: In the autonomous driving mode, the autonomous driving algorithm controls the autonomous driving device to drive automatically based on the autonomous driving logic; during the automatic driving process, operation instructions input through the human-machine interface on the autonomous driving device are collected, and the operation instructions are used to label the control intention of the human for the autonomous driving device during the automatic driving process; Using the driving data of the automatic driving process as samples and the control intention corresponding to the operation instructions as labels, training samples are generated, and the training samples are used to optimize the autonomous driving logic of the autonomous driving algorithm according to the control intention; The step of using the driving data of the automatic driving process as samples and the control intention corresponding to the operation instructions as labels to generate training samples includes: Based on the driving state of the autonomous driving device when the operation instruction is received and the operation instruction, determining the control intention; Collecting the driving data of the automatic driving process; Generating training samples, where the training samples include the driving data and the control intention, the operation instructions are invalid control instructions in the autonomous driving mode, and the autonomous driving device does not respond to operation inputs from humans in the autonomous driving mode.

2. The method according to claim 1, characterized in that, The step of determining the control intention based on the driving state of the autonomous driving device when the operation instruction is received and the operation instruction includes: In response to receiving a lane change operation in the lane keeping state, determining the control intention as lane change.

3. The method according to claim 2, wherein The method further includes: Determining the moment when the lane change operation is received as the lane change initiation timing.

4. The method according to claim 1, wherein The step of determining the control intention based on the driving state of the autonomous driving device when the operation instruction is received and the operation instruction includes: In response to receiving a reverse lane change operation in the lane change state, determining the control intention as lane change cancellation; wherein, the lane change state is used to change the driving lane of the autonomous driving device.

5. The method according to claim 4, wherein The method further includes: Determining the moment when the reverse lane change operation is received as the lane change cancellation timing.

6. The method according to claim 2, wherein The step of determining the control intention based on the driving state of the autonomous driving device when the operation instruction is received and the operation instruction includes: In response to receiving a duration control operation in the lane change state, determining the control intention as changing the lane change duration; wherein, the lane change state is used to change the driving lane of the autonomous driving device.

7. The method according to claim 2, characterized in that, The step of determining the control intention based on the driving state of the autonomous driving device when the operation instruction is received and the operation instruction includes: In response to receiving a lane borrowing operation in the lane keeping state, determining the control intention as lane borrowing.

8. The method according to claim 7, wherein The method further includes: Determining the moment when the lane borrowing operation is received as the lane borrowing initiation timing.

9. The method according to claim 1, wherein The step of determining the control intention based on the driving state of the autonomous driving device when the operation instruction is received and the operation instruction includes: In response to receiving a distance adjustment operation in the lane borrowing state, determining the control intention as lane borrowing distance adjustment; wherein, the lane borrowing state is used to control the autonomous driving device to perform lateral avoidance.

10. The method according to claim 1, characterized in that, Determining the control intention based on the driving state of the autonomous driving device and the operation instruction when the operation instruction is received, includes: In response to receiving a lane-changing completion operation in the lane-changing state, determining that the control intention is lane-changing completion; Wherein, the lane-changing state is used to control the autonomous driving device to perform lateral avoidance.

11. According to the method described in any one of claims 1 to 10, characterized in that, The method further includes: Sending the training sample to a computer, where the computer is used to optimize the autonomous driving logic of the autonomous driving algorithm according to the training sample.

12. A computer device, characterized in that, The computer includes: a processor and a memory, and at least one instruction, at least one program, a code set or an instruction set is stored in the memory, and the at least one instruction, the at least one program, the code set or the instruction set is loaded and executed by the processor to implement the method for generating an autonomous driving training sample according to any one of claims 1 to 10.

13. A computer-readable storage medium, characterized in that, At least one instruction, at least one program, a code set or an instruction set is stored in the storage medium, and the at least one instruction, the at least one program, the code set or the instruction set is loaded and executed by a processor to implement the method for generating an autonomous driving training sample according to any one of claims 1 to 10.

Images