High-precision map human-in-the-loop method and system for autonomous driving decision control

Abstract

The present invention discloses a high-precision map human-in-the-loop method and system for autonomous driving decision control, the method comprising: obtaining a high-precision map; based on the obtained high-precision map, providing human-in-the-loop support for one or more stages in the vehicle intelligent body reinforcement learning cycle; wherein the human-in-the-loop intervention comprises: expanding the perceived content through the high-precision map perceived content information of the observable state stage in the reinforcement learning cycle, interfering with the action stage in the reinforcement learning cycle through the high-precision map path planning information, and / or shaping the reward function through the high-precision map rule constraints of the reward stage in the reinforcement learning cycle. By embedding the high-precision map features obtained from the high-precision map information into the reinforcement learning training cycle, the present invention can maximize the auxiliary role of the human-in-the-loop method and achieve the improvement of the autonomous driving decision control performance.

Description

Technical Field

[0001] The present invention relates to the field of autonomous driving reinforcement learning, and in particular to a high-precision map human-in-the-loop method and system for autonomous driving decision control. Background Art

[0002] In the field of autonomous driving, reinforcement learning (RL), as an important machine learning method, has important application value. It can be combined with the increasingly improved simulation environment to solve decision-making problems in complex environments through end-to-end learning, breaking through the limitations of traditional methods. Existing reinforcement learning performs well in autonomous driving decision-making and control, but is still limited by problems such as perception field, cold start, local optimal traps, and sparse reward functions. Summary of the invention

[0003] In order to overcome the deficiencies of the above-mentioned prior art, the present invention provides a high-precision map human-in-the-loop method and system for autonomous driving decision control. By embedding high-precision map features obtained from high-precision map information into a reinforcement learning training loop, the auxiliary role of the human-in-the-loop method can be maximized to achieve improved autonomous driving decision control performance.

[0004] According to one aspect of the present invention, a high-precision map human-in-the-loop method for autonomous driving decision control is provided, comprising:

[0005] Get high-precision maps;

[0006] Based on the acquired high-precision map, human-in-the-loop support is provided for one or more stages of the vehicle agent reinforcement learning cycle;

[0007] Among them, the human-in-the-loop intervention includes: expanding the perception content of the observable state stage in the reinforcement learning cycle through the perception content information of the high-precision map, interfering with the action stage in the reinforcement learning cycle through the path planning information of the high-precision map, and / or shaping the reward function of the reward stage in the reinforcement learning cycle through the rule constraints of the high-precision map.

[0008] As a further technical solution, the observable state stage in the reinforcement learning cycle is expanded through the perception content information of the high-precision map, including:

[0009] Obtain high-precision map perception content information including image information and attribute information;

[0010] Superimposing the image information with the vehicle's self-carried forward image, and obtaining a forward fusion feature through training based on the superposition result;

[0011] Obtaining attribute features through training based on the attribute information;

[0012] The attribute features are concatenated with the previous fusion features to achieve feature-level fusion.

[0013] As a further technical solution, high-precision map perception content information including image information and attribute information is obtained, including:

[0014] Matching is performed on the high-precision map according to the vehicle's driving position, and a top-view semantic segmentation image represented by a grid is captured in real time with the vehicle position as the center in the high-precision map. The top-view semantic segmentation image forms a time-series image according to the change of the vehicle position;

[0015] Obtain attribute information based on the high-precision map platform.

[0016] As a further technical solution, the image information is superimposed with the vehicle's self-carried forward image, and a front fusion feature is obtained by training based on the superposition result, including:

[0017] Acquire a forward-looking time-sequence image of the vehicle, and crop the forward-looking time-sequence image and the overhead semantic segmentation image into images of the same size;

[0018] The cropped forward-looking time-series image and the downward-looking semantic segmentation image are superimposed on the channel to obtain the forward-fused image. Noise is added to the superimposed forward-fused image and then input into the reinforcement learning neural network to obtain the forward-fused features.

[0019] As a further technical solution, action intervention is performed on the action stage in the reinforcement learning cycle through high-precision map path planning information, including:

[0020] Control the vehicle's trajectory based on high-precision map road planning information;

[0021] Assign probabilities to the high-precision map-guided actions and the decision-making actions output by the reinforcement learning neural network, and select the actions actually performed by the agent in the current state based on the probabilities;

[0022] As the learning progresses, the probability weight of the high-precision map-guided action is continuously reduced, amplifying the influence of the decision-making action output by the agent's neural network.

[0023] As a further technical solution, the high-precision map guidance action is obtained based on the high-precision map guidance line and the deceleration zone, including:

[0024] Extract vehicle guide lines from high-precision maps, and obtain vehicle guidance actions based on the high-precision map guide lines in combination with the autonomous driving control algorithm;

[0025] When the vehicle is in the deceleration zone, braking operation is added until the vehicle speed meets the speed limit of the zone to obtain vehicle deceleration action.

[0026] As a further technical solution, the reward function of the reward stage in the reinforcement learning cycle is shaped by high-precision map rule constraints, including:

[0027] Obtain high-precision map rule constraint information;

[0028] For each rule constraint information, a reward function is set according to traffic rules, and the rule feature information is obtained and updated in real time from the high-precision map platform, returning the reward value under the current state.

[0029] According to one aspect of the present invention, a high-precision map human-in-the-loop system for autonomous driving decision control is provided, comprising:

[0030] Acquisition module, used to obtain high-precision maps;

[0031] An execution module, for providing human-in-the-loop support for one or more stages in the vehicle agent reinforcement learning cycle based on the acquired high-precision map;

[0032] Among them, the human-in-the-loop intervention includes: expanding the perception content of the observable state stage in the reinforcement learning cycle through the perception content information of the high-precision map, interfering with the action stage in the reinforcement learning cycle through the path planning information of the high-precision map, and / or shaping the reward function of the reward stage in the reinforcement learning cycle through the rule constraints of the high-precision map.

[0033] According to one aspect of the present invention specification, there is provided an electronic device, comprising: at least one processor, at least one memory and a communication interface; wherein the processor, memory and communication interface communicate with each other; the memory stores program instructions executed by the processor, and the processor calls the program instructions to execute the high-precision map human-in-the-loop method for autonomous driving decision control described in any implementation manner.

[0034] According to one aspect of the present specification, a non-transitory computer-readable storage medium is provided, wherein the non-transitory computer-readable storage medium stores computer instructions, wherein the computer instructions enable the computer to execute the high-precision map human-in-the-loop method for autonomous driving decision control described in any implementation manner.

[0035] Compared with the prior art, the present invention has the following beneficial effects:

[0036] The present invention performs perception content expansion, action interference and reward function shaping operations on the perception content information, path planning information and rule constraint information of high-precision map information in the observable state stage, action stage and reward stage in the reinforcement learning cycle of the vehicle agent, and uses the reward function results to evaluate and verify the effect of autonomous driving decision control, thereby achieving an improvement in the performance of autonomous driving decision control. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, a brief introduction is given below to the drawings used in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0038] Figure 1 This is a human-in-the-loop technology roadmap for high-precision maps according to an embodiment of the present invention.

[0039] Figure 2 This is an example diagram of the expansion of the field of view of high-precision map perception content information according to an embodiment of the present invention.

[0040] Figure 3 This is an example diagram of a high-precision map bird's-eye view semantic segmentation image according to an embodiment of the present invention.

[0041] Figure 4 This is an example diagram of a high-precision map intervention method according to an embodiment of the present invention.

[0042] Figure 5 2 is a schematic diagram of the structure of a high-precision map human-in-the-loop system for autonomous driving decision control according to an embodiment of the present invention.

[0043] Figure 6 FIG. 4 is a schematic diagram of a physical structure of an electronic device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0044] It should be noted that:

[0045] Existing reinforcement learning has performed well in autonomous driving decision-making and control, but it is still limited by problems such as perception field, cold start, local optimal trap and sparse reward function. These problems reflect the lack of the existing traditional autonomous driving reinforcement learning process: a wider range of information perception field, expanding and enriching the observable environment content; the guidance of human experts' prior knowledge, replacing the intelligent agent to make auxiliary action decisions in the early stage of training; a reward function form that is more in line with human cognition and real traffic rules, improving and increasing the information feedback rate given by the reward function during the training process.

[0046] High-Definition Maps (HD Maps) are highly accurate digital maps that contain rich geographic information and topological structures, and are used to assist autonomous vehicles in tasks such as positioning, perception, and path planning. Its content includes not only basic information such as road networks, buildings, and landmarks in traditional maps, but also more detailed geographic features, and is updated regularly to maintain accuracy and practicality. In autonomous driving, HD maps can be used as an effective prior data source to make up for the perception range of intelligent agents. Combining the time-sensitive information in HD maps with vehicle sensor information can effectively expand the scope of observable environments in reinforcement learning, thereby improving the training effect of reinforcement learning. At the same time, as a data and decision-making platform in the field of autonomous driving, HD maps contain a large amount of expert experience and knowledge in the field of transportation. For example, path control based on guide lines and speed limits in deceleration zones can guide and plan vehicle driving training. For example, the HD map platform can evaluate driving behaviors in simulation scenarios in combination with real traffic rules, thereby obtaining a reward function form with high coverage, human cognition, and good interpretability. These high-precision map information make up for the shortcomings of reinforcement learning, such as slow startup, difficulty in convergence, and easy to be trapped in local optimality. This method embeds the high-precision map features obtained from high-precision map information into the reinforcement learning training loop to maximize the auxiliary role of the human-in-the-loop method and achieve improved autonomous driving decision-making and control performance.

[0047] Human in the Loop refers to the process of intervening in the reinforcement learning cycle of vehicle agents in autonomous driving through human evaluation, human demonstration, and human intervention, guiding them to absorb and adopt the prior knowledge of human experts, and improving the learning rate and learning effect. As a product of rule constraints, high-precision maps contain a lot of human expert experience in model construction and use methods. They can serve as experts in the loop to guide the reinforcement learning training of vehicle agents, thus forming a human-in-the-loop solution (HD-map in the Loop) unique to high-precision maps.

[0048] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments in the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention. In addition, the technical features in the various embodiments or single embodiments provided by the present invention are arbitrarily combined with each other to form a technical solution. This combination is not restricted by the sequence of steps and / or the structural composition mode, but must be based on the ability of ordinary technicians in this field to implement. When the combination of technical solutions is contradictory or cannot be implemented, it should be considered that such a combination of technical solutions does not exist and is not within the scope of protection required by the present invention.

[0049] It should be noted that in the description of the present invention, the term "and / or" is used to describe the association relationship of associated objects, indicating that there can be multiple relationships between associated objects. For example, A and / or B can represent three situations: A exists alone, B exists alone, and A and B exist at the same time. A, B and / or C can represent six situations: A exists alone, B exists alone, C exists alone, A and B exist at the same time, A and C exist at the same time, B and C exist at the same time, and ABC exist at the same time. In addition, the character " / " generally indicates that the previous and next associated objects are in an "or" relationship.

[0050] This invention focuses on the observable state, action and reward stages in the vehicle agent reinforcement learning cycle. Through high-precision maps, it performs operations such as expanding the perception range, interfering with guide lines and shaping the traffic rule reward function, maximizing the high-precision map's support for human-in-the-loop reinforcement learning. The overall technical route is as follows: Figure 1 The three human-in-the-loop intervention methods provided by the present invention can be used together or separately. In the embodiment of the present invention, the CARLA platform is used as the simulation environment. In principle, all simulation environments that support high-precision maps and reinforcement learning training belong to the scope of the embodiment of the present invention.

[0051] The embodiment of the present invention provides a field of view expansion method based on high-precision map perception content information for vehicle intelligent body reinforcement learning observable state, including: obtaining high-precision map perception content information including image information and attribute information in the high-precision map; pre-fusing the high-precision map image and the vehicle self-carrying forward image in the neural network by superimposing them, and adding them as input to the neural network training; splicing the attribute information after training with the features learned from the superimposed image to achieve feature-level fusion and add them to the neural network intermediate layer training. The technical route of the field of view expansion method based on high-precision map perception content information is as follows: Figure 2 shown. Figure 2The technical flow chart of the field of view expansion method is shown by taking the DQN algorithm as an example, but the embodiments of the present invention are not limited to the DQN algorithm in principle, and all reinforcement learning algorithms that support human-in-the-loop embedding belong to the scope of implementation cases of the present invention.

[0052] Based on the content of the above method embodiment, the embodiment of the present invention provides a field of view expansion method based on high-precision map perception content information for vehicle intelligent body reinforcement learning observable state, wherein the high-precision map perception content information containing image information and attribute information is obtained in the high-precision map, including: matching the high-precision map according to the vehicle driving position, and capturing the top-view semantic segmentation image represented by a grid in real time with the vehicle position as the center in the high-precision map, and the top-view semantic segmentation image forms a time series image according to the change of the vehicle position, such as Figure 3 As shown. The high-precision map platform directly obtains the attributes such as the vehicle speed, vehicle position, other vehicle speed, other vehicle position, road section speed limit, road section direction and lane center distance, among which the dynamic time series information such as the vehicle speed, vehicle position, other vehicle speed and other vehicle position is the projection element of dynamic data on the high-precision map. The high-precision map attribute information in the embodiment of the present invention can be modified or increased or decreased in combination with different high-precision map models.

[0053] Based on the content of the above method embodiment, an embodiment of the present invention provides a field of view expansion method based on high-precision map perception content information for observable states of vehicle intelligent body reinforcement learning, wherein the high-precision map image and the vehicle's self-carried forward image are front-fused by superimposition in the neural network and added to the training, including: obtaining a forward-view time-series image according to the front camera of the vehicle, cropping the forward-view time-series image and the overhead semantic segmentation image to the same size, converting them into Tensors under the PyTorch framework, and then performing channel superposition operations to obtain a front-fused image, adding noise to the superimposed front-fused image to enhance the robustness of the neural network; and inputting the front-fused image into the reinforcement learning neural network to obtain the learning features of the front-fused image.

[0054] On the basis of the content of the above method embodiment, the embodiment of the present invention provides a means of expanding the field of view based on high-precision map perception content information for the observable state of vehicle intelligent body reinforcement learning, wherein the attribute information is spliced ​​with the features learned from the superimposed image after training to achieve feature-level fusion, including: converting the attributes such as the vehicle speed, road section speed limit, road section direction and distance from the lane center into Tensor under the PyTorch framework and then inputting the learning features of the attribute information into the neural network to obtain the learning features of the attribute information, splicing the learning features of the attribute information with the learning features of the previous fusion image and then continuing to input the neural network for learning to achieve feature-level fusion. The high-precision map perception content attribute information that is subjected to feature-level fusion in the implementation case of the present invention is not fixed. In principle, most of the non-image information obtained in the high-precision map can be trained for feature-level fusion, thereby reducing the number of parameters of the neural network as much as possible while providing information.

[0055] In view of the action interference through high-precision map guide lines and deceleration zones in the action stage of the reinforcement learning cycle, an embodiment of the present invention provides a map intervention method for the reinforcement learning action execution stage of a vehicle agent, including: controlling the vehicle action trajectory on a simulation platform relying on path planning information such as high-precision map guide lines and deceleration zone elements; assigning probabilities to the high-precision map guiding actions and the decision actions output by the reinforcement learning neural network, and selecting the actions actually performed by the agent in the current state based on the probabilities; and continuously reducing the probability weights of the guide line control actions and the deceleration actions as the learning progresses, thereby amplifying the influence of the decision actions output by the agent neural network.

[0056] Based on the content of the above method embodiment, the embodiment of the present invention provides a map intervention method for the execution stage of reinforcement learning action of vehicle intelligent body, which relies on high-precision map guide lines to control vehicle actions in the simulation platform, including: extracting vehicle guide lines in the high-precision map and importing them into the simulation platform, and realizing vehicle guide line tracking through the simulation platform automatic driving control algorithm. When the vehicle is in the deceleration zone, the control module adds a braking operation until the vehicle speed meets the speed limit of the interval, and the operation can be output as a vehicle deceleration action. In the embodiment, the vehicle guidance action and the deceleration action can be combined into a high-precision map guidance action.

[0057] Based on the content of the above method embodiments, an embodiment of the present invention provides a map intervention method for the reinforcement learning action execution stage of a vehicle intelligent body, wherein the high-precision map guided action and the decision action output by the reinforcement learning neural network are respectively assigned probabilities, and the action actually performed by the intelligent body in the current state is selected according to the probability, including: for the current state, the vehicle intelligent body can obtain the high-precision map guided action and the decision action output by the intelligent body neural network, assign probabilities to them, and sample and execute the corresponding action according to the probability whenever an action decision is selected.

[0058] Based on the content of the above method embodiment, the embodiment of the present invention provides a map intervention method for the execution stage of the vehicle intelligent body reinforcement learning action, wherein the probability weight of the high-precision map guided action is continuously reduced as the learning progresses, and the influence of the intelligent body neural network output decision action is amplified, including: setting a larger initial probability for the high-precision map guided action, and setting a smaller initial probability for the reinforcement learning neural network output decision action. Set a discount factor γ, and multiply the initial probability of the high-precision map guided action by the discount factor in each round of training, so as to achieve the dependence on high-precision map guidance in the early stage of training, and gradually increase the influence of the intelligent body neural network's self-judgment in the later stage of training.

[0059] An embodiment of the present invention provides a means for shaping a reward function based on map elements in the reward feedback stage of reinforcement learning for a vehicle agent, including: obtaining high-precision map rule constraint information; setting a reward function for each rule constraint information and obtaining and updating feature information in real time from a high-precision map platform and returning the reward value in the current state.

[0060] Based on the content of the above method embodiment, the embodiment of the present invention provides a means of shaping a reward function based on map elements for the reward feedback stage of reinforcement learning of vehicle agents, wherein the acquisition of high-precision map rule constraint information includes: acquiring static guide line elements, lane geometry, lane direction, lane speed limit, deceleration area location, and variable lane attributes and other high-precision map rule constraint information from the high-precision map platform, using a certain distance of the vehicle agent as the perception range, filtering out redundant global elements outside the perception range and retaining local feature information. In principle, the perception range should be larger than the field of view of the vehicle, that is, it includes all feature elements involved in the calculation of the reward function within the field of view of the vehicle. In the implementation case of the present invention, the number of feature elements in the local perception range is not fixed.

[0061] Based on the content of the above method embodiment, the embodiment of the present invention provides a means of shaping a reward function based on map elements for the reward feedback stage of vehicle agent reinforcement learning, wherein the reward function is set for each high-precision map rule constraint information and feature information is obtained and updated in real time from the high-precision map platform and the reward value under the current state is returned, including: setting different reward functions for different high-precision map rule constraint elements, thereby reflecting the current state evaluation in different aspects, to guide the training of reinforcement learning agents. The details of this step are:

[0062] The acquired high-precision map features include: guide line elements, lane line elements, lane direction, lane speed limit, obstacle elements and lane change information. 1) According to the position of the guide line elements, the shortest distance between the vehicle agent and the guide line is calculated after each step of the action, and its reciprocal is multiplied by the weight parameter as the guide line distance reward. The closer the step is to the guide line, the higher the single-step reward value is. 2) According to the geometric shape of the lane line elements, after each step of the agent, it is calculated whether the vehicle agent conflicts with the lane line. If the lane change information is prohibited when a conflict occurs, the reward of the predetermined value is subtracted as a penalty for crossing the lane. 3) According to the lane direction, the angle between the direction of the vehicle and the lane direction is determined. If it is an acute angle, it means that they are in the same direction, and a fixed reward value is added as a same-direction reward. 4) According to the ratio of the vehicle speed to the lane speed limit, it is determined whether the current vehicle is speeding. When it is less than 1, it is multiplied by the weight parameter as a speed reward. When it is greater than 1, it is multiplied by a larger weight parameter as a speeding penalty, aiming to encourage the vehicle agent to drive as fast as possible without exceeding the speed limit. 5) According to the geometric shape of the obstacle element, after each step of the intelligent agent's action, calculate whether the vehicle intelligent agent collides with the obstacle element. If a conflict occurs, subtract a predetermined reward as a collision penalty. Based on the content of the above method embodiment, the reward function set based on the high-precision map rule constraint information provided in the embodiment of the present invention can be added as the high-precision map perception content information to the observable state stage for human-in-the-loop training for perception content expansion.

[0063] The implementation basis of each embodiment of the present invention is to implement programmed processing through a device with a processor function. Therefore, in engineering practice, the technical solutions and functions of each embodiment of the present invention are encapsulated into various modules. Based on this reality, on the basis of the above embodiments, the embodiments of the present invention provide a high-precision map human-in-the-loop system for autonomous driving decision control, which is used to execute the high-precision map human-in-the-loop method for autonomous driving decision control in the above method embodiment. Figure 5 The device includes: an acquisition module for acquiring a high-precision map; an execution module for providing human-in-the-loop support for one or more stages in a reinforcement learning cycle of a vehicle agent according to the acquired high-precision map; wherein the human-in-the-loop intervention includes: expanding the perceived content of the observable state stage in the reinforcement learning cycle through the perceived content information of the high-precision map, intervening in the action stage in the reinforcement learning cycle through the path planning information of the high-precision map, and / or shaping the reward function of the reward stage in the reinforcement learning cycle through the rule constraints of the high-precision map.

[0064] The high-precision map human-in-the-loop system for autonomous driving decision control provided by the embodiment of the present invention adopts Figure 5Several modules in the reinforcement learning loop perform perception content expansion, action interference and reward function shaping operations on the perception content information, path planning information and rule constraint information of high-precision map information in the observable state stage, action stage and reward stage respectively. The reward function result can be used to evaluate and verify the effect of autonomous driving decision control, thereby improving the performance of autonomous driving decision control.

[0065] It should be noted that the system embodiment provided by the present invention is used to implement the method in the above method embodiment as well as the method in other method embodiments provided by the present invention. The only difference is that the corresponding functional modules are set. The principle is basically the same as the principle of the above system embodiment provided by the present invention. As long as the technical personnel in this field refer to the specific technical solutions in other method embodiments on the basis of the above system embodiment, obtain the corresponding technical means and the technical solutions composed of these technical means by combining the technical features, and on the premise of ensuring the practicality of the technical solutions, they can improve the system in the above system embodiment to obtain the corresponding system class embodiment, which is used to implement the methods in other method class embodiments. For example:

[0066] Based on the content of the above system embodiment, as an optional embodiment, the high-precision map human-in-the-loop system for autonomous driving decision control provided in the embodiment of the present invention further includes:

[0067] The first execution submodule is used to expand the perceived content of the observable state stage in the reinforcement learning cycle through the high-precision map perceived content information, and executes the following instructions:

[0068] Obtain high-precision map perception content information including image information and attribute information;

[0069] Superimposing the image information with the vehicle's self-carried forward image, and obtaining a forward fusion feature through training based on the superposition result;

[0070] Obtaining attribute features through training based on the attribute information;

[0071] The attribute features are concatenated with the previous fusion features to achieve feature-level fusion.

[0072] Furthermore, when the first execution submodule acquires the high-precision map perception content information including image information and attribute information, it includes:

[0073] Matching is performed on the high-precision map according to the vehicle's driving position, and a top-view semantic segmentation image represented by a grid is captured in real time with the vehicle position as the center in the high-precision map. The top-view semantic segmentation image forms a time-series image according to the change of the vehicle position;

[0074] Obtain attribute information based on the high-precision map platform.

[0075] Furthermore, when the first execution submodule superimposes the image information with the vehicle's self-carried forward image and obtains the front fusion feature through training based on the superposition result, it includes:

[0076] Acquire a forward-looking time-sequence image of the vehicle, and crop the forward-looking time-sequence image and the overhead semantic segmentation image into images of the same size;

[0077] The cropped forward-looking time-series image and the downward-looking semantic segmentation image are superimposed on the channel to obtain the forward-fused image. Noise is added to the superimposed forward-fused image and then input into the reinforcement learning neural network to obtain the forward-fused features.

[0078] Based on the content of the above system embodiment, as an optional embodiment, the high-precision map human-in-the-loop system for autonomous driving decision control provided in the embodiment of the present invention further includes:

[0079] The second execution submodule is used to perform action intervention on the action phase of the reinforcement learning cycle through the high-precision map path planning information, including:

[0080] Control the vehicle's trajectory based on high-precision map road planning information;

[0081] Assign probabilities to the high-precision map-guided actions and the decision-making actions output by the reinforcement learning neural network, and select the actions actually performed by the agent in the current state based on the probabilities;

[0082] As the learning progresses, the probability weight of the high-precision map-guided action is continuously reduced, amplifying the influence of the decision-making action output by the agent's neural network.

[0083] Furthermore, when the second execution submodule obtains the high-precision map guidance action based on the high-precision map guide line and the deceleration zone, it includes:

[0084] Extract vehicle guide lines from high-precision maps, and obtain vehicle guidance actions based on the high-precision map guide lines in combination with the autonomous driving control algorithm;

[0085] When the vehicle is in the deceleration zone, braking operation is added until the vehicle speed meets the speed limit of the zone to obtain vehicle deceleration action.

[0086] Based on the content of the above system embodiment, as an optional embodiment, the high-precision map human-in-the-loop system for autonomous driving decision control provided in the embodiment of the present invention further includes:

[0087] The third execution submodule is used to shape the reward function of the reward stage in the reinforcement learning cycle through high-precision map rule constraints, including:

[0088] Obtain high-precision map rule constraint information;

[0089] For each rule constraint information, a reward function is set according to traffic rules, and the rule feature information is obtained and updated in real time from the high-precision map platform, returning the reward value under the current state.

[0090] The method of the embodiment of the present invention is implemented by relying on electronic devices, so it is necessary to introduce the relevant electronic devices. Based on this purpose, the embodiment of the present invention provides an electronic device, such as Figure 6 As shown, the electronic device includes: at least one processor, a communication interface, at least one memory, and a communication bus, wherein at least one processor, the communication interface, and at least one memory communicate with each other through the communication bus. At least one processor calls the logic instructions in at least one memory to execute all or part of the steps of the method provided by the aforementioned various method embodiments.

[0091] In addition, when the logic instructions in the above-mentioned at least one memory are implemented in the form of a software functional unit and sold or used as an independent product, they are stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention is essentially or partly in other words, the part that contributes to the prior art or part of the technical solution is embodied in the form of a software product, and the computer software product is stored in a storage medium, including several instructions for enabling a computer device (a personal computer, a server, or a network device) to perform all or part of the steps of the method described in each method embodiment of the present invention. The aforementioned storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a disk or an optical disk, and various media for storing program codes.

[0092] The system embodiments described above are merely illustrative, wherein the units described as separate components are or are not physically separated, and the components shown as units are or are not physical units, located in one place, or distributed on multiple network units. Some or all of the modules are selected according to actual conditions to achieve the purpose of the solution of this embodiment. Those of ordinary skill in the art can understand and implement without creative effort.

[0093] It will be appreciated by those skilled in the art that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0094] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0095] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0096] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process in the computer or other programmable device. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0097] In summary, the present invention focuses on the field of autonomous driving decision control. The vehicle agent uses high-precision maps to expand the perception content, interfere with guide lines, and shape the reward function based on traffic rules in the three stages of the observable state (statement), action (action), and reward (reward) in the vehicle agent's reinforcement learning cycle, so as to maximize the high-precision map's support for its human-in-the-loop, thereby realizing the improvement of decision-making control means in autonomous driving.

[0098] It should be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "includes..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.

[0099] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the technical solutions of the embodiments of the present invention.

Claims

1. A high-precision map human-in-the-loop method for autonomous driving decision control, characterized by: include: Get high-precision maps; Based on the acquired high-precision map, human-in-the-loop support is provided for one or more stages of the vehicle agent reinforcement learning cycle; Wherein, the human-in-the-loop support includes: expanding the perception content through high-precision map perception content information in the observable state stage of the reinforcement learning cycle, interfering with the action phase of the reinforcement learning cycle through high-precision map path planning information, and / or shaping the reward function through high-precision map rule constraints in the reward phase of the reinforcement learning cycle; The observable state stage in the reinforcement learning cycle is expanded through the high-precision map perception content information, including: obtaining the high-precision map perception content information including image information and attribute information; superimposing the image information and the vehicle's self-carried forward image, and training to obtain the front fusion feature based on the superposition result; training to obtain the attribute feature based on the attribute information; splicing the attribute feature and the front fusion feature to achieve feature-level fusion; Perform action intervention on the action stage of the reinforcement learning cycle through high-precision map path planning information, including: controlling the vehicle's action trajectory according to the high-precision map path planning information; assigning probabilities to the high-precision map guided action and the decision action output by the reinforcement learning neural network, and selecting the action actually performed by the agent in the current state based on the probability; and continuously reducing the probability weight of the high-precision map guided action as the learning progresses, amplifying the influence of the decision action output by the agent's neural network; The reward function of the reward stage in the reinforcement learning cycle is shaped through high-precision map rule constraints, including: obtaining high-precision map rule constraint information; for each rule constraint information, setting a reward function according to traffic rules and obtaining and updating rule feature information from the high-precision map platform in real time, and returning the reward value under the current state.

2. The high-precision map human-in-the-loop method for autonomous driving decision control according to claim 1, characterized in that: Obtain high-precision map perception content information containing image information and attribute information, including: Matching is performed on the high-precision map according to the vehicle's driving position, and a top-view semantic segmentation image represented by a grid is captured in real time with the vehicle position as the center in the high-precision map. The top-view semantic segmentation image forms a time-series image according to the change of the vehicle position; Obtain attribute information based on the high-precision map platform.

3. The high-precision map human-in-the-loop method for autonomous driving decision control according to claim 2, characterized in that: The image information is superimposed with the vehicle's self-carried forward image, and a front fusion feature is obtained by training based on the superposition result, including: Acquire a forward-looking time-sequence image of the vehicle, and crop the forward-looking time-sequence image and the overhead semantic segmentation image into images of the same size; The cropped forward-looking time-series image and the downward-looking semantic segmentation image are superimposed on the channel to obtain the forward-fused image. Noise is added to the superimposed forward-fused image and then input into the reinforcement learning neural network to obtain the forward-fused features.

4. The high-precision map human-in-the-loop method for autonomous driving decision control according to claim 1, characterized in that: The high-precision map guidance action is obtained based on the high-precision map guidance line and the deceleration zone, including: Extract vehicle guide lines from high-precision maps, and obtain vehicle guidance actions based on the high-precision map guide lines in combination with the autonomous driving control algorithm; When the vehicle is in a deceleration zone, braking operations are added until the vehicle speed meets the speed limit of the zone to obtain vehicle deceleration action.

5. High-precision map human-in-the-loop system for autonomous driving decision control, characterized by: include: Acquisition module, used to obtain high-precision maps; An execution module, for providing human-in-the-loop support for one or more stages of the vehicle agent reinforcement learning cycle based on the acquired high-precision map; Wherein, the human-in-the-loop support includes: expanding the perception content through high-precision map perception content information in the observable state stage of the reinforcement learning cycle, interfering with the action phase of the reinforcement learning cycle through high-precision map path planning information, and / or shaping the reward function through high-precision map rule constraints in the reward phase of the reinforcement learning cycle; The observable state stage in the reinforcement learning cycle is expanded through the high-precision map perception content information, including: obtaining the high-precision map perception content information including image information and attribute information; superimposing the image information and the vehicle's self-carried forward image, and training to obtain the front fusion feature based on the superposition result; training to obtain the attribute feature based on the attribute information; splicing the attribute feature and the front fusion feature to achieve feature-level fusion; Perform action intervention on the action stage of the reinforcement learning cycle through high-precision map path planning information, including: controlling the vehicle's action trajectory according to the high-precision map path planning information; assigning probabilities to the high-precision map guided action and the decision action output by the reinforcement learning neural network, and selecting the action actually performed by the agent in the current state based on the probability; and continuously reducing the probability weight of the high-precision map guided action as the learning progresses, amplifying the influence of the decision action output by the agent's neural network; The reward function of the reward stage in the reinforcement learning cycle is shaped through high-precision map rule constraints, including: obtaining high-precision map rule constraint information; for each rule constraint information, setting a reward function according to traffic rules and obtaining and updating rule feature information from the high-precision map platform in real time, and returning the reward value under the current state.

6. An electronic device, characterized in that: include: At least one processor, at least one memory and a communication interface; wherein the processor, memory and communication interface communicate with each other; The memory stores program instructions executed by the processor, and the processor calls the program instructions to execute the method according to any one of claims 1 to 4.

7. A non-transitory computer-readable storage medium, characterized in that: The non-transitory computer-readable storage medium stores computer instructions, which cause the computer to execute the method of any one of claims 1 to 4.

Images