Autonomous driving method and system based on offline reinforcement learning and teacher-student agent

Abstract

The application relates to the technical field of automatic driving, in particular to an automatic driving method and system based on offline reinforcement learning and teacher-student agents, wherein the method comprises the following steps: obtaining a training data set, wherein the training data set comprises driving data of a real traffic scene; constructing a teacher agent, training the teacher agent by using offline reinforcement learning based on the training data set; constructing a student agent, training the student agent by using imitation learning according to the output of the trained teacher agent and the training data set; and controlling a vehicle to automatically drive based on the trained student agent. Therefore, the problems of insufficient generalization ability, limitations of supervised learning methods, high risks of reinforcement learning in practical application and differences between simulation and reality in the prior art are solved.

Description

Technical Field

[0001] This application relates to the field of autonomous driving technology, and in particular to an autonomous driving method and system based on offline reinforcement learning and teacher-student intelligent agents. Background Technology

[0002] In the field of autonomous driving, planning and control are core technologies, and their challenges cannot be ignored. Traditional rule-based methods, while stable, struggle to cope with all complex and ever-changing scenarios. Supervised learning-based methods, although capable of mimicking expert trajectories to solve driving problems in some complex scenarios, show limited generalization ability when faced with scenarios outside the training set, making it difficult to adapt to the constantly changing traffic environment in the real world.

[0003] Reinforcement learning, as a technology with self-updating capabilities, can theoretically adapt to complex and ever-changing traffic scenarios through interactive learning. However, training reinforcement learning in real-world environments carries extremely high risks. During the exploration and learning process of robots or autonomous vehicles, errors can lead to hardware damage or threaten the safety of surrounding objects and even humans. Therefore, current reinforcement learning strategies often involve training in simulation environments first, followed by gradual deployment to real-world environments. However, due to the data differences between simulation and real-world environments, policies learned in simulation environments often fail to achieve satisfactory performance in actual deployments.

[0004] Offline reinforcement learning can utilize interactive data from human experts in real traffic environments for offline training. Since the training data is collected in advance in the real scene, there is no need to interact with the environment during training. Therefore, the offline reinforcement learning method avoids the dangers of training directly in the real environment online. Moreover, offline reinforcement learning introduces data collected in the real environment into the simulation environment for training, eliminating the need for post-processing due to the gap between simulation and reality, and can eliminate the differences between the data collected in the simulation and the real environment.

[0005] During the training process, information such as road traffic environment is an important basis for decision-making. This kind of privileged information can be explicitly told to the agent in the simulation environment, but the agent cannot directly obtain this kind of information in the real environment. Therefore, training and actual deployment can be completed through a two-level agent system of teacher and student. The teacher agent is trained with privileged information as input, and the student agent learns by imitation with the output of the teacher agent as the supervision signal. Finally, the student agent is deployed in the real vehicle. Summary of the Invention

[0006] This application provides an autonomous driving method, apparatus, device, and medium to address issues such as insufficient generalization ability, limitations of supervised learning methods, high risks of reinforcement learning in practical applications, and differences between simulation and reality in the prior art.

[0007] This application provides an autonomous driving method based on offline reinforcement learning and teacher-student intelligent agents, including the following steps: acquiring driving data of a driver operating a vehicle in a real-world scenario; processing the driving data into a state set, action set, and reward set required for intelligent agent training; and storing this data as a training trajectory. The intelligent agents include a teacher intelligent agent and a student intelligent agent, wherein the state set of the teacher intelligent agent is... The state set of the student agent is: The action set of the teacher agent is A. l,t The reward function is R. l,t Where l is the number of trajectories and t is the time step for each trajectory; the teacher agent is constructed, comprising a teacher policy neural network, a teacher value neural network, and a teacher estimation neural network, wherein the teacher agent is trained using offline reinforcement learning based on the state set, action set, and reward function to obtain the teacher policy neural network, the teacher value neural network, and the teacher estimation neural network, wherein the vehicle control parameters output by the teacher policy network are: The teacher estimates the road privilege information output by the neural network as follows: The student agent is constructed, comprising a student policy neural network and a student estimation neural network. The student agent is trained using imitation learning based on its state set, the output of the teacher policy network, and the output of the teacher estimation neural network to obtain the student policy neural network and the student estimation neural network. The vehicle control parameters output by the student policy neural network are... The student policy neural network and the student estimation neural network are deployed on a real vehicle. The state set of the student agent is acquired on the vehicle, and this state set is input into the student agent. The output of the student policy neural network is used to control the vehicle. The state set of the student agent is...

[0008] Optionally, the state set of the teacher intelligent agent includes navigation commands. Vehicle Information BEV image Privileged information on road traffic environment Right now The state set of the student agent includes navigation commands. Vehicle Information BEV image and historical information of navigation commands and vehicle information Right now Wherein, the common state set of the teacher agent and the student agent is

[0009] The teacher agent's action set includes steering wheel turning. and acceleration Right now in, Left turns are recorded as positive values, right turns as negative values; Acceleration is represented by positive values, and deceleration by negative values.

[0010] The reward function R l,t for:

[0011]

[0012] in, This is the weighting coefficient for action quality. For the quality of the action, For comfort weighting coefficient, For comfort, w energy Energy weighting coefficient, For energy consumption, This is the termination condition.

[0013] Optionally, the motion quality for:

[0014]

[0015] Where Δp, Δv, and Δθ are the position difference, velocity difference, and orientation angle difference between the trajectory formed by the human-driven vehicle and the trajectory planned by the convex optimization method, respectively. The reward value is the position difference. The reward value for the speed difference, The return value is the difference in orientation angle. The weighting coefficient for the position difference. The weighting coefficient for the speed difference. The weighting coefficient for the difference in orientation angle;

[0016] The comfort for:

[0017]

[0018] in, It is the reward value corresponding to the change in steering wheel angle at the previous and next moments. It is the reward value corresponding to the change in acceleration at different times. The weighting coefficients corresponding to changes in steering wheel angle. These are the weighting coefficients corresponding to the changes in acceleration;

[0019] energy consumption for:

[0020]

[0021] in, It is the acceleration value at time t. It is the acceleration value at time t-1. It is the acceleration value at time t-2, and Δt is the corresponding time difference;

[0022] The termination condition for:

[0023]

[0024] Optionally, constructing the teacher agent includes: based on the fitted function The teacher estimation neural network is constructed to construct the teacher agent, wherein the state input of the teacher estimation neural network is... It is privileged information about the road traffic environment. It is privileged information Feature estimation; based on the fitting function Construct the teacher policy neural network for the teacher agent, wherein, These are the vehicle control parameters output by the teacher strategy network; based on the fitting function Construct the teacher value neural network of the teacher agent, wherein, Vehicle control parameters output by the teacher strategy network Evaluation.

[0025] Optionally, the update formulas for the teacher policy neural network and the teacher estimation neural network are:

[0026]

[0027] in, These are the parameters of the teacher strategy neural network. The parameters of the neural network are estimated by the teacher, L represents the total number of training trajectories, and λ1 is the scaling factor.

[0028] The update formula for the teacher value neural network is:

[0029]

[0030] in, These are the parameters of the teacher value neural network, where γ is the proportionality coefficient. This is the output of the teacher value neural network at time t+1. The navigation command at time t+1. This refers to the vehicle information at time t+1. The image of BEV at time t+1. It is a feature estimation of privileged information at time t+1. It is the evaluation of the vehicle control parameters output by the teacher's value neural network at time t. The vehicle control parameters output by the teacher's strategy network at time t+1.

[0031] Optionally, constructing the student agent includes: based on the fitted function The student estimation neural network is constructed to construct the student agent, wherein the state input of the student estimation neural network is... It contains historical information about navigation commands and vehicle information. It is a feature estimation of privileged information; based on the fitting function Construct the student policy neural network for the student agent.

[0032] Optionally, the update formulas for the student policy neural network and the student estimation neural network are:

[0033]

[0034] in, These are the parameters of the student policy neural network. λ2 represents the parameters of the neural network estimated by the student, and λ2 is the scaling factor.

[0035] This application embodiment also provides an autonomous driving system based on offline reinforcement learning and teacher-student intelligent agents, including: a data acquisition system for acquiring driving data of a driver operating a vehicle in a real-world scenario, processing the driving data into a state set, action set, and reward set required for intelligent agent training, and storing it as a training trajectory, wherein the intelligent agents include a teacher intelligent agent and a student intelligent agent, wherein the state set of the teacher intelligent agent is... The state set of the student agent is: The action set of the teacher agent is A. l,t The reward function is R. l,tWhere l is the number of trajectories and t is the time step for each trajectory; A teacher learning system is used to construct the teacher agent, which includes a teacher policy neural network, a teacher value neural network, and a teacher estimation neural network. The teacher agent is trained using offline reinforcement learning based on its state set, action set, and reward function to obtain the teacher policy neural network, the teacher value neural network, and the teacher estimation neural network. A student learning system is used to construct the student agent, which includes a student policy neural network and a student estimation neural network. The student agent is trained using imitation learning based on its state set, the output of the teacher policy network, and the output of the teacher estimation neural network to obtain the student policy neural network and the student estimation neural network. The vehicle control parameters output by the student policy neural network are... A deployment system is used to deploy the student policy neural network and the student estimation neural network onto a real vehicle, acquire the state set of the student agent on the real vehicle, input the state set of the student agent into the student agent, and control the real vehicle using the output of the student policy neural network. The state set of the student agent is as follows:

[0036] This application provides an electronic device, including: a memory, a processor, and a computer program stored in the memory and executable on the processor. The processor executes the program to implement the autonomous driving method based on offline reinforcement learning and teacher-student intelligent agents as described in the above embodiments.

[0037] This application provides a computer-readable storage medium storing a computer program that is executed by a processor to implement the autonomous driving method based on offline reinforcement learning and teacher-student intelligent agents as described in the above embodiments.

[0038] Therefore, this application has at least the following beneficial effects:

[0039] This application's embodiments construct a learning system involving teacher and student intelligent agents. The teacher agent is trained using real traffic data and then guides the student agent in learning and optimization. This significantly improves the robustness, generalization ability, and safety of actual deployment of the autonomous driving system, while eliminating the gap between simulation and real-world scenarios. Thus, it solves technical problems in existing technologies such as insufficient generalization ability, limitations of supervised learning methods, high risks in practical applications of reinforcement learning, and discrepancies between simulation and reality.

[0040] Additional aspects and advantages of this application will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of this application. Attached Figure Description

[0041] The above and / or additional aspects and advantages of this application will become apparent and readily understood from the following description of the embodiments taken in conjunction with the accompanying drawings, wherein:

[0042] Figure 1 The flowchart shows an autonomous driving method based on offline reinforcement learning and teacher-student intelligent agents provided according to some embodiments of this application;

[0043] Figure 2 This is a teacher estimation network structure diagram provided according to some embodiments of this application;

[0044] Figure 3 This is a network structure diagram of a teacher strategy provided according to some embodiments of this application;

[0045] Figure 4 This is a teacher value network structure diagram provided according to some embodiments of this application;

[0046] Figure 5 This is a student estimation network structure diagram provided according to some embodiments of this application;

[0047] Figure 6 This is a student policy network structure diagram provided according to some embodiments of this application;

[0048] Figure 7 A schematic diagram of an autonomous driving method based on offline reinforcement learning and teacher-student intelligent agents provided according to some embodiments of this application;

[0049] Figure 8 This is a schematic diagram of an autonomous driving system based on offline reinforcement learning and teacher-student intelligent agents provided according to some embodiments of this application;

[0050] Figure 9 This is a schematic diagram of the structure of an electronic device according to an embodiment of this application. Detailed Implementation

[0051] The embodiments of this application are described in detail below. Examples of these embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and intended to explain this application, and should not be construed as limiting this application.

[0052] The following description, with reference to the accompanying drawings, describes an autonomous driving system and its embodiments based on offline reinforcement learning and teacher-student intelligent agents. Addressing the issue of data discrepancies between simulation and reality mentioned in the background section, this application provides an autonomous driving system based on offline reinforcement learning and teacher-student intelligent agents. In this method, a learning system for teacher and student intelligent agents is constructed. The teacher intelligent agent is trained using real traffic data, and then guides the student intelligent agent to learn and optimize. This significantly improves the robustness, generalization ability, and safety of actual deployment of the autonomous driving system, while eliminating the gap between simulation and real-world scenarios. Thus, it solves the problems of insufficient generalization ability, limitations of supervised learning methods, high risks in practical applications of reinforcement learning, and discrepancies between simulation and reality in the prior art.

[0053] Specifically, Figure 1 This is a schematic diagram of an autonomous driving process based on offline reinforcement learning and teacher-student intelligent agents, provided as an embodiment of this application.

[0054] like Figure 1 As shown, this autonomous driving method based on offline reinforcement learning and teacher-student intelligent agents includes the following steps:

[0055] In step S101, the driving data of the driver driving the vehicle in a real scene is obtained, the driving data of the vehicle is processed into a state set, action set and reward set required for the training of the intelligent agent, and stored as a training trajectory.

[0056] The intelligent agents include teacher intelligent agents and student intelligent agents; the state set of the teacher intelligent agent is as follows: The state set of the student agent is Let A be the action set of the teacher agent. l,t The reward function is R. l,t , where l is the number of trajectories and t is the time step for each trajectory.

[0057] In this embodiment, the state set of the teacher agent includes navigation commands. Vehicle Information BEV image Privileged information on road traffic environment Right now The state set of the student agent includes navigation commands. Vehicle Information BEV image and historical information of navigation commands and vehicle information Right now

[0058] It should be noted that navigation commands Includes left turns, right turns, going straight, changing lanes to the right, and changing lanes to the left; vehicle information. Includes pose, velocity, acceleration, angular velocity, and angular acceleration; BEV feature map Obtained by the BEVFormer algorithm, it can represent dynamic targets such as vehicles and pedestrians, and static targets such as lane lines and road edges; privileged information of the road traffic environment. This includes real-time lane information, traffic light and sign information; historical information on navigation commands and vehicle data. Take 6 time steps and Numerical value.

[0059] In this embodiment, the teacher agent's action set includes steering wheel turning. and acceleration Right now in, Left turns are recorded as positive values, right turns as negative values; Acceleration is represented by positive values, and deceleration by negative values.

[0060] Reward function R l,t for:

[0061]

[0062] in, This is the weighting coefficient for action quality. For the quality of the action, For comfort weighting coefficient, For comfort, w energy Energy weighting coefficient, For energy consumption, This is the termination condition.

[0063] In the embodiments of this application, motion quality for:

[0064]

[0065]

[0066] Where Δp, Δv, and Δθ are the position difference, velocity difference, and orientation angle difference between the trajectory formed by the human-driven vehicle and the trajectory planned by the convex optimization method, respectively. The reward value is the position difference. The reward value for the speed difference, The return value is the difference in orientation angle. The weighting coefficient for the position difference. The weighting coefficient for the speed difference. The weighting coefficient for the difference in orientation angle;

[0067] Comfort for:

[0068]

[0069] in, It is the reward value corresponding to the change in steering wheel angle at the previous and next moments. It is the reward value corresponding to the change in acceleration at different times. The weighting coefficients corresponding to changes in steering wheel angle. These are the weighting coefficients corresponding to the changes in acceleration;

[0070] Energy consumption for:

[0071]

[0072] in, It is the acceleration value at time t. It is the acceleration value at time t-1. It is the acceleration value at time t-2, and Δt is the corresponding time difference;

[0073] Termination conditions for:

[0074]

[0075] Among them, a penalty of -100 will be imposed for causing a collision or violating traffic rules such as running a red light, and a reward of 100 will be given upon reaching the destination.

[0076] In step S102, a teacher agent is constructed, which includes a teacher policy neural network, a teacher value neural network, and a teacher estimation neural network. The teacher agent is trained using offline reinforcement learning based on its state set, action set, and reward function to obtain the teacher policy neural network, teacher value neural network, and teacher estimation neural network.

[0077] Among them, the vehicle control parameters output by the teacher strategy network are: The teacher estimates the feature estimates of the road information output by the neural network as follows:

[0078] In this embodiment of the application, constructing a teacher intelligent agent includes: based on a fitting function Construct a teacher estimation neural network for the teacher agent, wherein the state input of the teacher estimation neural network is... It is privileged information about the road traffic environment. It is privileged information Feature estimation; based on the fitting function Construct a teacher strategy neural network for the teacher agent, in which... These are the vehicle control parameters output by the teacher strategy network; based on the fitting function Constructing a teacher value neural network for teacher intelligent agents, in which... Vehicle control parameters output by the teacher strategy network Evaluation.

[0079] Specifically, such as Figure 2 As shown, the teacher estimation neural network uses a fully connected network with ReLU activation function, and the input layer contains privileged information about the road traffic environment. The hidden layer has two layers, both of which are ReLU(256), and the output layer is a feature estimate of the privileged information.

[0080] like Figure 3 As shown, the teacher policy neural network uses a fully connected network with ReLU activation function and an input layer of... There are 6 hidden layers, namely ReLU(512), ReLU(256), ReLU(256), ReLU(256), ReLU(128), and ReLU(2). The output layer contains vehicle control parameters.

[0081] like Figure 4 As shown, the teacher value neural network uses a fully connected network with ReLU activation function and an input layer of... The hidden layer has 5 layers, namely ReLU(512), ReLU(256), ReLU(256), ReLU(256), and ReLU(1), and the output layer contains vehicle control parameters. Evaluation

[0082] In this embodiment, the update formulas for the teacher policy neural network and the teacher estimation neural network are as follows:

[0083]

[0084] in, These are the parameters of the teacher strategy neural network. The parameters of the neural network are estimated by the teacher, L represents the total number of training trajectories, and λ1 is the scaling factor.

[0085] The update formula for the teacher value neural network is:

[0086]

[0087] in, These are the parameters of the teacher value neural network, where γ is the proportionality coefficient. This is the output of the teacher value neural network at time t+1. The navigation command at time t+1. This refers to the vehicle information at time t+1. The image of BEV at time t+1. It is a feature estimation of privileged information at time t+1. It is the evaluation of the vehicle control parameters output by the teacher's value neural network at time t. The vehicle control parameters output by the teacher's strategy network at time t+1.

[0088] In step S103, a student agent is constructed, which includes a student policy neural network and a student estimation neural network. The student agent is trained using imitation learning based on the state set of the student agent, the output of the teacher policy network, and the output of the teacher estimation neural network to obtain the student policy neural network and the student estimation neural network.

[0089] Among them, the vehicle control parameters output by the student policy neural network are:

[0090] In this embodiment of the application, constructing a student intelligent agent includes: based on a fitting function Construct a student estimation neural network for the student agent, where the state input of the student estimation neural network is... It contains historical information about navigation commands and vehicle information. It is a feature estimation of privileged information; based on the fitting function Construct a student policy neural network for student agents.

[0091] Specifically, such as Figure 5 As shown, the student estimation neural network uses a fully connected network with ReLU activation function, and the input layer consists of historical information from navigation commands and vehicle data. The hidden layer has two layers, both of which are ReLU(128) and the output layer is a feature estimate of privileged information.

[0092] like Figure 6 As shown, the student policy neural network uses a fully connected network with ReLU activation function and an input layer of... There are 6 hidden layers, namely ReLU(512), ReLU(256), ReLU(256), ReLU(256), ReLU(128), and ReLU(2). The output layer contains vehicle control parameters.

[0093] In this embodiment, the update formulas for the student policy neural network and the student estimation neural network are as follows:

[0094]

[0095] in, These are the parameters of the student policy neural network. λ2 represents the parameters of the neural network estimated by the student, and λ2 is the scaling factor.

[0096] In step S104, the student policy neural network and the student estimation neural network are deployed on the actual vehicle, the state set of the student agent is obtained on the actual vehicle, the state set of the student agent is input into the student agent, and the output of the student policy neural network is used to control the actual vehicle.

[0097] Among them, the state set of the student agent is

[0098] Specifically, this involves deploying student intelligent agents on actual vehicles and using the state sets acquired from those vehicles. Information is input into the student policy neural network and the student estimation neural network, according to the formula. and Output vehicle control parameters The vehicle's autonomous driving control is achieved by using vehicle control parameters.

[0099] The autonomous driving method based on offline reinforcement learning and teacher-student intelligent agents proposed in this application significantly improves the robustness, generalization ability, and safety of actual deployment of the autonomous driving system by constructing a learning system for teacher and student intelligent agents, training the teacher intelligent agent using real traffic data, and then having the teacher guide the student intelligent agent to learn and optimize. This also eliminates the gap between simulation and real-world scenarios. Therefore, it solves problems such as insufficient generalization ability, limitations of supervised learning methods, high risks in practical applications of reinforcement learning, and discrepancies between simulation and reality in existing technologies.

[0100] The following will combine Figure 7 This paper elaborates on an autonomous driving method based on offline reinforcement learning and teacher-student agents, as follows: The driver operates the vehicle and collects data in various real-world environments. This data is stored as training trajectories and provided to teacher and student agents for offline training. Specifically, the teacher agent is trained using offline reinforcement learning to obtain a teacher estimation network and a teacher policy network. The student agent learns by imitating the teacher agent's estimation and policy networks to obtain a student estimation network and a student policy network, which are then deployed on the actual vehicle.

[0101] Next, referring to the accompanying drawings, we describe the autonomous driving system based on offline reinforcement learning and teacher-student intelligent agents proposed according to embodiments of this application.

[0102] Figure 8 This is a block diagram of an autonomous driving system based on offline reinforcement learning and teacher-student intelligent agents, according to an embodiment of this application.

[0103] like Figure 8 As shown, the autonomous driving system 10 based on offline reinforcement learning and teacher-student intelligent agents includes: a data acquisition system 100, a teacher learning system 200, a student learning system 300, and a deployment system 400.

[0104] The data acquisition system 100 is used to acquire driving data of the driver in real-world scenarios, process the driving data into a state set, action set, and reward set required for intelligent agent training, and store it as a training trajectory. The intelligent agents include a teacher intelligent agent and a student intelligent agent. The state set of the teacher intelligent agent is... The state set of the student agent is Let A be the action set of the teacher agent. l,t The reward function is R. l,t Where l is the number of trajectories and t is the time step for each trajectory;

[0105] The teacher learning system 200 is used to construct a teacher agent, which includes a teacher policy neural network, a teacher value neural network, and a teacher estimation neural network. The teacher agent is trained using offline reinforcement learning based on the state set, action set, and reward function of the teacher agent to obtain the teacher policy neural network, teacher value neural network, and teacher estimation neural network.

[0106] The student learning system 300 is used to construct a student agent, which includes a student policy neural network and a student estimation neural network. The student agent is determined based on the norm of the difference between the outputs of the teacher estimation neural network and the student estimation neural network. and the norm of the output difference between the teacher's policy neural network and the student's policy neural network. Supervised training was conducted to obtain the student policy neural network model and the student estimation neural network model.

[0107] The deployment system 400 is used to deploy student policy neural networks and student estimation neural networks onto a real vehicle. It acquires the state set of the student agent on the vehicle, inputs the student agent's state set into the student agent, and uses the output of the student policy neural network to control the vehicle. The state set of the student agent is as follows:

[0108] It should be noted that the foregoing explanation of the autonomous driving method embodiment based on offline reinforcement learning and teacher-student intelligent agents also applies to the autonomous driving system based on offline reinforcement learning and teacher-student intelligent agents in this embodiment, and will not be repeated here.

[0109] The autonomous driving system based on offline reinforcement learning and teacher-student intelligent agents proposed in this application significantly improves the robustness, generalization ability, and safety of actual deployment of the autonomous driving system by constructing a learning system for teacher and student intelligent agents. This system trains the teacher intelligent agent using real traffic data and then guides the student intelligent agent to learn and optimize. Simultaneously, it eliminates the gap between simulation and real-world scenarios. Therefore, it solves problems in existing technologies such as insufficient generalization ability, limitations of supervised learning methods, high risks in practical applications of reinforcement learning, and discrepancies between simulation and reality.

[0110] Figure 9 A schematic diagram of the structure of an electronic device provided in an embodiment of this application. The electronic device may include:

[0111] The memory 901, the processor 902, and the computer program stored on the memory 901 and capable of running on the processor 902.

[0112] When the processor 902 executes the program, it implements the autonomous driving method based on offline reinforcement learning and teacher-student intelligent agents provided in the above embodiments.

[0113] Furthermore, electronic devices also include:

[0114] Communication interface 903 is used for communication between memory 901 and processor 902.

[0115] The memory 901 is used to store computer programs that can run on the processor 902.

[0116] The memory 901 may include high-speed RAM (Random Access Memory) memory, and may also include non-volatile memory, such as at least one disk storage.

[0117] If the memory 901, processor 902, and communication interface 903 are implemented independently, then the communication interface 903, memory 901, and processor 902 can be interconnected via a bus to complete communication between them. The bus can be an ISA (Industry Standard Architecture) bus, a PCI (Peripheral Component Interconnect) bus, or an EISA (Extended Industry Standard Architecture) bus, etc. The bus can be divided into address bus, data bus, control bus, etc. For ease of representation, Figure 9 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.

[0118] Optionally, in a specific implementation, if the memory 901, processor 902, and communication interface 903 are integrated on a single chip, then the memory 901, processor 902, and communication interface 903 can communicate with each other through an internal interface.

[0119] The processor 902 may be a CPU (Central Processing Unit), an ASIC (Application Specific Integrated Circuit), or one or more integrated circuits configured to implement the embodiments of this application.

[0120] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described autonomous driving method based on offline reinforcement learning and teacher-student intelligent agents.

[0121] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of this application. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0122] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this application, "N" means at least two, such as two, three, etc., unless otherwise explicitly specified.

[0123] Any process or method described in the flowchart or otherwise herein can be understood as representing a module, segment, or portion of code comprising one or N executable instructions for implementing custom logic functions or processes, and the scope of the preferred embodiments of this application includes additional implementations in which functions may be performed not in the order shown or discussed, including substantially simultaneously or in reverse order depending on the functions involved, as should be understood by those skilled in the art to which embodiments of this application pertain.

[0124] It should be understood that the various parts of this application can be implemented using hardware, software, firmware, or a combination thereof. In the above embodiments, the N steps or methods can be implemented using software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (FPGAs), field-programmable gate arrays (FPGAs), etc.

[0125] Those skilled in the art will understand that all or part of the steps of the methods described in the above embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, it includes one or a combination of the steps of the method embodiments.

[0126] Although embodiments of this application have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting this application. Those skilled in the art can make changes, modifications, substitutions and variations to the above embodiments within the scope of this application.

Claims

1. An automatic driving method based on offline reinforcement learning and teacher-student agent, characterized in that, The method comprises the following steps: Obtaining driving data of a driver driving a vehicle in a real scene, processing the driving data of the driving vehicle into a state set, an action set and a reward set required for agent training, and storing as a training trajectory, wherein the agent includes a teacher agent and a student agent, wherein the state set of the teacher agent is , the state set of the student agent is , the action set of the teacher agent is , the reward function is , wherein l is the number of trajectories, t is the time step of each trajectory. The teacher agent is constructed, and the teacher agent comprises a teacher policy neural network, a teacher value neural network and a teacher estimation neural network, wherein the teacher agent is trained by using offline reinforcement learning according to a state set of the teacher agent, an action set of the teacher agent and the reward function, so as to obtain the teacher policy neural network, the teacher value neural network and the teacher estimation neural network, wherein a vehicle control parameter output by the teacher policy network is , and a feature estimation of road information output by the teacher estimation neural network is . constructing the student agent, the student agent comprising a student policy neural network and a student estimation neural network, comprising a fitting function constructing the teacher estimation neural network of the teacher agent, wherein the state input of the teacher estimation neural network is privileged information of a road traffic environment, is a feature estimation of the privileged information ; a fitting function constructing the teacher policy neural network of the teacher agent, wherein, is a vehicle control parameter output by the teacher policy network; a fitting function constructing the teacher value neural network of the teacher agent, wherein, is an evaluation of the vehicle control parameter output by the teacher policy network ; wherein the student agent is trained by imitation learning according to the state set of the student agent, the output of the teacher policy network and the output of the teacher estimation neural network, to obtain the student policy neural network and the student estimation neural network, wherein the vehicle control parameter output by the student policy neural network is ; According to the student policy neural network and the student estimation neural network, a state set of the student agent is obtained on the real vehicle, the state set of the student agent is input into the student agent, and the output of the student policy neural network is used to control the real vehicle, wherein the state set of the student agent is . 2.The autonomous driving method based on offline reinforcement learning and teacher-student agent according to claim 1, wherein, The state set of the teacher agent comprises navigation commands , ego information , BEV images and privileged information of the road traffic environment , i.e. ); the state set of the student agent comprises navigation commands , ego information , BEV images and historical information of the navigation commands and the ego information , i.e. ); wherein the common state set of the teacher agent and the student agent is ) ; The action set of the teacher agent includes steering of a steering wheel and acceleration i.e. wherein, left turns take positive values and right turns take negative values; acceleration takes positive values and deceleration takes negative values; The reward function is: wherein, is a motion quality weight coefficient, is a motion quality, is a comfort weight coefficient, is a comfort, is an energy weight coefficient, is an energy consumption, is a termination condition. 3.The autonomous driving method based on offline reinforcement learning and teacher-student agent according to claim 2, wherein, the quality of the action is: wherein, , , are position difference, speed difference and orientation angle difference between the trajectory formed by the human-driven vehicle and the trajectory planned by the convex optimization method, respectively, is a reward value of the position difference, is a reward value of the speed difference, is a reward value of the orientation angle difference, is a weight coefficient of the position difference, is a weight coefficient of the speed difference, is a weight coefficient of the orientation angle difference; The comfort Is: wherein, is a reward value corresponding to a change in steering wheel angle between the previous and the current time, is a reward value corresponding to a change in acceleration between the previous and the current time, is a weight coefficient corresponding to a change in steering wheel angle, is a weight coefficient corresponding to a change in acceleration; The energy consumption Is: wherein is the acceleration value at the time instant is the acceleration value at the time instant is the acceleration value at the time instant is the corresponding time difference; The termination condition is: 。 4.The autonomous driving method based on offline reinforcement learning and teacher-student agent according to claim 1, wherein, The update formula of the teacher strategy neural network and the teacher estimation neural network is: wherein, are parameters of the teacher policy neural network, are parameters of the teacher estimation neural network, denotes the total number of trajectories trained, is a proportionality coefficient; The update formula of the teacher value neural network is: in, These are the parameters of the teacher value neural network. It is a proportionality coefficient. It is the teacher value neural network in Output at any moment for Navigation commands at any time, for Real-time vehicle information, for BEV image at time of moment, yes Feature estimation of privileged information at any time. yes Evaluation of vehicle control parameters output by the teacher value neural network at any given time. for The vehicle control parameters output by the teacher strategy network at any time. 5.The autonomous driving method based on offline reinforcement learning and teacher-student agent according to claim 1, wherein, The student agent is constructed, comprising: According to the fitting function constructing the student estimation neural network of the student agent, wherein the state input of the student estimation neural network is historical information of navigation commands and ego information, is a feature estimation of privileged information; According to the fitting function constructing the student policy neural network of the student agent. 6.The autonomous driving method based on offline reinforcement learning and teacher-student agent according to claim 5, wherein, The update formula of the student strategy neural network and the student estimation neural network is: wherein, are parameters of the student policy neural network, are parameters of the student estimation neural network, is a proportionality coefficient.

7. An electronic device, comprising: Comprising: A memory, a processor and a computer program stored on the memory and running on the processor, the processor executing the program to implement the off-line reinforcement learning and teacher-student agent based automatic driving method of any one of claims 1-6.

8. A computer-readable storage medium having stored thereon a computer program, characterized in that, The program is executed by the processor for implementing the off-line reinforcement learning and teacher-student agent based automatic driving method of any one of claims 1-6.

Images