Automatic driving method and system based on offline reinforcement learning and teacher-student intelligent agent

Through the method based on offline reinforcement learning and teacher-student agents, using real traffic data to train teacher agents and guide students to learn, the problem of insufficient generalization ability of the autonomous driving system and the high-risk problem of reinforcement learning in practical applications is solved. It significantly improves robustness and security, eliminating the gap between simulation and real scenes.

CN118915537AActive Publication Date: 2024-11-08CHERY AUTOMOBILE CO LTD

Patent Information

Application Number
CN202410990688.6
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-23
Publication Date
2024-11-08
Estimated Expiration
2044-07-23

AI Technical Summary

Technical Problem

Insufficient generalization capabilities in existing autonomous driving technologies, limitations of supervised learning methods, high risks of reinforcement learning in practical applications, and differences between simulation and real scenarios.

Method used

Using the method based on offline reinforcement learning and teacher-student intellectuals, the teacher and student intellectuals are constructed by obtaining drivers' driving data in real scenarios, and using offline reinforcement learning and imitation learning to train teachers and student intellectuals, the teacher intelligence is The system uses road privilege information as input, and the student agent uses the output of the teacher agent as supervision signal for learning, and is finally deployed to the actual vehicle for control.

Benefits of technology

It significantly improves the robustness, generalization capability and actual deployment security of the autonomous driving system, eliminates the gap between simulation and real scenarios, solves the insufficient generalization capability in the existing technology, the limitations and enhances the supervised learning method. High-risk problems in practical applications of learning.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118915537A_ABST
    Figure CN118915537A_ABST
Patent Text Reader

Abstract

The invention relates to the technical field of automatic driving, in particular to an automatic driving method and system based on offline reinforcement learning and a teacher-student agent, and the method comprises the steps: obtaining a training data set which comprises the driving data of a real traffic scene; constructing a teacher agent, and training the teacher agent by using offline reinforcement learning based on the training data set; constructing a student agent, and training the student agent by utilizing imitation learning according to the output of the trained teacher agent and the training data set; and controlling the vehicle to perform automatic driving based on the trained student intelligent agent. Therefore, the problems of insufficient generalization ability, limitation of a supervised learning method, high risk of reinforcement learning in practical application, difference between simulation and reality and the like in the prior art are solved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of autonomous driving technology, and in particular to an autonomous driving method and system based on offline reinforcement learning and teacher-student intelligent agents. Background Art

[0002] In the field of autonomous driving, planning and control are core technologies, and their challenges cannot be ignored. Although the traditional rule-based method is stable, it is difficult to cope with all complex and changing scenarios. Although the supervised learning-based method can imitate the expert trajectory to solve driving problems in some complex scenarios, its generalization ability is stretched when facing non-training set scenarios, and it is difficult to adapt to the ever-changing traffic environment in the real world.

[0003] As a self-updating technology, reinforcement learning can theoretically adapt to complex and ever-changing traffic scenarios through interactive learning with the environment. However, there are extremely high risks in conducting reinforcement learning training in a real environment. In the process of exploration and learning of robots or self-driving cars, once an error occurs, it may cause hardware damage or threaten the safety of surrounding objects or even human life. Therefore, current reinforcement learning mostly adopts the strategy of first training in a simulation environment and then gradually deploying it to a real environment. However, due to the data difference between the simulation environment and the actual environment, the strategies learned by reinforcement learning in the simulation environment are often difficult to achieve satisfactory performance when actually deployed.

[0004] Offline reinforcement learning can use the interaction data of human experts in real traffic environments for offline training. Since the data of the training set is collected in advance in a real scene at one time, there is no need to interact with the environment during the training process. Therefore, the method based on offline reinforcement learning avoids the danger of online training directly in the real environment. In addition, offline reinforcement learning introduces the data collected in the real environment into the simulation environment for training. There is no gap between simulation and reality that requires post-processing, and the difference between the data collected in the simulation and the real environment can be eliminated.

[0005] During the training process, information such as the road traffic environment is an important basis for decision-making. This type of privileged information can be explicitly informed to the intelligent agent in a simulation environment, but in a real environment, the intelligent agent cannot directly obtain this type of information. Therefore, training and actual deployment can be completed through a two-level teacher-student agent method, in which the teacher agent is trained with privileged information as input, and the student agent imitates and learns with the output of the teacher agent as a supervision signal, and finally the student agent is deployed in a real vehicle. Summary of the invention

[0006] The present application provides an autonomous driving method, apparatus, device and medium to solve the problems in the prior art such as insufficient generalization ability, limitations of supervised learning methods, high risks of reinforcement learning in practical applications, and differences between simulation and reality.

[0007] The embodiment of the present application provides an automatic driving method based on offline reinforcement learning and teacher-student agents, comprising the following steps: obtaining driving data of a driver driving a vehicle in a real scene, processing the driving data of the vehicle into a state set, an action set and a reward set required for agent training, and storing them as a training trajectory, wherein the agent includes a teacher agent and a student agent, wherein the state set of the teacher agent is The state set of the student agent is The action set of the teacher agent is A l,t , the reward function is R l,t , where l is the number of trajectories and t is the time step of each trajectory; construct the teacher agent, which includes a teacher strategy neural network, a teacher value neural network and a teacher estimation neural network, wherein the teacher agent is trained using offline reinforcement learning according to the state set of the teacher agent, the action set of the teacher agent and the reward function to obtain the teacher strategy neural network, the teacher value neural network and the teacher estimation neural network, wherein the vehicle control parameters output by the teacher strategy network are The estimate of the road privilege information output by the teacher estimation neural network is The student agent is constructed, and the student agent includes a student strategy neural network and a student estimation neural network. The student agent is trained by imitation learning according to the state set of the student agent, the output of the teacher strategy network and the output of the teacher estimation neural network to obtain the student strategy neural network and the student estimation neural network. The vehicle control parameter output by the student strategy neural network is The student strategy neural network and the student estimation neural network are deployed on a real vehicle, the state set of the student agent is obtained on the real vehicle, the state set of the student agent is input into the student agent, and the real vehicle is controlled using the output of the student strategy neural network, wherein the state set of the student agent is

[0008] Optionally, the state set of the teacher agent includes navigation commands Car information BEV Image and privileged information about the road traffic environment Right now The state set of the student agent includes navigation commands Car information BEV Image and historical information of navigation commands and ego vehicle information Right now Among them, the common state set of the teacher agent and the student agent is

[0009] The action set of the teacher agent includes steering wheel and acceleration Right now in, A left turn takes a positive value, a right turn takes a negative value; Acceleration takes positive values, deceleration takes negative values;

[0010] The reward function R l,t for:

[0011]

[0012] in, is the action quality weight coefficient, For the quality of action, is the comfort weight coefficient, For comfort, w energy is the energy weight coefficient, For energy consumption, is the termination condition.

[0013] Optionally, the action quality for:

[0014]

[0015] Among them, Δp, Δv, and Δθ are the position difference, speed difference, and heading angle difference between the trajectory formed by the human driving vehicle and the trajectory planned by the convex optimization method, respectively. is the return value of the position difference, is the return value of the speed difference, is the return value of the heading angle difference, is the weight coefficient of the position difference, is the weight coefficient of the speed difference, is the weight coefficient of the heading angle difference;

[0016] The comfort for:

[0017]

[0018] in, is the return value corresponding to the change in steering wheel angle before and after, is the return value corresponding to the change in acceleration before and after. is the weight coefficient corresponding to the change of steering wheel angle, is the weight coefficient corresponding to the acceleration change;

[0019] The energy consumption for:

[0020]

[0021] in, is the acceleration value at time t, is the acceleration value at time t-1, is the acceleration value at time t-2, and Δt is the corresponding time difference;

[0022] The termination conditions for:

[0023]

[0024] Optionally, constructing the teacher agent includes: according to the fitting function Constructing the teacher estimation neural network of the teacher agent, wherein the state input of the teacher estimation neural network It is the privileged information of the road traffic environment. Privileged information The characteristic estimation of Construct the teacher strategy neural network of the teacher agent, wherein: is the vehicle control parameter output by the teacher strategy network; according to the fitting function Construct the teacher value neural network of the teacher agent, wherein: The vehicle control parameters output by the teacher policy network 's evaluation.

[0025] Optionally, the update formulas of the teacher strategy neural network and the teacher estimation neural network are:

[0026]

[0027] in, are the parameters of the teacher strategy neural network, are the parameters of the teacher estimation neural network, L represents the total number of trajectories trained, and λ1 is the scaling coefficient;

[0028] The updating formula of the teacher value neural network is:

[0029]

[0030] in, is the parameter of the teacher value neural network, γ is the proportional coefficient, is the output of the teacher value neural network at time t+1, is the navigation command at time t+1, is the vehicle information at time t+1, is the BEV image at time t+1, is the feature estimate of the privileged information at time t+1, is the evaluation of the vehicle control parameters output by the teacher value neural network at time t, are the vehicle control parameters output by the teacher strategy network at time t+1.

[0031] Optionally, constructing the student agent includes: according to the fitting function Constructing the student estimation neural network of the student agent, wherein the state input of the student estimation neural network It is the historical information of navigation commands and vehicle information. is a feature estimate of the privileged information; according to the fitting function Construct the student strategy neural network of the student agent.

[0032] Optionally, the update formulas of the student strategy neural network and the student estimation neural network are:

[0033]

[0034] in, are the parameters of the student strategy neural network, are the parameters of the student-estimated neural network and λ2 is the scaling coefficient.

[0035] The embodiment of the present application also provides an automatic driving system based on offline reinforcement learning and teacher-student agents, including: a data acquisition system for obtaining driving data of a driver driving a vehicle in a real scene, processing the driving data of the driving vehicle into a state set, an action set and a reward set required for agent training, and storing them as a training trajectory, wherein the agent includes a teacher agent and a student agent, wherein the state set of the teacher agent is The state set of the student agent is The action set of the teacher agent is A l,t , the reward function is R l,t, where l is the number of trajectories and t is the time step of each trajectory; a teacher learning system, used to construct the teacher agent, the teacher agent includes a teacher policy neural network, a teacher value neural network and a teacher estimation neural network, wherein the teacher agent is trained using offline reinforcement learning according to the state set of the teacher agent, the action set of the teacher agent and the reward function to obtain the teacher policy neural network, the teacher value neural network and the teacher estimation neural network; a student learning system, used to construct the student agent, the student agent includes a student policy neural network and a student estimation neural network, wherein the student agent is trained using imitation learning according to the state set of the student agent, the output of the teacher policy network and the output of the teacher estimation neural network to obtain the student policy neural network and the student estimation neural network, wherein the vehicle control parameters output by the student policy neural network are A deployment system is used to deploy the student strategy neural network and the student estimation neural network on a real vehicle, obtain the state set of the student agent on the real vehicle, input the state set of the student agent into the student agent, and control the real vehicle using the output of the student strategy neural network, wherein the state set of the student agent is

[0036] An embodiment of the present application provides an electronic device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the autonomous driving method based on offline reinforcement learning and teacher-student agents as described in the above embodiment.

[0037] An embodiment of the present application provides a computer-readable storage medium on which a computer program is stored, and the program is executed by a processor to implement the autonomous driving method based on offline reinforcement learning and teacher-student agents as described in the above embodiment.

[0038] Therefore, this application has at least the following beneficial effects:

[0039] The embodiment of the present application constructs a learning system of teacher agents and student agents, uses real traffic data to train teacher agents, and then guides student agents to learn and optimize, which significantly improves the robustness, generalization ability and safety of actual deployment of the autonomous driving system, and eliminates the gap between simulation and real scenes. Thus, technical problems such as insufficient generalization ability, limitations of supervised learning methods, high risks of reinforcement learning in practical applications, and differences between simulation and reality in the prior art are solved.

[0040] Additional aspects and advantages of the present application will be given in part in the description below, and in part will become apparent from the description below, or will be learned through the practice of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] The above and / or additional aspects and advantages of the present application will become apparent and easily understood from the following description of the embodiments in conjunction with the accompanying drawings, in which:

[0042] Figure 1 A flowchart of an autonomous driving method based on offline reinforcement learning and teacher-student agents provided according to some embodiments of the present application;

[0043] Figure 2 A diagram of a teacher estimation network structure provided according to some embodiments of the present application;

[0044] Figure 3 A teacher strategy network structure diagram provided according to some embodiments of the present application;

[0045] Figure 4 A teacher value network structure diagram provided according to some embodiments of the present application;

[0046] Figure 5 A student estimation network structure diagram provided according to some embodiments of the present application;

[0047] Figure 6 A student strategy network structure diagram provided according to some embodiments of the present application;

[0048] Figure 7 A schematic diagram of an autonomous driving method based on offline reinforcement learning and teacher-student agents provided according to some embodiments of the present application;

[0049] Figure 8 A schematic diagram of an autonomous driving system based on offline reinforcement learning and teacher-student agents provided according to some embodiments of the present application;

[0050] Fig. 9 Schematic diagram of the structure of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0051] Embodiments of the present application are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present application, and should not be construed as limiting the present application.

[0052] The following describes the automatic driving and system based on offline reinforcement learning and teacher-student agents in the embodiments of the present application with reference to the accompanying drawings. In view of the problem of data differences between simulation and reality mentioned in the above background technology, the present application provides an automatic driving based on offline reinforcement learning and teacher-student agents. In this method, by constructing a learning system of teacher agents and student agents, the teacher agent is trained using real traffic data, and then the teacher agent guides the student agent to learn and optimize, which significantly improves the robustness, generalization ability and safety of actual deployment of the automatic driving system, and eliminates the gap between simulation and real scenes. As a result, the problems of insufficient generalization ability, limitations of supervised learning methods, high risks of reinforcement learning in practical applications, and differences between simulation and reality in the prior art are solved.

[0053] Specifically, Figure 1 A flowchart of autonomous driving based on offline reinforcement learning and teacher-student agents provided in an embodiment of the present application.

[0054] like Figure 1 As shown, the autonomous driving method based on offline reinforcement learning and teacher-student agents includes the following steps:

[0055] In step S101, the driving data of the driver driving the vehicle in the real scene is obtained, and the driving data of the vehicle is processed into a state set, an action set and a reward set required for the training of the intelligent agent, and stored as a training trajectory.

[0056] Among them, the agents include teacher agents and student agents; the state set of the teacher agent is The state set of the student agent is The action set of the teacher agent is A l,t , the reward function is R l,t , where l is the number of trajectories and t is the time step of each trajectory.

[0057] In the embodiment of the present application, the state set of the teacher agent includes navigation commands Car information BEV Image and privileged information about the road traffic environment Right now The state set of the student agent includes navigation commands Car information BEV Image and historical information of navigation commands and ego vehicle information Right now

[0058] It should be noted that the navigation command Including left turn, right turn, straight ahead, lane change to the right and lane change to the left, etc.; vehicle information Including posture, velocity, acceleration, angular velocity and angular acceleration, etc.; BEV feature map Obtained by the BEVFormer algorithm, it can represent dynamic targets such as vehicles and pedestrians and static targets such as lane lines and road edges; privileged information of the road traffic environment Including real lane information, traffic lights and traffic signs; historical information of navigation commands and vehicle information Take 6 time steps and Numeric value.

[0059] In the embodiment of the present application, the action set of the teacher agent includes steering wheel turning and acceleration Right now in, A left turn takes a positive value, a right turn takes a negative value; Acceleration takes positive values, deceleration takes negative values;

[0060] Reward function R l,t for:

[0061]

[0062] in, is the action quality weight coefficient, For the quality of action, is the comfort weight coefficient, For comfort, w energy is the energy weight coefficient, For energy consumption, is the termination condition.

[0063] In the embodiment of the present application, the action quality for:

[0064]

[0065]

[0066] Among them, Δp, Δv, and Δθ are the position difference, speed difference, and heading angle difference between the trajectory formed by the human driving vehicle and the trajectory planned by the convex optimization method, respectively. is the return value of the position difference, is the return value of the speed difference, is the return value of the heading angle difference, is the weight coefficient of the position difference, is the weight coefficient of the speed difference, is the weight coefficient of the heading angle difference;

[0067] Comfort for:

[0068]

[0069] in, is the return value corresponding to the change in steering wheel angle before and after, is the return value corresponding to the change in acceleration before and after. is the weight coefficient corresponding to the change of steering wheel angle, is the weight coefficient corresponding to the acceleration change;

[0070] Energy consumption for:

[0071]

[0072] in, is the acceleration value at time t, is the acceleration value at time t-1, is the acceleration value at time t-2, and Δt is the corresponding time difference;

[0073] Termination Condition for:

[0074]

[0075] Among them, the penalty for violating traffic rules such as collision or running a red light is -100, and a reward of 100 is given when reaching the destination.

[0076] In step S102, a teacher agent is constructed, which includes a teacher strategy neural network, a teacher value neural network and a teacher estimation neural network, wherein the teacher agent is trained using offline reinforcement learning according to the state set of the teacher agent, the action set of the teacher agent and the reward function to obtain the teacher strategy neural network, the teacher value neural network and the teacher estimation neural network.

[0077] Among them, the vehicle control parameters output by the teacher strategy network are The feature estimate of the road information output by the teacher estimation neural network is

[0078] In the embodiment of the present application, constructing a teacher agent includes: according to the fitting function Construct the teacher estimation neural network of the teacher agent, where the state input of the teacher estimation neural network It is the privileged information of the road traffic environment. Privileged information The characteristic estimation of Construct the teacher strategy neural network of the teacher agent, where is the vehicle control parameter output by the teacher strategy network; according to the fitting function Construct the teacher value neural network of the teacher agent, where The vehicle control parameters output by the teacher policy network 's evaluation.

[0079] Specifically, if Figure 2 As shown in Figure 2, the teacher estimation neural network uses a fully connected network, the activation function is ReLU, and the input layer is the privileged information of the road traffic environment. There are 2 hidden layers, both of which are ReLU (256), and the output layer is the feature estimation of privileged information

[0080] like Figure 3 As shown, the teacher strategy neural network uses a fully connected network, the activation function is ReLU, and the input layer is There are 6 hidden layers, namely ReLU(512), ReLU(256), ReLU(256), ReLU(256), ReLU(128) and ReLU(2), and the output layer is the vehicle control parameters.

[0081] like Figure 4 As shown, the teacher value neural network uses a fully connected network, the activation function is ReLU, and the input layer is There are 5 hidden layers, namely ReLU(512), ReLU(256), ReLU(256), ReLU(256) and ReLU(1), and the output layer is the vehicle control parameters. Reviews

[0082] In the embodiment of the present application, the update formulas of the teacher strategy neural network and the teacher estimation neural network are:

[0083]

[0084] in, are the parameters of the teacher strategy neural network, are the parameters of the teacher estimation neural network, L represents the total number of trajectories trained, and λ1 is the scaling coefficient;

[0085] The update formula of the teacher value neural network is:

[0086]

[0087] in, is the parameter of the teacher value neural network, γ is the proportional coefficient, is the output of the teacher value neural network at time t+1, is the navigation command at time t+1, is the vehicle information at time t+1, is the BEV image at time t+1, is the feature estimate of the privileged information at time t+1, is the evaluation of the vehicle control parameters output by the teacher value neural network at time t, are the vehicle control parameters output by the teacher strategy network at time t+1.

[0088] In step S103, a student agent is constructed, which includes a student strategy neural network and a student estimation neural network, wherein the student agent is trained using imitation learning according to the state set of the student agent, the output of the teacher strategy network and the output of the teacher estimation neural network to obtain the student strategy neural network and the student estimation neural network.

[0089] Among them, the vehicle control parameters output by the student strategy neural network are

[0090] In the embodiment of the present application, constructing a student agent includes: according to the fitting function Construct a student estimation neural network of the student agent, where the state input of the student estimation neural network It is the historical information of navigation commands and vehicle information. is a feature estimate of the privileged information; according to the fitting function Construct the student strategy neural network of the student agent.

[0091] Specifically, if Figure 5 As shown in Figure 2, the student estimation neural network uses a fully connected network, the activation function is ReLU, and the input layer is the historical information of the navigation command and the vehicle information. There are 2 hidden layers, both of which are ReLU (128), and the output layer is the feature estimation of privileged information

[0092] like Figure 6 As shown, the student strategy neural network uses a fully connected network, the activation function is ReLU, and the input layer is There are 6 hidden layers, namely ReLU(512), ReLU(256), ReLU(256), ReLU(256), ReLU(128) and ReLU(2), and the output layer is the vehicle control parameters.

[0093] In the embodiment of the present application, the update formulas of the student strategy neural network and the student estimation neural network are:

[0094]

[0095] in, are the parameters of the student strategy neural network, are the parameters of the student-estimated neural network and λ2 is the scaling coefficient.

[0096] In step S104, the student strategy neural network and the student estimation neural network are deployed on the actual vehicle, the state set of the student agent is obtained on the actual vehicle, the state set of the student agent is input into the student agent, and the output of the student strategy neural network is used to control the actual vehicle.

[0097] Among them, the state set of the student agent is

[0098] Specifically, the student agent is deployed on the real vehicle, and the state set obtained on the real vehicle is Information is input to the student strategy neural network and the student estimation neural network according to the formula and Output vehicle control parameters The automatic driving control of the real vehicle is realized by the vehicle control parameters.

[0099] According to the automatic driving method based on offline reinforcement learning and teacher-student agents proposed in the embodiment of the present application, by constructing a learning system of teacher agents and student agents, using real traffic data to train teacher agents, and then letting them guide student agents to learn and optimize, the robustness, generalization ability and safety of actual deployment of the automatic driving system are significantly improved, while eliminating the gap between simulation and real scenes. Thus, the problems of insufficient generalization ability, limitations of supervised learning methods, high risks of reinforcement learning in practical applications, and differences between simulation and reality in the prior art are solved.

[0100] The following will be combined Figure 7 The automatic driving method based on offline reinforcement learning and teacher-student agents is specifically described as follows: the driver drives the vehicle in various real environments and collects data, and stores the data as training trajectories for offline training of the teacher-student two-level agents. Among them, the teacher agent is trained using offline reinforcement learning to obtain the teacher estimation network and the teacher policy network; the student agent imitates the teacher agent's estimation network and policy network to obtain the student estimation network and the student policy network and deploy them on the real vehicle.

[0101] Secondly, the autonomous driving system based on offline reinforcement learning and teacher-student agents proposed in accordance with an embodiment of the present application is described with reference to the accompanying drawings.

[0102] Figure 8 It is a block diagram of an autonomous driving system based on offline reinforcement learning and teacher-student agents according to an embodiment of the present application.

[0103] like Figure 8 As shown, the autonomous driving system 10 based on offline reinforcement learning and teacher-student agents includes: a data acquisition system 100, a teacher learning system 200, a student learning system 300 and a deployment system 400.

[0104] The data acquisition system 100 is used to obtain the driving data of the driver in the real scene, process the driving data into the state set, action set and reward set required for the agent training, and store them as training trajectories, wherein the agent includes the teacher agent and the student agent, wherein the state set of the teacher agent is The state set of the student agent is The action set of the teacher agent is A l,t , the reward function is R l,t , where l is the number of trajectories and t is the time step of each trajectory;

[0105] The teacher learning system 200 is used to construct a teacher agent, which includes a teacher strategy neural network, a teacher value neural network and a teacher estimation neural network, wherein the teacher agent is trained using offline reinforcement learning according to a state set of the teacher agent, an action set of the teacher agent and a reward function to obtain a teacher strategy neural network, a teacher value neural network and a teacher estimation neural network;

[0106] The student learning system 300 is used to construct a student agent, which includes a student strategy neural network and a student estimation neural network, wherein the output difference norm of the teacher estimation neural network and the student estimation neural network is And the output difference norm of the teacher strategy neural network and the student strategy neural network Conduct supervised training to obtain a student strategy neural network model and a student estimation neural network model;

[0107] The deployment system 400 is used to deploy the student strategy neural network and the student estimation neural network to the real vehicle, obtain the state set of the student agent on the real vehicle, input the state set of the student agent into the student agent, and control the real vehicle using the output of the student strategy neural network, wherein the state set of the student agent is

[0108] It should be noted that the aforementioned explanation of the embodiment of the autonomous driving method based on offline reinforcement learning and teacher-student intelligent agents is also applicable to the autonomous driving system based on offline reinforcement learning and teacher-student intelligent agents of this embodiment, and will not be repeated here.

[0109] According to the automatic driving system based on offline reinforcement learning and teacher-student agents proposed in the embodiment of the present application, by constructing a learning system of teacher agents and student agents, using real traffic data to train teacher agents, and then letting them guide student agents to learn and optimize, the robustness, generalization ability and safety of actual deployment of the automatic driving system are significantly improved, while eliminating the gap between simulation and real scenes. Thus, the problems of insufficient generalization ability, limitations of supervised learning methods, high risks of reinforcement learning in practical applications, and differences between simulation and reality in the prior art are solved.

[0110] Fig. 9 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. The electronic device may include:

[0111] A memory 901 , a processor 902 , and a computer program stored in the memory 901 and executable on the processor 902 .

[0112] When the processor 902 executes the program, the autonomous driving method based on offline reinforcement learning and teacher-student agents provided in the above embodiment is implemented.

[0113] Furthermore, the electronic device further comprises:

[0114] The communication interface 903 is used for communication between the memory 901 and the processor 902 .

[0115] The memory 901 is used to store computer programs that can be executed on the processor 902 .

[0116] The memory 901 may include a high-speed RAM (Random Access Memory) memory, and may also include a non-volatile memory, such as at least one disk memory.

[0117] If the memory 901, the processor 902 and the communication interface 903 are implemented independently, the communication interface 903, the memory 901 and the processor 902 can be connected to each other through a bus and communicate with each other. The bus can be an ISA (Industry Standard Architecture) bus, a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Fig. 9 Only one thick line is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0118] Optionally, in a specific implementation, if the memory 901, the processor 902 and the communication interface 903 are integrated on a chip, the memory 901, the processor 902 and the communication interface 903 can communicate with each other through an internal interface.

[0119] The processor 902 may be a CPU (Central Processing Unit), or an ASIC (Application Specific Integrated Circuit), or one or more integrated circuits configured to implement the embodiments of the present application.

[0120] An embodiment of the present application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the above-mentioned autonomous driving method based on offline reinforcement learning and teacher-student intelligent agents.

[0121] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, the schematic representations of the above terms are not necessarily directed to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or N embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, without contradiction.

[0122] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include at least one of the features. In the description of this application, "N" means at least two, such as two, three, etc., unless otherwise clearly and specifically defined.

[0123] Any process or method description in a flowchart or otherwise described herein may be understood to represent a module, fragment or portion of code comprising one or N executable instructions for implementing the steps of a custom logical function or process, and the scope of the preferred embodiments of the present application includes alternative implementations in which functions may not be performed in the order shown or discussed, including performing functions in a substantially simultaneous manner or in reverse order depending on the functions involved, which should be understood by technicians in the technical field to which the embodiments of the present application belong.

[0124] It should be understood that the various parts of the present application can be implemented by hardware, software, firmware or a combination thereof. In the above-mentioned embodiment, the N steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array, a field programmable gate array, etc.

[0125] A person skilled in the art may understand that all or part of the steps in the method for implementing the above-mentioned embodiment may be completed by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, which, when executed, includes one or a combination of the steps of the method embodiment.

[0126] Although the embodiments of the present application have been shown and described above, it can be understood that the above embodiments are exemplary and cannot be understood as limitations on the present application. Ordinary technicians in this field can change, modify, replace and modify the above embodiments within the scope of the present application.

Claims

1. An autonomous driving method based on offline reinforcement learning and teacher-student agents, characterized in that: The following steps are involved: Acquire the driving data of the driver in the real scene, process the driving data into the state set, action set and reward set required for agent training, and store them as training trajectories, wherein the agent includes a teacher agent and a student agent, wherein the state set of the teacher agent is The state set of the student agent is The action set of the teacher agent is A l,t , the reward function is R l,t , where l is the number of trajectories and t is the time step of each trajectory; The teacher agent is constructed, and the teacher agent includes a teacher strategy neural network, a teacher value neural network and a teacher estimation neural network. The teacher agent is trained using offline reinforcement learning according to the state set of the teacher agent, the action set of the teacher agent and the reward function to obtain the teacher strategy neural network, the teacher value neural network and the teacher estimation neural network. The vehicle control parameters output by the teacher strategy network are The feature estimate of the road information output by the teacher estimation neural network is The student agent is constructed, and the student agent includes a student strategy neural network and a student estimation neural network. The student agent is trained by imitation learning according to the state set of the student agent, the output of the teacher strategy network and the output of the teacher estimation neural network to obtain the student strategy neural network and the student estimation neural network. The vehicle control parameter output by the student strategy neural network is The student strategy neural network and the student estimation neural network are deployed on a real vehicle, the state set of the student agent is obtained on the real vehicle, the state set of the student agent is input into the student agent, and the real vehicle is controlled using the output of the student strategy neural network, wherein the state set of the student agent is 2. The autonomous driving method based on offline reinforcement learning and teacher-student agents according to claim 1, characterized in that: The state set of the teacher agent includes navigation commands Car information BEV Image and privileged information on road traffic environment Right now The state set of the student agent includes navigation commands Car information BEV Image and historical information of navigation commands and ego vehicle information Right now Among them, the common state set of the teacher agent and the student agent is The action set of the teacher agent includes steering wheel and acceleration Right now in, A left turn takes a positive value, a right turn takes a negative value; Acceleration takes positive values, deceleration takes negative values; The reward function R l,t for: in, is the action quality weight coefficient, For the quality of action, is the comfort weight coefficient, For comfort, w energy is the energy weight coefficient, For energy consumption, is the termination condition.

3. The autonomous driving method based on offline reinforcement learning and teacher-student agents according to claim 2, characterized in that: The quality of the action for: Among them, Δp, Δv, and Δθ are the position difference, speed difference, and heading angle difference between the trajectory formed by the human driving vehicle and the trajectory planned by the convex optimization method, respectively. is the return value of the position difference, is the return value of the speed difference, is the return value of the heading angle difference, is the weight coefficient of the position difference, is the weight coefficient of the speed difference, is the weight coefficient of the heading angle difference; The comfort for: in, is the return value corresponding to the change in steering wheel angle before and after, is the return value corresponding to the change in acceleration before and after. is the weight coefficient corresponding to the change of steering wheel angle, is the weight coefficient corresponding to the acceleration change; The energy consumption for: in, is the acceleration value at time t, is the acceleration value at time t-1, is the acceleration value at time t-2, and Δt is the corresponding time difference; The termination conditions for:

4. The autonomous driving method based on offline reinforcement learning and teacher-student agents according to claim 1, characterized in that: Constructing the teacher agent, including: According to the fitting function Constructing the teacher estimation neural network of the teacher agent, wherein the state input of the teacher estimation neural network It is the privileged information of the road traffic environment. Privileged information Estimation of the characteristics of According to the fitting function Construct the teacher strategy neural network of the teacher agent, wherein: are the vehicle control parameters output by the teacher policy network; According to the fitting function Construct the teacher value neural network of the teacher agent, wherein: The vehicle control parameters output by the teacher policy network 's evaluation.

5. The autonomous driving method based on offline reinforcement learning and teacher-student agents according to claim 4, characterized in that: The update formula of the teacher strategy neural network and the teacher estimation neural network is: in, are the parameters of the teacher strategy neural network, are the parameters of the teacher estimation neural network, L represents the total number of trajectories trained, and λ1 is the scaling coefficient; The updating formula of the teacher value neural network is: in, is the parameter of the teacher value neural network, γ is the proportional coefficient, is the output of the teacher value neural network at time t+1, is the navigation command at time t+1, is the vehicle information at time t+1, is the BEV image at time t+1, is the feature estimate of the privileged information at time t+1, is the evaluation of the vehicle control parameters output by the teacher value neural network at time t, are the vehicle control parameters output by the teacher strategy network at time t+1.

6. The autonomous driving method based on offline reinforcement learning and teacher-student agents according to claim 1, characterized in that: Constructing the student agent, including: According to the fitting function Constructing the student estimation neural network of the student agent, wherein the state input of the student estimation neural network It is the historical information of navigation commands and vehicle information. It is a characteristic estimate of privileged information; According to the fitting function Construct the student strategy neural network of the student agent.

7. The autonomous driving method based on offline reinforcement learning and teacher-student agents according to claim 6, characterized in that: The update formulas of the student strategy neural network and the student estimation neural network are: in, are the parameters of the student strategy neural network, are the parameters of the student-estimated neural network and λ2 is the scaling coefficient.

8. An autonomous driving system based on offline reinforcement learning and teacher-student agents, characterized in that: include: A data acquisition system is used to obtain driving data of a driver driving a vehicle in a real scene, process the driving data of the vehicle into a state set, an action set and a reward set required for agent training, and store them as training trajectories, wherein the agent includes a teacher agent and a student agent, wherein the state set of the teacher agent is The state set of the student agent is The action set of the teacher agent is A l,t , the reward function is R l,t , where l is the number of trajectories and t is the time step of each trajectory; A teacher learning system is used to construct the teacher agent, wherein the teacher agent includes a teacher strategy neural network, a teacher value neural network and a teacher estimation neural network, wherein the teacher agent is trained using offline reinforcement learning according to the state set of the teacher agent, the action set of the teacher agent and the reward function to obtain the teacher strategy neural network, the teacher value neural network and the teacher estimation neural network; A student learning system is used to construct the student agent, wherein the student agent includes a student strategy neural network and a student estimation neural network, wherein the student agent is trained using imitation learning according to the state set of the student agent, the output of the teacher strategy network and the output of the teacher estimation neural network to obtain the student strategy neural network and the student estimation neural network, wherein the vehicle control parameter output by the student strategy neural network is A deployment system is used to deploy the student strategy neural network and the student estimation neural network on a real vehicle, obtain the state set of the student agent on the real vehicle, input the state set of the student agent into the student agent, and control the real vehicle using the output of the student strategy neural network, wherein the state set of the student agent is 9. An electronic device, characterized in that: include: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the autonomous driving method based on offline reinforcement learning and teacher-student agents as described in any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: The program is executed by a processor to implement the autonomous driving method based on offline reinforcement learning and teacher-student agents as described in any one of claims 1 to 7.

Citation Information

Patent Citations

  • Automatic driving control method, system and equipment based on reinforcement learning and storage medium

    CN117734715A

  • Deployment method and device of intelligent driving strategy, equipment, storage medium and program product

    CN118171723A

  • Training distilled machine learning models using a pre-trained feature extractor

    EP4086817A1

  • System, electronic equipment, server device, and program, or the like

    JP2024093229A

  • Training of student neural network with switched teacher neural networks

    US20200034702A1

Cited By

  • Evaluation network design method of unmanned clamping vehicle reinforcement learning regulation control algorithm

    CN120046690A

  • An Evaluation Network Design Method for a Reinforcement Learning-Based Regulation Algorithm for Unmanned Clamping Vehicles

    CN120046690B