Autonomous driving lane selection decision method and system based on inverse reinforcement learning

Abstract

The application discloses an automatic driving lane selection decision method and system based on inverse reinforcement learning, which comprises the following steps: constructing an expert strategy by using a planning algorithm to simulate in a highway-env environment to obtain an expert data set; then, reconstructing a local reward function on the basis of an actor-critic network architecture of reinforcement learning; measuring the uncertainty of an agent for a state by using an extended mixed critic network pool during the reinforcement learning process, and then selecting a course subgoal; reconstructing a local matching reward function based on the subgoal to guide the agent to act according to an expert trajectory; finally, updating a local intrinsic reward generator by using a meta-imitation target during the reinforcement learning process, guiding the agent to explore near the expert data, reasoning and modeling the expert intention from the perspective of the reward function, and further guiding the agent to be close to the expert strategy.

Description

Technical Field

[0001] This invention belongs to the field of autonomous driving lane selection decision-making and reinforcement learning, and relates to an autonomous driving lane selection decision-making method and system. Background Technology

[0002] Autonomous driving technology utilizes computer technology and various sensors and control devices to automate vehicle driving. It leverages advanced technologies to enable vehicles to drive autonomously, safely, and accurately without a human driver. Decision planning and autonomous driving control are crucial components in the development of autonomous driving technology. The decision planning module uses deep learning, machine learning, and other technologies to determine the vehicle's trajectory and path based on the surrounding environment and driving needs. It also assesses relationships with other vehicles and resolves collision issues. The autonomous driving control module, through control devices and algorithms, enables the vehicle to drive autonomously and perform various operations, including braking, acceleration, and turning. Autonomous driving lane selection is a critical aspect of autonomous driving technology and is essential for achieving truly driverless driving.

[0003] As a solution to sequence control problems, deep reinforcement learning focuses on extracting features from input states and providing response actions in an end-to-end manner. This learning paradigm has already achieved significant success in tasks such as games, robotics, and power grid scheduling. The success of deep reinforcement learning relies on a well-designed reward function. However, current reinforcement learning theory offers little practical guidance on how to design reward functions. Furthermore, in the practical implementation of reinforcement learning agents for automatic path selection, designing a reasonable reward function that simultaneously considers factors such as high performance and safety is extremely difficult.

[0004] In reality, human driving trajectories are readily available, making them suitable for using inverse reinforcement learning algorithms to solve the lane selection decision-making problem in autonomous driving. Currently, a large amount of work in the field of autonomous driving attempts to use inverse reinforcement learning to improve the vehicle's autonomous decision-making capabilities. However, as tasks become longer and more complex, the performance of current inverse reinforcement learning algorithms shows a significant decline, failing to meet the requirements of long-term driving and safe driving. Therefore, there is an urgent need for new inverse reinforcement learning algorithms that can achieve better results in long-term tasks. Summary of the Invention

[0005] To address the aforementioned problems, this invention provides an autonomous driving lane selection decision-making method and system based on inverse reinforcement learning.

[0006] This invention collects expert trajectories in autonomous driving lane selection decisions and proposes an inverse reinforcement learning method for autonomous driving lane selection decisions based on the existing actor-evaluator architecture of reinforcement learning. The technical solution of this invention is:

[0007] The autonomous driving lane selection decision-making method based on inverse reinforcement learning includes the following steps:

[0008] S1. Construct an expert dataset for autonomous driving lane selection decisions;

[0009] Based on the autonomous driving simulation environment highway-env, an expert autonomous vehicle agent is constructed using optimal planning methods for deterministic systems. Simulations are performed in various environments within highway-env, and data on interactions with the environment is recorded. Each interaction requires recording information including the current environment state, the action taken by the expert agent, the next environment state, and whether the task is completed. K data trajectories are recorded for each environment, and the corresponding data is saved to form an expert dataset.

[0010] S2. Construct a reinforcement learning model for autonomous driving lane selection decision-making;

[0011] An actor-evaluator architecture is used to build a reinforcement learning model for autonomous driving lane selection decisions, in which the actor π θ The evaluator is a hybrid evaluation pool used to generate the lane selection decision action corresponding to the current road condition. It includes n evaluator network entities used to calculate the value of the current road condition and lane selection decision actions. The role of the evaluator is twofold: firstly, to reduce the high variance of the agent's gradient estimation while ensuring that its bias remains constant; secondly, the use of multiple network entities can be used to evaluate the uncertainty of the agent.

[0012] S3. Construct an intrinsic reward generator model;

[0013] A multilayer perceptron model is used to construct an intrinsic reward generator model for the environment. The intrinsic reward generator r ω The intrinsic reward function is used to specify the current road condition. The main purpose of this intrinsic reward function is to guide the agent to explore near expert trajectories, thereby enabling better imitation and learning.

[0014] S4. Iterate through the selection of sub-goals for curriculum development;

[0015] First, assume that the curriculum sub-goal g belongs to the expert trajectory set. And select a sub-objective based on the current decision uncertainty of the agent. In the current state s, agent g selects a sub-objective based on its policy π. θ If (·|s) is chosen as action a, then the decision uncertainty is... The calculation method is as follows:

[0016] U(s)=μ(Q φ (s, a)), (1)

[0017] Where μ is the operation of calculating the variance of the outputs of different evaluation network entities in the mixed evaluation pool.

[0018] To select course sub-objectives of appropriate difficulty, we specify an expert trajectory in the expert trajectory dataset. This allows for analysis and further decoupling of the task. At the beginning of each training round, we assume the agent has learned a policy π for achieving the previous sub-goal g. θ And the evaluation network Q φ Each expert state is input into the agent policy network π. θ and Value Network Q φ The trajectory uncertainty U(τ) can be obtained. e ):

[0019]

[0020] Next, the uncertainty sequence U(τ) of the entire trajectory is scanned sequentially from front to back. e The goal is to identify states with high uncertainty. If the uncertainty of a state-action pair and its subsequent state-action pairs is significantly higher than the uncertainty of its preceding state-action pairs, then that state g is designated as the curriculum sub-objective for the next training phase. A soft average of the uncertainty values ​​is maintained throughout this process. The soft average at position p is denoted as... The value was initially set to In scanned Updated regularly:

[0021]

[0022] Where σ is the soft update coefficient, and during the scanning process, the first condition that satisfies... The state of the condition The sub-objective *g* is detected as the next training round's target, where δ is the uncertainty threshold. Once the threshold δ is exceeded, it is selected as the sub-objective for the next iteration. If there is no state uncertainty satisfying the condition during this process, training can stop because U(τ) e When the uncertainty of all elements in the process has been reduced to an acceptable range, it can be considered that the agent has mastered the complete capabilities required to complete the entire complex task.

[0023] S5. Construct a matching reward function based on the curriculum sub-objectives;

[0024] Constructing a local reward function based on reward reshaping theory:

[0025] r m (s,a,s′,g)=ψ(s′,g)-γψ(s,g), (4)

[0026] Wherein, the agent takes action a at state s and dynamically reaches state s′ according to the environment, γ is a discount factor, and ψ is a state evaluation function that can be expressed in the following form:

[0027] ψ(s,g)=||h(s)-h(g)||, (5)

[0028] Where h is the state representation function. The representation function h extracts the features of the current autonomous vehicle, i.e., the agent itself, from all observed vehicles. Based on this matching reward function, the agent is trained using the SAC reinforcement learning algorithm, and the hybrid evaluation pool Q required for the next round of sub-target selection and training is obtained. φ And policy network π θ Through continuous iterative training, the agent gradually masters the complex individual tasks corresponding to each curriculum sub-goal.

[0029] S6. Train the intrinsic reward generator;

[0030] In the state space, the course sub-goals are represented by the local matching reward r. m Implicitly guide agents toward them. m Directly measuring the similarity between the current state and sub-goals ignores the difficulty of accessing different sub-goals. Therefore, an intrinsic reward model is needed to model the expert's specific preference information. Consider the reconstructed local reward function as r(s, a, s', g) = r m (s, a, s', g) + r ω (s, a, g), where r ω (s, a, g) are generated by the intrinsic reward generator. Its loss function is defined as:

[0031]

[0032] Where π θ This is the policy of the imitator agent. To update ω, a meta-learning method is used during the reinforcement learning process. Let ω be the policy evaluation process, where each evaluator network entity... Its parameter is φ i Through soft Bellman residual loss Optimize:

[0033]

[0034] in It is an experience replay pool. The update formula of the network is evaluated. The partial derivative of φ′ with respect to ω is defined as follows:

[0035]

[0036] Here, φ′ represents the parameters of the updated evaluator network entity in the mixed evaluator pool, and α1 is the learning rate of the evaluator network. The policy improvement step implicitly constructs the partial derivative of the updated actor θ′ with respect to the updated evaluator φ′. The actor parameters θ are optimized through policy improvement.

[0037]

[0038] Using the updated judge Q φ′ Update strategy π θ The dependency between the updated actor θ′ and the updated commentator φ′ can be represented as:

[0039]

[0040] Where α2 is the learning rate of the actor network. Finally, the gradient of the meta-imitation objective with respect to intrinsic reward can be calculated using the chain rule:

[0041]

[0042] Where ω′ is the updated intrinsic reward model, and α3 is the learning rate of the intrinsic reward. This meta-learning optimization method captures how local rewards influence the hidden machine of agent learning; it also models expert preferences and encourages agent policies to more closely resemble expert policies.

[0043] A system for implementing the autonomous driving lane selection decision-making method based on inverse reinforcement learning as described in this invention is characterized by comprising:

[0044] The autonomous driving lane selection decision expert data generation module is used to construct an expert dataset for autonomous driving lane selection decisions.

[0045] The module for constructing a reinforcement learning model for autonomous driving lane selection decision is used to build a reinforcement learning model for autonomous driving lane selection decision.

[0046] The intrinsic reward generator building block is used to construct intrinsic reward generator models;

[0047] The curriculum sub-goal selection module is used to iteratively select curriculum sub-goals.

[0048] The matching reward function construction module is used to construct matching reward functions based on curriculum sub-goals;

[0049] The intrinsic reward generator training module is used to train the intrinsic reward generator.

[0050] The autonomous driving lane selection decision-making method based on inverse reinforcement learning firstly constructs an expert policy using a planning algorithm and simulates it in a highway-env environment to obtain an expert dataset. Then, based on the actor-judge network architecture of reinforcement learning, the local reward function is reconstructed: during reinforcement learning, an extended hybrid judge network pool is used to measure the agent's uncertainty about the state and select a curricular sub-objective; based on the sub-objective, a locally matching reward function is reconstructed to guide the agent to act according to the expert data. A local intrinsic reward generator based on meta-mimicking objective optimization guides the agent to explore near the expert data, inferring the expert's intent from the perspective of the reward function. This invention also includes a system for autonomous driving lane selection decision-making based on inverse reinforcement learning.

[0051] This invention explicitly decomposes a complex single task into several low-level sub-tasks. The selected sub-objectives are used as temporary task abstractions on the expert trajectory, providing a series of simplified sub-tasks for agent learning. This allows the agent to interpret the expert trajectory in stages based on its current policy capabilities and further recover the different local reward functions at each stage. The local reward functions at different stages are more compact and precise than a single global reward function, and can effectively evaluate the agent's behavior based on its continuously improving capabilities. Therefore, the agent can use the allocated local rewards to progressively learn a final policy that matches the expert demonstration. In addition, the generated curriculum-based sub-objectives also have a certain degree of interpretability and can effectively represent the agent's capabilities. At the same time, the intrinsic reward function based on meta-imitation learning can also effectively infer and interpret the expert's preference information for state actions.

[0052] The advantages of this invention are: sub-objectives enable the agent to make decisions by rapidly imitating the corresponding parts of the expert driving route trajectory in stages according to its own strategic capabilities, which greatly improves the learning speed and learning ability of the autonomous driving lane selection decision agent; the intrinsic reward based on meta-imitation learning is used to reason and analyze the expert vehicle's lane selection behavior, which greatly improves the stability and overall performance of lane selection decision learning; the selection of sub-objectives also reasonably reflects the decision-making ability at different stages of the scenario, which can reasonably explain the confidence level of the agent's lane selection behavior output and present it to the user for reference, making it convenient for the user to supervise and improving the user experience of autonomous driving. Attached Figure Description

[0053] Figure 1 This is a schematic diagram of the autonomous driving lane selection decision-making process based on inverse reinforcement learning of the present invention.

[0054] Figure 2 This is a schematic diagram illustrating the principle of the curriculum sub-objectives of this invention. Detailed Implementation

[0055] The technical solution of the present invention will be clearly and completely explained and described below.

[0056] Example 1

[0057] Reference Figure 1 , Figure 2 An autonomous driving lane selection decision-making method based on inverse reinforcement learning includes the following steps:

[0058] S1. Construct expert data for autonomous driving lane selection decisions;

[0059] Based on the autonomous driving simulation environment highway-env, an expert autonomous vehicle agent is constructed using optimal planning methods for deterministic systems. Simulations are performed in various environments within highway-env, and data on interactions with the environment is recorded. Each interaction requires recording information including the current environment state, the action taken by the expert agent, the next environment state, and whether the task is completed. K data trajectories are recorded for each environment, and the corresponding data is saved to form an expert dataset. In this invention, K is set to 25.

[0060] S2. Construct a reinforcement learning model for autonomous driving lane selection decision-making;

[0061] An actor-evaluator architecture is used to build a reinforcement learning model for autonomous driving lane selection decisions, in which the actor π θ The evaluator is a hybrid evaluation pool used to generate the lane selection decision action corresponding to the current road condition. The system comprises n evaluator network entities used to calculate the value of the current road condition and lane selection decision actions. In this invention, n = 5. The evaluator serves two purposes: firstly, it reduces the high variance of the actor's gradient estimation while ensuring that its bias remains constant; secondly, the use of multiple network entities enables the evaluation of the agent's uncertainty.

[0062] S3. Construct an intrinsic reward generator model;

[0063] A multilayer perceptron model is used to construct an intrinsic reward generator model for the environment. The intrinsic reward generator r ω The intrinsic reward function is used to specify the current road condition. The main purpose of this intrinsic reward function is to guide the agent to explore near expert trajectories, thereby enabling better imitation and learning.

[0064] S4. Iterate through the selection of sub-goals for curriculum development;

[0065] First, assume that the curriculum sub-goal g belongs to the expert trajectory set. And select a sub-objective based on the current decision uncertainty of the agent. In the current state s, agent g selects a sub-objective based on its policy π. θ If (·|s) is chosen as action a, then the decision uncertainty is... The calculation method is as follows:

[0066] U(s)=μ(Q φ (s, a)), (1)

[0067] Where μ is the operation of calculating the variance of the outputs of different evaluation network entities in the mixed evaluation pool.

[0068] To select course sub-objectives of appropriate difficulty, we specify an expert trajectory in the expert trajectory dataset. To analyze and further decouple the task, in this invention, e = 1. At the beginning of each training round, we assume that the agent has learned the policy π for achieving the previous sub-goal g. θ And the evaluation network Q φ Each expert state is input into the agent policy network π. θ and Value Network Q φ The trajectory uncertainty U(τ) can be obtained. e ):

[0069]

[0070] Next, the uncertainty sequence U(τ) of the entire trajectory is scanned sequentially from front to back. e The goal is to identify states with high uncertainty. If the uncertainty of a state-action pair and its subsequent state-action pairs is significantly higher than the uncertainty of its preceding state-action pairs, then that state g is designated as the curriculum sub-objective for the next training phase. A soft average of the uncertainty values ​​is maintained throughout this process. The soft average at position p is denoted as... The value was initially set to In scanned Updated regularly:

[0071]

[0072] Where σ is the soft update coefficient, which is taken as 0.2 in this invention. During the scanning process, the first one that satisfies... The state of the condition The sub-target g is detected as the next training round, where δ is the uncertainty threshold, which is taken as δ = 5.0 in this invention. Once the threshold δ is exceeded, it is selected as the sub-target for the next iteration. If there is no state uncertainty that meets the conditions during this process, training can be stopped because U(τ) e When the uncertainty of all elements in the process has been reduced to an acceptable range, it can be considered that the agent has mastered the complete capabilities required to complete the entire complex task.

[0073] S5. Construct a matching reward function based on the curriculum sub-objectives.

[0074] Constructing a local reward function based on reward reshaping theory:

[0075] r m (s,a,s′,g)=ψ(s′,g)-γψ(s,g), (4)

[0076] In this process, the agent takes action a at state s and dynamically reaches state s′ based on the environment. γ is a discount factor, which is taken as 0.99 in this invention. ψ is a state evaluation function that can be expressed as follows:

[0077] ψ(s,g)=||h(s)-h(g)||, (5)

[0078] Where h is the state representation function. The representation function h extracts the features of the current autonomous vehicle, i.e., the agent itself, from all observed vehicles. Based on this matching reward function, the agent is trained using the SAC reinforcement learning algorithm, and the hybrid evaluation pool Q required for the next round of sub-target selection and training is obtained. φ And policy network π θ Through continuous iterative training, the agent gradually masters the complex individual tasks corresponding to each curriculum sub-goal.

[0079] S6. Train the intrinsic reward generator;

[0080] In the state space, the course sub-goals are represented by the local matching reward r. m Implicitly guide agents toward them. m Directly measuring the similarity between the current state and sub-goals ignores the difficulty of accessing different sub-goals. Therefore, an intrinsic reward model is needed to model the expert's specific preference information. Consider the reconstructed local reward function as r(s, a, s', g) = r m (s, a, s', g) + r ω (s, a, g), where r ω (s, a, g) are generated by the intrinsic reward generator. Its loss function is defined as:

[0081]

[0082] Where π θ This is the policy of the imitator agent. To update ω, a meta-learning method is used during the reinforcement learning process. Let ω be the policy evaluation process, where each evaluator network entity... Its parameter is φ i Through soft Bellman residual loss Optimize:

[0083]

[0084] in It is an experience replay pool. The update formula of the network is evaluated. The partial derivative of φ′ with respect to ω is defined as follows:

[0085]

[0086] Where φ′ represents the parameters of the updated evaluator network entity in the mixed evaluator pool, and α1 is the learning rate of the evaluator network, which is taken as α1 = 0.0003 in this invention. The policy improvement step implicitly constructs the partial derivative of the updated actor θ′ with respect to the updated evaluator φ′. The actor parameters θ are optimized through policy improvement.

[0087]

[0088] Using the updated judge Q φ′ Update strategy π θ The dependency between the updated actor θ′ and the updated commentator φ′ can be represented as:

[0089]

[0090] Where α2 is the learning rate of the actor network, which is taken as α2 = 0.0003 in this invention. Finally, the gradient of the meta-imitation target with respect to the intrinsic reward can be calculated using the chain rule:

[0091]

[0092] Where ω′ is the updated intrinsic reward model, and α3 is the learning rate of the intrinsic reward, which is taken as α3 = 0.001 in this invention. This meta-learning optimization method captures the hidden machine of how local rewards affect agent learning; it also models expert preferences and makes the agent policy closer to the expert policy.

[0093] Example 2

[0094] The system implementing the autonomous driving lane selection decision-making method based on inverse reinforcement learning in Example 1 includes:

[0095] The autonomous driving lane selection decision expert data generation module is used to construct an expert dataset for autonomous driving lane selection decisions.

[0096] The module for constructing a reinforcement learning model for autonomous driving lane selection decision is used to build a reinforcement learning model for autonomous driving lane selection decision.

[0097] The intrinsic reward generator building block is used to construct intrinsic reward generator models;

[0098] The curriculum sub-goal selection module is used to iteratively select curriculum sub-goals.

[0099] The matching reward function construction module is used to construct matching reward functions based on curriculum sub-goals;

[0100] The intrinsic reward generator training module is used to train the intrinsic reward generator.

[0101] The above modules, in sequence, contain steps 1 to 6 of the method of Embodiment 1.

[0102] The embodiments described in this specification are merely examples of implementations of the inventive concept. The scope of protection of this invention should not be considered as limited to the specific forms described in the embodiments. The scope of protection of this invention also extends to equivalent technical means that can be conceived by those skilled in the art based on the inventive concept.

Claims

1. An autonomous driving lane selection decision-making method based on inverse reinforcement learning, comprising the following steps: Step 1. Construct an expert dataset for autonomous driving lane selection decisions; Based on the autonomous driving simulation environment highway-env, an expert autonomous vehicle agent is constructed using optimal planning methods for deterministic systems. Simulations are performed in various environments within highway-env, and data on interactions with the environment is recorded. Each interaction requires recording information including the current environment state, the action taken by the expert agent, the next environment state, and whether the task is completed. Records are generated for each environment. After each data trajectory is recorded, the corresponding data is saved to form an expert dataset. ; Step 2. Construct a reinforcement learning model for autonomous driving lane selection decisions; An actor-evaluator architecture is used to build a reinforcement learning model for autonomous driving lane selection decisions. In this architecture, the actor... The evaluator is a hybrid evaluation pool used to generate the lane selection decision action corresponding to the current road condition. ,include Each network entity is used to calculate the value of the current road condition and the lane selection decision action. The role of the judge is, on the one hand, to reduce the high variance of the gradient estimation of the agent while ensuring that its bias remains unchanged; on the other hand, using multiple network entities can be used to evaluate the uncertainty of the agent. Step 3. Construct the intrinsic reward generator model; A multilayer perceptron model is used to construct an intrinsic reward generator model for the environment. The intrinsic reward function is used to determine the current road condition. The main purpose of the intrinsic reward function is to guide the agent to explore near the expert's trajectory, thereby enabling better imitation and learning. Step 4. Iteratively select sub-goals for the curriculum; First, let's assume the sub-goal of curriculum development. Belongs to the expert trajectory set And select a sub-objective based on the current decision uncertainty of the agent; in the current state At that time, the agent g acts according to its policy. Select Action Then the decision uncertainty The calculation method is as follows: in This is an operation that calculates the variance of the outputs of different evaluation network entities in a mixed evaluation pool; To select course sub-objectives of appropriate difficulty, an expert trajectory is specified in the expert trajectory dataset. This is used to analyze and further decouple the task; at the beginning of each training round, it is assumed that the agent has learned about achieving the previous sub-objectives. strategy and evaluation network Each expert state is input into the agent policy network. and value network It is possible to obtain trajectory uncertainty. : Next, the uncertainty sequence of the entire trajectory is scanned sequentially from front to back. The goal is to identify states with high uncertainty values; if the uncertainty values ​​of a state-action pair and its subsequent state-action pair are significantly higher than the uncertainty values ​​of its preceding state-action pair, then that state is designated. The curriculum sub-objectives for the next stage of training; maintaining the soft average of the uncertainty values ​​during the modification process; at the position The soft average is denoted as The value was initially set to In the scan Updated regularly: in It is a soft update coefficient; during the scanning process, the first one to satisfy... The state of the condition Detected as a sub-target for the next round of training ,in It is an uncertainty threshold; once the threshold is exceeded... If the condition is not met during this process, then it is selected as the sub-objective for the next iteration; if there is no state uncertainty that meets the conditions during this process, then training can be stopped because... When the uncertainty of all elements has been reduced to an acceptable range, it is believed that the agent has mastered the complete capabilities required to complete the entire complex task. Step 5. Construct a matching reward function based on the curriculum sub-goals; Constructing a local reward function based on reward reshaping theory: in, The agent is in state Take action And reach a state dynamically based on the environment. , It is a discount factor. It is a state evaluation function that can be represented as follows: in It is a state representation function; representation function Extract the features of the current autonomous vehicle, i.e., the agent itself, from the state of all observed vehicles. Based on this matching reward function, the agent is trained using the SAC reinforcement learning algorithm, and the hybrid evaluation pool required for the next round of sub-target selection and training is obtained. and policy network Through continuous iterative training, the agent gradually masters the complex individual tasks corresponding to each course-specific sub-goal. Step 6. Train the intrinsic reward generator; In the state space, the course sub-goals use locally matched rewards. Implicitly guide agents toward them; Directly measuring the similarity between the current state and sub-goals ignores the difficulty of accessing different sub-goals. Therefore, an intrinsic reward model is needed to model the expert's specific preference information. Considering the reconstructed local reward function as follows: ,in Generated by an internal reward generator; Its loss function is defined as: in It is the strategy of the imitator agent; in order to achieve updates It employs meta-learning methods to learn during the reinforcement learning process; In the strategy evaluation process, each network entity is evaluated. Its parameters are ; Through soft Bellman residual loss Optimize: in It is an experience replay pool; By evaluating the network's update formula Defined about Partial derivatives: in These are the parameters of the updated judging network entities in the mixed judging pool. It is used to evaluate the learning rate of a network; The strategy improvement steps implicitly construct an update actor. Regarding the updated judge Partial derivatives; actuator parameters Optimize through strategy improvement : Use the updated judges Update strategy Updated Actions and updated critics The dependency relationship between them can be represented as: in It is the learning rate of the actor network; finally, the gradient of the meta-imitation target with respect to intrinsic reward can be calculated using the chain rule: in It is an updated intrinsic reward model. The learning rate is the intrinsic reward; this meta-learning optimization method captures the hidden mechanism of how local rewards affect agent learning; it also models expert preferences and prompts agent policies to be closer to expert policies.

2. A system for implementing the autonomous driving lane selection decision-making method based on inverse reinforcement learning as described in claim 1, characterized in that, include: The autonomous driving lane selection decision expert data generation module is used to construct an expert dataset for autonomous driving lane selection decisions. The module for constructing a reinforcement learning model for autonomous driving lane selection decision is used to build a reinforcement learning model for autonomous driving lane selection decision. The intrinsic reward generator building block is used to construct intrinsic reward generator models; The curriculum sub-goal selection module is used to iteratively select curriculum sub-goals. The matching reward function construction module is used to construct matching reward functions based on curriculum sub-goals; The intrinsic reward generator training module is used to train the intrinsic reward generator.

Images