Improved imitation learning control method based on Markov chain

Abstract

The invention discloses an improved imitation learning control method based on a Markov chain, and relates to the technical field of autonomous control, and the method comprises the following steps: 1, collecting and marking automatic driving videos in a plurality of scenes, and constructing an automatic driving data set; wherein the annotation of the automatic driving data comprises a given strategy, a state and an action; 2, calculating a state transition probability according to the automatic driving data; 3, constructing a reward function according to the state transition probability; 4, calculating value estimation according to the state transition probability and a given strategy; 5, collecting real-time automatic driving data, updating a given strategy in combination with value estimation, a state transition probability and a reward function, if the value estimation meets a termination condition, taking the current given strategy as an optimal strategy, and otherwise, returning to the step 2; and step 6, generating a decision action according to the optimal strategy. The learning efficiency and the autonomous control performance are improved by introducing a strategy iteration method of sudden obstacles and a reward function design.

Description

Technical Field

[0001] The present invention relates to the field of autonomous control technology, and more specifically to an improved imitation learning control method based on Markov chain. Background Art

[0002] With the continuous development of artificial intelligence technology, imitation learning, as an important machine learning method, has achieved remarkable results in many fields. Imitation learning, also known as demonstration learning or apprenticeship learning, is based on the core idea of ​​learning how to perform specific tasks by observing the behavior of experts. This method has shown great potential in many fields, such as autonomous driving, robot control, game intelligence, etc.

[0003] In imitation learning, Markov chain is a key concept. Markov chain is a mathematical model that describes the state transition of a system. It assumes that the future state of the system depends only on the current state and has nothing to do with the past state. This property makes Markov chain very useful in describing and simulating various random processes. In imitation learning, Markov chain can be used to model the state transition process of an agent, thereby helping the agent learn how to transition from the current state to the target state.

[0004] However, existing imitation learning algorithms still have some limitations when dealing with complex tasks. For example, in the case of autonomous driving control in urban areas with strong constraints, accurate modeling models can be used to deal with mobile scenarios. Even if other pedestrians or vehicles that do not follow traffic rules are encountered, they can react quickly and form a solution strategy. However, in natural environments, such as deserts, Gobi and other special environments, and even on the surface of the moon, traditional algorithms will face the problem of insufficient training or insufficient constraints, resulting in control failure.

[0005] Therefore, how to improve the autonomous control performance in complex environments is an urgent problem that technical personnel in this field need to solve. Summary of the invention

[0006] In view of this, the present invention provides an improved imitation learning control method based on Markov chain, which aims to improve the performance of autonomous driving in complex environments. By introducing a strategy iteration method and reward function design for sudden obstacles, it copes with the difficulties of autonomous driving in natural environments, imitates the control solution strategy of people in the loop, realizes an autonomous control management closed loop, and improves learning efficiency and autonomous control performance.

[0007] In order to achieve the above object, the present invention adopts the following technical solution:

[0008] An improved imitation learning control method based on Markov chain includes the following steps:

[0009] Step 1: Collect and annotate autonomous driving videos in various scenarios to build an autonomous driving dataset. The annotation content of the autonomous driving data in the autonomous driving dataset includes given strategies, states, and actions.

[0010] Step 2: Calculate the state transition probability based on the autonomous driving data;

[0011] Step 3: Construct a reward function based on the state transition probability;

[0012] Step 4: Calculate the value estimate based on the state transition probability and the given strategy;

[0013] Step 5: Update the given strategy based on the value estimate, state transition probability and reward function. If the value estimate meets the termination condition, the current given strategy is used as the optimal strategy, otherwise return to step 2;

[0014] Step 6: Collect real-time autonomous driving data and generate decision actions based on the optimal strategy.

[0015] Preferably, in step 1, each video frame of the autonomous driving video is labeled with a given strategy, state and action; the state includes driving state, energy state, ground support state and vehicle health state, the driving state includes speed, acceleration, pitch angle, azimuth angle, degree of bumpiness, etc., the energy state includes battery power, storage state, etc.; the ground support state includes communication link stability, communication bandwidth, etc.; the vehicle health state includes fault mode, fault level, fault location, etc.; the state transition condition for the current state s of the given strategy to transfer to the next state s' indicates that the current state is changed to the next state based on the action, including surrounding environment information, state and corresponding one or more actions, the surrounding environment information includes obstacle position and terrain slope, etc.

[0016] Preferably, a state set S is constructed based on the statistical state of the autonomous driving data, an action set A is constructed based on the statistical action, and the state transition probability is calculated by a counting method, which is expressed as:

[0017]

[0018] Among them, P(s'|s,a) represents the probability of taking action a in the current state s and transitioning to the next state s'; N(s,a,s') represents the number of times action a is taken in the current state s and transitions to the next state s'; N(s,a) represents the total number of times action a is taken in the current state s.

[0019] Preferably, the reward function is expressed as:

[0020] R(s,a)=-logP(a|s)

[0021] Among them, R(s,a) represents the reward for taking action a in the current state s; P(a|s) is the probability of taking action a in the current state s, P(a|s) = P(s′,s,a) / [P(s′|s,a)P(s)], P(s′,s,a) is the probability that the current state s and the next state s′ and action a are simultaneously established, and P(s) is the probability of entering the current state s.

[0022] Preferably, the value function is obtained by iterating the Bellman equation and the value estimate is calculated, which is expressed as:

[0023]

[0024] in, Represents the estimated value of the current state s at the k+1th iteration; represents the value estimate of the current state s at the kth iteration; γ is the discount factor; R(s,a) represents the reward for taking action a in the current state s; P(s'|s,a) represents the probability of taking action a in the current state s to transfer to the next state s'; π represents a given strategy; A represents an action set; S represents a state set;

[0025] Setting the Initial Value Estimate M s Represents the number of all states in the state set, L a Indicates the number of all actions in the action set.

[0026] Preferably, the expression for updating the strategy in step 5 is:

[0027]

[0028] Among them, π k+1 (a|s) represents the updated strategy for the k+1th iteration, represents the value estimate of the current state s at the kth iteration; γ is the discount factor; R(s,a) represents the reward for taking action a in the current state s; P(s'|s,a) represents the probability of taking action a in the current state s to transfer to the next state s'; π represents a given strategy; A represents an action set; S represents a state set.

[0029] Preferably, the termination condition is expressed as:

[0030]

[0031] Among them, δ represents the set threshold; Represents the estimated value of the current state s at the kth iteration; It represents the estimated value of the current state s at the k+1th iteration; S represents the state set.

[0032] It can be seen from the above technical solutions that, compared with the prior art, the present invention discloses an improved imitation learning control method based on Markov chain, which first uses the examples of experts to train an initial strategy, then uses this strategy to generate new examples, and uses these examples to update the strategy. This process is iterated continuously until the strategy converges, and by learning a strategy, the behavior of the agent in the environment is made as similar as possible to the behavior of the expert. The present invention improves the learning efficiency and autonomous control performance by introducing strategy iteration and reward function design with sudden obstacles; has better generalization ability, and by learning state transition probability and reward function, the designed algorithm can better generalize to unseen states and actions; has higher data efficiency, and the algorithm can more effectively use limited historical data to learn strategies; has good convergence, and the performance is stable at different numbers of iterations. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.

[0034] Figure 1 A schematic diagram of the structure of the improved imitation learning control method based on Markov chain provided by the present invention;

[0035] Figure 2 Characteristic schematic diagrams of different scenarios provided by the present invention;

[0036] Figure 3 A schematic diagram for comparing key indicators of different algorithms provided by the present invention;

[0037] Figure 4 This is a schematic diagram for comparing key indicators under different numbers of iterations provided by the present invention. DETAILED DESCRIPTION

[0038] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0039] The embodiment of the present invention discloses an improved imitation learning control method based on Markov chain, comprising the following steps:

[0040] S1: Collect and annotate autonomous driving videos in various scenarios to build an autonomous driving dataset. The annotation content of autonomous driving data in the autonomous driving dataset includes given strategies, states, and actions.

[0041] S2: Calculate the state transition probability based on the autonomous driving data;

[0042] S3: Construct a reward function based on the state transition probability;

[0043] S4: Calculate value estimates based on state transition probabilities and given strategies;

[0044] S5: Collect real-time autonomous driving data, and update the given strategy based on the value estimation, state transition probability and reward function. If the value estimation meets the termination condition, the current given strategy is used as the optimal strategy, otherwise return to S2;

[0045] S6: Generate decision actions based on the optimal strategy.

[0046] Furthermore, S1 annotates each video frame of the autonomous driving video with a given strategy, state and action; the state includes driving state, energy state, ground support state and vehicle health state; the driving state includes speed, acceleration, pitch angle, azimuth angle and degree of bumpiness; the energy state includes battery power and storage state; the ground support state includes communication link stability and communication bandwidth; the vehicle health state includes fault mode, fault level and fault location; the given strategy is a state transition condition for the current state s to transfer to the next state s', indicating that the current state is changed to the next state according to the action, including surrounding environment information, state and corresponding one or more actions, and the surrounding environment information includes obstacle position and terrain slope, etc.

[0047] Furthermore, the state set S is constructed based on the statistical state of the autonomous driving data, and the action set A is constructed based on the statistical action. The state transition probability is calculated by the counting method, which is expressed as:

[0048]

[0049] Among them, P(s'|s,a) represents the probability of taking action a in the current state s and transferring to the next state s'; N(s,a,s') represents the number of times taking action a in the current state s and transferring to the next state s', and N(s,a) represents the total number of times taking action a in the current state s. The current state and the next state identify the operating state of the vehicle during the autonomous driving process, including forward driving, acceleration, deceleration, reverse driving, left turn, right turn, parking, etc., which can be expanded to high-speed forward driving, medium-speed forward driving, and low-speed forward driving according to the driving speed, and can be expanded to small-angle left / right turn and large-angle left / right turn according to the steering angle.

[0050] Furthermore, the reward function is expressed as:

[0051] R(s,a)=-logP(a|s)

[0052] Among them, R(s,a) represents the reward for taking action a in the current state s; P(a|s) is the probability of taking action a in the current state s, P(a|s)=P(s′,s,a) / [P(s′|s,a)P(s)], P(s′,s,a) is the probability that the current state s and the next state s' and action a are simultaneously true, and P(s) is the probability of entering the current state s. The goal of the reward function is to minimize the difference between the agent's behavior and the expert's behavior. The probability is obtained by counting the number of times. For example, the screen fails to display 100 times in history, the parking action switches the forward driving state to the parking state 32 times in history, the deceleration action switches the forward driving state to the deceleration state 27 times in history, the left turn action switches the left turn state to the small-angle left turn state 2 times, the parking action switches the reverse driving state to the parking state 5 times, and no operation is performed 34 times in history. Different P(a|s) statistical data can be calculated based on the above historical data statistics.

[0053] Furthermore, the value function is obtained by iterating the Bellman equation and the value estimate is calculated, which is expressed as:

[0054]

[0055] in, Represents the estimated value of the current state s at the k+1th iteration; represents the value estimate of the current state s at the kth iteration; γ is the discount factor; R(s,a) represents the reward for taking action a in the current state s; P(s'|s,a) represents the probability of taking action a in the current state s to transfer to the next state s'; π represents a given strategy; A represents an action set; S represents a state set; set the initial value estimate M s Represents the number of all states in the state set, L a Represents the number of all actions in the action set. The value function represents the expected return that can be obtained by following the strategy π starting from the current state s; the initial value estimate indicates that the state transition value under different strategies is the same.

[0056] Furthermore, the expression for the strategy update in S5 is:

[0057]

[0058] Among them, π k+1(a|s) represents the updated strategy for the k+1th iteration, represents the estimated value of the current state s at the kth iteration; γ is the discount factor; R(s,a) represents the reward for taking action a in the current state s; P(s'|s,a) represents the probability of taking action a in the current state s to transfer to the next state s'; π represents a given strategy; A represents an action set; S represents a state set. The update of the strategy ensures that the strategy is optimal at each step, that is, it follows a greedy strategy.

[0059] Furthermore, the termination condition is expressed as:

[0060]

[0061] Among them, δ represents the set threshold; Represents the estimated value of the current state s at the kth iteration; Represents the value estimate of the current state s at the k+1th iteration; S represents the state set. The termination condition is based on the convergence of the value function. Whether to stop the iteration is determined by comparing the difference in the value estimates of two consecutive iterations. The threshold δ corresponding to the difference is generally a relatively small positive number, which is used to control the accuracy of the iteration and ensure the stability of the strategy evaluation. That is, the change in the value estimate is very small. At this time, it can be considered that the strategy has converged to the optimal solution. In order to balance the computational cost and the degree of optimization of the strategy, a reasonable δ value is usually set. If δ is set too small, it may cause unnecessary computational overhead; if it is set too large, it may cause the strategy to fail to fully converge.

[0062] In a specific embodiment, in order to verify the effectiveness of the method proposed in the present invention in the field of autonomous driving, an experiment was conducted on an autonomous driving dataset containing a simulated lunar environment. The experimental steps include:

[0063] S1: Collect data sets; use the autonomous driving data set collected in a simulated lunar environment (based on the lunar simulator software) to conduct experiments. This data set is generated by simulating the driving environment on the lunar surface. It aims to provide training and test data for autonomous driving systems such as lunar rovers and to be used for verification. The characteristics of different scenarios are as follows: Figure 2 As shown;

[0064] The main features of the dataset are as follows:

[0065] Data volume: The dataset contains more than 20 hours of driving videos, covering a variety of simulated lunar driving scenarios and conditions;

[0066] Scene diversity: The dataset includes various terrains simulating the lunar surface, such as plains, crater areas, mountainous areas, and rugged roads;

[0067] Annotation information: Each video frame is annotated with the vehicle’s speed, direction, acceleration, and information about the surrounding environment, such as the location of obstacles and the slope of the terrain.

[0068] This dataset is very suitable for training and testing autonomous driving systems that simulate the lunar environment, because it provides a wealth of simulated lunar driving scenarios, which can help the algorithm better learn and adapt to the complex lunar environment;

[0069] S2: Experimental setup;

[0070] Training and testing split: The dataset is divided into training and testing sets, where 80% of the data is used for training and 20% of the data is used for testing;

[0071] Evaluation indicators: Accuracy and mean squared error (MSE) are used as evaluation indicators;

[0072] Comparison algorithm: The improved imitation learning control method of the present invention is compared with the following existing comparison algorithms, including behavior cloning18 (Behavior Cloning), inverse reinforcement learning19 (Inverse Reinforcement Learning, IRL), and generative adversarial imitation learning20 (Generative Adversarial Imitation Learning, GAIL).

[0073] S3: Comparative experiment;

[0074] like Figure 3 The figure shows a comparison of key indicators of different algorithms, comparing the performance of the method of the present invention and the comparison algorithm on a simulated autonomous driving data set, mainly comparing the accuracy and mean square error (MSE). Figure 3 (a) is the accuracy comparison, Figure 3 (b) is the comparison of mean square error (MSE). Through the statistical comparison after the test, it can be seen that the method of the present invention has higher generalization ability in the lunar simulation environment, can better extract key features from the training data, and can apply these features in new and different scenarios, thereby improving the overall accuracy. Due to the use of the state transition probability method, the feature extraction method can learn richer and more discriminative features from the input data, thereby achieving higher accuracy in classification or decision-making tasks. At the same time, compared with other methods, it has stronger robustness to noisy data and outliers, which means that even in the case of poor data quality in an unstructured lunar environment, it can maintain a high accuracy.

[0075] like Figure 4The figure shows the comparison of key indicators of the method of the present invention under different numbers of iterations, and compares the performance of the method of the present invention under different numbers of iterations. Figure 4 (a) is the accuracy comparison, Figure 4 (b) is the comparison of mean square error (MSE). It can be seen that the accuracy index can reach a balance after 140 iterations, while the MSE needs to reach 340 times. Considering that the control cycle is in milliseconds and the convergence time does not exceed 0.5 seconds, it fully demonstrates the effectiveness of the present invention and can achieve safe driving of autonomous driving on the lunar surface.

[0076] The experimental results show that on the lunar environment autonomous driving dataset, the method of the present invention outperforms the comparison algorithm in terms of accuracy and mean square error. This shows that the improved imitation learning algorithm based on Markov decision has better generalization ability and data efficiency when dealing with natural environment autonomous driving tasks. In addition, the performance of the algorithm is stable at different numbers of iterations, indicating good convergence. Specifically, the accuracy of the algorithm on the autonomous driving dataset reached 95%, and the mean square error was 0.02, while the accuracy of the comparison algorithm was up to 90%, and the mean square error was 0.05, which shows that the present invention can more accurately learn and imitate the driving behavior of experts.

[0077] In this specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part.

[0078] The above description of the disclosed embodiments enables one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. An improved imitation learning control method based on Markov chain, characterized in that: The following steps are involved: Step 1: Collect and annotate autonomous driving videos in various scenarios to build an autonomous driving dataset. The annotation content of the autonomous driving data in the autonomous driving dataset includes given strategies, states, and actions. Step 2: Calculate the state transition probability based on the autonomous driving data; Step 3: Construct a reward function based on the state transition probability; Step 4: Calculate the value estimate based on the state transition probability and the given strategy; Step 5: Collect real-time autonomous driving data, and update the given strategy based on the value estimation, state transition probability, and reward function. If the value estimation meets the termination condition, the current given strategy is used as the optimal strategy, otherwise return to step 2. Step 6: Generate decision actions based on the optimal strategy.

2. The improved imitation learning control method based on Markov chain according to claim 1, characterized in that: In step 1, each video frame of the autonomous driving video is labeled with a given strategy, state, and action; the state includes driving state, energy state, ground support state, and vehicle health state; the driving state includes speed, acceleration, pitch angle, azimuth angle, and bumpiness; the energy state includes battery power and storage state; The ground support status includes the stability of the communication link and the communication bandwidth; the vehicle health status includes the fault mode, fault level and fault location; the given strategy is the state transition condition for the current state s to transfer to the next state s', which means changing the current state to the next state according to the action.

3. The improved imitation learning control method based on Markov chain according to claim 1, characterized in that: According to the statistics of the state of the autonomous driving data, the state set S is constructed, and the action set A is constructed by the statistics of the actions. The state transition probability is calculated by the counting method, which is expressed as: Among them, P(s'|s,a) represents the probability of taking action a in the current state s and transitioning to the next state s'; N(s,a,s') represents the number of times action a is taken in the current state s and transitions to the next state s'; N(s,a) represents the total number of times action a is taken in the current state s.

4. The improved imitation learning control method based on Markov chain according to claim 3 is characterized in that: The reward function is expressed as: R(s,a)=-logP(a|s) Among them, R(s,a) represents the reward for taking action a in the current state s; P(a|s) is the probability of taking action a in the current state s, P(a|s) = P(s′,s,a) / [P(s′|s,a)P(s)], P(s′,s,a) is the probability that the current state s and the next state s′ and action a are simultaneously established, and P(s) is the probability of entering the current state s.

5. The improved imitation learning control method based on Markov chain according to claim 4, characterized in that: The value function is obtained by iterating the Bellman equation and the value estimate is calculated, which is expressed as: in, Represents the estimated value of the current state s at the k+1th iteration; represents the value estimate of the current state s at the kth iteration; γ is the discount factor; π represents the given strategy; Setting the Initial Value Estimate M s Represents the number of all states in the state set, L a Indicates the number of all actions in the action set.

6. The improved imitation learning control method based on Markov chain according to claim 5, characterized in that: The expression for the policy update in step 5 is: Among them, π k+1 (a|s) represents the updated strategy for the k+1th iteration.

7. The improved imitation learning control method based on Markov chain according to claim 5, characterized in that: The termination condition is expressed as: Among them, δ represents the set threshold; Represents the estimated value of the current state s at the kth iteration; It represents the estimated value of the current state s at the k+1th iteration; S represents the state set.

Images