A random exploration method for intelligent agents based on flight strategy

Abstract

The present invention discloses a random exploration method for an intelligent agent based on a flight strategy, which includes collecting the current information of the intelligent agent through the world environment, processing the image or vector information, obtaining the quantified position information of the robot, the obstacle distribution information, and the given destination information, obtaining the difference from the destination through data quantitative analysis, taking the position of the intelligent agent as the state s of the intelligent agent, and the movement direction of the intelligent agent as the action a; and realizing the movement control of the intelligent agent according to the movement strategy for reaching the destination determined by the learning process of the intelligent agent.

Description

Technical Field

[0001] The present invention relates to the field of intelligent body motion control, and more specifically, to an intelligent body random exploration method based on flight strategy. Background Art

[0002] With the progress of deep reinforcement learning agent research, proposing more effective exploration methods has become an important key to agent exploration.

[0003] For example, Chinese invention patent application CN201910549586.X discloses an efficient exploration method based on reinforcement learning, which discloses an efficient exploration method for reinforcement learning, and the steps of the method are as follows: 1) pre-training count estimation function; 2) using the pre-trained count estimation function to perform efficient exploration of reinforcement learning. The present invention mainly focuses on the balance problem between exploration and utilization in reinforcement learning. In continuous space tasks, the number of occurrences of the state encountered by the agent is estimated by pre-training count estimation function, and the reward is calculated using the number of occurrences of the state. The reward is used to guide the agent to explore those states that are less encountered, thereby achieving efficient exploration. The present invention avoids the influence of the reward signal on the agent's action strategy by using an independent exploration strategy to process the reward signal, making the exploration process more stable.

[0004] Chinese invention patent application CN201810071032.9 discloses a method for exploring the discretization of high-dimensional continuous action space in reinforcement learning tasks. The continuous action space is converted into a discrete action space through quantization operations, and then the dictionary values ​​in the discrete action space are encoded and counted through the autoencoder implemented by the deep neural network. The number of occurrences of the encoding codewords corresponding to each dictionary value in a certain number of strategy updates is counted, and the dictionary values ​​that rarely appear are removed from the action dictionary according to probability, thereby continuously removing the redundancy in the action dictionary, thereby improving the search efficiency when the intelligent agent strategy is updated.

[0005] Deep reinforcement learning is a method for solving goal-oriented sequential decision-making problems. It is inspired by the behaviorist theory in psychology, which is how organisms gradually form expectations of stimuli under the stimulation of rewards or punishments given by the environment, and produce habitual behaviors that can obtain maximum benefits. This method is universal, so it has been studied in many other fields, such as game theory, control theory, operations research, information theory, simulation optimization methods, multi-agent system learning, swarm intelligence, statistics, and genetic algorithms. In recent years, as an important research hotspot in the field of machine learning, DRL has been widely used in game games, robot control, simulation and other fields, and has gradually become an important branch of machine learning.

[0006] In most reinforcement learning tasks, the true value function of the environment is difficult to obtain, and its distribution is nonlinear and conforms to a multi-peak distribution. Therefore, the agent is very likely to fall into one of the peaks during training and converge to a local optimum. In some tasks, this problem directly leads to the failure of the agent's training. Therefore, alleviating the exploration problem of the agent in a multi-peak environment is one of the challenges of reinforcement learning.

[0007] The balance between exploration and exploitation is an important challenge in deep reinforcement learning. Although exploitation is crucial for agents to obtain higher immediate rewards, agents can only obtain higher cumulative rewards through sufficient exploration. The value function of reinforcement learning tasks conforms to a multi-peak distribution. Under this premise, agents can only learn better strategies by exploring more peaks. However, increasing exploration means longer convergence time. Under existing problems and exploration strategies, deep reinforcement learning algorithms require very high costs to explore most peaks. ε-greedy is a simple and general exploration strategy that has a certain probability of taking random actions and takes the actions output by the agent strategy in other cases. The advantage of this exploration strategy is that it is simple to implement and is very effective in tasks with relatively simple environments and good reward settings. However, for tasks with large environmental space dimensions, the ε-greedy strategy cannot be effectively explored, that is, the agent cannot cross more value function peaks when interacting with the environment, which will cause the agent strategy to converge to a local optimum. Therefore, finding a more effective exploration strategy is the primary task in solving multi-dimensional state spaces. Summary of the invention

[0008] In order to solve at least one of the above technical problems, the present invention proposes a random exploration method of an intelligent agent based on a flight strategy.

[0009] The first aspect of the present invention provides a random exploration method of an intelligent agent based on a flight strategy, comprising the following steps:

[0010] S1, collects the current information of the agent through the world environment, processes the image or vector information, obtains the quantized position information of the agent (the current position of the robot: represented by a multi-dimensional matrix, the obstacles near the robot are represented by a multi-dimensional matrix), obstacle distribution information, and given destination information;

[0011] S2, through data quantitative analysis, obtain the difference from the destination, take the location of the agent as the state s of the agent, and the movement direction of the agent as the action a;

[0012] S3, select the corresponding current action a from the strategy network in the current state s, select the flight step Step according to the flight method, execute action a, get the corresponding reward r and migrate to the next state s', then select the next action a' corresponding to the next state s' according to the behavior strategy μ(a|s), and repeat the above process Step times;

[0013] S4, based on the agent evaluation function Q maxa′ (s t ,a′) obtain the value function size of the optimal action of the agent in the current state (the value function is expressed as the score of the agent in the current state), generate the path adjustment condition, and adjust the path according to the path adjustment condition;

[0014] S5, action a after adjusting according to the path fly , the experience (s t ,a t ,r t+1 ,s t+1 ,d) for storage, where d is the state termination judgment, which means that the agent cannot continue to perform actions in the environment according to the feedback given by the environment;

[0015] S6, sample training data from the experience pool storing experience, and obtain sample samples from the environment e = (s i ,a i ,r i+1 ,s i+1 ), calculate the predicted value y of the current state value function i ,

[0016] S7, based on the information learned in steps S3 to S6, iteratively update the policy network according to the loss function L t (θ t )=E[δ 2 ] calculates its gradient, the loss function is the predicted value y i The gradient represents the update direction of each parameter of the prediction network. It is used to converge the agent network through multiple gradient descents and update the parameters using the formula of the gradient descent method:

[0017] S8, based on the above S1-S7 intelligent agent learning process, determine the motion strategy to reach the destination, that is, the action value of the intelligent agent in each state (the action value is usually expressed as a discrete value), to achieve the motion control of the intelligent agent.

[0018] In a preferred embodiment of the present invention, the flight step length calculation formula in step S3 is as follows:

[0019] In a preferred embodiment of the present invention, the behavior strategy (the agent action value output by the network) μ(a|s) in step S3 adopts a completely random strategy in the initial state (i.e., when the agent is not trained), that is, the action selection is completely random and does not pass through the network output. The selection probabilities of all actions in any state are equal and the sum is 1. The purpose is to increase the exploration ability of the agent.

[0020] In a preferred embodiment of the present invention, in step S3, β is a symmetric Lévy stable distribution coefficient, where S max is the maximum flight step length, u and v are both sampled from the normal distribution:

[0021] Where Γ(1+β) is the Gamma function.

[0022] In a preferred embodiment of the present invention, in step S4, Q(s t ,a fly ) is the value function for maintaining the flight action, and Q maxa′ (s t ,a′) is the optimal value function in the current state, ρ is the flight threshold, the larger the ρ is, the smaller the tolerance of the current agent to the flight action is, and the more difficult it is to have a bad flight action: the smaller the ρ is, the greater the tolerance is, and the more likely it is to have a bad flight action.

[0023] In a preferred embodiment of the present invention, the value function is calculated by linear function approximation.

[0024]

[0025] In a preferred embodiment of the present invention, the intelligent body flight judgment is performed according to the path adjustment condition, and the path adjustment condition formula is as follows:

[0026] Q(s t ,a fly )≥Q maxa′ (s t ,a′)

[0027]

[0028] If any of the above formulas is satisfied, the path adjustment is performed, where Q(s t ,a fly ) is the value function for maintaining the flight action, and Q maxa′ (s t ,a′) is the optimal value function in the current state, ρ is the flight threshold, the larger the ρ is, the smaller the tolerance of the current agent to the flight action is, and the more difficult it is to have a bad flight action: the smaller the ρ is, the greater the tolerance is, and the more likely it is to have a bad flight action.

[0029] In a preferred embodiment of the present invention, in step S7, the parameter value is updated using a gradient descent method, and the update form is:

[0030]

[0031] The above formula uses the gradient descent method to update the parameter θ t .

[0032] The above technical solution of the present invention has the following advantages compared with the prior art:

[0033] 1. The present invention increases the ability of the agent to explore a larger area by transforming the exploration strategy from a one-step random walk to a multi-step random walk framework.

[0034] 2. The path adjustment method reduces the problem of repeated exploration of the agent, balances the relationship between exploration and utilization in reinforcement learning, and enables the agent to obtain higher rewards.

[0035] 3. The intelligent agent obtains the reward value of the current behavior through interaction with the environment. Starting from the perspective of walking step length, the intelligent agent uses the flight module to jump out of the 1-step random walking framework and solves many exploration problems that may arise when the intelligent agent flies in a complex environment. For example, the intelligent agent may fly to a known poor or dangerous state during flight, and the data validity will be reduced due to the truncation step length being too large. This exploration framework can explore a larger area and avoid the repeated area walking of the traditional exploration algorithm. On the premise of ensuring data validity, it improves the exploratory nature of the algorithm. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] In order to more clearly illustrate the specific implementation methods of the present invention or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, some of the drawings described below are some implementation methods of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0037] Figure 1 This is a flow chart of the robot control method in a preferred embodiment of the present invention. DETAILED DESCRIPTION

[0038] In order to more clearly understand the above-mentioned purpose, features and advantages of the present invention, the present invention is further described in detail below in conjunction with specific embodiments. It should be noted that the embodiments of the present application and the features in the embodiments can be combined with each other without conflict.

[0039] In the following description, many specific details are set forth to facilitate a full understanding of the present invention. However, the present invention may also be implemented in other ways different from those described herein. Therefore, the protection scope of the present invention is not limited to the specific embodiments disclosed below.

[0040] like Figure 1 As shown, the present invention provides an intelligent agent exploration method based on multi-step random walks, which enables the intelligent agent to interact with the environment through a world model, obtains the current state of the intelligent agent and the target of the intelligent agent through image information or feature matrix; obtains a map through analysis of video data, takes the position of the robot as the state s of the robot, and the movement direction of the robot as the action a; Figure 1 As shown, the specific method flight strategy and path adjustment.

[0041] The learning process includes the following steps:

[0042] (1) Collect the current information of the agent through the world environment, process the image or vector information, and obtain the quantized location information of the agent, obstacle distribution information, and given destination information;

[0043] (2) Through data quantitative analysis, the difference from the destination is obtained, and the location of the agent is taken as the state s of the agent, and the direction of movement of the agent is taken as the action a;

[0044] (3) Flight selection: Select the corresponding current action a from the policy network under the current state s, and select the flight step length Step according to the flight method. Execute action a, get the corresponding reward r and migrate to the next state s', then select the next action a' corresponding to the next state s' according to the behavior strategy μ(a|s), and repeat the above process Step times. Among them, the behavior strategy μ(a|s) uses a completely random strategy in the initial state, and the selection probability of all actions in any state is equal and the sum is 1;

[0045] (4) Path adjustment: Based on the agent evaluation function Q maxa′ (s t ,a′) to obtain the value function size of the optimal action of the agent in the current state. According to the path adjustment condition formula Q(s t ,a fly )≥Q maxa′ (s t ,a′) and Make flight judgment calls.

[0046] (5) Storage experience: Action a adjusted according to the path fly , the experience (s t ,a t ,r t+1 ,st+1 ,d), where d is the state termination judgment.

[0047] (6) Sampling training: Sampling training data from the experience pool stored in step (5), sampling sample e = (s i ,a i ,r i+1 ,s i+1 ) and calculate y i ,

[0048] (7) Update the policy network: Based on the information learned in steps (3) to (6), iteratively update the policy network according to the loss function L t (θ t )=E[δ 2 ] Calculate its gradient;

[0049] (8) Update parameters using gradient descent method:

[0050] (9) Realize the motion control of the intelligent agent based on the motion strategy to reach the destination determined by the intelligent agent learning process.

[0051] In the above technical solution, in step (2), the state value is discrete or continuous, and the action value is discrete. In step (3), the flight step selection is added, so that the original one-step random flight framework is transformed into a multi-step random walk framework, which increases the ability of the agent to explore a larger area. In step (4), path adjustment is used to avoid the problem of over-exploration of the agent. Path adjustment can greatly avoid the agent flying to a worse state or dangerous boundary under sufficient training. The path adjustment method can adjust the relationship between exploration and utilization in the algorithm. In step (7), time difference is used to evaluate whether the new strategy obtained by the current update is better or worse than the strategy before the update. Therefore, the time difference is defined as the difference between the value of the current state-action pair and the value of the state-action pair in the previous time step. If the difference is positive, it means that the current update is conducive to the development of the strategy in a better direction, and the update should be effectively utilized. On the contrary, if the difference is negative, it means that the current update makes the strategy develop in a bad direction, which is not conducive to the improvement of the strategy, so the update should be suppressed. In step (8), the method uses the training method of the neural network to approximate the true policy function, so the gradient descent method is used to gradually converge the policy.

[0052] In step (3), β is the symmetric Lévy stable distribution coefficient, where S max is the maximum flight step length, u and v are both sampled from the normal distribution: Where Γ(1+β) is the Gamma function, and the step size distribution follows the expected Lévy |s|≥|s0|, where s0 is the minimum step size, and s0=1 is set in the ALF exploration strategy. In reinforcement learning, action selection replaces the choice of flight direction, so the step size s does not have a negative case.

[0053] In step (4), Q(s t ,a fly ) is the value function to maintain the flight action, and Q maxa′ (s t ,a′) is the optimal value function in the current state, ρ is the flight threshold, the larger the ρ is, the smaller the tolerance of the current agent to the flight action is, and the more difficult it is to have a bad flight action: the smaller the ρ is, the greater the tolerance is, and the more likely it is to have a bad flight action.

[0054] The value function is calculated by linear function approximation.

[0055]

[0056] In step (8), the gradient descent method is used to update the parameter value, and the update form is,

[0057]

[0058] The above formula uses the gradient descent method to update the parameter θ t .

[0059] The intelligent agent obtains the reward value of the current behavior through interaction with the environment. Starting from the perspective of walking step length, the intelligent agent uses the flight module to jump out of the 1-step random walk framework, and solves many exploration problems that may arise when the intelligent agent flies in a complex environment, such as flying to a known poor or dangerous state during flight, and the reduction of data validity due to excessive truncation step length. This exploration framework can explore a larger area, avoiding the repeated area walking of the traditional exploration algorithm, and improving the exploratory nature of the algorithm while ensuring data validity.

[0060] The technical features of the above-described embodiments may be arbitrarily combined. To make the description concise, not all possible combinations of the technical features in the above-described embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0061] The above description of the disclosed embodiments enables one skilled in the art to implement or use the present invention. Various modifications to the above embodiments will be apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the above embodiments shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.

[0062] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any person skilled in the art can easily think of changes or substitutions within the technical scope disclosed by the present invention, which should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.

Claims

1. A random exploration method of an intelligent agent based on a flight strategy, characterized in that: The steps include: S1, collects the current information of the agent through the world environment, processes the image or vector information, and obtains the quantized location information of the agent, obstacle distribution information, and given destination information; S2, through data quantitative analysis, obtain the difference from the destination, take the location of the agent as the state s of the agent, and the movement direction of the agent as the action a; S3, select the corresponding current action a from the strategy network in the current state s, select the flight step Step according to the flight method, execute action a, get the corresponding reward r and migrate to the next state s', then select the next action a' corresponding to the next state s' according to the behavior strategy μ(a|s), and repeat the above process Step times; S4, based on the agent evaluation function Q max a′ (s t ,a′) obtain the value function size of the optimal action of the agent in the current state, generate the path adjustment condition, and adjust the path according to the path adjustment condition; S5, action a after adjusting according to the path fly , the experience (s t ,a t ,r t+1 ,s t+1 ,d) for storage, where d is the state termination judgment, which means that the agent cannot continue to perform actions in the environment according to the feedback given by the environment; S6, sample training data from the experience pool storing experience, and obtain sample samples from the environment e = (s i ,a i ,r i+1 ,s i+1 ), calculate the predicted value y of the current state value function i ; S7, based on the information learned in steps S3 to S6, iteratively update the policy network according to the loss function L t (θ t )=E[δ 2 ] calculates its gradient, the loss function is the predicted value y i The gradient represents the update direction of each parameter of the prediction network. It is used to converge the agent network through multiple gradient descents and update the parameters using the formula of the gradient descent method: S8, based on the above S1-S7 intelligent agent learning process, determine the motion strategy to reach the destination, that is, the action value of the intelligent agent in each state, to achieve the motion control of the intelligent agent.

2. The method of random exploration of intelligent agents based on flight strategy according to claim 1, characterized in that: The flight step length calculation formula in step S3 is as follows: β is the symmetric Lévy stable distribution coefficient, where S max is the maximum flight step length, u and v are both sampled from the normal distribution: Where Γ(1+β) is the Gamma function.

3. The method of random exploration of an intelligent agent based on flight strategy according to claim 1, characterized in that: In step S3, the behavior strategy μ(a|s) adopts a completely random strategy in the initial state, and the selection probabilities of all actions in any state are equal and the sum is 1.

4. The method of random exploration of intelligent agents based on flight strategy according to claim 1, characterized in that: The value function is used to determine the excellence of the current state of the agent. The value function is calculated using a linear function approximation.

5. The method of random exploration of intelligent agents based on flight strategy according to claim 1, characterized in that: The flight judgment of the intelligent body is carried out according to the path adjustment conditions. The formula of the path adjustment conditions is as follows: Q(s t ,a fly )≥Q max a′ (s t ,a′) If any of the above formulas is satisfied, the path adjustment is performed, where Q(s t ,a fly ) is the value function to maintain the flight action, and Q max a′ (s t ,a′) is the optimal value function in the current state, ρ is the flight threshold, the larger the ρ is, the smaller the tolerance of the current agent to the flight action is, and the more difficult it is to have a bad flight action: the smaller the ρ is, the greater the tolerance is, and the more likely it is to have a bad flight action.

Images