A path coverage method based on a retrieval enhancement mechanism

Abstract

The present invention proposes a path coverage method based on a retrieval enhancement mechanism. The method includes: obtaining a pre-collected data set, constructing a policy model based on the Decision Transformer architecture, regarding the robot as an agent, using the policy model to predict the action distribution of the agent to obtain the action distribution predicted by the policy model at the current moment, based on the pre-collected data set, sequentially processing the input trajectory through the FH mechanism and the pre-trained language model to obtain a set of key-value pairs, retrieving similar sub-trajectories from the set of key-value pairs based on the current input sub-trajectory to obtain a set of retrieved similar sub-trajectories. By introducing an external memory bank and adding a similar trajectory retrieval mechanism, the present invention can quickly construct a path planning strategy in the initial stage of training the model, effectively reduce the effect of redundant coverage path planning, significantly reduce the computational resources consumed in training, and improve the decision-making ability of the model in a sparse reward environment.

Description

Technical Field

[0001] The present invention relates to the fields of deep reinforcement learning and coverage path planning, and particularly relates to a path coverage method based on a retrieval enhancement mechanism. Background Art

[0002] The coverage path planning technology has a wide range of application scenarios in many production and life fields. In the service industry: autonomous cleaning, weed removal, snow shoveling, etc. In agriculture: pesticide spraying, automatic reclamation, automatic farming, etc. In environmental exploration tasks: map surveying, interstellar exploration, oil and other mineral surveys. In similar application fields, the goal of the coverage path planning algorithm is to efficiently and comprehensively plan a path so that the mobile robot can effectively and without dead ends cover the specified area to perform tasks. An efficient coverage path planning algorithm can improve the operation efficiency and reduce labor costs and resource waste. With the development and innovation of artificial intelligence and robot technology, the coverage path planning algorithm is constantly being optimized to adapt to more complex environments and diverse application requirements. However, the traditional reinforcement learning-based methods still have some limitations:

[0003] First, the sample utilization rate is low. Existing agents based on Deep Reinforcement Learning (DRL) often require a large amount of interaction data to learn a suitable strategy, and the data sampling cost in reinforcement learning is high, especially in practical applications, it is difficult to collect training data on a large scale.

[0004] Second, the environmental rewards are sparse. In the coverage path planning task, the reward signal is usually sparse, that is, the agent can only obtain effective feedback after completing a large-scale coverage, resulting in low learning efficiency and easy to fall into local optima.

[0005] Finally, the long episode tasks lead to difficult model convergence. Many reinforcement learning tasks require the agent to perform thousands of steps of interaction to complete the task, and most traditional reinforcement learning methods (such as DQN, PPO, SAC) are difficult to effectively handle ultra-long time step tasks. In addition, although Transformer, as a powerful sequence modeling tool, can process long sequence data, its computational cost is high, and its application in reinforcement learning tasks is limited. Summary of the Invention

[0006] In view of the above situation, the main purpose of the present invention is to propose a path coverage method based on a retrieval enhancement mechanism to solve the above technical problems.

[0007] The present invention proposes a path coverage method based on a retrieval enhancement mechanism, and the method includes the following steps:

[0008] Step 1: Obtain the pre-collected dataset, construct an external memory bank based on the pre-collected dataset, construct a policy model based on the Decision Transformer architecture, regard the robot as an agent, and use the policy model to predict the action distribution of the agent to obtain the action distribution predicted by the policy model at the current moment;

[0009] Step 2: Based on the pre-collected dataset, sequentially process the input trajectory through the FH mechanism and the pre-trained language model to obtain a set of key-value pairs, and save the set of key-value pairs to the external memory bank;

[0010] Step 3: Obtain the current input sub-trajectory, retrieve similar sub-trajectories from the set of key-value pairs based on the current input sub-trajectory to obtain a set of retrieved similar sub-trajectories;

[0011] Step 4: Based on the set of retrieved similar sub-trajectories, re-weight the similar sub-trajectories to obtain a sorted set of re-weighted sub-trajectories;

[0012] Step 5: Based on the current input sub-trajectory and the sorted set of re-weighted sub-trajectories, adjust the action distribution predicted by the policy model at the current moment to obtain the final action distribution predicted by the policy model, and use the final action distribution predicted by the policy model to optimize the policy model to obtain an optimized policy model;

[0013] Use the optimized policy model to confirm the recommendation rate of each action executed by the agent.

[0014] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0015] 1. By introducing an external memory bank and adding a similar trajectory retrieval mechanism, the present invention quickly constructs a path planning strategy at the initial stage of training the model, effectively reduces the effect of redundant coverage path planning, significantly reduces the computational resources consumed during training, and improves the decision-making ability of the model in a sparse reward environment;

[0016] 2. By introducing an external memory bank and adding a similar trajectory retrieval mechanism, and learning the retrieved similar paths, the present invention realizes the pertinence of training samples, further removes redundant paths, enables the coverage path planning model to converge earlier under limited computing resources, improves the performance of the coverage path planning, and enhances the coverage ability of the model in a complex environment;

[0017] 3. By designing and improving the external memory mechanism, the present invention can store the trajectory data that has existed in the historical trajectory, significantly reducing the demand of the model for the amount of training data. When performing the coverage path planning task, query the data highly relevant to the current task in the stored historical data, improving the utilization rate of the data, effectively alleviating the problem of the model's dependence on the context length, and making the robot complete the coverage path planning task more efficiently.

[0018] Additional aspects and advantages of the present invention will be given in part in the following description, will become apparent in part from the following description, or will be understood through the embodiments of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 is a flowchart of the steps of a path coverage method based on a retrieval enhancement mechanism proposed by the present invention;

[0020] Figure 2 is a schematic diagram of the method framework of a path coverage method based on a retrieval enhancement mechanism proposed by the present invention Figure 1 ;

[0021] Figure 3 is a schematic diagram of the method framework of a path coverage method based on a retrieval enhancement mechanism proposed by the present invention Figure 2 . DETAILED DESCRIPTION OF THE EMBODIMENTS

[0022] The embodiments of the present invention will be described in detail below. Examples of the embodiments are shown in the drawings, where the same or similar reference numerals indicate the same or similar elements or elements having the same or similar functions throughout. The embodiments described below by referring to the drawings are exemplary and are only used to explain the present invention and should not be construed as a limitation of the present invention.

[0023] Referring to the following description and drawings, these and other aspects of the embodiments of the present invention will be clear. In these descriptions and drawings, some specific embodiments of the embodiments of the present invention are specifically disclosed to represent some ways of implementing the principles of the embodiments of the present invention, but it should be understood that the scope of the embodiments of the present invention is not limited thereto.

[0024] Please refer to Figure 1 , this embodiment provides a path coverage method based on a retrieval enhancement mechanism, and the method includes the following steps:

[0025] Step 1: Obtain the pre-collected data set, construct an external memory bank based on the pre-collected data set, construct a policy model based on the Decision Transformer architecture, regard the robot as an agent, and use the policy model to predict the action distribution of the agent to obtain the action distribution predicted by the policy model at the current moment.

[0026] See also Figure 2 In step 1, we obtain the pre-collected dataset, build an external memory based on the pre-collected dataset, and build a policy model based on the Decision Transformer architecture. The following relationship exists in the corresponding process:

[0027] ;

[0028] in, represents a pre-collected dataset, Indicates the Tracks, represents the index of the trajectory, represents the total number of trajectories, Represents groups 0 to The group's state, actions, rewards, and returns, represents the length of the trajectory;

[0029] Considering the robot as an intelligent agent, the policy model is used to predict the action distribution of the intelligent agent, and the action distribution predicted by the policy model at the current moment is obtained. The corresponding process has the following relationship:

[0030] ;

[0031] in, Indicates the action at the current moment. represents the reinforcement learning strategy at the current moment, Indicates the current moment, represents a sequence of states, represents the return sequence, Represents a sequence of actions, represents the reward sequence, represents the context length and .

[0032] It should be noted that the context length is used to define the sub-trajectory range. Figure 2 middle, Indicates the current moment action at each moment; the nearest neighbor retrieval algorithm is a parameter-free machine learning method based on similarity. Its core idea follows the inductive principle that "neighboring samples have similar properties". Given a query vector and a data set, the goal of the nearest neighbor retrieval algorithm is to find the required number of samples in the data set that are closest to the query vector, and form these samples into a neighborhood set.

[0033] Step 2: Based on the pre-collected dataset, the input trajectory is processed by the FH mechanism and the pre-trained language model in sequence to obtain a set of key-value pairs, and the key-value pair set is saved in the external memory.

[0034] In step 2, based on the pre-collected dataset, the input trajectory is successively processed by the FH mechanism and the pre-trained language model to obtain a set of key-value pairs, and the set of key-value pairs is saved in the external memory bank, which specifically includes the following sub-steps:

[0035] Based on the pre-collected dataset, use the FH mechanism to transform all sub-trajectories from time to to obtain the transformed sub-trajectories from time to During the corresponding process, there are the following relational expressions:

[0036] ;

[0037] Among them, represents all sub-trajectories from the transformed time to ; represents being processed by the FH mechanism, represents all sub-trajectories from time to ; represents the embedding matrix, represents the transpose, represents the scaling factor, represents the random matrix, represents being processed by the softmax activation function;

[0038] Use the pre-trained language model to extract features from all sub-trajectories from the transformed time to to obtain the embedding vectors of all sub-trajectories from time to During the corresponding process, there are the following relational expressions:

[0039] ;

[0040] Among them, represents the embedding vectors of all sub-trajectories from time to ; represents being successively processed by the FH mechanism and the pre-trained language model, represents feature extraction by the pre-trained language model;

[0041] Based on the embedding vectors of all sub-trajectories from time to and the embedding vectors of all sub-trajectories from time to Construct all sub-trajectories to obtain a set of key-value pairs, and save the set of key-value pairs to an external memory bank. The following relational expressions exist during the corresponding process:

[0042] ;

[0043] Among them, represents the set of key-value pairs, represents the key set, represents the value set, represents from time to all sub-trajectories.

[0044] Step 3: Obtain the current input sub-trajectory, and retrieve similar sub-trajectories from the set of key-value pairs based on the current input sub-trajectory to obtain a set of retrieved similar sub-trajectories.

[0045] In Step 3, obtain the current input sub-trajectory, and retrieve similar sub-trajectories from the set of key-value pairs based on the current input sub-trajectory to obtain a set of retrieved similar sub-trajectories, which specifically includes the following sub-steps:

[0046] Obtain the current input sub-trajectory, and process the current input sub-trajectory through the FH mechanism and the pre-trained language model in sequence to obtain a query vector. The following relational expressions exist during the corresponding process:

[0047] ;

[0048] Among them, represents the query vector, represents the current input sub-trajectory;

[0049] Based on the set of key-value pairs, calculate the similarity between the key in each key-value pair and the query vector to obtain a set of retrieved similar sub-trajectories. The following relational expressions exist during the corresponding process:

[0050] ;

[0051] Among them, represents the set of retrieved similar sub-trajectories, represents after similarity calculation, represents the key, that is, the embedding vector of each sub-trajectory in the external memory index library.

[0052] Step 4: Based on the set of retrieved similar sub-trajectories, re-weight the similar sub-trajectories to obtain a set of re-weighted sub-trajectories sorted.

[0053] Please refer to Figure 3, in step 4, based on the set of retrieved similar sub-trajectories, re-weight the similar sub-trajectories to obtain a sorted set of re-weighted sub-trajectories, specifically including the following sub-steps:

[0054] Based on the key-value pair set, calculate the cosine similarity between the query vector and the key to obtain the correlation score. There is the following relational expression in the corresponding process:

[0055] ;

[0056] Among them, represents the correlation score, represents after cosine similarity calculation;

[0057] In the training stage, calculate the similarity between the current input sub-trajectory and the previous relevant sub-trajectories to obtain the similarity between the input trajectory and the similar task. There is the following relational expression in the corresponding process:

[0058] ;

[0059] Among them, represents the similarity between the input trajectory and the similar task, represents the previous relevant sub-trajectories, represents after processing by the task index function, represents after processing by the indication function, that is, output 1 when the condition is satisfied, otherwise 0;

[0060] In the inference stage, the policy model re-weights based on the retrieved sub-trajectories to obtain the magnitude of the reward in the input sub-trajectory and the similar sub-trajectories. There is the following relational expression in the corresponding process:

[0061] ;

[0062] Among them, represents the magnitude of the reward in the input sub-trajectory and the similar sub-trajectories, represents the reward for each step in the trajectory;

[0063] Re-weight and fuse the correlation score and the similarity between the input trajectory and the similar task to obtain the final correlation score. There is the following relational expression in the corresponding process:

[0064] ;

[0065] Among them, represents the final correlation score, represents the weight, represents the sub-trajectory utility score adjusted by the weight ;

[0066] Based on the final relevance scores, reorder the sub-trajectories in the set of retrieved similar sub-trajectories to obtain a set of reordered re-weighted sub-trajectories. The following relational expressions exist in the corresponding process:

[0067] ;

[0068] Among them, represents the set of reordered re-weighted sub-trajectories, represents the top sub-trajectories sorted by similarity, represents the operation of sorting by relevance magnitude.

[0069] It should be noted that in Figure 3 , represents the state at the current moment, represents the reward at the current moment; in the present invention, by designing and improving the external memory mechanism, the historical selections and historical reward situations of the agent in the task are recorded for subsequent retrieval and speculation in similar tasks; during the training process of the agent, by retrieving the state space similar to the current situation in the external memory mechanism, the past training information is fully utilized, and finally, combined with the situation of the current task, the decision-making ability of the agent in a sparse reward environment is improved, thereby solving problems such as low sample utilization rate and difficult model convergence faced by traditional reinforcement learning methods.

[0070] Step 5: Based on the current input sub-trajectory and the set of reordered re-weighted sub-trajectories, adjust the action distribution predicted by the policy model at the current moment to obtain the final predicted action distribution of the policy model. Use the final predicted action distribution of the policy model to optimize the policy model to obtain an optimized policy model;

[0071] Use the optimized policy model to confirm the recommendation rates for the agent to execute each action.

[0072] In Step 5, based on the current input sub-trajectory and the set of reordered re-weighted sub-trajectories, adjust the action distribution predicted by the policy model at the current moment to obtain the final predicted action distribution of the policy model. The following relational expressions exist in the corresponding process:

[0073] ;

[0074] Among them, represents the final predicted action distribution of the policy model, represents the reinforcement learning policy.

[0075] It should be noted that in each layer of the self-attention layer of the policy model of the present invention, a cross-attention layer is added to fuse the retrieved sub-trajectory information, avoiding the computational bottleneck of traditional Transformers in processing long sequences; the retrieved sub-trajectories are encoded through an independent embedding layer, and different embedding layers are used for each token type (state, dynamics, reward, and return). The encoded sub-trajectories are passed to the cross-attention layer.

[0076] It should be understood that although the steps in the flowcharts of the various embodiments of the present invention are shown sequentially as indicated by the arrows, these steps do not necessarily have to be executed in the order indicated by the arrows. Unless otherwise clearly stated in this document, there is no strict order restriction for the execution of these steps, and these steps can be executed in other orders. Moreover, at least some of the steps in each embodiment may include multiple sub-steps or multiple stages. These sub-steps or stages do not necessarily have to be executed at the same time, but can be executed at different times. The execution order of these sub-steps or stages does not necessarily have to be sequential either, but can be executed alternately or in rotation with at least a part of other steps or sub-steps or stages of other steps.

[0077] It should be understood that each part of the present invention can be implemented by hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, any one or a combination of the following techniques well known in the art can be used: discrete logic circuits with logic gate circuits for implementing logical functions on data signals, application-specific integrated circuits with appropriate combinational logic gate circuits, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0078] In the description of this specification, the description with reference to terms such as "one embodiment", "some embodiments", "example", "specific example", or "some examples" means that the specific features, structures, materials, or characteristics described in connection with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials, or characteristics described can be combined in a suitable manner in any one or more embodiments or examples.

[0079] The above-described embodiments merely represent several implementation manners of the present invention. The description thereof is relatively specific and detailed, but it should not be construed as a limitation on the scope of the patent for the present invention. It should be noted that for those of ordinary skill in the art, without departing from the concept of the present invention, several modifications and improvements can still be made, and these all fall within the protection scope of the present invention. Therefore, the protection scope of the patent for the present invention shall be subject to the appended claims.

Claims

1. A path coverage method based on a retrieval enhancement mechanism, characterized in that The method includes the following steps: Step 1: Obtain a pre-collected dataset, construct an external memory bank based on the pre-collected dataset, construct a policy model based on the Decision Transformer architecture, regard the robot as an agent, and use the policy model to predict the action distribution of the agent to obtain the action distribution predicted by the policy model at the current moment; Step 2: Based on the pre-collected dataset, sequentially process the input trajectory through the FH mechanism and the pre-trained language model to obtain a set of key-value pairs, and save the set of key-value pairs to the external memory bank; Step 3: Obtain the current input sub-trajectory, and retrieve similar sub-trajectories from the set of key-value pairs based on the current input sub-trajectory to obtain a set of retrieved similar sub-trajectories; Step 4: Based on the set of retrieved similar sub-trajectories, re-weight the similar sub-trajectories to obtain a set of re-weighted sub-trajectories after sorting; Step 5: Based on the current input sub-trajectory and the set of re-weighted sub-trajectories after sorting, adjust the action distribution predicted by the policy model at the current moment to obtain the finally predicted action distribution of the policy model, and use the finally predicted action distribution of the policy model to optimize the policy model to obtain an optimized policy model; Use the optimized policy model to confirm the recommendation rate of each action executed by the agent.

2. The path coverage method based on a retrieval enhancement mechanism according to claim 1, wherein In Step 1, when obtaining the pre-collected dataset, constructing the external memory bank based on the pre-collected dataset, and constructing the policy model based on the Decision Transformer architecture, the following relational expressions exist in the corresponding process: ； Among them, represents a pre-collected dataset, represents the th trajectory, represents the index of the trajectory, represents the total number of trajectories, respectively represent the state, action, reward, and return from the 0th group to the th group, represents the length of the trajectory; In the step of regarding the robot as an agent, using the policy model to predict the action distribution of the agent, and obtaining the action distribution predicted by the policy model at the current moment, the following relational expressions exist in the corresponding process: ； Among them, represents the action at the current moment, represents the reinforcement learning policy at the current moment, represents the current moment, represents the state sequence, represents the reward sequence, represents the action sequence, represents the reward sequence, represents the context length.

3. The path coverage method based on the retrieval enhancement mechanism according to claim 2, wherein In Step 2, based on the pre-collected dataset, sequentially process the input trajectory through the FH mechanism and the pre-trained language model to obtain a set of key-value pairs, and save the set of key-value pairs to the external memory bank, which specifically includes the following sub-steps: Based on a pre-collected dataset, use the FH mechanism to transform all sub-trajectories from time to to obtain all transformed sub-trajectories from time to ; Use a pre-trained language model to extract features from all sub-trajectories from the converted starting time to to obtain the embedding vectors of all sub-trajectories from the starting time to ; Based on the embedding vectors of all sub-trajectories from time to and all sub-trajectories from time to are constructed to obtain a set of key-value pairs, and the set of key-value pairs is saved in an external memory bank.

4. The path coverage method based on a retrieval enhancement mechanism according to claim 3, wherein Based on the pre-collected data set, use the FH mechanism to transform all sub-trajectories from time to , and obtain all sub-trajectories from time to after transformation. There are the following relational expressions in the corresponding process: ； Among them, represents all sub-trajectories from the converted moment to ; represents being processed by the FH mechanism, represents all sub-trajectories from the moment to ; represents the embedding matrix, represents the transpose, represents the scaling factor, represents the random matrix, represents being processed by the softmax activation function; In the step of extracting features from all sub-trajectories from the converted starting time to to obtain the embedding vectors of all sub-trajectories from the starting time to in the corresponding process, there is the following relational expression: ； Among them, represents the embedding vectors of all sub-trajectories from time to ; represents being processed successively through the FH mechanism and the pre-trained language model, represents feature extraction through the pre-trained language model. In the step of constructing a set of key-value pairs based on the embedding vectors of all sub-trajectories from time to and all sub-trajectories from time to and saving the set of key-value pairs to an external memory bank, there is the following relational expression in the corresponding process: ； Among them, represents a set of key-value pairs, represents a set of keys, represents a set of values, represents all sub-trajectories from time to ​ 5. The path coverage method based on a retrieval enhancement mechanism according to claim 4, wherein In Step 3, when obtaining the current input sub-trajectory and retrieving similar sub-trajectories from the set of key-value pairs based on the current input sub-trajectory to obtain a set of retrieved similar sub-trajectories, it specifically includes the following sub-steps: Obtain the current input sub-trajectory, and sequentially process the current input sub-trajectory through the FH mechanism and the pre-trained language model to obtain a query vector; Based on the set of key-value pairs, calculate the similarity between the key in each key-value pair and the query vector to obtain a set of retrieved similar sub-trajectories.

6. The path coverage method based on a retrieval enhancement mechanism according to claim 5, wherein When obtaining the current input sub-trajectory and sequentially processing the current input sub-trajectory through the FH mechanism and the pre-trained language model to obtain a query vector, the following relational expressions exist in the corresponding process: ； Among them, represents the query vector, represents the current input sub-trajectory; In the step of calculating the similarity between the key in each key-value pair and the query vector based on the set of key-value pairs to obtain a set of retrieved similar sub-trajectories, the following relational expressions exist in the corresponding process: ； Among them, represents the set of retrieved similar sub-trajectories, indicates that after similarity calculation, represents a key.

7. The path coverage method based on the retrieval enhancement mechanism according to claim 6, wherein In Step 4, based on the set of retrieved similar sub-trajectories, re-weight the similar sub-trajectories to obtain a set of re-weighted sub-trajectories after sorting, which specifically includes the following sub-steps: Based on the key-value pair set, calculate the cosine similarity between the query vector and the key to obtain the correlation score; In the training stage, the current input sub-trajectory is compared with the previous relevant sub-trajectories to calculate the similarity between the input trajectory and the similar tasks; In the inference stage, the policy model re-weights based on the retrieved sub-trajectories to obtain the magnitudes of the rewards in the input sub-trajectories and the similar sub-trajectories; Re-weight and fuse the correlation score with the similarity between the input trajectory and the similar tasks to obtain the final correlation score; Based on the final correlation score, perform a re-ordering operation on the sub-trajectories in the set of retrieved similar sub-trajectories to obtain the set of re-weighted sub-trajectories after sorting; 8. The path coverage method based on a retrieval enhancement mechanism according to claim 7, wherein Based on the key-value pair set, calculate the cosine similarity between the query vector and the key to obtain the correlation score. There is the following relational expression in the corresponding process: ； Among them, represents the relevance score, indicating that it is calculated by cosine similarity; In the training phase, when calculating the similarity between the current input sub-trajectory and the previous relevant sub-trajectories to obtain the similarity between the input trajectory and the similar tasks, the following relational expressions exist in the corresponding process: ； Among them, represents the similarity between the input trajectory and the similar tasks, represents the first related sub-trajectories, represents being processed by the task index function, represents being processed by the indication function; In the step where, in the inference stage, the policy model re-weights based on the retrieved sub-trajectories to obtain the magnitudes of the rewards in the input sub-trajectories and the similar sub-trajectories, there is the following relational expression in the corresponding process: ； Among them, represents the magnitude of the rewards in the input sub-trajectory and the similar sub-trajectory, represents the reward for each step in the trajectory; In the step of re-weighting and fusing the correlation score with the similarity between the input trajectory and the similar tasks to obtain the final correlation score, there is the following relational expression in the corresponding process: ； Among them, represents the final relevance score, represents the weight, represents the sub-trajectory utility score adjusted by the weight ; In the step of, based on the final correlation score, performing a re-ordering operation on the sub-trajectories in the set of retrieved similar sub-trajectories to obtain the set of re-weighted sub-trajectories after sorting, there is the following relational expression in the corresponding process: ； Among them, represents the set after re-weighted sub-trajectory sorting, represents the top sub-trajectories sorted by similarity, represents the operation of sorting by the magnitude of relevance.

9. The path coverage method based on a retrieval enhancement mechanism according to claim 8, wherein In step 5, based on the current input sub-trajectory and the set of re-weighted sub-trajectories after sorting, adjust the action distribution predicted by the policy model at the current moment to obtain the final predicted action distribution of the policy model. There is the following relational expression in the corresponding process: ； Among them, represents the action distribution finally predicted by the policy model, represents the reinforcement learning policy.

Images