A navigation decision method based on an ARPPO model of attention and a recurrent neural network

Abstract

The application discloses a navigation decision method based on an ARPPO model of attention and a recurrent neural network, and adopts the following steps when the ARPPO model is constructed: step 1, a multi-core convolutional neural network module is constructed to perform preliminary feature extraction on environmental information; step 2, an attention module combined with an LSTM network is constructed to screen and compress feature information, and important and key information is acquired; and step 3, important historical information extracted through the model is input into an actor network to acquire an action and into a critic network to evaluate a value.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, specifically to an ARPPO model based on attention and recurrent neural networks. This invention is a divisional application of the invention patent entitled "A Navigation Decision-Making Method Based on Attention and Recurrent PPO" (application number 2023104559254). Background Technology

[0002] Path planning is an essential technology for robot navigation, aiming to enable robots to find the shortest, obstacle-free path in their environment with minimal cost. Traditional path planning techniques include A... Algorithms such as particle swarm optimization and simulated annealing are used. However, these algorithms are only applicable to path planning in fully observable environments. In reality, most environments are only partially observable, and there are challenging scenarios with incomplete state information and sparse rewards, which greatly increases the difficulty of exploration for intelligent agents.

[0003] Thanks to the advancements in deep learning and reinforcement learning technologies, many scholars have begun to consider combining the perception capabilities of deep learning with the decision-making abilities of reinforcement learning, providing solutions for path planning problems in complex environments. Patent application CN115469683A discloses a method and system for UAV formation flight based on the PPO algorithm. This method uses a deep reinforcement learning PPO algorithm to analyze the distance and direction information of surrounding objects collected by each UAV and then makes decisions, achieving autonomous formation flight training for UAVs. The trained UAVs can form formations, accurately avoid surrounding obstacles and other UAVs, and successfully reach the designated location. Patent application CN113255890A discloses a reinforcement learning agent training method based on the PPO algorithm. This method can analyze key environmental information and provide automatic model generation. However, none of the above methods can solve the exploration and obstacle avoidance problems in some observable environments, or can only solve some observable exploration problems less affected by field of view.

[0004] With the development of deep learning and reinforcement learning, the memory function of recurrent neural network algorithms can effectively solve exploration tasks in partially observable environments, and it also has a certain generalization ability, but it suffers from slow training speed. Similarly, the introduction of attention mechanisms can better extract important key information from the environment, and obtaining high-value information in partially observable environments undoubtedly improves the training speed. Recurrent neural networks and attention mechanisms have significant advantages in exploration tasks in partially observable environments.

[0005] Therefore, this patent proposes a navigation decision-making method based on attention and cyclic PPO. Summary of the Invention

[0006] The purpose of this invention is to address the technical problems of existing technologies, such as poor ability to analyze state information in partially observable environment exploration tasks, reliance on a large amount of state information data, and inability to solve relatively complex partially observable environment exploration tasks using only the PPO algorithm. The invention provides a navigation decision-making method based on attention and cyclic PPO.

[0007] To solve the above-mentioned technical problems, the technical solution adopted by the present invention is as follows: A navigation decision-making method based on attention and recurring point of interest (PPO) implementation includes the following steps: Step 1: The intelligent agent interacts with the environment to acquire environmental image data information; Step 2: Encode the acquired image information and compress the state information to improve training efficiency; Step 3: Construct a multi-kernel convolutional neural network module to perform preliminary feature extraction of environmental information; Step 4: Construct a network combining an attention module and LSTM to filter and compress feature information, obtaining important and crucial information; Step 5: Input the important historical information extracted by the model into the actor network to obtain actions, and input it into the critic network to evaluate the value; Step 6: The agent selects actions based on the probability matrix obtained from the actor network, interacts with the environment a certain number of times, and then stores the data in the experience pool. Step 7: Randomly select batch_size data for training, calculate the loss function, perform gradient clipping, and finally update the parameters through backpropagation.

[0008] Step 2 specifically includes the following sub-steps: Step 2-1) Perform overall encoding processing on the image information in the environmental data obtained in Step 1; Step 2-2) Design the encoding based on the color and state information of the target unit in the image; Steps 2-3) Integrate state information and compress the image data information into encoded information to improve training efficiency.

[0009] In step 3, a two-layer CNN convolutional network is constructed, using multiple convolutional kernels to perform feature processing on the encoded environmental information. The CNN convolutional network formula used is as follows:

[0010] This indicates the current environmental state characteristics. This represents the weight matrix of the filter. This represents the bias vector. It is an activation function.

[0011] Step 4 specifically includes the following sub-steps: Step 4-1) The features extracted by the convolutional network module are input into the attention network module. The multi-head attention network enables the model to focus on information from different subspaces at multiple locations. Finally, the information obtained from each space is concatenated, which can better increase the weight of important information. The formula for the multi-head attention network is:

[0012]

[0013]

[0014]

[0015]

[0016]

[0017] in the formula Represents the query matrix. Represents the key matrix. Represents value matrices, which are derived from the input eigenvectors. respectively with the corresponding weight matrix The result of multiplication, The attention formula is represented by a matrix. With matrix The result of the transposed multiplication divided by the matrix The square root of the dimension, then multiplied by the matrix. The result, This represents the activation function. Indicates the first Attention information of each head, Indicates the first Each head corresponds to The weight matrix, Indicates the first Each head corresponds to The weight matrix, Indicates the first Each head corresponds to The weight matrix, Indicates passage The join function integrates multi-head attention feature information that is important from each head. The matrix representing the linear transformation of the head attention instance; 4-2) Input the output of the attention module into the LSTM network; the LSTM neural network controls the flow and loss of information by introducing 3 gating structures and 1 long-term memory unit, and its calculation formula is as follows:

[0018]

[0019]

[0020]

[0021]

[0022] in express Forget the door information at all times express Enter door information at all times. express Output door information at all times. express Memory cells remember the state of cells in the previous time period, including their latent state. with sequence The input is fed into the network, simultaneously updating the hidden state and the state of memory cells. This represents the weight matrix corresponding to each gating structure. This represents the bias vector. and This represents the weight matrix within the memory cells. Represents the Sigmoid activation function. It represents the Hadamah accumulation. It is the hyperbolic tangent function.

[0023] In step 5, an Actor-Critic-based network layer is constructed. The Actor network uses a fully connected layer to compress the feature information and generate an action probability distribution matrix. The Critic network uses a fully connected layer to obtain the evaluation value of the current state. In step 6, the agent selects an action based on probability, inputs the selected action into the step function of the environment, obtains data such as the currently selected reward, and stores the obtained data into the experience pool after a certain number of interactions.

[0024] Step 7 includes the following sub-steps: Randomly select a batch size of sample data for training, and use importance sampling to evaluate the difference between the new and old strategies. The importance sampling formula is as follows:

[0025] Indicated in Take action in a state The old strategy after that, Indicates in Take action in a state The new strategy, through gradient pruning, limits the update magnitude of the policy parameters, as shown in the formula:

[0026]

[0027] in For hyperparameters, For the dominant function, This represents the cumulative reward value for taking action a in state s. ) is the state estimate, when the advantage function When this occurs, it indicates that the action is better than the average action, and the probability of selecting this action needs to be increased. In such cases, it is necessary to reduce the probability of selecting that action. Represents the expectation function, To find the minimum value function, The importance sampling ratio is obtained from the above formula. As a truncation function, the PPO algorithm is more conservative than other deep reinforcement learning algorithms, setting an upper limit of 0. The lower limit is The interval limit for policy parameter update ensures that the new policy is not significantly different from the old policy. The policy gradient parameter update formula is as follows:

[0028] The above-mentioned Both represent policy parameters, and the essence of updating policy parameters is to find the truncation loss function. The strategy parameter with the highest expected value To find the parameter-finding function that maximizes the function.

[0029] An ARPPO model based on attention and recurrent neural networks has the following structure: The sequence is as follows: First convolutional layer of the encoding feature extraction convolutional module → Second convolutional layer of the encoding feature extraction convolutional module → Multi-head attention layer of the encoding feature extraction attention module → First fully connected layer of the encoding feature extraction attention module → First dropout layer of the encoding feature extraction attention module → Second fully connected layer of the encoding feature extraction attention module → First normalized layer of the encoding feature extraction attention module → Second normalized layer of the encoding feature extraction attention module → Second dropout layer of the encoding feature extraction attention module → Third dropout layer of the encoding feature extraction attention module → LSTM layer of the encoding feature extraction recurrent neural network module. The LSTM layer of the recurrent neural network module for encoding feature extraction → the first fully connected layer of the policy selection Actor network module → the second fully connected layer of the policy selection Actor network module; The sequence is: LSTM layer of the recurrent neural network module for encoding feature extraction → first fully connected layer of the Critic network module for state value evaluation → second fully connected layer of the Critic network module for action value evaluation.

[0030] The model operates using the following steps: 1) Initial state information obtained from the environment is used to perform preliminary feature extraction on the image encoding information using a convolutional network module. Deep multidimensional information of the data is extracted through the first and second convolutional network layers. 2) The extracted deep multidimensional information is input into the attention module. First, it is input into a multi-head attention network to capture the correlation of information. Important and key information is extracted from the feature information at multiple different locations and concatenated. Then, feature processing is performed through the first fully connected layer and the first dropout layer, and some data is selected to be discarded to prevent overfitting. Next, the second fully connected layer realizes the residual connection of the previous fully connected layer. The first normalization layer and the second normalization layer are used to perform layer normalization to solve the problem of gradient stability between layers. Finally, some data is discarded through the second dropout layer and the third dropout layer to prevent overfitting. 3) Input the feature information output by the attention module into the LSTM layer of the recurrent neural network module. By introducing the LSTM network to extract the temporal characteristics of the data, the agent can form long-term memory during the exploration process. 4) Finally, the output features of the recurrent neural network module are input into the Actor network module and the Critic network module. The Actor network module integrates feature information through the first fully connected layer, and then obtains the actions taken by the agent and their probability distribution through the second fully connected layer. The Critic network module uses the first fully connected layer to integrate feature information, and finally uses the second fully connected layer to score the value of the agent's current state.

[0031] Compared with the prior art, the present invention has the following technical effects: 1) The deep reinforcement learning PPO algorithm based on the combination of attention network and recurrent neural network in this invention is helpful to enable the agent to acquire more valuable information, enable the agent to form long-term memory in partially observable environment, and has a fast training convergence speed, which helps guide the agent to quickly complete the mapless exploration task. 2) This invention establishes temporal dependencies for sample data by adding an LSTM network, while the introduction of an attention mechanism strengthens the dependencies between important and critical sample data over long distances, enabling the agent to form long-term memory and solve the dynamic and random POMDP exploration task. 3) This invention does not require relying on excessive sample data for decision-making. The agent retains memory information each round based on the time step, avoiding repeated invalid exploration actions, improving exploration efficiency, accelerating the convergence speed of the algorithm, and achieving precise obstacle avoidance and completion of exploration tasks through memory information. Attached Figure Description

[0032] The present invention will be further described below with reference to the accompanying drawings and embodiments: Figure 1 This is a schematic diagram of the ARPPO model based on attention and recurrent neural networks in this invention; Figure 2 This is a flowchart of the model training process in step 7 of the present invention. Detailed Implementation

[0033] A navigation decision-making method based on attention and recurring point of interest (PPO) includes the following steps: Step 1. The intelligent agent interacts with the environment to acquire environmental image data information; Step 2. Encode the acquired image information and compress the state information to improve training efficiency; Step 3. Construct a multi-kernel convolutional neural network module to perform preliminary feature extraction of environmental information; Step 4. Construct a network combining an attention module and LSTM to filter and compress feature information, obtaining important and crucial information; Step 5. Input the important historical information extracted by the model into the actor network to obtain actions, and input it into the critic network to evaluate the value; Step 6. The agent selects actions based on the probability matrix obtained from the actor network, interacts with the environment a certain number of times, and then stores the data in the experience pool.

[0034] Step 7. Randomly select batch_size data for training, calculate the loss function, perform gradient clipping, and finally update the parameters through backpropagation.

[0035] Step 2 specifically includes: 2.1 The image information in the environmental data obtained in step 1 is encoded as a whole.

[0036] 2.2 Design special detail encoding based on the unique information of a certain unit in the image.

[0037] 2.3 Integrate state information and compress image data into encoded information to improve training efficiency.

[0038] Step 3 specifically includes: Construct a two-layer CNN convolutional network, using multiple convolutional kernels to perform feature processing on the encoded environmental information. The CNN convolutional network formula is as follows:

[0039] This indicates the current environmental state characteristics. This represents the weight matrix of the filter. This represents the bias vector. It is an activation function.

[0040] Step 4 specifically includes: 4.1 The features extracted by the convolutional network module are input into the attention module. Multi-head attention allows the model to focus on information from different subspaces at multiple locations. Finally, the information from each space is concatenated, which better weights important information. The multi-head attention network formula is:

[0041]

[0042]

[0043]

[0044]

[0045]

[0046] in the formula These are represented as the query matrix, key matrix, and value matrix, respectively, which are derived from the input... respectively with the corresponding weight matrix The result of multiplication. Depend on With matrix The result of the transposed multiplication is divided by The square root of the dimension, then multiplied by the matrix. The result. This represents the matrix used to compute the linear transformation of the head attention instance.

[0047] 4.2 Input the output of the attention module into the LSTM network. The LSTM neural network controls the flow and loss of information by introducing three gating structures and one long-term memory unit. Its calculation formula is as follows:

[0048]

[0049]

[0050]

[0051]

[0052] in Represents the Gate of Oblivion Indicates the input gate. Indicates the output gate. This indicates the state of memory cells. The previous hidden state. with sequence Input is fed into the network, simultaneously updating the hidden state and memory cell state. Represents the weight matrix. This represents the bias vector. Represents the Sigmoid activation function. It represents the Hadamah accumulation. It is the hyperbolic tangent function.

[0053] Step 5 specifically includes: A network layer based on Actor-Critic is constructed. The Actor network uses fully connected layers to compress feature information and generate an action probability distribution matrix. The Critic network uses fully connected layers to obtain the evaluation value of the current state.

[0054] Step 6 specifically includes: The agent selects actions based on probability, inputs the selected actions into the step function of the environment, obtains data such as the reward of the current selection, and stores the obtained data into the experience pool after a certain number of interactions.

[0055] Step 7 specifically includes: A batch size of sample data is randomly selected for training, and importance sampling is used to evaluate the difference between the new and old strategies. The importance sampling formula is:

[0056] Gradient pruning limits the update magnitude of policy parameters, as shown in the formula:

[0057]

[0058] in For hyperparameters, For the dominant function, This represents the cumulative reward value for taking action a in state s. ) represents the state estimate. When the advantage function When this occurs, it indicates that the action is better than the average action, and the probability of selecting this action needs to be increased. In such cases, the probability of selecting that action needs to be reduced. The PPO algorithm is more conservative than other deep reinforcement learning algorithms, setting an upper limit of [value missing]. The lower limit is The range of strategy parameter updates is limited to ensure that the new strategy is not significantly different from the old strategy.

[0059] Calculate the policy loss entropy and value loss entropy separately. Multiply the policy loss entropy and value loss entropy by their respective weights to obtain the final loss function value. Finally, backpropagate to update the policy parameters. The policy gradient parameter update formula is:

[0060] The present invention will be further explained below with reference to specific embodiments; A navigation decision-making method based on attention and recurrent point of interest (PPO) implementation includes the following steps: starting a simulation environment to explore and acquire image information, encoding the image information, extracting features from the state information, having the agent interact with the environment to obtain a batch of data and saving it to an experience pool, randomly selecting data for training, and observing the model's performance through visualization tools and model evaluation.

[0061] Step 2: Encode the image pixel information, compress the information space, and obtain the encoded data to improve the algorithm training speed. Specific environmental information is shown in Table 1-3: Table 1 Environmental Color Coding Information

[0062] Table 2 Environmental Target Object Type Coding Information

[0063] Table 3 Environmental Target Object Status Coding Information

[0064] Step 3: Process environmental information using the ARPPO model proposed in this patent. Its structure consists of a convolutional network module, an attention network module, a recurrent neural network module, and an Actor-Critic module based on the PPO algorithm. The structure diagram of the ARPPO model is shown below. Figure 1 As shown in Table 4, the environmental information acquired by the agent is first input into a CNN network, where multiple convolutional kernels process the information. Then, a multi-head attention network module highlights important and crucial information. Next, the information is input into an LSTM network to obtain the temporal graph of the data. An ARPPO model performs feature processing, enabling the agent to acquire important and crucial information and form long-term memory. Finally, the information is input into an Actor network to obtain actions and their corresponding probabilities, and then into a Critic network to evaluate the agent's performance. The action space of this environment is shown in Table 4.

[0065] Table 4. List of Environmental Action Spaces

[0066] The convolutional module consists of two convolutional network layers. The attention module consists of a multi-head attention network layer, a fully connected layer, a Norm layer, and a Dropout layer. The calculation formula for the fully connected layer is:

[0067] in, These are the weight coefficients in the fully connected layer. It is the next level The value of each neuron, This is the corresponding bias. The Norm layer normalizes the data, and its specific formula is:

[0068] The standard deviation of the sample data. Let Variance be the variance of the data, and Minimum. This is to prevent overflow caused by a standard deviation of 0. The Dropout layer prevents the model from overfitting, thereby improving the model's generalization ability. The specific formula is:

[0069]

[0070]

[0071]

[0072] in The function is based on probability. Generate probability vector Then multiply the probability matrix by the input matrix. The calculation is performed, and then the result is obtained through a fully connected layer. Finally, the activation function is used to obtain the result of the next iteration.

[0073] The output of the attention module is input into the LSTM network to extract the temporal characteristics of the data, and then the final result of feature processing is input into the Actor network and the Critic network.

[0074] Step 4: After each action is acquired, the agent feeds the action back to the environment, obtaining corresponding new environmental state information and immediate reward data. After the agent interacts with the environment 128 times, the training data (…) The data is stored in the experience pool for subsequent training.

[0075] Step 5: Randomly select a batch of sample data of size `batch_size` from the experience pool, calculate the policy function loss entropy, value function loss entropy, and advantage function, etc., and process them through gradient pruning and generalization advantage estimation. Finally, integrate the data to obtain the final loss value, and then backpropagate to update the policy parameters. To increase the amount of sample data, parallel operation in multiple environments is used to provide training data for the agent. The formula for calculating the loss function is as follows:

[0076]

[0077]

[0078] To use the action value function With value function The calculated advantage function, The clip function sets an upper limit to assign importance sampling weights. lower limit Limit the strategy to prevent drastic changes. The policy parameters are determined by finding the truncation loss function. The strategy parameter with the highest expected value is updated. The Actor-Critic method based on the advantage function for reward estimation suffers from low variance but high bias. This patent also employs Generalized Advantage Estimation (GAE) to balance variance and bias; its formula is as follows:

[0079] In this case, the advantage's GAE representation degenerates into a one-step time difference method (TD). When =1, the GAE representation of advantage degenerates into the Monte Carlo method; When the value is in the (0,1) interval, it represents a trade-off between bias and variance. Finally, the policy parameters are updated through backpropagation to achieve the agent's exploration task. Figure 2 This is a flowchart of the entire training process for the model.

[0080] Step 6: To evaluate the performance of the trained model, an evaluation module was set up. This module extracts the 10 worst-performing reward values ​​from 100 episodes of interaction between the agent and the model, and compares them with the PPO model, Attention-PPO model, and LSTM-PPO model to analyze the strengths and weaknesses of each model. A corresponding visualization evaluation was also included to observe each step of the ARPPO model's decision-making in the environment and analyze the model's performance.

[0081] This invention also includes an ARPPO model based on attention and recurrent neural networks, the structure of which is as follows: The sequence is as follows: First convolutional layer 1 of the encoding feature extraction convolutional module → Second convolutional layer 2 of the encoding feature extraction convolutional module → Multi-head attention layer 3 of the encoding feature extraction attention module → First fully connected layer 4 of the encoding feature extraction attention module → First dropout layer 5 of the encoding feature extraction attention module → Second fully connected layer 6 of the encoding feature extraction attention module → First normalized layer 7 of the encoding feature extraction attention module → Second normalized layer 8 of the encoding feature extraction attention module → Second dropout layer 9 of the encoding feature extraction attention module → Third dropout layer 10 of the encoding feature extraction attention module → LSTM layer 11 of the encoding feature extraction recurrent neural network module. LSTM layer 11 of the recurrent neural network module for encoding feature extraction → first fully connected layer 12 of the Actor network module for policy selection → second fully connected layer 13 of the Actor network module for policy selection; The LSTM layer 11 of the recurrent neural network module for encoding feature extraction → the first fully connected layer 14 of the Critic network module for state value evaluation → the second fully connected layer 15 of the Critic network module for action value evaluation.

[0082] The model operates using the following steps: 1) Initial state information obtained from the environment is used to perform preliminary feature extraction on the image encoding information using a convolutional network module. Deep multidimensional information of the data is extracted through the first convolutional network layer 1 and the second convolutional network layer 2. 2) The extracted deep multidimensional information is input into the attention module. First, it is input into the multi-head attention network 3 to capture the correlation of information. Important and key information is extracted from the feature information at multiple different locations and concatenated. Then, feature processing is performed through the first fully connected layer 4 and the first dropout layer 5, and some data is selected to be discarded to prevent overfitting. Next, the second fully connected layer 6 realizes the residual connection of the previous fully connected layer 5. The first normalization layer 7 and the second normalization layer 8 are used to perform layer normalization to solve the problem of gradient stability between layers. Finally, some data is discarded through the second dropout layer 9 and the third dropout layer 10 to prevent overfitting. 3) Input the feature information output by the attention module into the LSTM layer 11 of the recurrent neural network module. By introducing the LSTM network to extract the temporal characteristics of the data, the agent can form long-term memory during the exploration process. 4) Finally, the output features of the recurrent neural network module are input into the Actor network module and the Critic network module. The Actor network module integrates feature information through the first fully connected layer 12, and then obtains the actions taken by the agent and their probability distribution through the second fully connected layer 13. The Critic network module integrates feature information using the first fully connected layer 14, and finally uses the second fully connected layer 15 to score the value of the agent's current state.

Claims

1. A navigation decision-making method based on the ARPPO model of attention and recurrent neural networks, characterized in that, The following steps are adopted: Step 1) Obtain environmental image data information, encode the image information in the obtained environmental data as a whole, and encode the target unit in the image according to its color, type and status; the type includes wall, floor and door, and the status includes open, closed and locked. Use the convolutional network module to perform preliminary feature extraction on the image encoding information, and extract the deep multidimensional information of the data through the first convolutional layer (1) and the second convolutional layer (2); Step 2) Input the extracted deep multidimensional information into the attention module; first, input it into the multi-head attention layer (3) to capture the correlation of information, extract important and key information from multiple different positions and splice it, then process the features through the first fully connected layer (4) and the first dropout layer (5) and select to discard some data to prevent overfitting, then the second fully connected layer (6) realizes the residual connection of the previous fully connected layer, and uses the first normalization layer (7) and the second normalization layer (8) to perform layer normalization processing to solve the problem of gradient stability between layers, and finally discard some data through the second dropout layer (9) and the third dropout layer (10) to prevent overfitting; Step 3) Input the feature information output by the attention module into the LSTM layer (11) of the encoding feature extraction recurrent neural network module. By introducing the temporal characteristics of the data extracted by the LSTM network, the agent can form long-term memory during the exploration process. Step 4) Finally, the output features of the recurrent neural network module are input into the Actor network module and the Critic network module. The Actor network module integrates feature information through the first fully connected layer (12), and then obtains the actions taken by the agent and their probability distribution through the second fully connected layer (13). The Critic network module integrates feature information using the first fully connected layer (14), and finally uses the second fully connected layer (15) to score the value of the agent's current state. The ARPPO model based on attention and recurrent neural networks is as follows: The system includes the first convolutional layer (1) of the encoding feature extraction convolutional module, the second convolutional layer (2) of the encoding feature extraction convolutional module, the multi-head attention layer (3) of the encoding feature extraction attention module, the first fully connected layer (4) of the encoding feature extraction attention module, the first dropout layer (5) of the encoding feature extraction attention module, the second fully connected layer (6) of the encoding feature extraction attention module, the first normalized layer (7) of the encoding feature extraction attention module, the second normalized layer (8) of the encoding feature extraction attention module, the second dropout layer (9) of the encoding feature extraction attention module, the third dropout layer (10) of the encoding feature extraction attention module, and the LSTM layer (11) of the encoding feature extraction recurrent neural network module. The output of the LSTM layer (11) of the encoding feature extraction recurrent neural network module is connected to the input of the first fully connected layer (12) of the policy selection Actor network module and the first fully connected layer (14) of the state value evaluation Critic network module, respectively. The output of the first fully connected layer (12) of the policy selection Actor network module is connected to the input of the second fully connected layer (13) of the policy selection Actor network module. The output of the first fully connected layer (14) of the state value evaluation Critic network module is connected to the input of the second fully connected layer (15) of the action value evaluation Critic network module.

2. The method according to claim 1, characterized in that, The following steps are used to construct the ARPPO model based on attention and recurrent neural networks: Step 1: Construct a multi-kernel convolutional neural network module to perform preliminary feature extraction of environmental information; Step 2: Construct a network combining attention module and LSTM to filter and compress feature information, and obtain important and key information; Step 3: Input the important historical information extracted by the model into the actor network to obtain actions, and input it into the critic network to evaluate the value.

3. The method according to claim 2, characterized in that, In step 1, a two-layer CNN convolutional network is constructed, using multiple convolutional kernels to perform feature processing on the encoded environmental information. The CNN convolutional network formula used is as follows: ； This indicates the current environmental state characteristics. This represents the weight matrix of the filter. This represents the bias vector. It is an activation function.

4. The method according to claim 2, characterized in that, Step 2 specifically includes the following sub-steps: Step 2-1) The features extracted by the convolutional network module are input into the attention network module. The multi-head attention network enables the model to focus on information from different subspaces at multiple locations. Finally, the information obtained from each space is concatenated, which can better increase the weight of important information. The formula for the multi-head attention network is: ； ； ； ； ； ； in the formula Represents the query matrix. Represents the key matrix. Represents value matrices, which are derived from the input eigenvectors. respectively with the corresponding weight matrix The result of multiplication, The attention formula is represented by a matrix. With matrix The result of the transposed multiplication divided by the matrix The square root of the dimension, then multiplied by the matrix. The result, This represents the activation function. Indicates the first Attention information of each head, Indicates the first Each head corresponds to The weight matrix, Indicates the first Each head corresponds to The weight matrix, Indicates the first Each head corresponds to The weight matrix, Indicates passage The join function integrates multi-head attention feature information that is important from each head. The matrix representing the linear transformation of the head attention instance; 2-2) Input the output of the attention module into the LSTM network; the LSTM neural network controls the flow and loss of information by introducing 3 gating structures and 1 long-term memory unit, and its calculation formula is as follows: ； ； ； ； ； in express Forget the door information at all times express Enter door information at all times. express Output door information at all times. express Memory cells remember the state of cells in the previous time period, including their latent state. with sequence The input is fed into the network, simultaneously updating the hidden state and the state of memory cells. This represents the weight matrix corresponding to each gating structure. This represents the bias vector. and This represents the weight matrix within the memory cells. Represents the Sigmoid activation function. It represents the Hadamah accumulation. It is the hyperbolic tangent function.

5. The method according to claim 2, characterized in that, In step 3, an Actor-Critic-based network layer is constructed. The Actor network uses a fully connected layer to compress the feature information and generate an action probability distribution matrix. The Critic network uses a fully connected layer to obtain the evaluation value of the current state.

Images