Deep reinforcement learning method and system based on collaborative attention auto-encoder

Abstract

The invention discloses a deep reinforcement learning method based on a collaborative attention auto-encoder, and the method comprises the steps: collecting an environment image, carrying out the preprocessing, and generating an environment observation frame sequence; performing feature extraction on the environment observation frame through a shared weight encoder; taking the first frame of the environment observation frame sequence as a main frame, taking the characteristics of the main frame as a reference, and calculating the collaborative attention characteristics of the main frame and subsequent frames; fusing the collaborative attention features through a collaborative attention mechanism to obtain cross-frame features of the environment observation frames; splicing the main frame feature and the cross-frame feature to generate a potential feature of the environment observation frame; and finally, reconstructing a tail end frame of the environment observation frame sequence through a decoder, calculating reconstruction loss, splicing potential features and static features, inputting the spliced features into a strategy network, and outputting action distribution for guiding decision of a reinforcement learning agent, thereby realizing the deep reinforcement learning method based on the collaborative attention auto-encoder. And the learning efficiency and the strategy performance can be obviously improved.

Description

Technical Field

[0001] This invention relates to the field of deep reinforcement learning technology, and in particular to a deep reinforcement learning method and system based on a collaborative attention autoencoder. Background Technology

[0002] With the rapid development of artificial intelligence technology, deep reinforcement learning (DRL) has shown great application potential in fields such as games, robot control, and autonomous driving. However, when dealing with complex visual environments, such as Atari 2600 games, reinforcement learning agents face high-dimensional and highly redundant observation state information, which greatly increases the difficulty of learning and computational cost.

[0003] Traditional deep reinforcement learning methods, such as Deep Q-Network (DQN), typically use frame stacking to combine consecutive game frames into an observation state to address the partially observable Markov decision process (POMDP) ​​problem. However, this approach suffers from significant information redundancy because there is often a large amount of repetitive information between consecutive game frames. Furthermore, the frame stacking method does not explicitly utilize the transition information between frame sequences, such as object position and velocity, which is crucial for the learning agent to make decisions.

[0004] To overcome the limitations of traditional methods, some approaches have emerged that utilize auxiliary tasks or autoencoders to extract useful information from observed states. However, these methods either require additional labeled data or have complex and unstable training processes, making them difficult to generalize in practical applications.

[0005] To address the aforementioned problems, this invention proposes a deep reinforcement learning method and system based on the Collaborative Attention Autoencoder (CARL). Summary of the Invention

[0006] The purpose of this invention is to provide a deep reinforcement learning method and system based on Collaborative Attention Autoencoder (CARL). By introducing a collaborative attention mechanism, it delves into the intrinsic connections between consecutive observed state frames. Simultaneously, it reduces information redundancy, generates low-dimensional latent features containing rich transition information, and then inputs these low-dimensional latent features into a policy network to output action distributions, thereby assisting the reinforcement learning agent in decision-making. CARL can significantly improve learning efficiency and policy performance.

[0007] To solve the above technical problems, the technical solution of the present invention is as follows:

[0008] In a first aspect, the present invention provides a deep reinforcement learning method based on a collaborative attention autoencoder, comprising the following steps:

[0009] Step 1: Acquire environmental images and perform preprocessing to generate an environmental observation frame sequence;

[0010] Step 2: Based on the environmental observation frame sequence, feature extraction is performed on the environmental observation frames using a shared weight encoder;

[0011] Step 3: Take the first frame of the environmental observation frame sequence as the main frame, use the features of the main frame as the benchmark, and calculate the collaborative attention features between the main frame and subsequent frames based on the features extracted in Step 2.

[0012] Step 4: Fuse the collaborative attention features obtained in Step 3 through a collaborative attention mechanism to obtain cross-frame features of the environmental observation frame;

[0013] Step 5: Concatenate the main frame features extracted in Step 2 with the cross-frame features obtained in Step 4 to generate the latent features of the environmental observation frame.

[0014] Step 6: Based on the latent features of the environmental observation frames, reconstruct the last frame of the environmental observation frame sequence through the decoder, calculate the reconstruction loss, complete the training of the collaborative attention autoencoder model, and realize the deep reinforcement learning method based on the collaborative attention autoencoder.

[0015] Preferably, in step 2, the shared weight encoder consists of multiple convolutional layers, each using a convolutional kernel of a different size to extract feature maps of the environmental observation frames, as shown in the following formula:

[0016]

[0017] in, for Time of the first Frame feature map, and , This is the feature map after the convolution operation. The number of channels in the feature map. The width of the feature map. The height of the feature map, For a convolutional encoder with shared weights, After normalization Time of the first Frame environment image.

[0018] Preferably, in step 3, the collaborative attention features between the main frame and subsequent frames are calculated using the following formula:

[0019]

[0020] in, for Time main frame and the first Inter-frame collaborative attention features for Time of the first Frame feature map For normalization operations, , The main frame and the first frame are respectively The learnable diagonal matrix of the frame feature map, and .

[0021] Preferably, in step 4, the collaborative attention features obtained in step 3 are fused through a collaborative attention mechanism. Specifically, the collaborative attention features among different frames in the environmental observation frame sequence are weighted and averaged, and the calculation formula is as follows:

[0022]

[0023] in, for Time-lapse features The total number of environmental observation frames. for Time main frame and the first Cooperative attention features between frames.

[0024] Preferably, in step 5, the latent features of the environmental observation frame are calculated using the following formula:

[0025]

[0026] in, for Latent features at any time, and , This represents the depth of the feature map in the convolutional network, i.e., the number of channels in the output of the convolutional layer. It is a fully connected layer. For flattening operation, For splicing operations, for Timing of main frame feature map for Time-spanning frame features.

[0027] Preferably, in step 6, the reconstructed end frame is calculated using the following formula:

[0028]

[0029] in, for The environmental image of the final frame is reconstructed in real time. The total number of environmental observation frames. For decoder, for Potential characteristics at any given moment.

[0030] Preferably, in step 6, calculating the reconstruction loss specifically involves: constructing a reconstruction loss function with the objective of minimizing the reconstruction error, and optimizing the reconstruction parameters of the decoder; the reconstruction loss function is expressed by the following formula:

[0031]

[0032] in, To reconstruct the loss, for The actual end-frame environment image at any given moment. For decoder, for Potential characteristics at any moment It is an L2 norm.

[0033] Preferably, the method further includes joint reinforcement learning training:

[0034] Step 7: Concatenate the latent features obtained in Step 5 with the static features to obtain fused features, input them into the policy network, and output the action distribution to guide the decision-making of the reinforcement learning agent; the static features are extracted through a downsampling network.

[0035] Preferably, during the joint reinforcement learning training process, a joint loss function is constructed by combining the reconstruction loss function with the goal of optimizing the balance between policy performance and temporal information extraction. This optimizes the learning efficiency and decision-making ability of the learning agent. The joint loss function is expressed by the following formula:

[0036]

[0037]

[0038] in, For joint losses, To reinforce learning loss, , , and These are reconstruction loss, policy loss, value function loss, and entropy loss, respectively. , , , These are the weight coefficients for reconstruction loss, policy loss, value function loss, and entropy loss, respectively. .

[0039] Secondly, a deep reinforcement learning system based on a collaborative attention autoencoder is provided to implement a deep reinforcement learning method based on a collaborative attention autoencoder. The system includes an image acquisition module, an autoencoder module, a collaborative attention module, a reinforcement learning decision module, and a joint training module.

[0040] The image acquisition module is used to acquire environmental images and preprocess them to generate an environmental observation frame sequence, which is then input into the autoencoder module.

[0041] The autoencoder module is used to extract feature maps of the environment observation frame through a convolutional encoder with shared weights, which are then used as input to the collaborative attention module. Based on the latent features of the environment observation frame output by the collaborative attention module, the final frame of the environment observation frame is reconstructed by the decoder.

[0042] The collaborative attention module is used to obtain the latent features of the environmental observation frame based on the feature map of the extracted environmental observation frame through a collaborative attention mechanism.

[0043] The reinforcement learning decision module is used to concatenate the latent features and static features of the environmental observation frames as input to the policy network and output the action distribution to guide the decision-making of the reinforcement learning agent.

[0044] The joint training module is used to optimize the losses of the autoencoder module, collaborative attention module, and reinforcement learning decision module through a joint loss function.

[0045] Compared with the prior art, the beneficial effects of the present invention are as follows: (1) The present invention explicitly extracts inter-frame transition information, such as object motion trajectory, through a collaborative attention mechanism. Experimental results in an Atari environment (such as StarGunner) show that, compared with traditional methods, the agent using CARL can significantly improve policy performance, with a reward improvement of 4.97%. This improvement is based on the average results in multiple experimental settings, verifying the effectiveness of the collaborative attention mechanism in reducing information redundancy and improving learning efficiency; (2) The present invention extracts image features as visual feature input through a convolutional encoder with shared weights, and then further processes these visual features through a collaborative attention mechanism, thereby reducing redundancy and effectively extracting cross-frame temporal transition information. By using a learnable diagonal matrix to dynamically suppress redundant visual features, high redundancy information between consecutive frames is reduced. Compared with the traditional frame stacking method, CARL significantly reduces information redundancy in this way. This result was determined through comparative experiments in multiple Atari environments. Experimental data show that the learning efficiency and policy performance of the agent are improved through this optimization; (3) The present invention pioneers the self-supervised training mechanism of "predicting the end frame", which enables the model to learn the temporal dynamic changes between frames during the training process. By minimizing the reconstruction error, the model can effectively capture and strengthen the temporal dynamic changes, thereby improving its learning and decision-making ability in complex environments. The training efficiency is 7.90% in environments such as Breakout; (4) The end-to-end joint training architecture of the present invention can seamlessly integrate mainstream reinforcement learning algorithms such as A2C and PPO. In 6 Atari environments, and in all 6 environments, it surpasses the benchmark model, with an average improvement of 10.68%.

[0046] The CARL model of this invention reconstructs the last frame, or end frame, in the current observation sequence through a decoder. Specifically, the CARL model encodes the previous few frames (e.g., the first three frames) and generates latent feature representations, then uses these latent features to reconstruct the end frame. During the reconstruction process, the model learns how to capture temporal transition information between frames, such as dynamic features like object motion and position changes. This self-supervised training task provides the model with a powerful temporal learning signal without relying on external labels or supervisory data. Attached Figure Description

[0047] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the description of the embodiments of the present invention will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0048] Figure 1This is a flowchart illustrating the deep reinforcement learning method based on a collaborative attention autoencoder of the present invention.

[0049] Figure 2 This is a schematic diagram of the deep reinforcement learning system based on a collaborative attention autoencoder according to the present invention.

[0050] Figure 3 This is a schematic diagram of the collaborative attention module of the present invention. Detailed Implementation

[0051] The present invention will now be described in further detail with reference to the accompanying drawings. The terminal technical solutions of the embodiments of the present invention will be clearly and completely described. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort are within the scope of protection of the present invention.

[0052] This embodiment uses a 4-frame continuous environmental image sequence as an example. In practical applications, this invention can be extended to more consecutive image sequences; the specific number of frames can be adjusted according to task requirements and environmental complexity. For longer continuous image sequences, the CARL model can effectively capture inter-frame temporal dynamic information using a collaborative attention mechanism, reducing redundancy and enhancing model performance.

[0053] like Figure 1 As shown, this invention provides a deep reinforcement learning method based on a collaborative attention autoencoder, comprising the following steps:

[0054] Step 1: Collect environmental images and preprocess them to generate an environmental observation frame sequence.

[0055] Learning agents in time steps At any given moment, four consecutive environmental images are captured from the environment. The data is then normalized and generated by an environment simulator or game engine (such as Atari 2600) for each frame. A grayscale image of pixels yields a continuous sequence of environmental observation frames. This sequence contains information about changes in the environmental state received by the learning agent, including changes in the position and velocity of objects, changes in the scene, and changes in the interaction state.

[0056] The formula for normalizing the acquired environmental images is as follows:

[0057]

[0058] in, for Time of the first Frame environment image, For the first The mean of the frame environment image. For the first The standard deviation of the frame environment image After normalization Time of the first Frame environment image, , This represents the total number of environmental observation frames.

[0059] By standardizing each frame of the image, we can ensure that the image data is consistent in scale, which makes subsequent data processing more stable and efficient.

[0060] Step 2: Based on the environmental observation frame sequence, feature extraction is performed on the environmental observation frames using a shared weight encoder.

[0061] The standardized environmental observation frame sequence is used as the input to a convolutional neural network encoder with shared weights. The encoder consists of multiple convolutional layers, each of which uses convolutional kernels of different sizes to extract feature maps of the environmental observation frames. The feature maps include spatial features and visual features. Spatial features represent spatial structural information in the image, such as the position and shape of objects, while visual features include texture and color.

[0062] The feature map of the environmental observation frame is calculated using the following formula:

[0063]

[0064] in, for Time of the first Frame feature map, and , The feature map after convolution contains spatial structural information and high-level visual features of the image. The number of channels in the feature map. 8, The width of the feature map. , The height of the feature map, , For a convolutional encoder with shared weights, After normalization Time of the first Frame environment image.

[0065] The specific process of feature extraction by the shared-weight convolutional neural network encoder is as follows:

[0066] Each frame of the sequence is used as input to the first convolutional layer, using 32... The convolution kernel has a stride of 4. After passing through the ReLU activation function, low-level spatial information such as edges and textures in the image is extracted to obtain preliminary spatial and visual features.

[0067] The output features of the first layer are used as the input to the second convolutional layer, using 64 features. The convolution kernel, with a stride of 2, is passed through the ReLU activation function to extract more complex spatial information (such as shape and object outline) and high-level visual information from the image, resulting in more abstract visual features.

[0068] The output features of the second layer are used as the input to the third convolutional layer, using 64 features. The convolution kernel, with a stride of 1, is followed by a ReLU activation function to extract finer-grained visual information, such as object details and texture variations, resulting in high-dimensional visual features; the size of these high-dimensional visual features is... It contains the spatial information and important visual features of the image.

[0069] The encoder converts each frame of the image into a low-dimensional feature representation. These feature representations contain key information about the image, effectively supporting subsequent temporal information extraction and reinforcement learning decision-making tasks. In this embodiment, a convolutional encoder using shared weights can effectively capture common features between frames while avoiding excessive redundant features.

[0070] Step 3: As Figure 3 As shown, the first frame of the environmental observation frame sequence is taken as the main frame, and the features of the main frame are used as the basis for the analysis. As a baseline, based on the features extracted in step 2, the main frame and subsequent frames are calculated. frame Collaborative attention features.

[0071] The collaborative attention features between the main frame and subsequent frames are calculated using the following formula:

[0072]

[0073] in, for Time main frame and the first Inter-frame collaborative attention features for Time of the first Frame feature map For normalization operations, , The main frame and the first frame are respectively The learnable diagonal matrix of the frame feature map, and .

[0074] Affinity Matrix This represents the similarity between the main frame and subsequent frames, calculated using the following formula:

[0075]

[0076] Affinity Matrix go through Normalization yields the collaborative attention matrix, as shown in the formula:

[0077]

[0078] Affinity matrix and collaborative attention matrix These factors work together on the model, and the attention intensity across different frames is dynamically adjusted through the model's learnable diagonal matrix. The collaborative attention mechanism enables the model to learn temporal dependencies between frames, rather than relying solely on the single feature representation of each frame. By combining the collaborative attention mechanism with the learnable diagonal matrix to dynamically suppress redundant features, the model can more accurately extract cross-frame temporal transition information (such as the dynamic behavior of objects), improving computational efficiency and enhancing generalization ability.

[0079] Step 4: Fuse the collaborative attention features obtained in Step 3 using a collaborative attention mechanism to obtain cross-frame features of the environment observation frames. Specifically, this involves fusing the collaborative attention features between different frames in the environment observation frame sequence. and The weighted average is calculated using the following formula:

[0080]

[0081] in, for Time-lapse features for Time main frame and the first Inter-frame collaborative attention features for Time main frame and the first Cooperative attention features between frames.

[0082] All collaborative attention features The average is calculated to fuse inter-frame temporal transition information, integrating collaborative attention features from different frames into a unified feature representation. This reduces redundancy while preserving inter-frame information. Cross-frame features It contains dynamic transfer information across frames, which will be passed to subsequent latent features and reinforcement learning decisions.

[0083] Step 5: Extract the main frame features from Step 2. The cross-frame features obtained in step 4 By stitching the images together, latent features of the environmental observation frames are generated. The calculation formula is as follows:

[0084]

[0085] in, for Latent features at any time, and , This represents the depth of the feature map in the convolutional network, i.e., the number of channels in the output of the convolutional layer. It is a fully connected layer. To perform the flattening operation, the output of the convolutional layer is... The 3D feature map is flattened into a 1D vector. For the splicing operation, multiple tensors are spliced ​​together along a certain dimension. for Timing of main frame feature map for Time-spanning frame features.

[0086] Potential characteristics As a low-dimensional feature representation, it not only includes information from the current frame but also temporal transition information across frames, ensuring that richer temporal information is considered during reinforcement learning decision-making.

[0087] Step 6: Based on the latent features of the environmental observation frames, reconstruct the final frames of the environmental observation frame sequence using a decoder, calculate the reconstruction loss, and complete the training of the collaborative attention autoencoder model, thus realizing a deep reinforcement learning method based on collaborative attention autoencoders. The final output reconstructed final frames can provide effective estimates of future states and enhance the model's temporal reasoning ability.

[0088] The reconstructed final frame is calculated using the following formula:

[0089]

[0090] in, for The fourth frame of the environment image, which is reconstructed over time, is the final frame. For decoder, for Potential characteristics at any given moment.

[0091] The convolutional neural network managing the decoder consists of multiple convolutional layers, used to reconstruct the final frame from latent features. The specific process is as follows:

[0092] latent features As input to the first convolutional layer, use 24 The convolution kernel, with a stride of 1 and padding of 1, is used in a convolution operation. After passing through the ReLU activation function, lower-level spatial information, such as the edges and rough shapes of objects, is extracted to obtain preliminary spatial features.

[0093] The output features of the first layer are used as the input to the second convolutional layer, using 24... A convolutional kernel with a stride of 1 and padding of 1 is used to extract more complex spatial information, such as the outline and texture changes of objects, to obtain abstract spatial and visual features.

[0094] The output features of the second layer are used as the input to the third convolutional layer, using one... A convolutional operation with a kernel size of 1, stride of 1, and padding of 1 extracts detailed information, such as subtle object features or small-scale texture variations, yielding refined visual features; and maps these features back to... The image size is used to obtain the reconstructed end frame.

[0095] In this embodiment, the output feature map is upsampled using bilinear interpolation. The resolution is adjusted to restore the original image resolution, resulting in an output that matches the actual image size.

[0096] With the goal of minimizing reconstruction error, a reconstruction loss function is constructed, the reconstruction parameters of the decoder are optimized, and the reconstruction loss is calculated. This can force the model to learn temporal information and effectively capture dynamic changes between frames.

[0097] The reconstruction loss function is expressed by the following formula:

[0098]

[0099] in, To reconstruct the loss, for The actual end-frame environment image at any given moment. For decoder, for Potential characteristics at any moment It is an L2 norm.

[0100] Step 7: Concatenate the latent features obtained in Step 5 with the static features to obtain fused features, input them into the policy network, and output the action distribution to guide the decision-making of the reinforcement learning agent.

[0101] The static information features are extracted through a downsampling network, including:

[0102] (1) Stack three consecutive frames of images to generate a stacked image sequence containing the three frames, denoted as: The size of each frame is also Pixel.

[0103] (2) Input the stacked image sequence into the downsampling network, and extract static features after multiple convolutions. The specific implementation is as follows:

[0104] The stacked image sequence is used as input to the first convolutional layer, using 24... The convolution kernel, with a stride of 1 and padding of 1, is used for convolution operations. After passing through the ReLU activation function, low-level spatial features such as edges and textures are extracted to obtain preliminary static features.

[0105] The output features of the first layer are used as the input to the second convolutional layer, also using 24 features. A convolutional kernel with a stride of 1 and padding of 1 is used to extract more complex spatial information, such as the outline and shape of an object, to obtain more abstract static features.

[0106] The output features of the second layer are used as the input to the third convolutional layer, using one... The convolutional kernel, with a stride of 1 and padding of 1, extracts fine-grained visual features, such as object details and texture variations, resulting in more accurate visual information. The output of this layer can be further processed by a non-linear activation function and finally mapped to the target space to obtain the final static features.

[0107] The static features represent static information in the environment, such as the background and stationary objects, with a size of [missing information]. .

[0108] The fused features represent dynamic temporal information and static environmental information in the environment, and the calculation formula is as follows:

[0109]

[0110] in, for Real-time fusion features for Potential characteristics at any moment for Static characteristics at any given time This is for splicing operations.

[0111] Fusion features As a policy network The state input, after being processed by network operations, outputs the action distribution. Specifically, the policy network Composed of multiple fully connected layers, the fused features undergo linear transformations through several fully connected layers to progressively extract high-dimensional feature representations. Each fully connected layer also requires non-linear transformations through activation functions to enhance the network's expressive power. Finally, the network's output layer generates an output corresponding to the size of the action space, i.e., the action distribution. The distributed action represents the action in a given state. Below, all possible actions The selection probabilities are used to guide the learning agent in choosing the best action to maximize cumulative reward.

[0112] Using the Adam optimizer, the learning rate is set to... In the joint reinforcement learning training process, a joint loss function is constructed to optimize the learning efficiency and decision-making ability of the learning agent by combining the reconstruction loss function and aiming at balancing policy performance and temporal information extraction. The joint loss function is expressed by the following formula:

[0113]

[0114]

[0115] in, For joint losses, To reinforce learning loss, , , and These are reconstruction loss, policy loss, value function loss, and entropy loss, respectively. , , , These are the weight coefficients for reconstruction loss, policy loss, value function loss, and entropy loss, respectively. .

[0116] This represents the total loss in the learning strategy. It is used to measure the difference between the current policy and the optimal policy, and is calculated using the policy gradient method; Used to measure the error between the current value function and the target value function, and to evaluate the long-term reward of each state; This helps the agent maintain policy diversity and avoid premature convergence to a local optimum.

[0117] Used to control the contribution of policy updates to the total loss; Used to control the contribution of value function loss to the total loss; Used to control the effect of entropy regularization on the total loss.

[0118] This represents the difference between the predicted frame output by the decoder and the real frame, serving as a self-supervised task to help learn useful representations; The weight used to measure the relationship between the reconstruction task and the reinforcement learning task is usually set to greater than or equal to 10 to ensure that the reconstruction loss accounts for a sufficient proportion of the total loss.

[0119] like Figure 2 As shown, the present invention also provides a deep reinforcement learning system based on a collaborative attention autoencoder, the system comprising an image acquisition module, an autoencoder module, a collaborative attention module, a reinforcement learning decision module, and a joint training module;

[0120] The image acquisition module is used to acquire environmental images and preprocess them to generate an environmental observation frame sequence, which is then input into the autoencoder module.

[0121] The autoencoder module is used to extract feature maps of the environment observation frame through a convolutional encoder with shared weights, which are then used as input to the collaborative attention module. Based on the latent features of the environment observation frame output by the collaborative attention module, the final frame of the environment observation frame is reconstructed by the decoder.

[0122] The collaborative attention module is used to obtain the latent features of the environmental observation frame based on the feature map of the extracted environmental observation frame through a collaborative attention mechanism.

[0123] The reinforcement learning decision module is used to concatenate the latent features and static features of the environmental observation frames as input to the policy network and output the action distribution to guide the decision-making of the reinforcement learning agent.

[0124] The joint training module is used to jointly optimize the losses in the autoencoder module, collaborative attention module, and reinforcement learning decision module through a joint loss function, enabling the system to perform joint learning effectively.

[0125] In an Atari environment, the performance of CARL of this invention was compared with that of other encoders. The experimental results show that CARL (CA_A2C) outperforms A2C and A2C_AE, as shown in Table 1. Each value in Table 1 corresponds to the average seizure reward of 100 evaluation seizures after training for 20M time steps. In all six environments, the average improvement rate was 10.68%, demonstrating the effectiveness of CARL in enhancing the learning ability of RL agents.

[0126] Table 1: Performance Comparison of CARL with Other Encoders in Atari Environment

[0127]

[0128] All parts not covered in this invention are the same as or implemented using existing technologies.

[0129] The above description, in conjunction with specific embodiments, provides a further detailed explanation of the present invention. It should not be construed that the specific implementation of the present invention is limited to these descriptions. For those skilled in the art, various simple deductions or substitutions can be made without departing from the concept of the present invention, and all such modifications and substitutions should be considered within the scope of protection of the present invention.

Claims

1. A deep reinforcement learning method based on a cooperative attention autoencoder, characterized in that, Includes the following steps: Step 1: Acquire environmental images and perform preprocessing to generate an environmental observation frame sequence; Step 2: Based on the environmental observation frame sequence, feature extraction is performed on the environmental observation frames using a shared weight encoder; Step 3: Take the first frame of the environmental observation frame sequence as the main frame, use the features of the main frame as the benchmark, and calculate the collaborative attention features between the main frame and subsequent frames based on the features extracted in Step 2. Step 4: Fuse the collaborative attention features obtained in Step 3 through a collaborative attention mechanism to obtain cross-frame features of the environmental observation frame; Step 5: Concatenate the main frame features extracted in Step 2 with the cross-frame features obtained in Step 4 to generate the latent features of the environmental observation frame. Step 6: Based on the latent features of the environmental observation frames, reconstruct the last frame of the environmental observation frame sequence through the decoder, calculate the reconstruction loss, complete the training of the collaborative attention autoencoder model, and realize the deep reinforcement learning method based on the collaborative attention autoencoder.

2. The deep reinforcement learning method based on a collaborative attention autoencoder according to claim 1, characterized in that: In step 2, the shared weight encoder consists of multiple convolutional layers, each using a convolutional kernel of a different size to extract feature maps of the environmental observation frames, as shown in the following formula: in, for Time of the first Frame feature map, and , This is the feature map after the convolution operation. The number of channels in the feature map. The width of the feature map. The height of the feature map, For a convolutional encoder with shared weights, After normalization Time of the first Frame environment image.

3. The deep reinforcement learning method based on a collaborative attention autoencoder according to claim 1, characterized in that: In step 3, the collaborative attention features between the main frame and subsequent frames are calculated using the following formula: in, for Time main frame and the first Inter-frame collaborative attention features for Time of the first Frame feature map For normalization operations, , The main frame and the first frame are respectively The learnable diagonal matrix of the frame feature map, and .

4. The deep reinforcement learning method based on a collaborative attention autoencoder according to claim 1, characterized in that: In step 4, the collaborative attention features obtained in step 3 are fused through a collaborative attention mechanism. Specifically, the collaborative attention features among different frames in the environmental observation frame sequence are weighted and averaged, and the calculation formula is as follows: in, for Time-lapse frame features The total number of environmental observation frames. for Time main frame and the first Cooperative attention features between frames.

5. The deep reinforcement learning method based on a collaborative attention autoencoder according to claim 1, characterized in that: In step 5, the latent features of the environmental observation frame are calculated using the following formula: in, for Latent features at any time, and , This represents the depth of the feature map in the convolutional network, i.e., the number of channels in the output of the convolutional layer. It is a fully connected layer. For flattening operation, For splicing operations, for Timing of main frame feature map for Time-spanning frame features.

6. The deep reinforcement learning method based on a collaborative attention autoencoder according to claim 1, characterized in that: In step 6, the reconstructed end frame is calculated using the following formula: in, for The environmental image of the final frame is reconstructed in real time. The total number of environmental observation frames. For decoder, for Potential characteristics at any given moment.

7. A deep reinforcement learning method based on a collaborative attention autoencoder according to claim 1, characterized in that: In step 6, calculating the reconstruction loss specifically involves: constructing a reconstruction loss function with the objective of minimizing the reconstruction error, and optimizing the reconstruction parameters of the decoder; the reconstruction loss function is expressed by the following formula: in, To reconstruct the loss, for The actual end-frame environment image at any given moment. For decoder, for Potential characteristics at any moment It is an L2 norm.

8. The deep reinforcement learning method based on a collaborative attention autoencoder according to claim 1, characterized in that: The method also includes joint reinforcement learning training: Step 7: Concatenate the latent features obtained in Step 5 with the static features to obtain fused features, input them into the policy network, and output the action distribution to guide the decision-making of the reinforcement learning agent; the static features are extracted through a downsampling network.

9. A deep reinforcement learning method based on a collaborative attention autoencoder according to claim 8, characterized in that: In the joint reinforcement learning training process, a joint loss function is constructed to optimize the learning efficiency and decision-making ability of the learning agent by combining the reconstruction loss function with the goal of balancing policy performance and temporal information extraction. The joint loss function is expressed by the following formula: in, For joint losses, To reinforce learning loss, , , and These are reconstruction loss, policy loss, value function loss, and entropy loss, respectively. , , , These are the weight coefficients for reconstruction loss, policy loss, value function loss, and entropy loss, respectively. .

10. A deep reinforcement learning system based on a collaborative attention autoencoder, used to implement the deep reinforcement learning method based on a collaborative attention autoencoder as described in claims 1-9, characterized in that: The system includes an image acquisition module, an autoencoder module, a collaborative attention module, a reinforcement learning decision module, and a joint training module; The image acquisition module is used to acquire environmental images and preprocess them to generate an environmental observation frame sequence, which is then input into the autoencoder module. The autoencoder module is used to extract feature maps of the environment observation frame through a convolutional encoder with shared weights, which are then used as input to the collaborative attention module. Based on the latent features of the environment observation frame output by the collaborative attention module, the final frame of the environment observation frame is reconstructed by the decoder. The collaborative attention module is used to obtain the latent features of the environmental observation frame based on the feature map of the extracted environmental observation frame through a collaborative attention mechanism. The reinforcement learning decision module is used to concatenate the latent features and static features of the environmental observation frames as input to the policy network and output the action distribution to guide the decision-making of the reinforcement learning agent. The joint training module is used to optimize the losses of the autoencoder module, collaborative attention module, and reinforcement learning decision module through a joint loss function.

Images