Visual saliency reinforcement learning method based on neuron reset

Abstract

The invention provides a visual saliency reinforcement learning method based on neuron reset, and aims to solve the problem of deep reinforcement learning model overfitting caused by a traditional attention mechanism under the condition of limited data samples, thereby improving the performance and adaptability of an unmanned aerial vehicle inspection system in a complex environment. According to the method, a dynamic visual attention map generation and feature extraction mechanism is introduced, an input four-channel image is processed by using a neural network, an attention map with significant features is generated, and high-dimensional feature representation of an input state is extracted through an encoder module. Through the converter module, the feature representation of the current state is integrated with the action information, so that a plurality of possible subsequent states are predicted. According to the method, the attention mechanism of the intelligent agent is regularly reset, a two-stage training strategy is adopted, the design of the loss function is optimized, and the learning effect of the model and the overall performance of the system are further improved by adaptively updating the loss function.

Description

Technical Field

[0001] The present invention relates to the fields of visual reinforcement learning, artificial intelligence, computer vision, and in particular to a visual saliency reinforcement learning method based on neuron resetting. Background Art

[0002] Current drone systems still face many challenges in identifying key areas of interest in complex and dynamic environments. When performing missions, drones need to quickly locate and focus on key areas in a changing environment, such as equipment anomalies, sudden conditions in the target area, obstacles, and terrain markings. However, these areas of interest are interfered by a variety of uncertain factors, such as changes in equipment appearance, ambient lighting, and climate conditions, which limits the system's efficiency in identifying and responding to key information.

[0003] Existing drone systems typically rely on preset rule-guided systems or deep reinforcement learning (DRL) methods to identify and focus on these areas of interest. Although these methods are effective in specific situations, they have limited adaptability when dealing with rapidly changing areas of interest. In particular, when drones perform inspection tasks between different environments, it is difficult for the system to quickly adapt to new areas of interest. In addition, traditional reinforcement learning methods often rely on fixed task settings and areas of interest, which limits the system's flexible real-time adjustments in diverse inspection tasks.

[0004] In order to meet the above challenges, attention mechanisms have been gradually introduced into drone inspection systems in recent years to enhance their responsiveness to areas of interest. However, in the case of limited data, the attention mechanism can easily cause overfitting of the DRL model, resulting in a decrease in the system's generalization ability in new environments, which in turn affects the drone system's recognition and response to key areas of interest. Summary of the invention

[0005] The purpose of this invention is to propose a visual saliency reinforcement learning method based on neuron reset, aiming to improve the recognition and adaptation ability of the drone inspection system to dynamic attention areas. This invention enhances the dynamic response ability of the system in a changing environment by regularly resetting the attention mechanism of the intelligent agent, and effectively reduces the potential risks caused by the imbalance of attention areas.

[0006] The technical solution of the present invention is to provide a visual saliency reinforcement learning method based on neuron resetting, which is characterized by comprising the following steps:

[0007] Step 1: Generate dynamic visual attention map and complete feature extraction: The input four-channel image is processed by a neural network to generate a dynamic visual attention map, and then the features of the input state are extracted through the encoder module;

[0008] Step 2: Action integration and state mapping: The converter module is responsible for integrating the feature representation and action information of the current state and predicting the next multiple states; the projection module maps the encoded state representation to a specific feature space;

[0009] Step 3: Complete the neuron information reset and two-stage training design: introduce the visual attention mechanism and reset the relevant information; reset the neuron information of the encoder module, converter module and Q network head excluding the fully connected layer (Q-head) in the obtained dynamic visual attention map to ensure that the image will not be changed after adding visual attention.

[0010] Further, step 1 comprises:

[0011] Step 1.1 In the generation of the dynamic visual attention map, a neural network is used to input the four-channel image of the drone image and then process it through a series of convolutional layers and activation functions;

[0012] Step 1.2 sets the encoder module as the core feature extractor and the network structure as a three-layer convolutional neural network;

[0013] The first convolution layer uses an 8x8 convolution kernel with a step size of 4x4. While maintaining the local information of the features, it significantly reduces the size of the feature map and outputs 32 feature maps.

[0014] The second convolution layer uses a 4x4 convolution kernel and a 2x2 stride, outputting 64 feature maps;

[0015] The third convolutional layer uses a 3x3 convolution kernel and a 1x1 step size, which keeps the size of the feature map unchanged while increasing the depth of the network and outputs 64 feature maps.

[0016] Further, step 2 includes: the converter module is responsible for integrating the action information and the feature representation of the current state, and predicting the next multiple states by combining the feature map output by the converter module with the encoded feature representation of the current state and the action information through convolution operation, and compressing it into the feature representation z t :

[0017]

[0018] Where: z t+1 is the feature representation of the next moment, a t is the action at the current moment, θ t is the network parameter of the converter module at the current moment;

[0019] Projection module Responsible for further mapping the encoded state representation to a specific feature space, which is achieved through a fully connected layer. An additional mapping for prediction is set up during the online process.

[0020]

[0021] Among them, z is the feature representation, y is the image channel, and θ p are the projection module network parameters, are the prediction model network parameters.

[0022] Furthermore, in step 3:

[0023] In the first step of training, visual attention, encoder module, and converter module are trained to learn effective encoding feature expressions;

[0024] In the second step of training, the visual attention, Q-head module, projection module, and prediction model are trained to complete the learning of Q value and prediction ability. The beneficial effects of the present invention are: the present invention adopts a two-stage training strategy by periodically resetting the attention mechanism of the intelligent agent, and optimizes the loss function design. By adaptively updating the loss function, the learning effect of the model and the overall performance of the system are further improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Figure 1 It is a flowchart of the visual saliency reinforcement learning method based on neuron resetting described in the present invention. DETAILED DESCRIPTION

[0026] The following reference Figure 1 Specific embodiments of the present invention are described in detail.

[0027] like Figure 1 As shown, this embodiment provides a visual saliency reinforcement learning method based on neuron reset, comprising the following steps:

[0028] Step 1: Dynamic visual attention map generation and feature extraction: The input four-channel image is processed by a neural network to generate a dynamic visual attention map, and then the features of the input state are extracted through the Encoder module.

[0029] Step 1.1 In the generation of the dynamic visual attention map, a neural network is used to input the four-channel image of the drone image and then process it through a series of convolutional layers and activation functions.

[0030] First, the image passes through a series of 3x3 convolutional layers, gradually increasing from 32 channels to 64 channels and then reducing back to 32 channels.

[0031] Using a 3x3 convolutional layer to reduce the number of channels to 4, combined with the Sigmoid activation function, an image that can directly reflect the intensity of visual attention is generated, and its number of channels is consistent with the input. Sigmoid limits the output to between 0 and 1, and those close to 1 are considered important parts, and those close to 0 are considered unimportant parts.

[0032] Step 1.2 sets the Encoder module as the core feature extractor and the network structure as a three-layer convolutional neural network.

[0033] The first convolutional layer uses an 8x8 convolution kernel with a step size of 4x4, which can significantly reduce the size of the feature map while maintaining the local information of the feature and output 32 feature maps.

[0034] The second convolutional layer uses a 4x4 convolution kernel and a 2x2 step size to further refine the features and output 64 feature maps.

[0035] The last convolution layer uses a 3x3 convolution kernel and a 1x1 step size, which keeps the size of the feature map unchanged, increases the depth of the network, and outputs 64 feature maps. This hierarchical feature extraction process ensures that the network can capture feature representations from low-level to high-level.

[0036] The feature representation is mapped into a 512-dimensional feature space by the last layer of convolutional network, which is completed by a fully connected layer. This high-dimensional feature space provides rich information for subsequent decision making and state transfer. The Q-head module uses a fully connected layer, in which the 512-dimensional feature vector is mapped into a space defined by the number of actions and the number of value atoms to represent the value distribution of each action.

[0037] Step 2: Action integration and state mapping: The Transition module is responsible for integrating the feature representation and action information of the current state and predicting the next multiple states. The Projection module maps the encoded state representation to a specific feature space.

[0038] The Transition module is responsible for integrating the action information and the feature representation of the current state. It predicts the next multiple states by combining the feature map output by the encoder with the encoded feature representation of the current state and the action information through convolution operation and compressing it into the feature representation z. t , as shown in formula (1). t+1 is the feature representation of the next moment, a t is the action at the current moment, θ t It is the network parameter of the Transition module at the current moment.

[0039]

[0040] Projection Module Responsible for further mapping the encoded state representation to a specific feature space, which is achieved through a fully connected layer. An additional mapping for prediction is set up during the online process. As shown in Formula 2. Where z is the feature representation, y is the image channel, θ p is the Projection module network parameter, It is the network parameter of the predication model.

[0041]

[0042] Step 3: Neuron information reset and two-stage training design: Introduce the visual attention mechanism and reset the relevant information.

[0043] This step uses an algorithm to train the model. In order to ensure the stability of training and improve learning efficiency, a two-stage training strategy is designed.

[0044] The obtained dynamic visual attention map resets the neuron information of the encoder, transition and Q network feature extraction models to ensure that the image will not be changed after adding visual attention.

[0045] At the same time, in order to stabilize the training, this embodiment is divided into two steps during training. In the first step of training, visual attention, encoder, and transition are trained. This part is aimed at training the learning of effective coding feature expression; in the second step of training, visual attention, the feature extraction part of the Q network, projection, and prediction are trained to learn the Q value and prediction ability. During the training process, this embodiment continuously optimizes the feature extraction part of the Q network, projection, prediction, and the network loss function of the encoder, thereby ultimately realizing the recognition and adaptation capabilities of the drone inspection system to the dynamic focus area.

[0046] The above embodiments are specific embodiments of the present invention. It should be clear that the implementation methods of the present invention are not limited to the above embodiments, and any subsequent modifications, equivalent substitutions, improvements, etc. on the embodiments should be included in the protection scope of the present invention.

Claims

1. A visual saliency reinforcement learning method based on neuron resetting, characterized in that: The following steps are involved: Step 1: Generate dynamic visual attention map and complete feature extraction: The input four-channel image is processed by a neural network to generate a dynamic visual attention map, and then the features of the input state are extracted through the encoder module; Step 2: Action integration and state mapping: The converter module is responsible for integrating the feature representation and action information of the current state and predicting the next multiple states; the projection module maps the encoded state representation to a specific feature space; Step 3: Complete the neuron information reset and two-stage training design: introduce the visual attention mechanism and reset the relevant information; reset the neuron information of the encoder module, converter module and Q network head excluding the fully connected layer (Q-head) in the obtained dynamic visual attention map to ensure that the image will not be changed after adding visual attention.

2. The visual saliency reinforcement learning method based on neuron resetting according to claim 1, characterized in that: Step 1 includes: Step 1.1 In the generation of the dynamic visual attention map, a neural network is used to input the four-channel image of the drone image and then process it through a series of convolutional layers and activation functions; Step 1.2 sets the encoder module as the core feature extractor and the network structure as a three-layer convolutional neural network; The first convolution layer uses an 8x8 convolution kernel with a step size of 4x4. While maintaining the local information of the features, it significantly reduces the size of the feature map and outputs 32 feature maps. The second convolution layer uses a 4x4 convolution kernel and a 2x2 stride, outputting 64 feature maps; The third convolutional layer uses a 3x3 convolution kernel and a 1x1 step size, which keeps the size of the feature map unchanged while increasing the depth of the network and outputs 64 feature maps.

3. The visual saliency reinforcement learning method based on neuron resetting according to claim 1, characterized in that: Step 2 includes: The converter module is responsible for integrating the action information and the feature representation of the current state. The feature map output by the converter module is convolved and combined with the encoded feature representation z of the current state. t The action information a t , predict the next multiple states and compress them into feature representation z t : Where: z t+1 is the feature representation of the next moment, θ t is the network parameter of the converter module at the current moment; Projection module Responsible for further mapping the encoded state representation to a specific feature space, which is achieved through a fully connected layer. An additional mapping for prediction is set in the online process. Among them, z is the feature representation, y is the image channel, and θ p are the projection module network parameters, are the prediction model network parameters.

4. The visual saliency reinforcement learning method based on neuron resetting according to claim 1, characterized in that: In step 3: In the first step of training, visual attention, encoder module, and converter module are trained to learn effective encoding feature expressions; In the second step of training, visual attention, Q-head, projection module, and prediction model are trained to complete the learning of Q value and prediction ability.

Images