Visual saliency reinforcement learning method based on prior knowledge
By introducing a deep reinforcement learning method based on prior knowledge in the robot vision navigation system, using Gaussian filters and multi-head attention mechanisms to generate an optimized attention map, and training them in combination with deep Q networks, the problem of traditional methods identifying key areas in complex environments is solved, and the accuracy and autonomy of navigation decisions are improved.
Patent Information
- Application Number
- CN202411792771.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-12-07
- Publication Date
- 2025-04-25
AI Technical Summary
Traditional robot visual navigation methods are difficult to accurately identify areas that are critical to navigation tasks when dealing with complex backgrounds, dynamic obstacles, or the presence of a large amount of irrelevant information in the environment, and lack adaptability in unknown or dynamic environments.
A deep reinforcement learning method based on prior knowledge is adopted. By introducing a learnable Gaussian filter and a multi-head attention mechanism, an optimized attention map is generated, which is used to guide the robot to pay attention to important areas, and to combine the deep Q network for reinforcement learning training to dynamically adjust the attention area.
It improves the robot's decision accuracy and autonomy during navigation, enhances its adaptability in complex and dynamic environments, reduces noise interference, and improves the efficiency of image processing and decision optimization.
Smart Images
Figure CN119888437A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the fields of visual reinforcement learning, artificial intelligence, computer vision, and more particularly to a visual saliency deep reinforcement learning method based on prior knowledge. Background Art
[0002] Robot visual navigation is one of the key research directions in the current field of intelligent robots. In order to achieve autonomous mobility, robots need to rely on visual systems to perceive the environment and plan paths and perform obstacle avoidance operations through understanding visual information. However, in the face of complex and diverse environments, traditional visual navigation methods have certain limitations in information extraction and key area attention, especially when dealing with complex backgrounds, dynamic obstacles or a large amount of irrelevant information in the environment, it is difficult to accurately identify areas that are critical to navigation tasks.
[0003] In existing robot vision navigation systems, they usually rely on rule-based feature extraction methods or deep learning models to perceive and process scenes. Although these methods perform well in specific situations, their effectiveness and adaptability will be significantly reduced when dealing with complex and changing scenes. For example, rule-based feature extraction methods lack flexibility and are easily affected by factors such as ambient lighting changes and occlusion; while traditional deep learning models often require a large amount of labeled data for training. When the data samples are limited, the model is prone to overfitting, resulting in a decrease in the decision-making accuracy of the system.
[0004] In recent years, attention mechanisms have been gradually applied to robot vision systems to help models selectively focus on key areas in complex environments. However, in the absence of effective prior information, relying solely on attention mechanisms cannot fully improve the system's navigation capabilities, especially when dealing with unknown or dynamic environments. Robots need to have stronger adaptability to adjust their focus areas in real time.
[0005] Based on this, the present invention proposes a deep reinforcement learning method of visual saliency based on prior knowledge for robot visual navigation tasks. By introducing a learnable Gaussian filter combined with a general residual to generate a prior attention map, the present invention can effectively guide important areas in the early stage of visual information processing, so that the robot can more accurately identify key features in the environment. In addition, through the combination of multi-head attention mechanism and deep reinforcement learning, the robot can dynamically adjust the focus area during navigation. Summary of the invention
[0006] The present invention provides a visual saliency deep reinforcement learning method based on prior knowledge, which aims to enable a robot to dynamically adjust its attention area during navigation. The method first processes the input image through a Gaussian filter and a general residual to obtain a preliminary prior attention map. Then, the image is mapped to the query vector, key vector and value vector space in the attention mechanism through a convolution operation, and the prior attention map is optimized through multi-head attention calculation. Then, the optimized prior attention map is dot-multiplied with the original image to generate a final attention map, which is used to guide the robot to pay attention to important areas. On this basis, a deep Q network (DQN) reinforcement learning algorithm is used to train an intelligent agent, and the optimized attention map and the original image are used alternately. The intelligent agent can adjust its attention area in real time and optimize the decision-making results, thereby enhancing the autonomy and decision-making ability of the robot during navigation.
[0007] The technical solution of the present invention is to provide a method for deep reinforcement learning of visual saliency based on prior knowledge, which is characterized by comprising the following steps:
[0008] Step 1: Obtain the initial prior attention map: Use a learnable Gaussian filter combined with a general residual to process the input robot visual navigation image to obtain a prior attention map;
[0009] Step 2: Optimize the prior attention map: Map the original image to the three feature spaces corresponding to the attention mechanism through 2D convolution, namely Q is the query vector, K is the key vector, V is a value vector; Q It is a vector extracted from the input features to represent the target features; K It is a vector extracted from the input features and used to calculate the correlation between the query vector and the input features; V It is a vector representing the actual content extracted from the input features. The specific formula is:
[0010] V=f V (I; θ V ) (4)
[0011] Among them, f K (·),f Q (·) and f V (·) represents the convolution map, θ K ,θ Q and θ V Represents the weight corresponding to the mapping; calculates the attention weight through the inner product of K and Q; then applies the attention weight to the value vector V to obtain the final attention map O s (I), calculated as:
[0012] O s(I) = softmax(KQ T )V (5)
[0013] Step 3: Get the final attention map: multiply the optimized prior attention map by the original image to get the final attention image.
[0014] Step 4: Training with reinforcement learning algorithm: Use reinforcement learning algorithm to let the agent observe the attention image and the original image alternately, so that the robot can adjust its attention area in real time and give the decision result of the attention area.
[0015] Further, step 1.1, define a learnable Gaussian filter module, given an input image I f (x, y), the Gaussian filter module acts as a preprocessing step of the attention map to reduce the impact of image noise on the performance;
[0016] The Gaussian filter is defined as:
[0017]
[0018] Among them, G(x, y; θ μ ,θ σ ) represents the weight of the Gaussian kernel at position (x, y); θ μ and θ σ represents the mean and variance of the Gaussian kernel;
[0019] By combining the weighted operation of the input image and the Gaussian filter weights, the learnable Gaussian filter module is defined as:
[0020]
[0021] Among them, GF(I f (x, y); θ μ ,θ σ ) represents the input image I f Gaussian filtering result of (x, y); G(m, n; θ μ ,θ σ ) is the weight of the Gaussian filter, which is calculated based on formula (1) at each position (m, n) within the window size; B k is the size of the Gaussian filter, which indicates the side length of the filter window; I f (x+m, y+n) represents the input image I f (x, y) is the pixel value at position (x+m, y+n) in the two-dimensional plane; |x| and |y| represent the size of the Gaussian filter.
[0022] Step 1.2: Extract the features of the image in the frequency domain: First, use the fast Fourier transform to transform the input image I f(u, y) is converted to frequency domain representation:
[0023] F(I f (x, y)) = A (u, υ), φ (u, υ) (3)
[0024] Among them, A(u,υ) represents the amplitude information in the frequency domain, φ(u,υ) represents the phase information in the frequency domain; (u,υ) represents the coordinates in the frequency domain;
[0025] Then the features are enhanced by calculating the residual amplitude R(u,υ);
[0026] Then the calculation of the residual amplitude R(u,υ) is:
[0027] R(u, υ)=exp(log(1+A(u, υ))-M(log(1+A(u, υ)),M k )) (4)
[0028] Wherein, log(1+A(u,υ)) represents the logarithmic transformation of the amplitude; M(X,M k ) represents average pooling, X is the input matrix to be pooled, M k Indicates the size of the pooling filter;
[0029] Through inverse fast Fourier transform, the frequency domain amplitude A(u,υ) and the original phase φ(u,υ) are combined to reconstruct the image representation in the spatial domain, thereby generating an attention map in the spatial domain;
[0030] The saliency attention map S(x, y) after frequency domain optimization can be expressed as:
[0031] S(x,y)=IFFT(A(u,υ),φ(u,υ)). (5)
[0032] Furthermore, in step 3, the optimized prior attention map is dot-multiplied with the original image to obtain the final attention image. The composition of the optimized prior attention map is calculated as follows:
[0033]
[0034] in, Indicates that the optimized prior attention map is finally obtained; O s (I) represents the original image attention map; SR represents the general residual calculation module, which is expressed as:
[0035] SR(x, y) = F -1 (e R(x，y)+Θ(x，y) ) (7)
[0036] Among them, I f (x, y) is the input image; F and F-1 is the Fourier transform and inverse Fourier transform; h aυ is the mean filter; L(x, y) is the logarithm of the spectrum; R(x, y) is the general residual; SR(x, y) is the saliency map, and Θ(x, y) is the phase spectrum of the image.
[0037] Furthermore, in step 4:
[0038] Step 4.1 Use the deep Q network algorithm in the reinforcement learning algorithm to train and obtain the prior attention map; during the training process, the time difference method is used to learn the parameters θ of the nonlinear approximator network. d for:
[0039]
[0040] Among them, the Q function Q(s, a; θ) represents the action value function when executing action a in the current state s, which is used to evaluate the pros and cons of different actions; the parameter θ is the weight of the q network; A represents the complete action set of the agent, that is, all the optional actions in each state; s' represents the transfer of the agent to the next state after taking action a in state s; r represents the immediate reward obtained by the agent after executing action a in the current state s; γ is the discount factor, ranging from [0,1];
[0041] Step 4.2 In the reinforcement learning process, the image is enhanced by optimizing the attention The Q value of is approximated by the time difference target in the original DQN through formula (8) to train the weight of the Q network. The specific objective function is:
[0042]
[0043] The beneficial effects of the present invention are:
[0044] (1) The present invention combines Gaussian filters and multi-head attention mechanisms to enable the robot to dynamically adjust its focus area during navigation. By generating an optimized attention map, the robot can focus on the key areas related to the task in real time, thereby improving the accuracy and autonomy of decision-making.
[0045] (2) The present invention uses Gaussian filtering and residual processing technology to effectively reduce noise interference in the input image, thereby reducing the attention mechanism's focus on unimportant areas. In this way, the system can reduce redundant calculations and improve the efficiency of image processing and decision optimization.
[0046] (3) The present invention combines deep reinforcement learning with optimized prior attention maps to allow the agent to conduct effective exploration and decision-making in complex and dynamic environments. By alternating between the optimized attention maps and the original images, the robot can quickly adapt to different environmental changes, thereby improving its ability to pay attention in unknown or uncertain environments. BRIEF DESCRIPTION OF THE DRAWINGS
[0047] Figure 1 is a flowchart of the visual saliency deep reinforcement learning method based on prior knowledge;
[0048] Figure 2 It is the optimized prior attention map of the deep reinforcement learning method of visual saliency based on prior knowledge. DETAILED DESCRIPTION
[0049] The following reference Figure 1-2 Specific embodiments of the present invention are described in detail.
[0050] like Figure 1 As shown, this embodiment provides a visual saliency deep reinforcement learning method based on prior knowledge, comprising the following steps:
[0051] Step 1: Obtain the initial prior attention map: First, use a learnable Gaussian filter combined with a general residual to process the input robot vision navigation image to obtain a prior attention map.
[0052] Step 1.1, define a learnable Gaussian filter module, given the input image I f The (x, y) Gaussian filter module acts as a pre-processing step for the attention map to reduce the impact of image noise on the performance.
[0053] The Gaussian filter is defined as:
[0054]
[0055] Among them, θ μ and θ σ represents the mean and variance. Then the learnable Gaussian filter module can be defined as:
[0056]
[0057] Among them, I x，y is the input robot visual navigation image. Here, |x| and |y| represent the size of the Gaussian filter, which is defined as B in Formula 2. k G(x, y) is the weight of the convolution operation, which is defined in the present invention as B kRepresents the size of the Gaussian filter kernel. In the learnable Gaussian filter module of this embodiment, the mean and variance are learnable parameters, so that the Gaussian filter can adapt to various input images.
[0058] Step 1.2: Use fast Fourier transform to obtain the frequency domain of the image and express it as A(u,υ), φ(u,υ)=F(I f (x, y)). Where A(u, υ) represents the amplitude and φ(u, υ) represents the phase. Then the residual amplitude is calculated as:
[0059] R(u, υ)=exp(log(1+A(u, υ))-M(log(1+A(u, υ)),M k )) (3)
[0060] Among them, M(X,M k ) represents average pooling, which uses a size of M k The spatial attention map is then obtained by inverse fast Fourier transform and using amplitude A(u,υ) and phase φ(u,υ).
[0061] Step 2: Optimize the prior attention map: Map the image to the KQV value of the attention mechanism through convolution, perform multi-head attention calculation, and guide the network through fine-tuning to optimize the prior attention map, such as Figure 2 As shown, the specific steps include:
[0062] The original image is mapped to the three feature spaces corresponding to the attention mechanism through 2D convolution, namely query K, key Q and value V. The formula is:
[0063] V=f V (I; θ V ) (4)
[0064] Among them, f K (·),f Q (·) and f V (·) represents the convolution map, θ K ,θ Q and θ V Denotes the weight corresponding to the mapping. The attention weight is calculated by the inner product of K and Q. The attention weight is then applied to the value to obtain the final attention map O s (I), calculated as:
[0065] O s (I) = softmax(KQ T )V (5)
[0066] Step 3: Get the final attention map: multiply the optimized prior attention map by the original image to get the final attention image.
[0067] The optimized prior attention map is dot-multiplied with the original image to obtain the final attention image. Therefore, the composition of the optimized prior attention map is calculated as follows:
[0068]
[0069] in, It means that the optimized prior attention map is finally obtained. The parameter θ includes θ in formula 5 K ,θ Q and θ V , so these parameters are jointly trained during the training process. In addition, GF is obtained in step 1 of the claim, and SR is the general residual, which is expressed as:
[0070] S(x, y) = F -1 (e R(x，y)+iΘ(x，y) ) (7)
[0071] Where I(x, y) is the input image, F and F-1 are the Fourier transform and inverse Fourier transform, and h aυ is the mean filter, L(x, y) is the logarithm of the spectrum, R(x, y) is the general residual, S(x, y) is the saliency map, and Θ(x, y) is the phase spectrum of the image.
[0072] Step 4: Training with reinforcement learning algorithm: Use reinforcement learning algorithm to let the agent observe the attention image and the original image alternately, so that the robot can adjust its attention area in real time and give the decision result of the attention area.
[0073] Step 4.1 uses the deep Q-network (DQN) algorithm in the reinforcement learning algorithm to train the above-mentioned prior attention map. During the training process, DQN uses the temporal-difference method (TD target) to learn the parameters of the nonlinear approximator network:
[0074]
[0075] The Q function Q(·) represents the evaluation of the agent performing action a in state s, A represents the complete action space, and s' represents the next state after taking action a in state s. The present invention defines the dot product of an image and its corresponding attention map as During the inference process, this embodiment alternately uses I and for:
[0076]
[0077] Here, p represents the probability of inferring Q(I) directly from the original image, and 1-p represents the probability of inferring Q(I) from the weighted image obtained by element-wise multiplication of the image and the attention map. The information flow related to the task is retained, while the irrelevant information flow is weakened. Therefore, this embodiment assumes that the agent should be I and Receive the same Q value, that is
[0078] Step 4.2 However, directly define will make Infinitely approaching I, resulting in the attention map being all 1. Therefore, during the training process, this embodiment combines the TD target in the original DQN with Combined with the Q-value approximation in , the weight of the trained Q network is:
[0079]
[0080] The Q value of approaches the TD target of I, and the weight of the Q network is trained using prior knowledge:
[0081]
[0082] Since approximating the former will cause the TD target of the original image to change continuously, the agent is encouraged to continue training. Therefore, the loss function of the improved DQN of the present invention is:
[0083]
[0084] In summary, the present invention adopts reinforcement learning algorithm to continuously update network parameters and optimize loss function, allowing the intelligent agent to observe the attention image and the original image alternately, and finally allowing the robot to adjust its attention area in real time and give the decision result of the attention area.
[0085] The above embodiments are specific embodiments of the present invention. It should be clear that the implementation methods of the present invention are not limited to the above embodiments, and any subsequent modifications, equivalent substitutions, improvements, etc. on the embodiments should be included in the protection scope of the present invention.
Claims
1. A deep reinforcement learning method for visual saliency based on prior knowledge, characterized in that: The steps include: Step 1: Obtain the initial prior attention map: Use a learnable Gaussian filter combined with a general residual to process the input robot visual navigation image to obtain a prior attention map; Step 2: Optimize the prior attention map: Map the original image to the three feature spaces corresponding to the attention mechanism through 2D convolution, that is, Q is the query vector, K is the key vector, and V is the value vector; Q is the vector extracted from the input features to characterize the target features; K is the vector extracted from the input features to calculate the correlation between the query vector and the input features; V is the vector extracted from the input features to represent the actual content. The specific formula is: V=f V (I;θ V ) (4) Among them, f K (·),f Q (·) and f V (·) represents the convolution map, θ K ,θ Q and θ V Represents the weight corresponding to the mapping; calculates the attention weight through the inner product of K and Q; then applies the attention weight to the value vector V to obtain the final attention map O s (I), calculated as: O s (I)=softmax(KQ T )V (5) Step 3: Get the final attention map: multiply the optimized prior attention map by the original image to get the final attention image. Step 4: Training with reinforcement learning algorithm: Use reinforcement learning algorithm to let the agent observe the attention image and the original image alternately, so that the robot can adjust its attention area in real time and give the decision result of the attention area.
2. The method for deep reinforcement learning of visual saliency based on prior knowledge according to claim 1, characterized in that: Step 1.1: Define a learnable Gaussian filter module. Given an input image I f (x, y), the Gaussian filter module acts as a preprocessing step of the attention map to reduce the impact of image noise on the performance; The Gaussian filter is defined as: Among them, G(x, y; θ μ ,θ σ ) represents the weight of the Gaussian kernel at position (x, y); θ μ and θ σ represents the mean and variance of the Gaussian kernel; By combining the weighted operation of the input image and the Gaussian filter weights, the learnable Gaussian filter module is defined as follows: Its purpose is to denoise the input image through preprocessing operations to reduce the impact of image noise on performance: Among them, GF(I f (x, y); θ μ ,θ σ ) represents the input image I f Gaussian filtering result of (x, y); G(m, n; θ μ ,θ σ ) is the weight of the Gaussian filter, which is calculated based on formula (1) at each position (m, n) within the window size; B k is the size of the Gaussian filter, which indicates the side length of the filter window; I f (x+m, y+n) represents the input image I f (x, y) is the pixel value at position (x+m, y+n) in the two-dimensional plane; |x| and |y| represent the size of the Gaussian filter; Step 1.2: Extract the features of the image in the frequency domain: First, use the fast Fourier transform to transform the input image I f (x, y) is converted to frequency domain representation: F(I f (x,y))=A(u,v),φ(u,v) (3) Among them, A(u, v) represents the amplitude information in the frequency domain, φ(u, v) represents the phase information in the frequency domain; (u, v) represents the coordinates in the frequency domain; Then the features are enhanced by calculating the residual amplitude R(u, v); Then the calculation of the residual amplitude R(u, v) is: R(u,v)=exp(log(1+A(u,v))-M(log(1+A(u,v)),M k )) (4) Wherein, log(1+A(u,v)) represents the logarithmic transformation of the amplitude; M(X,M k ) represents average pooling, X is the input matrix to be pooled, M k Indicates the size of the pooling filter; Through the inverse fast Fourier transform, the frequency domain amplitude A(u, v) and the original phase φ(u, v) are combined to reconstruct the image representation in the spatial domain, thereby generating an attention map in the spatial domain; The saliency attention map S(x, y) after frequency domain optimization can be expressed as: S(x,y)=IFFT(A(u,v),φ(u,v)) (5).
3. The method for deep reinforcement learning of visual saliency based on prior knowledge according to claim 1, characterized in that: In step 3, the optimized prior attention map is dot-multiplied with the original image to obtain the final attention image. The composition of the optimized prior attention map is calculated as follows: in, Indicates that the optimized prior attention map is finally obtained; O s (I) represents the original image attention map; SR represents the general residual calculation module, which is expressed as: SR(x,y)=F -1 (e R(x，y)+Θ(x，y) ) (7) Among them, I f (x, y) is the input image; F and F -1 is the Fourier transform and inverse Fourier transform; h av is the mean filter; L(x, y) is the logarithm of the spectrum; R(x, y) is the general residual; SR(x, y) is the saliency map, and Θ(x, y) is the phase spectrum of the image.
4. The method for deep reinforcement learning of visual saliency based on prior knowledge according to claim 1, characterized in that: In step 4: Step 4.1 Use the deep Q network algorithm in the reinforcement learning algorithm to train and obtain the prior attention map; during the training process, the time difference method is used to learn the parameters θ of the nonlinear approximator network. d for: Among them, the Q function Q(s, a; θ) represents the action value function when executing action a in the current state s, which is used to evaluate the pros and cons of different actions; the parameter θ is the weight of the q network; A represents the complete action set of the agent, that is, all the optional actions in each state; s' represents the transfer of the agent to the next state after taking action a in state s; r represents the immediate reward obtained by the agent after executing action a in the current state s; γ is the discount factor, ranging from [0,1]; Step 4.2 In the reinforcement learning process, the image is enhanced by optimizing the attention The Q value of is approximated by the time difference target in the original DQN through formula (8) to train the weight of the Q network. The specific objective function is: