Agricultural robot navigation method based on masking residual Transform quantum reinforcement learning

Abstract

The invention provides an agricultural robot navigation method based on masked residual Transformer quantum reinforcement learning, and the method is composed of a masked residual Transformer (MRT) network model and the Qua-SAC quantum reinforcement learning supporting the model. The input of the MRT is an original RGB image in a target state under polar coordinates and an original RGB image in a visual state, the feature extraction part is mainly composed of a layer normalization operation layer, a mask multi-head self-attention block, a multi-layer sensor block and a full connection layer, and through the design of the MRT network, target key features can be extracted. The problem of low data efficiency caused by decoupling of a sensing module and target information in an agricultural unstructured environment is solved, and scene representation is promoted to be more focused on target related features. Quantum convolution is introduced in the Qua-SAC quantum reinforcement learning, so that the accuracy and the calculation efficiency of Q value estimation are improved to learn an optimal navigation strategy, and the navigation performance of an agricultural robot in an agricultural non-institution environment is improved.

Description

Technical Field

[0001] The present invention belongs to the fields of agricultural robots, deep reinforcement learning, and visual navigation, and relates to a masked residual Transformer quantum reinforcement learning method, which is applicable to the field of agricultural robot navigation. Background Art

[0002] In recent years, with the development of agricultural automation and intelligence, deep reinforcement learning (DRL) has received extensive attention and application in the field of agricultural robot navigation. The complexity and uncertainty of the agricultural environment, such as different soil types, crop growth cycles, climate change, etc., pose great challenges to the autonomous navigation of agricultural robots. Rule-based navigation methods are difficult to cope with these dynamic changes, while deep reinforcement learning can learn the optimal navigation strategy from a large amount of environmental data by simulating the biological nervous system, significantly improving the adaptability and navigation efficiency of agricultural robots. The main applications of DRL in agricultural robot navigation include crop planting, pest and disease monitoring, harvesting and picking, etc. However, the existing DRL methods still have some problems in map-free navigation tasks, such as low data utilization due to the decoupling of target information and perception modules, low navigation success rate and stability, etc. These problems limit the further application and promotion of DRL in agricultural robot navigation, especially the ability to perform efficient autonomous navigation in unknown environments.

[0003] Visual navigation is an important direction in agricultural robot navigation. It uses visual sensors such as cameras to obtain image information of the environment, perceive and understand the surrounding environment in real time, and thus achieve autonomous navigation. Existing visual navigation methods rely on accurate environmental maps and manually designed feature extraction methods, which have obvious limitations in agricultural environments. Agricultural environments are usually highly dynamic and unpredictable, such as changes in terrain during crop growth and differences in lighting conditions, making it difficult for traditional methods to update in real time and accurately build maps. Therefore, deep learning-based visual navigation methods have emerged. These methods can learn navigation strategies directly from pixel-level images in an end-to-end manner, which not only improves the accuracy and robustness of navigation, but also reduces the dependence on pre-built environmental maps. However, when processing high-dimensional image data, existing visual navigation methods still have problems such as high consumption of computing resources and long training time, which limits their practical application in agricultural robots.

[0004] Vision Transformer (VIT) is a self-attention model that has emerged in the field of computer vision in recent years. It significantly improves the efficiency and accuracy of image processing by dividing the image into multiple small blocks and then using the Transformer architecture for feature extraction and encoding. In agricultural robot navigation, VIT has been used to process complex environmental images to help robots better understand and analyze surrounding scenes. For example, different types of crops, soil types, and obstacles can be identified through VIT, thereby providing more semantic information for navigation decisions. However, the existing application of ViT in agricultural robot navigation still has some shortcomings, such as insufficient processing of target information and low efficiency of the attention mechanism in processing variable-length sequences. These problems lead to the robot's learning of target-driven navigation strategies in map-free navigation tasks. The navigation performance and interpretability also need to be improved.

[0005] Therefore, the present invention proposes an agricultural robot navigation method based on masked residual Transformer quantum reinforcement learning, which is used to achieve efficient target-driven agricultural robot autonomous navigation. The masked residual Transformer model is based on the Vision Transformer. By coupling the physical target state with the RGB image, the scene representation is mainly focused on the target-related features, which greatly improves the data efficiency of the deep reinforcement learning process. Specifically, the masked residual Transformer method adds Dropout to the residual connection part to help regularize the residual input and prevent overfitting, and uses PreLayerNorm instead of PostLayerNorm to reduce the gradient vanishing problem in the early stage of training. In addition, the attention mask (AM) is introduced to process variable-length sequences or ignore part of the sequence in some scenarios, so that the model can process high-dimensional image data more effectively.

[0006] In the decision-making system, the present invention designs a quantum reinforcement learning method Qua-SAC that supports the masked residual Transformer model. Qua-SAC introduces quantum convolution in the dual Q-value estimation network, and uses the characteristics of quantum convolution to improve the efficiency and accuracy of Q-value estimation. At the same time, when designing the reward function, multi-target rewards, adaptive rewards and sparse rewards are introduced to encourage agricultural robots to learn optimal strategies. This method not only improves navigation performance and robustness, but also enhances the interpretability of the system. The agricultural robot navigation method described in the present invention has obvious advantages in data efficiency, performance and robustness. Summary of the invention

[0007] 1. An agricultural robot navigation method based on masked residual Transformer quantum reinforcement learning, comprising the following steps:

[0008] (1) The navigation target position information is parameterized and a polar coordinate is used as the target position input, which is fed into the navigation system together with the RGB image of the fisheye camera stacked with 4 frames as the raw data input.

[0009] The specific method to obtain the image visual input that meets the requirements is as follows:

[0010] (1.1) Subscribe to the original image of the fisheye camera with a field of view of 220 degrees from ROS, and use cv_bridge in ROS to convert camera_image into an image in OpenCV format. The converted image is a two-dimensional Numpy array with a shape of (160, 120);

[0011] (1.2) Pixel-level noise is added to the input image, and then the four most recent frames are stacked together as the original image input;

[0012] (2) The polar coordinates are used as the target position input and coupled with the 4-frame stacked fisheye camera RGB image, and converted into a complete input mark;

[0013] (3) Using the Vision Transformer network model as the backbone, a masked residual Transformer network model is designed as the backbone of the perception system to process input tags with added position embeddings.

[0014] (4) Design a Qua-SAC quantum reinforcement learning method that supports the masked residual Transformer network model to learn the optimal navigation strategy and perform action input;

[0015] (5) Construct an expert prior action pair dataset to pre-train the masked residual Transformer network model to achieve hot start;

[0016] 2. The agricultural robot navigation method based on masked residual Transformer quantum reinforcement learning, the specific method of step (2) is as follows:

[0017] (2.1) The target polar coordinate information is input into the Multi-Layer Perceptron (MLP) network and converted into feature patches after encoding, while the original RGB image of the visual state is decomposed into 8×8 image patches. Then, the target state input patch and the image input patch are converted into complete input tokens through the linear projection layer of the block token;

[0018] (2.2) Add position embedding to each input token and fix the token encoding the target information to the first position.

[0019] 3. The agricultural robot navigation method based on masked residual Transformer quantum reinforcement learning, the specific method of step (3) is as follows:

[0020] The specific construction method of the network model of the masked residual Transformer is as follows:

[0021] (3.1) Construct a layer normalization operation layer (LN) to normalize the output of each layer, stabilize the training process, accelerate convergence and improve the generalization ability of the model;

[0022] (3.2) Construct a Masked Multi-Head Self-Attention (MSA) module, which uses the attention mechanism to capture the dependencies between different positions in the sequence and uses masks to control which information can be considered in the calculation;

[0023] (3.3) Construct a multi-layer perceptron (MLP), which can effectively handle complex nonlinear problems and learn navigation target related features;

[0024] (3.4) Construct the AvgPool2D layer. AvgPool2D uses two-dimensional average pooling to simplify the feature map.

[0025] (3.5) Construct Latent blocks to obtain and process latent features;

[0026] (3.6) Construct a fully connected layer as a linear transformation layer for further feature combination and output;

[0027] 4. The agricultural robot navigation method based on masked residual Transformer quantum reinforcement learning, the specific method of step (4) is as follows:

[0028] (4.1) The direct line distance between the real-time position coordinates of the agricultural robot and the target position coordinates is normalized and calculated as:

[0029]

[0030] in Represents the real-time position coordinates of the mobile robot, V <x,y> represents the target position coordinates of each round of training; ‖‖2 is the Euclidean

[0031] The norm operator, μ is a constant normalizer;

[0032] (4.2) The second dimension of the polar coordinate input is defined as the difference between the target position vector angle and the real-time heading angle of the agricultural robot, which is calculated as:

[0033]

[0034] in represents the target position vector angle, φ t is the real-time heading angle of the agricultural robot. The heading error Δφ t It is also normalized and calculated as:

[0035]

[0036] After receiving the above input, the masked residual quantum reinforcement learning network outputs a t =[v t ,ω t ] decision command, linear speed v t ∈[0,1], angular velocity And it is passed to the agricultural robot through the Robot Operating System (ROS).

[0037] (4.3) Design a reward function that combines multi-objective, adaptive, and continuous sparse rewards. The reward consists of four parts, and the final reward R t Calculated as:

[0038] R t =R h +R a +R g +R c (4)

[0039] Where R h is the heuristic reward, R a is the action reward, R g is the reward for reaching the target location, R c Penalty for collision. h The heuristic reward is to encourage the agent to move towards the target position and keep a certain distance from obstacles, R h Calculated as:

[0040]

[0041] χ is the distance between the robot and the obstacle scanned by the laser radar carried by the mobile robot, and ρ represents a safe distance limit between the robot and the obstacle; is the coordinate of the mobile robot at the last moment. At the same time, in order to encourage the smooth movement of the mobile robot, the action reward function R is designed a , R a Calculated as:

[0042] R a =3v t -abs(ω t ) (6)

[0043] v t and ω t represents the linear velocity and angular velocity of the mobile robot at time t, and abs is an absolute value operator. Finally, R is designed g , R c The two sparse rewards are as follows: the reward for reaching the target point is R g Calculated as:

[0044]

[0045] Collision Reward R c Calculated as:

[0046]

[0047] κ represents the critical distance value for the robot to successfully navigate to the target position. Given the potential feature extracted from MRT at a specific time step t is J t ,Qua-SAC learns the decision strategy π(a t |J t )

[0048] (4.4) Quantum SAC Q-Network is designed for Q-value estimation. This design introduces quantum convolution to improve the feature extraction capability of the model, thereby effectively reducing the risk of overfitting and improving the balance between exploration and utilization. By combining the update method of the dual Q network, Quantum SAC Q-Network can effectively reduce the overestimation bias and enhance the stability and adaptability of the algorithm. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] Figure 1 It is the overall framework of the navigation method of the present invention.

[0050] Figure 2 It is the masked residual Transformer architecture of the present invention.

[0051] Figure 3 It is a processing block diagram of the Quantum SAC Q-Network network of the present invention for Q value estimation. DETAILED DESCRIPTION

[0052] The agricultural robot navigation method based on masked residual Transformer quantum reinforcement learning proposed in the present invention is described in detail as follows in conjunction with the accompanying drawings:

[0053] Figure 1 This is the overall framework of the navigation method of the present invention. The target position information is parameterized, and a polar coordinate is used as the target position input, which is fed into the masked residual Transformer network together with the RGB image of the fisheye camera stacked with 4 frames, so as to deeply couple the target position information with the visual input. By training the masked residual Transformer network model, the features related to the target position are extracted. During the training process, the parameters are continuously optimized to enhance the correlation between the environmental features and the target position information, so that the target feature extraction is more efficient; the target-related feature information extracted by the masked residual Transformer is used as the input for the subsequent reinforcement learning decision training, so that the intelligent agent can learn the optimal decision-making strategy more quickly and accurately, and perform efficient target-driven autonomous navigation.

[0054] Specifically, the present invention designs a masked residual Transformer network based on the Vision Transformer architecture to learn target-oriented scene representation, which treats multimodal (i.e., physical target state and visual state) input as a sequence of continuous representations. The masked residual Transformer network is used to extract target-oriented latent features to motivate Qua-SAC to learn decision strategies close to the target location through interaction with the environment.

[0055] Figure 2 It is the masked residual Transformer architecture of the present invention. Drawing on advanced model design concepts, a target recognition embedding symbol H is designed, and H is calculated as:

[0056] H=MLP(p goal ) (9)

[0057] Target state of input Mapped through the MLP network D is the number of dimensions of the mapping. Therefore, the initial embedding y0′ of the masked residual Transformer is calculated as:

[0058] y0′=E input (p,H) (10)

[0059] The initial embedding y0 of the masked residual Transformer after adding the position embedding is calculated as:

[0060] y0=y0′+E pos (11)

[0061] Among them, Einput and E pos Represent the input embedding and position embedding respectively. In each layer, the network is refined by the masked residual Transformer strategy, y l-1 Input y after LN layer, MSA layer for residual connection l ' is calculated as:

[0062] y l ′=MSA(LN(y l-1 ))+y l-1 (12)

[0063] y l ' Output y after MLP, AvgPool2d, Latent, FC and residual connection l Calculated as:

[0064] y l =FC(LN(Lat(Avg(MLP(y l ′)+y l ′)))) (13)

[0065] Where l represents the lth block. The present invention determines the depth of the masked residual Transformer as two blocks, so the potential feature J can be obtained from the output of the second block and calculated as:

[0066]

[0067] in Indicates the parameters of MRT.

[0068] The target state of the two-dimensional shape is input to the MLP network and converted into feature patches after encoding, while the original RGB image of the visual state is decomposed into 8×8 image patches (drawing 8×8 image input patches takes up too much area, so 3×3 image patches are used in the figure to illustrate the process). The target state input patch and the image input patch are then converted into complete input tokens through the Linear Projection of Patch-Tokens layer. In addition, we add position embedding to each input token and fix the token encoding the target information to the first position. For the MRT encoder, it consists of layer normalization operations, MSA blocks, MLP, AvgPool2d, Latent, fully connected layers, and residual connections. The layer normalization operation normalizes the input data to stabilize the training. The MSA block uses a multi-head attention mechanism and adds a mask to control the information flow. MLP is used as a nonlinear transformation layer for feature extraction. AvgPool2D uses two-dimensional average pooling to simplify the feature map. The Latent block is used to obtain and process potential features. The fully connected layer is used as a linear transformation layer for further feature combination and output. Considering the computational efficiency and performance, the present invention is designed to use two encoder blocks, each with four heads. By using expert demonstration data to pre-train the parameters of the MRT model, a hot start effect is achieved for subsequent reinforcement learning training. In the standard DIL, the optimization goal of the function approximator in the goal-driven end-to-end navigation problem is:

[0069]

[0070] The function approximator F depends on the environment state p i and the target state p {goal,i} .

[0071] In the method proposed in the present invention, the target state is no longer a condition, but an input. Therefore, the goal of goal-oriented imitation learning is:

[0072]

[0073] is the loss function, which is used to measure the gap between the model prediction value and the target value. i is the state of the input environment sample; H is the target information; where ψ and N are the parameters and number of samples of the function approximator; represents the expert action, that is, the target action that the model is expected to output or imitate.

[0074] This design is intended to make the physical target information and scene representation more strongly coupled, so that the perception system can more effectively extract target-related features and provide a basis for subsequent decision-making systems.

[0075] Figure 3 This is a processing block diagram of the Quantum SAC Q-Network network of the present invention for Q-value estimation. In order to enhance the state representation capability and improve computational efficiency when processing complex high-dimensional inputs, the present invention designs a dual-Q network model of Quantum SAC Q-Network to perform Q-value estimation. This design introduces quantum convolution to improve the feature extraction capability of the model, thereby effectively reducing the risk of overfitting and improving the balance between exploration and utilization. In addition, by combining the update method of the dual-Q network, the Quantum SAC Q-Network can effectively reduce the overestimation bias and enhance the stability and adaptability of the algorithm. This enables it to handle diverse task requirements well in dynamic and complex environments, and improve the learning efficiency and decision-making quality of the strategy. Figure 3 As shown in the figure, isate is the input image data, pstate is the target state in polar coordinates, and next_action is the next action of the mobile robot. The network starts from the current input image data and target state, and samples the next action from the experience pool through two independent Q networks Q1 and Q2 to overcome the overestimation bias problem in traditional Q learning. Quanv2d is quantum convolution and Conv2d is traditional convolution. The two are processed in parallel to improve the efficiency of feature extraction and model expression ability.

[0076] Each Quantv2d can be represented as:

[0077] Quanv2d(I)=U(quantifying operation(I,W,b)) (17)

[0078] Where I is the input, W and b are the quantum convolution kernel weights and biases, respectively, and U is a nonlinear activation function. The quantum convolution network is an enhancement of the classical convolutional neural network, and its implementation details depend on the framework dedicated to quantum computing. In general, quantum convolution can utilize quantum superposition and entanglement properties to process large amounts of information with fewer computing resources.

[0079] By minimizing the mean Bellman-squared error (BMSE) loss function To update the critic network parameters of Qua-SAC, Calculated as:

[0080]

[0081] In the formula is the state action value of the next step of the dual-objective Q network, Calculated as:

[0082]

[0083] where α is a temperature parameter that trades off between the randomness of the optimal policy and the state-action value. Accordingly, the actor network updates its parameters by maximizing the soft state-action function, Calculated as:

[0084]

[0085] The above description is only a preferred embodiment of the present invention, and does not limit the patent scope of the present invention. Any equivalent structure or equivalent process transformation made by using the contents of the present invention specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present invention.

Claims

1. An agricultural robot navigation method based on masked residual Transformer quantum reinforcement learning, characterized in that: The following steps are involved: (1) The navigation target position information is parameterized and a polar coordinate is used as the target position input, which is fed into the navigation system together with the RGB image of the fisheye camera stacked with 4 frames as the raw data input. The specific method to obtain the image visual input that meets the requirements is as follows: (1.1) Subscribe to the original image of the fisheye camera with a field of view of 220 degrees from ROS, and use cv_bridge in ROS to convert camera_image into an image in OpenCV format. The converted image is a two-dimensional Numpy array with a shape of (160, 120); (1.2) Pixel-level noise is added to the input image, and then the four most recent frames are stacked together as the original image input. (2) The polar coordinates are used as the target position input and coupled with the 4-frame stacked fisheye camera RGB image, and converted into a complete input mark; (3) Using the Vision Transformer network model as the backbone, we design a masked residual Transformer network model as the backbone of the perception system to process input tags with added position embeddings; (4) Design a Qua-SAC quantum reinforcement learning method that supports the masked residual Transformer network model to learn the optimal navigation strategy and perform action input; (5) Construct an expert prior action pair dataset and pre-train the masked residual Transformer network model to achieve hot start.

2. According to claim 1, the agricultural robot navigation method based on masked residual Transformer quantum reinforcement learning is characterized in that: The specific method of step (2) is as follows: (2.1) The target polar coordinate information is input into the Multi-Layer Perceptron (MLP) network and converted into feature patches after encoding, while the original RGB image of the visual state is decomposed into 8×8 image patches. Then, the target state input patch and the image input patch are converted into complete input tokens through the linear projection layer of the block token; (2.2) Add position embedding to each input token and fix the token encoding the target information to the first position.

3. The agricultural robot navigation method based on masked residual Transformer quantum reinforcement learning according to claim 1 is characterized in that: The specific method of step (3) is as follows: The specific construction method of the network model of the masked residual Transformer is as follows: (3.1) Construct a layer normalization operation layer (LN) to normalize the output of each layer, stabilize the training process, accelerate convergence and improve the generalization ability of the model; (3.2) Construct a Masked Multi-Head Self-Attention (MSA) module, which uses the attention mechanism to capture the dependencies between different positions in the sequence and uses masks to control which information can be considered in the calculation; (3.3) Construct a multi-layer perceptron (MLP), which can effectively handle complex nonlinear problems and learn navigation target related features; (3.4) Construct the AvgPool2D layer. AvgPool2D uses two-dimensional average pooling to simplify the feature map. (3.5) Construct Latent blocks to obtain and process latent features; (3.6) Construct a fully connected layer as a linear transformation layer for further feature combination and output.

4. According to claim 1, an agricultural robot navigation method based on masked residual Transformer quantum reinforcement learning is characterized in that: The specific method of step (4) is as follows: (4.1) The direct line distance between the real-time position coordinates of the agricultural robot and the target position coordinates is normalized and calculated as: in Represents the real-time position coordinates of the mobile robot, V <x,y> represents the target position coordinates for each round of training; ‖‖2 is the Euclidean norm operator, and μ is a constant normalizer; (4.2) The second dimension of the polar coordinate input is defined as the difference between the target position vector angle and the real-time heading angle of the agricultural robot, which is calculated as: in represents the target position vector angle, φ t is the real-time heading angle of the agricultural robot. The heading error Δφ t It is also normalized and calculated as: After receiving the above input, the masked residual quantum reinforcement learning network outputs a t =[v t ,ω t ] decision command, linear speed v t ∈[0,1], angular velocity And it is passed to the agricultural robot through the Robot Operating System (ROS). (4.3) Design a reward function that combines multi-objective, adaptive, and continuous sparse rewards. The reward consists of four parts, and the final reward R t Calculated as: R t =R h +R a +R g +R c (4) Where R h is the heuristic reward, R a is the action reward, R g is the reward for reaching the target location, R c Penalty for collision. h The heuristic reward is to encourage the agent to move towards the target position and keep a certain distance from obstacles, R h Calculated as: χ is the distance between the robot and the obstacle scanned by the laser radar carried by the mobile robot, and ρ represents a safe distance limit between the robot and the obstacle; is the coordinate of the mobile robot at the last moment. At the same time, in order to encourage the smooth movement of the mobile robot, the action reward function R is designed a , R a Calculated as: R a =3v t -abs(ω t ) (6) v t and ω t represents the linear velocity and angular velocity of the mobile robot at time t, and abs is an absolute value operator. Finally, R is designed g , R c The two sparse rewards are as follows: the reward for reaching the target point is R g Calculated as: Collision Reward R c Calculated as: κ represents the critical distance value for the robot to successfully navigate to the target position. Given the potential feature extracted from MRT at a specific time step t is J t ,Qua-SAC learns the decision strategy π(a t |J t ); (4.4) Quantum SAC Q-Network is designed for Q-value estimation. This design introduces quantum convolution to improve the feature extraction capability of the model, thereby effectively reducing the risk of overfitting and improving the balance between exploration and utilization. By combining the update method of the dual Q network, Quantum SAC Q-Network can effectively reduce the overestimation bias and enhance the stability and adaptability of the algorithm.

Images