Efficient recognition control reinforcement learning method for agricultural picking

Abstract

The application provides a reinforcement learning method based on discrete contrast features in agricultural picking, which adopts discrete contrast feature learning to obtain state feature input, obtains anchor points, positive samples and negative samples through data enhancement, makes anchor point feature representation as close as possible to positive sample feature representation and as different as possible from negative sample feature representation, and then obtains good input feature representation through discretization. This method does not require high-cost label data and does not require a complex model that may cause unstable training, and realizes an end-to-end deep reinforcement learning algorithm with high sample efficiency, stable and rapid training process.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of intelligent agriculture, and more particularly, to a reinforcement learning method based on discrete contrastive features in agricultural picking. BACKGROUND

[0002] Reinforcement learning is an important field of machine learning, mainly studying how an agent learns the optimal control decision in the process of interacting with the environment. Existing reinforcement learning algorithms require good environmental state feature input or a large amount of data interaction with the environment to achieve a good learning effect, which greatly limits the application of reinforcement learning algorithms in some practical problems where it is difficult to obtain good environmental state feature input or the interaction data is expensive. The environment in the agricultural field is more complex than that in the industrial field. Specifically in the field of agricultural picking, the complex environment includes uneven roads, irregular crop growth conditions, light intensity and the shadow of nearby crops, etc., which makes it difficult to obtain good environmental state feature input or a large amount of data interaction in the agricultural field, thereby making it difficult for reinforcement learning algorithms with excellent performance in agent optimization to be applied.

[0003] Current research helps reinforcement learning algorithms to be applied in fields with complex environments through auxiliary tasks, but these auxiliary tasks have certain limitations or require a large cost. Reinforcement learning algorithms combined with auxiliary tasks can be roughly divided into two parts: model-based reinforcement learning algorithms and model-free reinforcement learning algorithms. The auxiliary task of the model-based reinforcement learning algorithm approximates an environment model through a neural network, which can predict state transitions and give rewards. However, the model-based reinforcement learning algorithm often contains many complex components, such as hidden variable models, reward models, etc., which can make the training process unstable or require a large amount of expert human assistance. The auxiliary task of the model-free reinforcement learning algorithm is mainly to improve the representation of the agent's perception input features, and the improvement methods include supervised learning and self-supervised learning. Supervised learning can effectively improve the input feature representation, but the cost is to obtain a pre-trained network with a large amount of labeled data. Self-supervised learning can avoid the cost of annotating large data sets, for example, an auxiliary task such as image reconstruction based on self-encoding, which can obtain a feature with a lower dimension than the input, but the obtained feature may not be significant for the specific downstream task. SUMMARY

[0004] The present application provides a reinforcement learning method based on discrete contrastive features in agricultural picking, which does not require high-cost labeled data and does not require complex models that may cause unstable training, realizing a sample-efficient, stable and rapid training process, and an end-to-end deep reinforcement learning method.

[0005] In order to achieve the above technical effects, the technical solutions of the present application are as follows:

[0006] A reinforcement learning method based on discrete contrast features in agricultural picking, comprising the following steps:

[0007] S1: data augmentation is performed on the collected pictures to obtain queries (Query) and keys (Key);

[0008] S2: these data are encoded into query features and key features by a convolutional neural network (CNN); ;

[0009] S3: tight discrete feature input is obtained through discrete contrast learning, wherein the convolutional neural network of the key (Key) is not updated during training, and is updated through the parameters of the convolutional neural network of the query (Query), and finally a Soft Actor-Critic strategy is used for evaluation feedback.

[0010] Further, in step S1, n sample transition tuples are read , wherein the observation input is used to generate queries and keys through random data augmentation methods; In step S2, the encoder CNN is used to become and , and then the sign function is used to become and .

[0011] Further, in step S3, discrete contrast learning is performed using the data obtained in step S2:

[0012] The query representation and the key representation are given, wherein the key representation includes the positive sample key representation and the negative sample key representation , and the following loss function formula is used for discrete contrast learning:

[0013] (1).

[0014] Wherein is the transpose vector of the key representation vector, is a vector matrix for linear conversion of the positive sample key representation , is the negative sample key representation with index . ​

[0015] Further, in step S3, define where is the column dimension of is a hyper-parameter, and are the th row and the th row of and respectively, can be seen as the inner product similarity between the th query representation and the th key representation, define A as the sigmoid function of

[0016]

[0017] Get the likelihood function of the similarity matrix, there is a formula:

[0018] (2)

[0019] where the matrix is an identity matrix, define as follows:

[0020] (3)

[0021] Get the log-likelihood function of and :

[0022] (4)

[0023] The task of step S3 is to maximize the log-likelihood function, the objective can be represented as follows:

[0024] (5).

[0025]

[0026] Further, in step S3, fix update , first calculate the gradient and Hessian matrix of formula (5) about :

[0027] (6)

[0028] where , is a diagonal matrix, define where is an identity matrix, a Lower bound of

[0029]

[0030] The optimization problem is converted to the following formula

[0031]

[0032] The solution of formula (8) is:

[0033]

[0034] Through formula (9), we can get .

[0035] Further, in step S3, fix Update , get the update formula of :

[0036]

[0037] Through formula (10), we can get .

[0038] Further, in step S3, when and are updated, they are taken as the regression target of the state encoder with parameters , and the encoder is updated by mean square error (MSE):

[0039]

[0040] The key encoder with parameters and the query encoder with parameters are given, and they are updated by momentum without gradient backpropagation, as follows:

[0041]

[0042] Further, in step S3, the policy evaluation part of Soft Actor-Critic, the critic network is updated by minimizing the soft Bellman residual formula:

[0043]

[0044] where is the buffer, is a sample tuple, This is the current state. It is the action taken at the current time step. It is the feedback from the current time step. This is the state at the next time step. It is a hyperparameter discount factor. It is the action-state value function approximated by the critic network. The objective value function is obtained through the Monte Carlo evaluation formula, which is as follows:

[0045]

[0046] in It is a delay parameter.

[0047] Furthermore, in step S3, the objective loss function of the actor network in the policy enhancement stage of Soft Actor-Critic is as follows:

[0048]

[0049] in The parameter is Action network, The parameter is The action-state value function approximated by the critic network.

[0050] Compared with the prior art, the beneficial effects of the technical solution of the present invention are:

[0051] This invention employs discrete contrastive feature learning to obtain state feature inputs. It uses data augmentation to obtain anchor points, positive samples, and negative samples, then makes the anchor point feature representation as close as possible to the positive sample feature representation and as different as possible from the negative sample feature representation. Subsequently, discretization is used to obtain a good input feature representation. This method does not require costly labeled data or complex models that might cause training instability, achieving a deep reinforcement learning method that is highly efficient in terms of sample usage, stable and rapid in training, and end-to-end. Attached Figure Description

[0052] Figure 1 This is a flowchart of the method of the present invention;

[0053] Figure 2 The results show the comparison between the method of the present invention and the prior art under six continuous control tasks. Detailed Implementation

[0054] The accompanying drawings are for illustrative purposes only and should not be construed as limiting the scope of this patent.

[0055] For better illustrating the embodiments, some components in the drawings can be omitted, enlarged or reduced, and do not represent the actual product size.

[0056] It is understandable for those skilled in the art that some known structures and their descriptions in the drawings can be omitted.

[0057] The technical solutions of the present application will be further described below in combination with the drawings and embodiments.

[0058] As shown in the drawings, the present application provides a reinforcement learning method based on discrete contrast features in agricultural picking, comprising the following steps: Figure 1

[0059] S1: data augmentation is performed on the collected pictures to obtain queries (Query) and keys (Key) ;

[0060] S2: these data are encoded into query features and key features by a convolutional neural network (CNN) ;

[0061] S3: tight discrete feature input is obtained by discrete contrast learning, wherein the convolutional neural network of the key (Key) is not updated during training, and is updated by the parameters of the convolutional neural network of the query (Query), and finally the policy of the Soft Actor-Critic is used for evaluation and feedback.

[0062] In step S1, n sample transition tuples are read, and the query and the key are generated by using a random data augmentation method through observation input ; in step S2, the encoder CNN is used to become encoding and encoding , and then the sign function is used to become and .

[0063] In step S3, the data obtained in step S2 are used for discrete contrast learning:

[0064] The query representation and the key representation are given, wherein the key representation contains the positive sample key representation and the negative sample key representation , and the discrete contrast learning is performed by using the following formula:

[0065] (1).

[0066] Wherein is the transpose vector of the key representation vector, ​is a positive sample key representation for a vector matrix for linear transformation, is a negative sample key representation for .

[0067] Further, in step S3, define where is the column dimension of , is a hyperparameter, and are the first and second rows of and , can be seen as the inner product similarity of the first query representation and the second key representation, and define A as the sigmoid function of : Get the likelihood function of the similarity matrix, and the formula is:

[0068]

[0069] where the matrix S is the similarity adjacency matrix, which is set to the identity matrix here, and define

[0070] as follows:

[0071] (3)

[0072] Get the log-likelihood function of and :

[0073] (4)

[0074] The task of step S3 is to maximize the log-likelihood function, and the objective can be represented as follows:

[0075] (5).

[0076]

[0077] In step S3, fix update , first calculate the gradient and Hessian matrix of formula (5) about :

[0078] (6)

[0079] where ​​​​​, It is a diagonal matrix, defined ,in It is an identity matrix. It is the number of samples. yes Column dimensions, It's a hyperparameter; you can construct one. The lower bound:

[0080]

[0081] The optimization problem can be transformed into the following formula:

[0082]

[0083] The solution to formula (8) is:

[0084]

[0085] We can obtain the result from formula (9). .

[0086] In step S3, fix renew ,get Update formula:

[0087]

[0088] We can obtain the result from formula (10). .

[0089] In step S3, when it indicates and After the update is received, it is treated as a parameter. State encoder The autoregressive objective is updated by the encoder using mean squared error (MSE):

[0090]

[0091] The given parameters are key encoder and parameters are Query Encoder , Momentum update is performed without gradient backpropagation. The momentum update is as follows:

[0092]

[0093] In step S3, the policy evaluation part of the Soft Actor-Critic, the critic network is updated by minimizing the soft Bellman residual formula:

[0094]

[0095] wherein is the cache, is a sampled tuple, is the current state, is the action taken at the current time step, is the reward feedback at the current time step, is the state at the next time step, is the hyperparameter discount factor, is the action-state value function approximated by the critic network, is the target value function obtained by the Monte Carlo evaluation formula, and the evaluation formula is as follows:

[0096]

[0097] wherein is the delay parameter.

[0098] In step S3, the policy improvement part of the Soft Actor-Critic, the target loss function of the actor network is as follows:

[0099]

[0100] The present application has carried out two groups of simulation experiments in DeepMind Control Suite (DMControl): 100K steps and 500K steps experiments in various motion environments (finger-spin, cartpole-swingup, reacher-easy, cheetah-run, walker-walk, and ball in cup-catch.).

[0101] The specific implementation steps are as follows:

[0102] S1: If the cache is empty, randomly sample data in the simulation environment, and if the cache is not empty, sample data The sampled data is an anchor point. The query picture input and the key picture input are obtained through image enhancement, and the inputs are both 256*256 pictures.

[0103] S2: Input the query image and input the key image into the encoder. The query image encoder CNN has 4 layers and encodes the image into a 128-dimensional vector. This network participates in the entire training process. The key image encoder does not participate in the training process, and its network parameters are updated through the query image encoder.

[0104] S3: Input the query vector and key vector encoded into 128 dimensions into the discrete optimization model, which is updated by formula (9) and formula (10).

[0105] S4: When indicating and After the update, they can be used as parameters. encoder The regression target is updated by the encoder using formula (11), with the parameters given as follows: key encoder and parameters are Query Encoder , Momentum updates are performed without gradient backpropagation, and the momentum update uses formula (12).

[0106] S5: In the policy evaluation stage of Soft Actor-Critic, the critic network is updated using the formula (13) that minimizes the soft Bellman residual. In the policy enhancement stage of Soft Actor-Critic, the target loss function of the actor network is formula (15), and gradient backpropagation is performed using this formula.

[0107] Before the present application, the latest methods are (1) SAC algorithm based on auto-encoder (SAC-AE), using auto-encoding based on picture reconstruction as auxiliary task; (2) reinforcement learning algorithm based on contrastive unsupervised representation (Contrastive Unsupervised Representations for Reinforcement Learning, CURL), taking contrastive learning as auxiliary task; (3) deep planning network (Deep Planning Network, PlaNet), using an explicit environment model as an auxiliary task; (4) implicit model AC (Rastic Potential Actor Critic, SLAC), using an implicit environment model as an auxiliary task; (5) data regularization Q algorithm (Data-regularized Q, DrQ), using data augmentation as an auxiliary task; (6) pixel-based SAC algorithm (Pixel-SAC), using low-dimensional physical features as input. (7) state-based SAC algorithm (State-SAC), providing better input state. The present method is a reinforcement learning algorithm based on discrete contrastive features (Discrete Contrastive Representation Learning for Reinforcement Learning, DCRL) using discrete contrastive features as auxiliary tasks, and the results are shown in the accompanying Figure 2 The horizontal coordinate of the accompanying Figure 2 is the average performance of each model in six continuous control environments, and the vertical coordinate is the name of each model. It can be seen that in the experiment of taking the original picture as input and improving the sample utilization rate of the auxiliary task, DCRL is significantly better than SAC-AE, CURL, PlaNet, SLAC, DrQ. In comparison with Pixel-SAC which uses low-dimensional physical feature input, the results are also very close, but the present application does not need to input the low-dimensional physical feature input which is relatively difficult to obtain, and has more advantages in complex environments such as agricultural fields.

[0108] The same or similar reference signs correspond to the same or similar parts;

[0109] The positional relationship described in the accompanying drawings is only for illustrative purposes and cannot be understood as a limitation on the present patent;

[0110] Obviously, the above embodiments of the present application are merely exemplary but not intended to limit the embodiments of the present application. Based on the above description, any other variations or changes can be made by those skilled in the art without departing from the spirit and principles of the present application. It is not necessary to list all the embodiments here. Any modifications, equivalent replacements, and improvements made within the spirit and principles of the present application should be included in the protection scope of the claims of the present application.

Claims

1. A highly efficient recognition and control reinforcement learning method for agricultural harvesting, characterized in that, Includes the following steps: S1: Perform data augmentation on the acquired images to obtain the query and key, including: reading the transition tuples of n samples. , which contains observation input Using random data augmentation methods to observe input Generate query s and keys ; S2: This data is encoded into query features using a convolutional neural network (CNN). Bond features Then, through a symbolic function, it becomes and ; S3: Obtain tightly packed discrete feature inputs through discrete contrastive learning. The convolutional neural network for the key is not updated during training, but updated through the parameters of the convolutional neural network for query. Finally, a soft actuation / critic strategy is used for evaluation and feedback. The discrete contrastive learning specifically refers to: Give the query representation and key representation The key represents the key containing positive samples. and negative sample key representation Discrete contrastive learning is performed using the following loss function formula: （1） in It is the transpose of the key representation vector. This is for representing positive sample keys. The vector matrix undergoing linear transformation. The sampling index is The negative sample key representation; definition ,in yes Column dimensions, It's a hyperparameter. and They are and The row and number OK, See as the first The query represents and the first query. The inner product similarity is represented by each key, and A is defined as... The sigmoid function: The likelihood function of the similarity matrix is ​​obtained by the following formula: （2） Where S is the similarity adjacency matrix, which is set here as the identity matrix, defined as follows. as follows: get and log-likelihood function: （4） The task of step S3 is to maximize the log-likelihood function, and the objective is expressed as follows: （5） In step S3, fix it renew First calculate the expression of formula (5) with respect to... Gradient and Hessian matrix: （6） in , It is a diagonal matrix, defined ,in It is an identity matrix. It is the number of samples. yes Column dimensions, It's a hyperparameter, constructing one The lower bound: The optimization problem can be transformed into the following formula: The solution to formula (8) is: We obtain it through formula (9). ; fixed renew ,get Update formula: in yes and The log-likelihood function, To optimize intermediate variables, where It is an identity matrix. It is the number of samples. yes Column dimensions, It's a hyperparameter; We obtain it through formula (10). ; When indicating and After the update is received, it is treated as a parameter. State encoder The autoregressive objective is updated by the encoder using mean squared error (MSE): The given parameters are key encoder and parameters are Query Encoder , Momentum update is performed without gradient backpropagation. The momentum update is as follows: 。 2. The efficient recognition and control reinforcement learning method for agricultural harvesting according to claim 1, characterized in that, In step S3, during the policy evaluation phase of Soft Actor-Critic, the critic network is updated using the formula for minimizing the soft Bellman residual: in It's a cache. It is a sample tuple. This is the current state. It is the action taken at the current time step. It is the feedback from the current time step. This is the state at the next time step. It is a hyperparameter discount factor. It is the action-state value function approximated by the critic network. The objective value function is obtained through the Monte Carlo evaluation formula, which is as follows: in It is a delay parameter.

3. The efficient recognition and control reinforcement learning method for agricultural harvesting according to claim 2, characterized in that, In step S3, the policy enhancement stage of Soft Actor-Critic, the objective loss function of the actor network is as follows: in The parameter is Action network, The parameter is The action-state value function approximated by the critic network.

Images