Robot-oriented partial observation reinforcement learning method, device and equipment

Abstract

The invention provides a robot-oriented partial observation reinforcement learning method, device and equipment, and relates to the technical field of reinforcement learning, and the method comprises the steps: obtaining a first training data set, the first training data being observation information at different moments and a historical observation information sequence; performing feature extraction on the first training data at the t-th moment to obtain a current observation feature sequence and a historical observation feature sequence at the t-th moment, inputting the current observation feature sequence and the historical observation feature sequence into a memory module to obtain a confidence state at the t-th moment, and predicting to obtain a first candidate confidence state; performing feature extraction on the first training data at the (t + 1) th moment to obtain a current observation feature and a historical observation feature sequence at the (t + 1) th moment; inputting the confidence state into a target memory module to obtain a second candidate confidence state; an MSE loss function value is calculated, parameter updating is carried out, and a memory module and an executor network which are trained are obtained; and determining an execution action at the current moment by using the trained memory module and the executor network.

Description

Technical Field

[0001] The present application relates to the technical field of reinforcement learning, and in particular to a partial observation reinforcement learning method, device and equipment for robots. Background Art

[0002] Deep reinforcement learning (DRL) is the product of the combination of deep learning and reinforcement learning. It has the ability of deep learning perception and representation and the ability of reinforcement learning decision-making, which can make up for the shortcomings of classical reinforcement learning in estimating value functions and fitting policy functions. At present, the main achievements of reinforcement learning are still concentrated in virtual environments such as games. This is because the observation space of the virtual environment constructed by training reinforcement learning agents has the advantages of accurate observation and cheap data set generation.

[0003] However, when reinforcement learning is applied in real-world scenarios, the environment is usually random, complex, and partially observable. Specifically, in real scenarios, the state information collected often contains noise due to various reasons (such as errors in the sensor's measurement of speed and displacement), that is, a partially observed Markov environment. Such inaccurate state information often leads the agent to the wrong decision-making route, ultimately leading to the failure of the model. Therefore, it is necessary to develop a partially observed reinforcement learning method, device, and equipment for robots, so as to achieve accurate decision-making on the execution action when the collected information is noisy and incomplete. Summary of the invention

[0004] In view of the above problems, the embodiments of the present application provide a partial observation reinforcement learning method, device and equipment for robots to overcome the above problems or at least partially solve the above problems.

[0005] In a first aspect of an embodiment of the present application, a partial observation reinforcement learning method for a robot is provided, which is applied to the robot, and the method comprises:

[0006] Obtain a first training data set, wherein each first training data in the first training data set is observation information at different times and a corresponding historical observation information sequence obtained in a partial observation Markov environment; the historical observation information sequence represents a sequence composed of observation information at the previous N times in chronological order;

[0007] Input the first training data at time t into the feature extraction network to extract features, and obtain the current observation features and historical observation feature sequences at time t;

[0008] Inputting the current observation feature and the historical observation feature sequence at the t-th moment into the memory module to obtain the confidence state at the t-th moment;

[0009] According to the confidence state at the tth moment, predicting a first candidate confidence state at the t+1th moment;

[0010] Inputting the first training data at time t+1 into the feature extraction network to perform feature extraction to obtain the current observation feature at time t+1 and the historical observation feature sequence;

[0011] Inputting the current observation feature and the historical observation feature sequence at the t+1th time into the target memory module to obtain the second candidate confidence state at the t+1th time;

[0012] Calculate the MSE loss function value according to the first candidate confidence state and the second candidate confidence state at the t+1th time;

[0013] According to the MSE loss function value, update the parameters of the memory module and the executor network;

[0014] Reselecting the first training data and performing iterative training until the MSE loss function value reaches a convergence state or reaches a preset number of training times, thereby obtaining the trained memory module and the executor network;

[0015] The trained memory module and the executor network are used to determine the execution action of the robot at the current moment according to the observation information at the current moment.

[0016] The second aspect of the embodiment of the present application further proposes a partial observation reinforcement learning device for a robot, which is used to execute the partial observation reinforcement learning method for a robot described in the first aspect, and the device includes:

[0017] A first training data set acquisition module is used to acquire a first training data set, wherein each first training data in the first training data set is observation information at different times and a corresponding historical observation information sequence acquired in a partial observation Markov environment; the historical observation information sequence represents a sequence composed of observation information at the previous N times in chronological order;

[0018] A first feature extraction module is used to input the first training data at the tth moment into the feature extraction network to perform feature extraction to obtain the current observation feature and the historical observation feature sequence at the tth moment;

[0019] A first confidence state generating module, used for inputting the current observation feature and the historical observation feature sequence at the t-th moment into the memory module to obtain the confidence state at the t-th moment;

[0020] A first confidence state prediction module, used to predict a first candidate confidence state at time t+1 according to the confidence state at time t;

[0021] A second feature extraction module is used to input the first training data at time t+1 into the feature extraction network to perform feature extraction to obtain the current observation feature at time t+1 and a historical observation feature sequence;

[0022] A second confidence state generating module, used for inputting the current observation feature and the historical observation feature sequence at the t+1th time into the target memory module to obtain a second candidate confidence state at the t+1th time;

[0023] A loss function calculation module, used to calculate the MSE loss function value according to the first candidate confidence state and the second candidate confidence state at the t+1th time;

[0024] A parameter updating module, used for updating the parameters of the memory module and the executor network according to the MSE loss function value;

[0025] A training end module, used for reselecting the first training data and performing iterative training until the MSE loss function value reaches a convergence state or reaches a preset number of training times, thereby obtaining the trained memory module and the executor network;

[0026] The execution action prediction module is used to use the trained memory module and the executor network to determine the execution action of the robot at the current moment according to the observation information at the current moment.

[0027] The third aspect of the embodiments of the present application also provides an electronic device, including a memory, a processor and a computer program stored on the memory, wherein the processor executes the computer program to implement the steps in the partial observation reinforcement learning method for robots described in the first aspect of the embodiments of the present application.

[0028] The fourth aspect of the embodiment of the present application also provides a computer-readable storage medium, on which a computer program / instruction is stored. When the computer program / instruction is executed by a processor, the steps in the partial observation reinforcement learning method for robots described in the first aspect of the embodiment of the present application are implemented.

[0029] The fifth aspect of the embodiments of the present application also provides a computer program product. When the computer program product is run on an electronic device, the processor implements the steps in the partial observation reinforcement learning method for robots as described in the first aspect of the embodiments of the present application.

[0030] A partial observation reinforcement learning method for robots provided in an embodiment of the present application is applied to robots, and the method includes: obtaining a first training data set, wherein each first training data in the first training data set is observation information at different times and a corresponding historical observation information sequence obtained in a partial observation Markov environment; the historical observation information sequence represents a sequence of observation information at the previous N times in chronological order; inputting the first training data at time t into a feature extraction network to perform feature extraction to obtain the current observation feature at time t and a historical observation feature sequence; inputting the current observation feature at time t and the historical observation feature sequence into a memory module to obtain a confidence state at time t; predicting a first candidate confidence state at time t+1 based on the confidence state at time t; inputting the first training data at time t+1 into the memory module to obtain a confidence state at time t+1; A feature extraction network is used to extract features and obtain the current observed features at the t+1th moment and a sequence of historical observed features; the current observed features at the t+1th moment and the sequence of historical observed features are input into a target memory module to obtain a second candidate confidence state at the t+1th moment; the MSE loss function value is calculated according to the first candidate confidence state and the second candidate confidence state at the t+1th moment; the parameters of the memory module and the executor network are updated according to the MSE loss function value; the first training data is reselected and iterative training is performed until the MSE loss function value reaches a convergence state or a preset number of training times is reached to obtain the trained memory module and the executor network; the trained memory module and the executor network are used to determine the execution action of the robot at the current moment according to the observation information at the current moment.

[0031] The specific beneficial effects are: the embodiment of the present application proposes to use contrastive auxiliary tasks to train the decision model (memory module and executor network) to promote the decision model to learn the similarity between the confidence state at the next moment obtained by inferring the current confidence state (first candidate confidence state) and the confidence state inferred by the memory module (second candidate confidence state). By using the idea of ​​contrastive learning, the similarity between observations is measured by calculating the mean square loss function between them, and it is used as the goal of the auxiliary task, thereby improving the robustness and denoising performance of the decision model, so that it can be applied to a partial observation Markov environment (that is, an environment where the collected information is not accurate enough and there is noise), improving the performance of offline reinforcement learning, thereby generating a more accurate understanding of the environmental state (confidence state), and then obtaining a more accurate decision (execution action at the current moment). BRIEF DESCRIPTION OF THE DRAWINGS

[0032] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the description of the embodiments of the present application will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0033] Figure 1 It is a flowchart of the steps of a partial observation reinforcement learning method for a robot provided in an embodiment of the present application;

[0034] Figure 2 It is a structural diagram of a decision model provided in an embodiment of the present application;

[0035] Figure 3 It is a flowchart of training a decision model using comparative auxiliary tasks provided in an embodiment of the present application;

[0036] Figure 4 It is a partial observation reinforcement learning device for robots provided in an embodiment of the present application;

[0037] Figure 5 It is a schematic diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0038] The exemplary embodiments of the present application will be described in more detail below in conjunction with the accompanying drawings in the embodiments of the present application. Although the exemplary embodiments of the present application are shown in the accompanying drawings, it should be understood that the present application can be implemented in various forms and should not be limited by the embodiments set forth herein. On the contrary, these embodiments are provided in order to enable a more thorough understanding of the present application and to enable the scope of the present application to be fully communicated to those skilled in the art.

[0039] Deep learning has rapidly promoted the development of the field of artificial intelligence. Algorithms such as image recognition, target detection, text generation, and language translation based on deep learning have gradually matured and gradually entered people's daily lives. As an important branch of artificial intelligence, reinforcement learning is a different deep learning paradigm from supervised learning and unsupervised learning. It learns the optimal decision-making strategy by obtaining rewards through interaction with the environment, and is applicable to general sequential decision-making processes.

[0040] Deep reinforcement learning (DRL) is the product of the combination of deep learning and reinforcement learning. It has the ability of deep learning perception and representation and the ability of reinforcement learning decision-making, which can make up for the shortcomings of classical reinforcement learning in estimating value functions and fitting policy functions. Deep reinforcement learning has many well-known applications. For example, the newly released large language model ChatGPT uses a reinforcement learning method with human feedback, which enables the model to better understand human language. Deep reinforcement learning is also applied to biological information. The AlphaFold algorithm can accurately predict the three-dimensional structure of the corresponding protein from the amino acid sequence. In addition, deep reinforcement learning has achieved many remarkable results in the fields of Go, game competition, and autonomous driving.

[0041] Offline reinforcement learning (Offline RL), also known as batch reinforcement learning (BRL), is a reinforcement learning paradigm that is different from online reinforcement learning. It requires the agent to learn from a fixed dataset without exploring. In other words, Offline RL studies how to maximize the use of static offline datasets to train reinforcement learning agents.

[0042] Despite the impressive achievements in many fields, the main achievements of reinforcement learning are still concentrated in virtual environments such as games. This is because the observation space of the virtual environment constructed by training reinforcement learning agents has the advantages of accurate observation and cheap data set generation. When applied in the real world, such as in the medical robot environment, it is inevitable to consider how to land the agent in the virtual environment. Such a challenging problem. In real-world scenarios (such as medical scenarios), the environment is usually random, complex and partially observable. For example, in a cardiac ultrasound robot, ultrasound images are used as inputs to the reinforcement learning algorithm, and accurate and noise-free ultrasound images are expected. However, in reality, ultrasound images often have great noise, and the quality of ultrasound images is limited by the characteristics of ultrasound and imaging equipment. Ultrasound images are often blurred or missing due to breathing and small body movements of the examinee. In actual scenarios, it is difficult for agents to obtain accurate state information (such as errors in the measurement of speed and displacement by sensors), and such inaccurate state information often leads to the agent taking the wrong decision route, ultimately leading to the failure of the model.

[0043] In view of the above problems, the embodiments of the present application propose a partial observation reinforcement learning method, device and equipment for robots, so as to achieve accurate decision-making on the execution action of the robot when the collected information is noisy and incomplete. The partial observation reinforcement learning method for robots provided by the embodiments of the present application is described in detail below through some embodiments and their application scenarios in conjunction with the accompanying drawings.

[0044] The first aspect of the embodiment of the present application provides a partial observation reinforcement learning method for a robot, which is applied to the robot, referring to Figure 1 , Figure 1 A flowchart of a partial observation reinforcement learning method for a robot provided in an embodiment of the present application is shown in FIG. Figure 1 As shown, the method includes:

[0045] Step S101, obtain a first training data set, each first training data in the first training data set is observation information at different times and a corresponding historical observation information sequence obtained in a partial observation Markov environment; the historical observation information sequence represents a sequence of observation information at the previous N times in chronological order.

[0046] Step S102, inputting the first training data at the tth moment into the feature extraction network to perform feature extraction to obtain the current observation feature and the historical observation feature sequence at the tth moment.

[0047] Step S103, inputting the current observation feature and the historical observation feature sequence at the tth moment into a memory module to obtain the confidence state at the tth moment.

[0048] Step S104: predicting a first candidate confidence state at time t+1 according to the confidence state at time t.

[0049] Step S105, inputting the first training data at the time t+1 into the feature extraction network to perform feature extraction to obtain the current observation features at the time t+1 and the historical observation feature sequence.

[0050] Step S106, inputting the current observation feature and the historical observation feature sequence at the t+1th moment into the target memory module to obtain the second candidate confidence state at the t+1th moment.

[0051] Step S107, calculating the MSE loss function value according to the first candidate confidence state and the second candidate confidence state at the t+1th time;

[0052] Step S108, updating the parameters of the memory module and the executor network according to the MSE loss function value;

[0053] Step S109, reselecting the first training data and performing iterative training until the MSE loss function value reaches a convergence state or reaches a preset number of training times, thereby obtaining the trained memory module and the executor network;

[0054] Step S110, using the trained memory module and the executor network, and according to the observation information at the current moment, determining the execution action of the robot at the current moment.

[0055] For situations where there is noise in the environment, or where the information obtained by the sensor is noisy or incomplete, a mathematical model has been proposed in the related art, namely the Partially Observed Markov Decision Process (POM DP). This mathematical model believes that since the observation is noisy or incomplete, it is unreliable for the agent to select and execute actions based only on the observations at the current moment. A method often used in POM DP is to encode its past observations and actions into a confidence state, and select the action that the robot needs to perform based on the confidence state. The embodiment of the present application is based on an offline reinforcement learning method to train a decision model (including a feature extraction network, a memory module, and an executor network) so that it can generate accurate decisions (the robot's execution action at the current moment) in POM DP (i.e., in an environment where the information obtained by the sensor is noisy and the information is incomplete).

[0056] The partially observed Markov decision process can be defined by a seven-tuple (S, A, O, P, R, Z, γ), where S represents the state space, A represents the action space, O represents the observation space, P represents the state transition probability, R represents the reward function, Z represents the observation function, and γ∈(0, 1] represents the discount factor. Assume that s∈S represents the state of the environment, α∈A represents the action of the agent (i.e., the decision model), and o∈O represents the observation of the agent. The difference between the partially observed Markov decision process and the Markov decision process is that at the same time t, in the Markov decision process, o t =s t , while in the partially observed Markov decision process, o t ≠s t , so the Markov decision process can be regarded as a special case of the partially observed Markov decision process.

[0057] Specifically, in step S101, a first training data set is first established, and the first training data set includes first training data at multiple different moments, and each first training data includes: the observation information at that moment and the corresponding historical observation information sequence obtained in a partial observation Markov environment. Among them, the observation information represents the information collected by the sensor in the POMDP at that moment, that is, the observation information in the first training data may contain some noise. The historical observation information sequence represents a sequence of observation information at N moments before that moment in chronological order. N is a positive integer greater than 1. Exemplarily, in a medical scenario, a cardiac ultrasound robot needs to perform the next surgical operation based on the cardiac ultrasound image collected by the image sensor. In this scenario, the observation information o t It represents the cardiac ultrasound image acquired by the image sensor at the tth moment. Assuming that a cardiac ultrasound image is acquired every 1s, the corresponding historical observation information sequence h t It represents the sequence of 60 cardiac ultrasound images collected within one minute before the current moment in chronological order (o t-N , o t-N+1 ,……,o t-1 ). If the current time t satisfies t ≥ N, then the historical observation information sequence h at time t t It is defined as a set of historical observation information with a length of N in the past. Otherwise, the observation vectors at time less than or equal to 0 will be filled with zero vectors consistent with the observation dimension.

[0058] In step S102, the first training data at the tth moment is input into the feature extraction network for feature extraction to obtain the current observation feature and the historical observation feature sequence at the tth moment. Specifically, a first training data is randomly selected from the first training data set, and t represents the moment corresponding to the first training data. Feature extraction is performed through the feature extraction network. Specifically, the first training data at the tth moment includes: the observation information at the tth moment, and the historical observation information sequence composed of the observation information at N moments before the tth moment. The observation information at the tth moment is input into the feature extraction network to obtain the current observation feature at the tth moment; the historical observation information sequence at the tth moment is input into the feature extraction network to obtain the historical observation feature sequence at the tth moment. The feature extraction network selected in this embodiment can be a feature extractor based on a convolutional neural network, such as ResNet34. Exemplarily, in the application scenario of a cardiac ultrasound robot, the cardiac ultrasound image (observation information) at the tth moment is input into the feature extraction network to extract the cardiac ultrasound image features at the tth moment (in this case, the feature extraction network extracts the image features).

[0059] Before executing step S102, in order to accurately extract the desired features, the feature extraction network needs to be trained. In a possible implementation, before training the memory module, the method further includes: training the feature extraction network; the training process of the feature extraction network is as follows:

[0060] Step S201: Acquire a second training data set, where each second training data in the second training data set includes: observation information at a corresponding moment and a historical observation information sequence.

[0061] Step S202: input the second training data into the feature extraction network for feature extraction to obtain current observation features and historical observation feature sequences of the second training data.

[0062] Step S203, input the current observation features and the historical observation feature sequence of the second training data into a classifier, perform classification prediction, and obtain a classification result. The classification result represents the probability distribution of the execution action corresponding to the second training data; specifically, the classifier can be a linear classifier, and its specific structure is not limited in this embodiment.

[0063] Step S204, calculating a loss function value according to the classification result and the label of the second training data. The label of the second training data indicates the execution action that should be taken at the moment corresponding to the second training data.

[0064] Step S205: updating parameters of the feature extraction network according to the loss function value.

[0065] Step S206, reselect the second training data to train the feature extraction network until the loss function value converges or reaches a preset number of training times, and then terminate the training to obtain the trained feature extraction network.

[0066] In the related art, the task of the agent (i.e., the decision model used to predict the execution action) is often to extract observation features or extract time series features. In order to allow the agent to fully learn the two features, the embodiment of the present application proposes a two-stage deep reinforcement learning framework, so that the agent learns the observation features and time series features in stages. That is, in the first stage, through steps S201-S206, the feature extraction network is trained to learn the observation features. In different application scenarios, the data form of the observation information is different. Specifically, the observation information can be measured numerical data, image data obtained by shooting, or audio data obtained by recording, etc. In the case where the observation feature is a cardiac ultrasound robot, the cardiac ultrasound image obtained by shooting is the observation information, and correspondingly, the feature extraction network needs to learn image features. After the second training data passes through the feature extraction network, it enters the linear classifier to obtain the probability distribution of the execution action at that moment. The purpose of this stage is to better train the feature extraction network, avoid training the feature extraction network and the memory module together to increase the difficulty of model learning, and lead to a decrease in the ability to extract features and history.

[0067] After the feature extraction network is trained in the first stage, the linear classifier is discarded and the parameters of the feature extraction network are frozen. Then, through steps S101-S109, the memory module and the executor network are trained. That is, the main training target in the second stage (steps S101-S109) is the memory module, and the auxiliary tasks are used to improve the memory module's ability to extract time series features.

[0068] In step S103, the current observation feature and the historical observation feature sequence at the tth moment extracted in step S102 are input into the memory module to obtain the confidence state b at the tth moment. t .

[0069] Specifically, in the POM DP environment, the strategy of the decision model to select actions only based on the current state usually performs poorly, because the observation information at the current moment cannot provide enough information to accurately describe the current true state. Related research shows that the distribution of the true state can be inferred through previous experience. This inference is called a confidence state, which refers to the decision model's belief in the current possible true state and is expressed through the Bayesian posterior distribution p(s t |o ≤t , a <t ) is formally defined. The posterior distribution combines the previous experience with the uncertainty of the unobserved world. In the embodiment of the present application, h is defined t :=(o <t , a ＜t) is the observation-action history (i.e., the historical observation information sequence composed of the observation information of the previous N moments, and the historical execution action sequence composed of the execution actions of the previous N moments), and the confidence state b is defined t :=φ(h t ) is h t If we can learn b t And take it as the sufficient statistics of the state, that is, p(s t | o ≤t , a <t ≈p(s t |b t ), then you can use b t As a representation of confidence, it is used to train the decision model in POMDP. Therefore, the POM DP problem can be solved by introducing the concept of confidence state.

[0070] In a possible implementation, the step S103, inputting the current observation feature and the historical observation feature sequence into the trained memory module to extract the time series feature to obtain the confidence state at the current moment, includes:

[0071] Step S1031, encoding is performed according to the current observation feature to obtain the current encoding state.

[0072] Step S1032: Encoding is performed according to the historical observation feature sequence to obtain a historical encoding state.

[0073] Step S1033, concatenating the current encoding state with the historical encoding state to obtain the confidence state at the current moment.

[0074] Specifically, the embodiments of this application refer to Figure 2 , Figure 2 A structural diagram of a decision model is shown, such as Figure 2 As shown in Figure 2, the decision model consists of a memory module and an executor network, where the memory module includes a memory neural network (e.g. Figure 2 The shown one is composed of a fully connected layer FC and an LSTM network and another fully connected layer FC in series) and a fully connected layer (such as Figure 2 The memory module has two inputs, namely the current observation feature and the historical observation feature sequence. The memory module uses the memory neural network to encode the historical observation feature sequence to obtain the historical encoding state; and uses the fully connected layer to encode the current observation feature to obtain the current encoding state. The historical encoding state and the current encoding state are output and then spliced ​​(as shown in Figure 2 Splice symbols shown ), the concatenated vector is used as the confidence state corresponding to the current moment, and then passed through the fully connected layer (such as Figure 2 As shown in the figure, the actor network is composed of multiple fully connected layers (FC connections), and the activation function between the FC layers is the tanh function) to obtain the action output. The purpose of this design (the memory module includes a memory neural network and a fully connected layer) is to balance the weights between historical observations and current observations, ensuring that the decision model can obtain important information from both historical observations and current observations. Since the current observation contains the latest information of the decision model in the environment, its importance increases accordingly. By increasing the weight of the current observation as input in the neural network, it can ensure that the algorithm can pay more attention to the characteristics of the current state, so that the decision of the intelligent agent is more accurate and timely.

[0075] In a possible implementation, the memory module and the target memory module include a long short-term memory network, and the long short-term memory network is composed of a fully connected layer-LSTM layer-fully connected layer, and the activation function between each layer is a ReLU function.

[0076] Specifically, in order to make the confidence state b t In order to fully represent historical information, the present application embodiment will use some models with memory functions to build memory modules, such as recurrent neural networks (RNN) and long short-term memory networks (LSTM). These models can memorize previous state information and incorporate it into the current state representation, thereby improving the performance and stability of the decision model. In this embodiment, Figure 2 As shown, a long short-term memory network is used as a memory neural network in the memory module (corresponding to step S1031 above, used to encode according to the current observation features to obtain the current encoding state) to extract information from historical observations. The long short-term memory network (LSTM) is a memory network improved from the recurrent neural network. It can solve the gradient disappearance and gradient explosion problems that are prone to occur in ordinary recurrent neural networks, and has better short-term and long-term memory capabilities. Since the rewards from the environment as signals for training reinforcement learning agents are usually sparse, the LSTM layer will have a weak ability to extract time series features, and thus cannot provide sufficient statistical information for the confidence state b t In order to improve the estimation of the confidence state, the embodiment of the present application adopts a comparative auxiliary task through steps S104-S108.

[0077] In step S104, based on the confidence state at the tth moment obtained in step S103, a first candidate confidence state at the t+1th moment is predicted.

[0078] In a possible implementation, the step S104, predicting a first candidate confidence state at time t+1 according to the confidence state at time t, includes:

[0079] A historical action information sequence at the t-th moment is obtained, where the historical action information sequence represents a sequence of action information at previous N moments including the t-th moment in chronological order.

[0080] The historical action information sequence at the t-th moment and the confidence state at the t-th moment are input into a state prediction module to predict a first candidate confidence state at the t+1-th moment.

[0081] The state prediction module consists of linear layers to generate continuous confidence states.

[0082] Specifically, N is a positive integer greater than 1. For example, in a medical scenario, a cardiac ultrasound robot needs to perform the next surgical operation based on the cardiac ultrasound image acquired by the image sensor. In this scenario, the observation information o t It represents the cardiac ultrasound image acquired by the image sensor at the tth moment. Assuming that a cardiac ultrasound image is acquired every 1s, the corresponding historical observation information sequence h t It represents the sequence of 60 cardiac ultrasound images collected within one minute before the current moment in chronological order (o t-N , o t-N+1 ,……,o t-1 Correspondingly, the action information represents the actions performed by the cardiac ultrasound robot during surgery. Assuming that an action is performed every 1s, the historical action information sequence represents the 60 actions performed by the cardiac ultrasound robot within one minute before the current moment, in chronological order (α t-N , α t-N+1 , ..., α t-1 In the auxiliary task, a new neural network (state prediction module) is created, which is a linear neural network that uses the input historical action information sequence and the confidence state (b t ) to predict the confidence state at the next moment, that is, the first candidate confidence state at the t+1th moment (b t+1 ).

[0083] In step S105, the first training data at the time t+1 is input into the feature extraction network to perform feature extraction to obtain the current observation features at the time t+1 and the historical observation feature sequence.

[0084] Specifically, in step S102, the first training data at the tth moment is input, and correspondingly in step S105, the first training data at the next moment, that is, the first training data at the t+1th moment, needs to be input, and feature extraction is performed through the feature extraction network. Specifically, the first training data at the t+1th moment includes: the observation information at the t+1th moment, and the historical observation information sequence composed of the observation information at the N moments before the t+1th moment. The observation information at the t+1th moment is input into the feature extraction network to obtain the current observation feature at the t+1th moment; the historical observation information sequence at the t+1th moment is input into the feature extraction network to obtain the historical observation feature sequence at the t+1th moment. Exemplarily, in the application scenario of the cardiac ultrasound robot, the cardiac ultrasound image (observation information) at the t+1th moment is input into the feature extraction network to extract the cardiac ultrasound image features at the t+1th moment (in this case, the feature extraction network extracts the image features).

[0085] In step S106, the current observation feature and the historical observation feature sequence at the t+1th moment are input into the target memory module to obtain the second candidate confidence state at the t+1th moment.

[0086] Reference Figure 3 , Figure 3 A schematic diagram of a process of training a decision model using contrastive auxiliary tasks is shown, such as Figure 3 As shown, the current observation feature and the historical observation feature sequence extracted in step S105 at the t+1th moment are input into the target memory module to predict the second candidate confidence state at the t+1th moment (such as Figure 3 The b shown t+1 ). The embodiment of the present application sets a target memory module The target memory module The initialization parameters of the memory module before training The target memory module is exactly the same, and its function is also to store the experience data of the decision model's past interactions with the environment. By using the target memory module, an independent target memory neural network can be used during the training process to output the current action and estimate the confidence state at the next moment, thereby avoiding the memory module parameters gradually converging to zero during the training process, reducing the instability during the training process, and improving the convergence speed and performance of the algorithm. The target memory module can be expressed as

[0087] In step S107, the MSE loss function value is calculated according to the first candidate confidence state and the second candidate confidence state at the time t+1.

[0088] The embodiment of the present application creates a linear connection layer and regards it as a forward model (such as Figure 3The state prediction module shown in Figure 2) takes the confidence state (bt) and action (α t-N , α t-N+1 , ..., α t-1 , α t ), the output is the prediction of the confidence state at the next moment (such as Figure 3 The first candidate confidence state shown in FIG. 1 ). This embodiment uses the mean square error (as shown in FIG. 1 ) between the confidence state predicted by the memory module (the second candidate confidence state) and the confidence state inferred by the forward model (the first candidate confidence state) to be minimized. Figure 3 MSE Loss shown in the figure is used to improve the information extraction ability of the memory module.

[0089] In a possible implementation, calculating the MSE loss function according to the first candidate confidence state and the second candidate confidence state at the t+1th time includes:

[0090] The MSE loss function is calculated according to the first candidate confidence state and the second candidate confidence state at the t+1th time according to the following formula:

[0091]

[0092] in, represents the first candidate confidence state at the t+1th time, represents the second candidate confidence state at the t+1th time. and To calculate the minimum mean squared error (MSE) loss function, the memory module is optimized so that it can learn to better extract historical information (time series features) and ignore the function of some incomplete observation features.

[0093] Step S108: updating the parameters of the memory module and the executor network according to the MSE loss function value.

[0094] like Figure 3 As shown, in the first stage, the feature extraction network is first trained to provide better feature representation for the reinforcement learning in the second stage. In the second stage, the focus is on the extraction of historical features, and the current artifacts and other incomplete observation information are corrected through historical information. That is, the memory module is trained through steps S101-S108 to learn to extract time series features. In an embodiment of the present application, the idea of ​​contrastive learning is introduced into the auxiliary task, and the noisy observations in the target data set are changed to a confidence state. The confidence state at time t+1 is represented by two neural networks (i.e., the first candidate confidence state and the second candidate confidence state ), and the two representations are closer, so that the model can extract some more common historical features and avoid noise interference. In order to prevent the decision model from collapsing, stop the gradient return to the target memory module, and use the soft update method to update its parameters (such as Figure 3 EMA shown).

[0095] In a possible implementation manner, the initialization parameters of the target memory module are the same as those of the memory module, and the training process further includes:

[0096] After the parameters of the memory module are updated, the parameters of the target memory module are updated according to the following formula:

[0097] φ′←(1-τ)φ′+τφ;

[0098] Among them, φ′ represents the parameters of the target memory module, φ represents the parameters of the memory module, and τ represents a pre-set hyperparameter (generally set to a constant close to 1 such as 0.99).

[0099] Specifically, if the target memory module is updated together with the memory module and minimizes the first candidate confidence state and the second candidate confidence state The network is easy to learn when the mean square error between them is small. When all network parameters tend to 0, the loss function tends to be minimum, which leads to network collapse and training failure. Therefore, the embodiment of the present application adopts a soft update method, so that the update speed of the target memory module is slower than that of the memory module, thereby improving the stability of the model.

[0100] Step S109, reselecting the first training data and performing iterative training until the MSE loss function value reaches a convergence state or reaches a preset number of training times, thereby obtaining the trained memory module and the executor network.

[0101] Step S110, using the trained memory module and the executor network, and according to the observation information at the current moment, determining the execution action of the robot at the current moment.

[0102] After training the memory module and the executor network through steps S101-S109, the model is used to predict the execution action that should be performed at the current moment. For example, in the application scenario of the cardiac ultrasound robot, the robot can determine the action that should be performed in the operation at the current moment based on the cardiac ultrasound image taken at the current moment, and perform the cardiac ultrasound robot that performs the corresponding surgical operation. In the embodiment of the present application, in a partially observed Markov environment (there is noise in the observed information), an offline reinforcement learning method is used to obtain higher decision-making performance, so as to utilize the observation information at the current moment to generate more accurate decisions (the robot's execution action). In order to achieve this goal, the embodiment of the present application adopts a memory-based reinforcement learning method, and also uses auxiliary tasks, contrastive learning and other methods to improve the decision model (memory module and executor network). The performance of the decision model finally trained is significantly improved, and it can be applied to various decision-making scenarios (such as cardiac ultrasound robots).

[0103] In a possible implementation, the step S110, using the trained memory module and the executor network, and according to the observation information at the current moment, determining the execution action of the robot at the current moment includes:

[0104] Step S1101, obtaining the observation information at the current moment and the corresponding historical observation information sequence. Specifically, in actual application, the obtained observation information (including the historical observation information sequence) may contain some noise.

[0105] Step S1102, inputting the observation information at the current moment and the corresponding historical observation information sequence into the feature extraction network for feature extraction, to obtain the current observation feature and the historical observation feature sequence.

[0106] Step S1103, input the current observation feature and the historical observation feature sequence into the trained memory module, perform time series feature extraction, and obtain the confidence state at the current moment; the confidence state represents the predicted real state at the current moment.

[0107] Step S1104, inputting the confidence state at the current moment into the trained executor network to determine the execution action of the robot at the current moment.

[0108] The embodiment of the present application adopts a memory-based reinforcement learning method, introduces a memory module, and uses auxiliary tasks, contrastive learning and other methods to improve the reinforcement learning algorithm, so that it can be applied to a partial observation Markov environment (i.e., an environment where the collected information is not accurate enough and there is noise), thereby improving the performance of offline reinforcement learning, thereby generating a more accurate understanding of the environmental state (confidence state), and then obtaining a more accurate decision (the execution action of the robot at the current moment).

[0109] In recent years, reinforcement learning has shown great application potential in many fields. For example, in tasks such as robot control, autonomous driving, and e-sports, reinforcement learning has achieved remarkable results. In addition, reinforcement learning has also shown broad application prospects in practical problems such as resource allocation, network optimization, and medical decision-making. Although the current research on reinforcement learning mainly focuses on the problem of rich and accurate observation space information, this assumption is not practical in real-world robotics applications, especially in the field of medical robots. Limited by problems such as sensor sensitivity, sensor noise, human physiological complexity, and some characteristics of medical examinations, the environmental information received by medical robots may often contain missing and inaccurate situations. Partially observable Markov decision processes (POMDPs) provide a way to model this incomplete observation situation. In POMDPs, the agent can only observe part of the environment and needs to obtain a more accurate understanding of the environment state through inference and estimation. In POMDPs, reinforcement learning algorithms can make decisions and learn by learning the model and inferred state of the environment, thereby adapting to the incomplete observation situation in the real world.

[0110] The embodiment of the present application proposes to use contrastive auxiliary tasks to train the decision model (memory module and executor network) to promote the decision model to learn the similarity between the confidence state at the next moment obtained by inferring the current confidence state (first candidate confidence state) and the confidence state inferred by the memory module (second candidate confidence state). Using the idea of ​​contrastive learning, the similarity between observations is measured by calculating the mean square loss function between them, and it is used as the goal of the auxiliary task, thereby improving the robustness and denoising performance of the decision model, so that it can still perform good decision-making performance in the POMDP environment.

[0111] The second aspect of the embodiment of the present application also provides a partial observation reinforcement learning device for a robot, referring to Figure 4 , Figure 4 A partial observation reinforcement learning device for robots is shown, such as Figure 4 As shown, the device for executing the partial observation reinforcement learning method for robots described in the first aspect includes:

[0112] A first training data set acquisition module is used to acquire a first training data set, wherein each first training data in the first training data set is observation information at different times and a corresponding historical observation information sequence acquired in a partial observation Markov environment; the historical observation information sequence represents a sequence composed of observation information at the previous N times in chronological order;

[0113] A first feature extraction module is used to input the first training data at the tth moment into the feature extraction network to perform feature extraction to obtain the current observation feature and the historical observation feature sequence at the tth moment;

[0114] A first confidence state generating module, used for inputting the current observation feature and the historical observation feature sequence at the t-th moment into the memory module to obtain the confidence state at the t-th moment;

[0115] A first confidence state prediction module, used to predict a first candidate confidence state at time t+1 according to the confidence state at time t;

[0116] A second feature extraction module is used to input the first training data at time t+1 into the feature extraction network to perform feature extraction to obtain the current observation feature at time t+1 and a historical observation feature sequence;

[0117] A second confidence state generating module, used for inputting the current observation feature and the historical observation feature sequence at the t+1th time into the target memory module to obtain a second candidate confidence state at the t+1th time;

[0118] A loss function calculation module, used to calculate the MSE loss function value according to the first candidate confidence state and the second candidate confidence state at the t+1th time;

[0119] A parameter updating module, used for updating the parameters of the memory module and the executor network according to the MSE loss function value;

[0120] A training end module, used for reselecting the first training data and performing iterative training until the MSE loss function value reaches a convergence state or reaches a preset number of training times, thereby obtaining the trained memory module and the executor network;

[0121] The execution action prediction module is used to use the trained memory module and the executor network to determine the execution action of the robot at the current moment according to the observation information at the current moment.

[0122] In a possible implementation, the execution action prediction module includes:

[0123] An observation information acquisition submodule, used to acquire the observation information at the current moment and the corresponding historical observation information sequence;

[0124] A feature extraction submodule is used to input the observation information at the current moment and the corresponding historical observation information sequence into the feature extraction network for feature extraction to obtain the current observation feature and the historical observation feature sequence;

[0125] A confidence state generation submodule is used to input the current observation feature and the historical observation feature sequence into the trained memory module to extract the time series feature to obtain the confidence state at the current moment; the confidence state represents the real state of the current moment of the prediction;

[0126] The execution action determination submodule is used to input the confidence state at the current moment into the trained executor network to determine the execution action of the robot at the current moment.

[0127] In a possible implementation manner, the initialization parameters of the target memory module are the same as those of the memory module, and the device further includes:

[0128] The target memory module parameter updating module is used to update the parameters of the target memory module according to the following formula after the parameters of the memory module are updated:

[0129] φ′←(1-τ)φ′+τφ;

[0130] Wherein, φ′ represents the parameters of the target memory module, φ represents the parameters of the memory module, and τ represents the pre-set hyperparameters.

[0131] In a possible implementation, the first confidence state prediction module includes:

[0132] A historical action information acquisition submodule is used to acquire a historical action information sequence at the t-th moment, wherein the historical action information sequence represents a sequence of action information at the previous N moments including the t-th moment in chronological order;

[0133] The first candidate confidence state prediction submodule is used to input the historical action information sequence at the t-th moment and the confidence state at the t-th moment into the state prediction module to predict the first candidate confidence state at the t+1-th moment.

[0134] In a possible implementation, the memory module and the target memory module include a long short-term memory network, and the long short-term memory network is composed of a fully connected layer-LSTM layer-fully connected layer, and the activation function between each layer is a ReLU function.

[0135] In a possible implementation, the loss function calculation module includes:

[0136] A calculation submodule, configured to calculate the MSE loss function according to the first candidate confidence state and the second candidate confidence state at the t+1th time according to the following formula:

[0137]

[0138] in, represents the first candidate confidence state at the t+1th time, represents the second candidate confidence state at the t+1th time.

[0139] In a possible implementation, the confidence state generating submodule includes:

[0140] A first encoding unit, used for encoding according to the current observation feature to obtain a current encoding state;

[0141] A second encoding unit is used to encode according to the historical observation feature sequence to obtain a historical encoding state;

[0142] The concatenation unit is used to concatenate the current encoding state with the historical encoding state to obtain the confidence state at the current moment.

[0143] In a possible implementation, the device further includes: a feature extraction network training module, which is used to train the feature extraction network before training the memory module; the training process of the feature extraction network is as follows:

[0144] Acquire a second training data set, wherein each second training data in the second training data set includes: observation information at a corresponding moment and a historical observation information sequence;

[0145] Inputting the second training data into the feature extraction network to perform feature extraction, thereby obtaining current observation features and historical observation feature sequences of the second training data;

[0146] Inputting the current observation feature and the historical observation feature sequence of the second training data into a classifier to perform classification prediction to obtain a classification result, wherein the classification result represents a probability distribution of the execution action corresponding to the second training data;

[0147] Calculating a loss function value according to the classification result and the label of the second training data;

[0148] According to the loss function value, updating the parameters of the feature extraction network;

[0149] Reselect the second training data to train the feature extraction network until the loss function value converges or reaches a preset number of training times, and then terminate the training to obtain the trained feature extraction network.

[0150] The present application also provides an electronic device, referring to Figure 5 , Figure 5 Schematic diagram of an electronic device proposed in an embodiment of the present application. Figure 5 As shown, the electronic device 100 includes: a memory 110 and a processor 120. The memory 110 and the processor 120 are connected via a bus communication. A computer program is stored in the memory 110. The computer program can be run on the processor 120 to implement the steps in the partial observation reinforcement learning method for robots disclosed in the embodiment of the present application.

[0151] The embodiment of the present application also provides a computer-readable storage medium on which a computer program / instruction is stored. When the computer program / instruction is executed by a processor, the steps in the partial observation reinforcement learning method for robots as disclosed in the embodiment of the present application are implemented.

[0152] The embodiment of the present application also provides a computer program product. When the computer program product is run on an electronic device, the processor implements the steps of the partial observation reinforcement learning method for robots as disclosed in the embodiment of the present application.

[0153] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0154] The embodiments of the present application are described with reference to the flowcharts and / or block diagrams of the methods, apparatuses, electronic devices, and computer program products according to the embodiments of the present application. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of the processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing terminal device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing terminal device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0155] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing terminal device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce a manufactured product including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0156] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal device so that a series of operating steps are executed on the computer or other programmable terminal device to produce a computer-implemented process, thereby providing instructions for executing on the computer or other programmable terminal device to implement the process. Figure 1 A process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0157] Although the preferred embodiments of the present application have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the embodiments of the present application.

[0158] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or terminal device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or terminal device. In the absence of further restrictions, the elements defined by the sentence "including one..." do not exclude the existence of other identical elements in the process, method, article or terminal device including the elements.

[0159] The above is a detailed introduction to the partial observation reinforcement learning method, device and equipment for robots provided by the present application. Specific examples are used in this article to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea; at the same time, for general technical personnel in this field, according to the idea of ​​the present application, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.

Claims

1. A partial observation reinforcement learning method for robots, characterized in that: Applied to a robot, the method comprises: Obtain a first training data set, wherein each first training data in the first training data set is observation information at different times and a corresponding historical observation information sequence obtained in a partial observation Markov environment; the historical observation information sequence represents a sequence composed of observation information at the previous N times in chronological order; Input the first training data at time t into the feature extraction network to extract features, and obtain the current observation features and historical observation feature sequences at time t; Inputting the current observation feature and the historical observation feature sequence at the t-th moment into the memory module to obtain the confidence state at the t-th moment; According to the confidence state at the tth moment, predicting a first candidate confidence state at the t+1th moment; Inputting the first training data at time t+1 into the feature extraction network to perform feature extraction to obtain the current observation feature at time t+1 and the historical observation feature sequence; Inputting the current observation feature and the historical observation feature sequence at the t+1th time into the target memory module to obtain the second candidate confidence state at the t+1th time; Calculate the MSE loss function value according to the first candidate confidence state and the second candidate confidence state at the t+1th time; According to the MSE loss function value, updating the parameters of the memory module and the executor network; Reselecting the first training data and performing iterative training until the MSE loss function value reaches a convergence state or reaches a preset number of training times, thereby obtaining the trained memory module and the executor network; The trained memory module and the executor network are used to determine the execution action of the robot at the current moment according to the observation information at the current moment.

2. The robot-oriented partial observation reinforcement learning method according to claim 1, characterized in that: The method of using the trained memory module and the executor network to determine the execution action of the robot at the current moment according to the observation information at the current moment includes: Obtaining the observation information at the current moment and the corresponding historical observation information sequence; Inputting the observation information at the current moment and the corresponding historical observation information sequence into the feature extraction network for feature extraction, respectively, to obtain the current observation feature and the historical observation feature sequence; The current observation feature and the historical observation feature sequence are input into the trained memory module to extract the time series feature to obtain the confidence state at the current moment; the confidence state represents the predicted real state at the current moment; The confidence state at the current moment is input into the trained executor network to determine the execution action of the robot at the current moment.

3. The robot-oriented partial observation reinforcement learning method according to claim 1, characterized in that: The initialization parameters of the target memory module are the same as those of the memory module, and the training process further includes: After the parameters of the memory module are updated, the parameters of the target memory module are updated according to the following formula: φ′←(1-τ)φ′+τφ; Wherein, φ′ represents the parameters of the target memory module, φ represents the parameters of the memory module, and τ represents the pre-set hyperparameters.

4. The robot-oriented partial observation reinforcement learning method according to claim 2, characterized in that: The predicting, based on the confidence state at the tth moment, to obtain a first candidate confidence state at the t+1th moment includes: Obtaining a historical action information sequence at the t-th moment, wherein the historical action information sequence represents a sequence of action information at the previous N moments including the t-th moment in chronological order; The historical action information sequence at the t-th moment and the confidence state at the t-th moment are input into a state prediction module to predict a first candidate confidence state at the t+1-th moment.

5. The robot-oriented partial observation reinforcement learning method according to claim 2, characterized in that: The memory module and the target memory module include a long short-term memory network, which is composed of a fully connected layer-LSTM layer-fully connected layer, and the activation function between each layer is a ReLU function.

6. The robot-oriented partial observation reinforcement learning method according to claim 2, characterized in that: The calculating the MSE loss function according to the first candidate confidence state and the second candidate confidence state at the t+1th time includes: The MSE loss function is calculated according to the first candidate confidence state and the second candidate confidence state at the t+1th time according to the following formula: in, represents the first candidate confidence state at the t+1th time, represents the second candidate confidence state at the t+1th time.

7. The robot-oriented partial observation reinforcement learning method according to claim 2, characterized in that: The step of inputting the current observation feature and the historical observation feature sequence into the trained memory module to extract the time series feature to obtain the confidence state at the current moment includes: Encoding is performed according to the current observation feature to obtain a current encoding state; Encoding is performed according to the historical observation feature sequence to obtain a historical encoding state; The current encoding state is concatenated with the historical encoding state to obtain the confidence state at the current moment.

8. The robot-oriented partial observation reinforcement learning method according to claim 1, characterized in that: Before training the memory module, the method further includes: training the feature extraction network; The training process of the feature extraction network is as follows: Acquire a second training data set, wherein each second training data in the second training data set includes: observation information at a corresponding moment and a historical observation information sequence; Inputting the second training data into the feature extraction network to perform feature extraction, thereby obtaining current observation features and historical observation feature sequences of the second training data; Inputting the current observation feature and the historical observation feature sequence of the second training data into a classifier to perform classification prediction to obtain a classification result, wherein the classification result represents a probability distribution of the execution action corresponding to the second training data; Calculating a loss function value according to the classification result and the label of the second training data; According to the loss function value, updating the parameters of the feature extraction network; Reselect the second training data to train the feature extraction network until the loss function value converges or reaches a preset number of training times, and then terminate the training to obtain the trained feature extraction network.

9. A partial observation reinforcement learning device for robots, characterized in that: The device is used to execute the partial observation reinforcement learning method for robots according to any one of claims 1 to 8, comprising: A first training data set acquisition module is used to acquire a first training data set, wherein each first training data in the first training data set is observation information at different times and a corresponding historical observation information sequence acquired in a partial observation Markov environment; the historical observation information sequence represents a sequence composed of observation information at the previous N times in chronological order; A first feature extraction module is used to input the first training data at the tth moment into the feature extraction network to perform feature extraction to obtain the current observation feature and the historical observation feature sequence at the tth moment; A first confidence state generating module, used for inputting the current observation feature and the historical observation feature sequence at the t-th moment into the memory module to obtain the confidence state at the t-th moment; A first confidence state prediction module, used to predict a first candidate confidence state at time t+1 according to the confidence state at time t; A second feature extraction module is used to input the first training data at time t+1 into the feature extraction network to perform feature extraction to obtain the current observation feature at time t+1 and a historical observation feature sequence; A second confidence state generating module, used for inputting the current observation feature and the historical observation feature sequence at the t+1th time into the target memory module to obtain a second candidate confidence state at the t+1th time; A loss function calculation module, used to calculate the MSE loss function value according to the first candidate confidence state and the second candidate confidence state at the t+1th time; A parameter updating module, used for updating the parameters of the memory module and the executor network according to the MSE loss function value; A training end module, used for reselecting the first training data and performing iterative training until the MSE loss function value reaches a convergence state or reaches a preset number of training times, thereby obtaining the trained memory module and the executor network; The execution action prediction module is used to use the trained memory module and the executor network to determine the execution action of the robot at the current moment according to the observation information at the current moment.

10. An electronic device, characterized in that: The invention comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein when the processor is executed, the steps in the partial observation reinforcement learning method for robots as described in any one of claims 1 to 8 are implemented.

Images