Vision-based robot control method, system, equipment and medium

Abstract

The invention discloses a vision-based robot control method, system, equipment and medium, and belongs to the field of robot control, the method comprises the following steps: 1, using a camera communicating with a robot to obtain three images including the robot and the environment according to a preset time interval, and stacking the three images by using a reinforcement learning frame stack to obtain input image data; step 2, encoding the input image data by using the trained convolutional feature encoder to obtain output representation data; and step 3, inputting the output representation data into a multilayer perception decision model trained based on additional rewards, mapping the output representation data into an optimal action, and enabling the robot to execute the optimal action to perform optimal control. According to the method, the multi-layer perception decision model trained based on the external rewards is adopted, task-related information can be coded from the long-term accumulated rewards, and task-independent features are removed by using information bottleneck constraints, so that the visual generalization ability of the robot is enhanced, and the control accuracy of the robot based on visual control is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of vision-assisted robot control, and in particular to a robot control method for enhancing the generalization of robot vision. Background Art

[0002] Visual reinforcement learning (VRL) algorithms have achieved great success in the fields of robot control and game intelligence. It is a reinforcement learning algorithm that directly uses high-dimensional vision as input and is designed to handle complex control tasks with high-dimensional perceptual data. Taking the classic deep Q-network (DQN) algorithm as an example, it uses raw images as input and combines deep neural networks to build an end-to-end training model, and has achieved excellent performance on 49 Atari games. In addition, some existing VRL algorithms, such as the IMPALA algorithm and the QT-Opt algorithm, have achieved good results in solving vision-based multi-task learning and vision-based robot grasping problems, respectively.

[0003] However, although these methods perform well in the training environment, the strategies they train are difficult to generalize to new environments. This is because the input images often contain many visual factors that are irrelevant to the decision-making task, such as the background of the image, the shooting angle of the image, the color of the obstacles, etc. When the robot is placed in a new environment and only these irrelevant visual factors are changed, although the actual meaning of the environment is the same (or semantically equivalent) from the decision-making level, from the data level, the distribution of the environment observed by the robot will be greatly offset due to slight changes in these visual factors. This will cause the robot to be unable to recognize the task-related information it originally learned, thereby limiting its generalization ability. Existing visual reinforcement algorithms can train to obtain the optimal strategy in a single environment, but if these strategies are applied to other similar environments, their performance is usually poor.

[0004] In order to improve generalization performance, researchers have turned their attention to representation learning, hoping that the trained representation will only encode the task-related features in the image input, while discarding the task-irrelevant information caused by visual factors, so that the robot optimized based on the learned representation has a strong generalization ability. To achieve this goal, some researchers have proposed similarity metrics, which are used to judge semantically equivalent images and assist representation learning by making the representations of these images similar. Other researchers use the unique properties of the Markov decision process (such as rewards, probability transfer functions, etc.) to design auxiliary tasks, so as to learn invariant representations that are not disturbed by irrelevant visual factors. In the process of learning representations, these existing methods all use reward functions and probability transfer functions to obtain task-related features.

[0005] However, these representation learning algorithms usually do not take into account the critical downstream task of decision making. This results in the learned representations failing to capture long-term task-related information, which is a key factor in the generalization ability of visual reinforcement learning agents. Although the representations learned by these previous methods are robust to visual perturbations, they are unable to effectively facilitate decision making and thus improve generalization performance.

[0006] In view of this, the present invention is proposed. Summary of the invention

[0007] The purpose of the present invention is to provide a vision-based robot control method, system, device and medium, which can improve the performance of the vision-based control robot in resisting unknown visual interference, improve the accuracy of control, and thus solve the above-mentioned technical problems existing in the prior art.

[0008] The objective of the present invention is achieved through the following technical solutions:

[0009] A vision-based robot control method, comprising:

[0010] Step 1, using a camera communicating with the robot to obtain three images containing the robot and the environment at a preset time interval, and stacking the three images using a reinforcement learning frame stack to obtain input image data;

[0011] Step 2, encoding the input image data using a trained convolutional feature encoder to obtain output representation data;

[0012] Step 3: input the output representation data into a multi-layer perception decision model trained based on additional rewards, and the multi-layer perception decision model outputs an optimal action, so that the robot executes the optimal action to perform optimal control.

[0013] A vision-based robot control system, used to implement the method of the present invention, comprising:

[0014] Input image processing unit, convolutional feature encoder and multi-layer perception decision model trained based on additional rewards;

[0015] The input image processing unit is connected to the robot for communication, and can use a camera communicating with the robot to obtain three images including the robot and the environment at preset time intervals, and stack the three images using a reinforcement learning frame stack to obtain input image data;

[0016] The convolution feature encoder is communicatively connected to the input image processing unit and can encode the input image data obtained by the input image processing unit to obtain output representation data;

[0017] The multi-layer perception decision model trained based on additional rewards is respectively connected to the convolutional feature encoder and the robot for communication. It can take the output representation data obtained by the convolutional feature encoder as input, map the output representation data to an optimal action, and send it to the robot to execute the optimal action for optimal control.

[0018] A processing device, comprising:

[0019] at least one memory for storing one or more programs;

[0020] At least one processor can execute one or more programs stored in the memory. When the one or more programs are executed by the processor, the processor can implement the method described in the present invention.

[0021] A readable storage medium stores a computer program, which can implement the method described in the present invention when the computer program is executed by a processor.

[0022] Compared with the prior art, the vision-based robot control method, system, device and medium provided by the present invention have the following beneficial effects:

[0023] By utilizing the additional reward features based on the information bottleneck, the value representation in the multi-layer perception evaluation model is robustly and efficiently learned, thereby training the multi-layer perception decision model. This means that the robot can directly use the reward signal to train the robust model, significantly enhancing the generalization ability and improving the ability of the intelligent agent controlling the robot to resist visual interference. The present invention encodes the value information in the multi-layer perception evaluation model without including irrelevant features, which is a robust value representation, thereby facilitating decision-making and improving generalization performance. It is a good solution to the problem that the existing representation learning methods focus on learning robust representations that can resist visual interference irrelevant to the task, but often do not consider long-term decision-related information, which may reduce generalization performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative work.

[0025] Figure 1 A flowchart of a vision-based robot control method provided by an embodiment of the present invention.

[0026] Figure 2 A specific control flow chart of a vision-based robot control method provided in an embodiment of the present invention.

[0027] Figure 3 A block diagram of a control system of a vision-based robot provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0028] The following is a clear and complete description of the technical solutions in the embodiments of the present invention in combination with the specific content of the present invention; it is obvious that the described embodiments are only part of the embodiments of the present invention, not all of the embodiments, which does not constitute a limitation of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the protection scope of the present invention.

[0029] First, the terms that may be used in this article are explained as follows:

[0030] The term “and / or” means that either or both of them can be realized at the same time. For example, X and / or Y means both “X” or “Y” and “X and Y”.

[0031] The terms "include", "comprises", "contains", "has" or other descriptions with similar semantics should be interpreted as non-exclusive inclusion. For example, including certain technical feature elements (such as raw materials, components, ingredients, carriers, dosage forms, materials, dimensions, parts, components, mechanisms, devices, steps, procedures, methods, reaction conditions, processing conditions, parameters, algorithms, signals, data, products or products, etc.) should be interpreted as including not only certain technical feature elements explicitly listed, but also other technical feature elements known in the art that are not explicitly listed.

[0032] The term "consisting of..." means excluding any technical feature elements not explicitly listed. If this term is used in a claim, it will make the claim closed, so that it does not contain technical feature elements other than the technical feature elements explicitly listed, except for the conventional impurities related to them. If this term only appears in a clause of a claim, it only limits the elements explicitly listed in the clause, and the elements recorded in other clauses are not excluded from the overall claim.

[0033] Unless otherwise specified or limited, the terms "installed", "connected", "connected", "fixed" and the like should be understood in a broad sense, for example: it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or it can be an indirect connection through an intermediate medium, or it can be the internal communication of two components. For ordinary technicians in this field, the specific meanings of the above terms in this article can be understood according to specific circumstances.

[0034] The orientation or position relationship indicated by terms such as "center", "longitudinal", "lateral", "length", "width", "thickness", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside", "clockwise", "counterclockwise", etc. are based on the orientation or position relationship shown in the drawings and are only for the convenience and simplification of description, and do not explicitly or implicitly indicate that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore should not be understood as a limitation of this document.

[0035] The scheme provided by the present invention is described in detail below. The contents not described in detail in the embodiments of the present invention belong to the prior art known to professionals in the field. If the specific conditions are not specified in the embodiments of the present invention, they are carried out according to the conventional conditions in the field or the conditions recommended by the manufacturer. If the manufacturers are not specified in the reagents or instruments used in the embodiments of the present invention, they are all conventional products that can be purchased commercially.

[0036] like Figure 1 , Figure 2 As shown, an embodiment of the present invention provides a vision-based robot control method, comprising:

[0037] Step 1, using a camera communicating with the robot to obtain three images containing the robot and the environment at a preset time interval, and stacking the three images using a reinforcement learning frame stack to obtain input image data;

[0038] Step 2, encoding the input image data using a trained convolutional feature encoder to obtain output representation data;

[0039] Step 3: input the output representation data into a multi-layer perception decision model trained based on additional rewards, and the multi-layer perception decision model outputs an optimal action, so that the robot executes the optimal action to perform optimal control.

[0040] Preferably, in the above control method, the training method of the multi-layer perception decision model based on additional rewards includes:

[0041] Step 31, using a camera communicating with the robot to obtain three images including the robot and the environment and process them into an input image sample;

[0042] Step 32, encoding the input image sample with a convolutional feature encoder to obtain output representation data, mapping the output representation data into decision action data for controlling the robot behavior by a multi-layer perception decision model, and calculating reward data corresponding to the input image sample and the decision action data using a universal reward function corresponding to the robot control task;

[0043] Step 33, storing the input image samples, decision action data and reward data collected in the above steps into the experience replay pool in the order of data collection time;

[0044] Step 34, take 256 input image samples, 256 decision action data and 256 reward data from the experience replay pool storage data as a batch data, input all image samples in a batch data into the convolutional feature encoder to obtain output representation data, input the output representation data and all reward data in a batch data into the additional reward feature extraction model, and calculate the additional reward feature loss based on the information bottleneck theory using the following formula for:

[0045]

[0046] Among them, ψ and η represent two parallel three-layer perceptrons in the additional reward feature extraction model, and the input data of these two three-layer perceptrons are the reward data and the image representation data output by the convolutional feature encoder; represents the KL divergence used to calculate the difference in the distribution of parameters ψ and η of two three-layer perceptrons; z t represents the additional reward feature output by the three-layer perceptron ψ in the additional reward feature extraction model; represents the additional reward feature label output by the three-layer perceptron η in the additional reward feature extraction model; Represents the feature z used to calculate the additional reward t and bonus feature tags The mutual information of; α is a hyperparameter, selected as 0.001; It is the experience replay pool; It means to seek the expectation of the data distribution of the experience replay pool;

[0047] Step 35, updating the extra reward feature extraction model according to the loss calculated by the extra reward feature extraction model;

[0048] Step 36, use the additional reward feature output by the updated additional reward feature extraction model as the label signal to calculate the loss function of the multi-layer perception evaluation model Updated multi-layer perception evaluation model;

[0049] Step 37, use multi-layer perception to evaluate the floating point number Q output by the model θ , calculate the loss function of the multi-layer perception decision model for:

[0050]

[0051] Among them, Q θ (o t ,a t ) is the output of the multi-layer perception evaluation model; It is the experience replay pool; It means to seek the expectation of the data distribution of the experience replay pool;

[0052] Step 38, using the loss function of the multi-layer perception decision model to update the multi-layer perception decision model;

[0053] Step 39, repeat steps 31 to 38 until the amount of data in the experience replay pool exceeds the preset training end data amount, then the training is completed.

[0054] Preferably, in the above control method, in step 35, the loss function of the additional reward feature extraction model calculation loss for:

[0055]

[0056] Among them, θ is the parameter of the multi-layer perception evaluation model, and its input data is the state-action pair (o t ,a t ), the output data is a floating point number Q θ (o t ,a t );Z θ To pay attention to the output of the second layer perceptron of the multi-layer perceptual evaluation model, that is, the value representation of the multi-layer perceptual evaluation model; is the output of the target multi-layer perceptual evaluation model, which is a copy of the multi-layer perceptual evaluation model before the parameter θ is updated; The output of the second layer perceptron for the target multi-layer perceptron evaluation model; is the experience replay pool; c represents the hyperparameter, which is selected as 1.0; It represents the expectation of the data distribution of the experience replay pool.

[0057] Preferably, in the above control method, in step 31, three images containing the robot and the environment are acquired at preset time intervals through a camera communicating with the robot, and the three images are stacked using a reinforcement learning frame stack to obtain an input image sample.

[0058] Preferably, in the above control method, the convolutional feature encoder adopts a four-layer convolutional neural network, the convolution kernel of each layer of the convolutional neural network uses a matrix of size 32×32, and no pooling operation is added between every two layers of the convolutional neural network.

[0059] Preferably, in the above control method, the multi-layer perception decision model adopts a multi-layer perceptron.

[0060] like Figure 3 As shown, the embodiment of the present invention further provides a vision-based robot control system for implementing the above method, including:

[0061] Input image processing unit, convolutional feature encoder and multi-layer perception decision model trained based on additional rewards;

[0062] The input image processing unit is connected to the robot for communication, and can use a camera communicating with the robot to obtain three images including the robot and the environment at preset time intervals, and stack the three images using a reinforcement learning frame stack to obtain input image data;

[0063] The convolution feature encoder is communicatively connected to the input image processing unit and can encode the input image data obtained by the input image processing unit to obtain output representation data;

[0064] The multi-layer perception decision model trained based on the additional reward is connected to the convolutional feature encoder and the robot for communication, and can map the output representation data obtained by the convolutional feature encoder into an optimal action, and send it to the robot to execute the optimal action for optimal control.

[0065] Preferably, in the above control system, the training method of the multi-layer perception decision model based on additional rewards includes:

[0066] Step 31, using a camera communicating with the robot to obtain three images including the robot and the environment and process them into an input image sample;

[0067] Step 32, encoding the input image sample with a convolutional feature encoder to obtain output representation data, mapping the output representation data into decision action data for controlling the robot behavior by a multi-layer perception decision model, and calculating reward data corresponding to the input image sample and the decision action data using a universal reward function corresponding to the robot control task;

[0068] Step 33, storing the input image samples, decision action data and reward data collected in the above steps into the experience replay pool in the order of data collection time;

[0069] Step 34, take 256 input image samples, 256 decision action data and 256 reward data from the experience replay pool storage data as a batch data, input all image samples in a batch data into the convolutional feature encoder to obtain output representation data, input the output representation data and all reward data in a batch data into the additional reward feature extraction model, and calculate the additional reward feature loss based on the information bottleneck theory using the following formula for:

[0070]

[0071] Wherein, ψ and η represent two parallel three-layer perceptrons in the additional reward feature extraction model, and the input data of the two three-layer perceptrons are the reward data and the image representation data output by the convolutional feature encoder; represents the KL divergence used to calculate the difference in the distribution of two three-layer perceptrons ψ and η; z t represents the additional reward feature output by the three-layer perceptron ψ in the additional reward feature extraction model; represents the additional reward feature label output by the three-layer perceptron η in the additional reward feature extraction model; Represents the feature z used to calculate the additional reward t and bonus feature tags The mutual information of; α is a hyperparameter, selected as 0.001; It is the experience replay pool; It means to seek the expectation of the data distribution of the experience replay pool;

[0072] Step 35, updating the extra reward feature extraction model according to the loss calculated by the extra reward feature extraction model;

[0073] Step 36, use the additional reward feature output by the updated additional reward feature extraction model as the label signal to calculate the loss function of the multi-layer perception evaluation model Updated multi-layer perception evaluation model;

[0074] Step 37, use multi-layer perception to evaluate the floating point number Q output by the model θ, calculate the loss function of the multi-layer perception decision model for:

[0075]

[0076] Among them, Q θ (o t ,a t ) is the output of the multi-layer perception evaluation model; It is the experience replay pool; It means to seek the expectation of the data distribution of the experience replay pool;

[0077] Step 38, using the loss function of the multi-layer perception decision model to update the multi-layer perception decision model;

[0078] Step 39, repeat steps 31 to 38 until the amount of data in the experience replay pool exceeds the preset training end data amount, then the training is completed.

[0079] The embodiment of the present invention further provides a processing device, including:

[0080] at least one memory for storing one or more programs;

[0081] At least one processor can execute one or more programs stored in the memory. When the one or more programs are executed by the processor, the processor can implement the above method.

[0082] The embodiments of the present invention further provide a readable storage medium storing a computer program, wherein the computer program can implement the above method when executed by a processor.

[0083] In summary, the method and system of the embodiment of the present invention, by adopting a multi-layer perception decision model trained based on external rewards, can encode task-related information from long-term cumulative rewards (i.e., value), use information bottleneck constraints to remove task-irrelevant features, and thus enhance the robot's visual generalization ability and improve the control accuracy of the robot based on visual control. This method and system can directly use one-step rewards to extract robust value representations by integrating information bottlenecks into the representation learning process under the temporal difference learning paradigm, significantly enhancing the generalization ability and improving the ability of the intelligent body that controls the robot to resist visual interference. The present invention encodes value information without including irrelevant features, and is a robust value representation, thereby promoting decision-making and improving generalization performance. It solves the problem that the existing representation learning methods focus on learning robust representations that can resist task-irrelevant visual interference, but often do not consider long-term decision-related information, which may reduce generalization performance.

[0084] In order to more clearly demonstrate the technical solution and technical effects provided by the present invention, the solution provided by the embodiment of the present invention is described in detail with specific embodiments below.

[0085] Example 1

[0086] This embodiment provides a control method for a vision-based robot, which is a method for improving the visual generalization performance of a vision-controlled robot.

[0087] (I) In general reinforcement learning, the agent controlling the robot can capture observation o, action a, and perform value evaluation to obtain Q π (o t ,a t ). Define the representation of the value function as H π (o t ,a t ), Φ is a mapping relationship, and the relationship between this representation and the general value function is:

[0088]

[0089] The robust value representation is Z π (o t ,a t ) indicates that the following constraints need to be met:

[0090]

[0091] in, Represents the mutual information constraint between two variables. Based on the temporal difference learning in traditional reinforcement learning, the following iterative learning form can be inferred:

[0092]

[0093] Through this iterative method, value representation can be learned and decisions can be made based on it, thereby improving the generalization performance of the intelligent agent.

[0094] The method for learning such representations proposed in the present invention realizes robust value representation learning through information bottlenecks, can encode long-term task-related information from value assessments, and can effectively promote decision-making that resists visual interference in unseen environments, improve the visual generalization performance of vision-based control robots, and improve control accuracy.

[0095] In order to learn a robust value representation without affecting the existing iterative framework, the present invention introduces an additional reward feature extraction model and updates it through constraints under the information bottleneck constraint. The additional reward feature extraction model can extract reward representation based on the temporal difference learning paradigm to achieve the learning of value (cumulative reward return) representation.

[0096] The present invention proposes a learning loss function for:

[0097]

[0098] Among them, θ is a multi-layer perception evaluation model, whose input data is the state-action pair (o t ,a t ), the output data is a floating point number Q θ (o t ,a t ); Note that the output of the second layer of the multi-layer perception evaluation model (using a three-layer perceptron) is Z θ , that is, the value representation of the multi-layer perception evaluation model; is the output of the target multi-layer perceptual evaluation model, which is a copy of the multi-layer perceptual evaluation model before the parameter θ is updated; the output of the second layer perceptron of the target multi-layer perceptual evaluation model is is the experience replay pool; c represents the hyperparameter, which is selected as 1.0; It represents the expectation of the data distribution of the experience replay pool. Through iterative updates, the intelligent agent controlling the robot can learn the value representation of removing long-term interference information, make decisions based on it, resist the influence of environmental interference, and improve the accuracy of controlling the robot.

[0099] (II) Prediction stage:

[0100] Step 1: three images of the robot and the environment are acquired at preset time intervals through cameras connected to the robot (including cameras inside the robot and cameras around the robot), and the three acquired images are stacked through reinforcement learning frame stacks to obtain input image data for subsequent control;

[0101] Step 2: Encode the input image data obtained in step 1 using the trained convolutional feature encoder;

[0102] Step 3: Input the image data into a multi-layer perception decision model trained based on additional rewards, map it into an optimal action, and make the robot execute the optimal action to optimally control the robot.

[0103] (III) Training phase:

[0104] Step 31: Obtain three images including the robot and the environment through an internal camera connected to the robot and an external camera in the surrounding area, and stack the three images into an input image sample through reinforcement learning frame stacking;

[0105] Step 32: Encode the input image sample obtained in step 1 by using a convolutional neural network as a convolutional feature encoder to obtain corresponding output representation data, and then use a multi-layer perceptron as a multi-layer perception decision model to map the output representation data of the convolutional feature encoder into decision action data for specifically controlling the robot behavior, and then use the universal reward function corresponding to the robot according to the specific control task of the robot to calculate the reward data corresponding to the input image sample and the decision action data;

[0106] Step 33: The input image samples, decision action data and reward data collected in steps 1 and 2 are stored in the experience replay pool in the order of data collection time;

[0107] Step 34: Take 256 input image samples, 256 decision action data and 256 reward data from the data stored in the experience replay pool as a batch of data, input all image samples in a batch of data into the convolutional feature encoder to obtain output representation data, input the image representation data and all reward data into the additional reward feature extraction model to calculate the loss;

[0108] Step 35: According to the loss calculated by the additional reward feature extraction model, the additional reward feature extraction model, the multi-layer perception evaluation model and the multi-layer perception decision model are updated in sequence;

[0109] Step 36: Repeat the operations from step 31 to step 35 until the amount of data in the experience replay pool exceeds the preset training end data amount and the training is terminated.

[0110] like Figure 1 As shown, the additional reward feature extraction model can extract the single-step reward representation to learn the value representation. The constraints introduced by the present invention enable the model to learn the value representation of removing long-term interference information, thereby improving the generalization of controlling the robot agent.

[0111] Table 1 shows the performance improvement comparison results of the robust value representation learning method of the present invention

[0112]

[0113] Based on the method of the present invention, two advanced baseline methods were improved and evaluated in multiple task environments. The same degree of interference from visually irrelevant information was introduced in the evaluation environments, and the methods of the present invention achieved significant performance improvements.

[0114] Example 2

[0115] This embodiment provides a vision-based robot control method, which is as follows:

[0116] Task environment settings:

[0117] Through the MuJoCo simulation environment (MuJoCo stands for Multi-Joint dynamics with Contact, which was mainly developed by Professor Emo Todorov of the University of Washington and is used in the fields of optimal control, state estimation, system identification, etc., it has obvious advantages in the application scenarios of dynamic multi-point contact of robots (such as multi-finger dexterous hand operation), which is widely used in the field of robot control for algorithm training and performance testing, the control method of the present invention is used to perform robot control tasks in the simulation environment.

[0118] This embodiment is based on the MuJoCo robot modeling platform. Google's Robotics Laboratory expands on DeepMind's open source DM Control benchmark and provides the Distracting Control Suite (DCS) platform, which encapsulates three types of visual interference (changes in background, color, and camera pose), providing a new test benchmark for vision-based control tasks.

[0119] The control method of the present invention is implemented and tested in 6 different simulation robot control environments. In these task environments, the observable states are different physical quantities (such as position, angle, speed, etc.) of various parts (such as legs, joints, etc.) of the simulation robot, and the controllable actions are the magnitude of the force used by a specific part (such as legs, head). Specifically, the 6 simulation robot control environments are:

[0120] (1) Ball In Cup Catch: Control a planar container to translate on a vertical plane so that it swings and catches a ball attached to its bottom. The ball catching task (catch) has a sparse reward, scoring 1 when the ball is in the cup and 0 otherwise. The observation space is 8-dimensional and the action space is 2-dimensional.

[0121] (2) Cartpole Swingup: Control the cart to swing up and balance a pole without external force. In the swingup task, the pole is initially pointing straight down. The observation space is 5-dimensional and the action space is 1-dimensional.

[0122] (3) Cheetah Run: Control a two-dimensional bipedal robot to run forward. The observation space is 17 dimensions and the action space is 6 dimensions.

[0123] (4) Finger Spin: Control a 2D finger to rotate a body on a hinge without external force. The state space is 12-dimensional and the action space is 2-dimensional. In the spin task, the body must rotate continuously and uninterruptedly.

[0124] (5) Reacher Easy: Control a two-dimensional double-link reacher so that the end-feeder on the reacher passes through the target sphere. In the simple task (easy), the target sphere is larger. The observation space is 7-dimensional and the action space is 2-dimensional.

[0125] (6) Walker Walk: Control a 2D humanoid robot to run forward. In the walk task (walk), the reward includes the robot's forward speed. The observation space is 24-dimensional and the action space is 6-dimensional.

[0126] The input images of the above six tasks are 3×84×84 pixels. In the experiment, the input is the superposition of 3 consecutive frames of images, that is, 9×84×84 pixels. In addition, in the experiment, it is set that the robot will repeat the action selected by the current strategy every few frames. The following Table 2 shows the number of frames in which the robot repeats the action in different environments.

[0127] Table 2 shows the number of frames of the robot's repeated actions in different environments

[0128]

[0129] In this example, the generalization performance of the robot control method with visual input is improved in unseen test environments with visual distractors. Without any prior environmental knowledge (such as data augmentation, downstream adjustments, and environmental labels), the robot is trained in multiple environments, and the performance of the robot trained by different control algorithms in unseen test environments is studied.

[0130] For each environment, the proposed control method is compared with the previous state-of-the-art method:

[0131] (1) CURL: an RL algorithm using contrastive learning as an auxiliary task;

[0132] (2) DrQv2: an effective method with state-of-the-art performance on the DeepMind Control (DMControl) task;

[0133] (3) SRM: using data augmentation in the frequency domain to facilitate the learning of robust policies;

[0134] (4) RAP: effectively enhances the robustness of representation by leveraging behavioral similarity;

[0135] (5) CRESP: Learning task-related representations to achieve generalization by predicting the feature function of reward sequences;

[0136] (6) TACO: Learn state and action representations that contain sufficient control information to improve sample efficiency.

[0137] The generalization ability of the control method of the present invention was tested for two visual interference conditions. In dynamic color interference, the color of the robot was changed continuously in each round. The color change was modeled as a Gaussian distribution, and its mean was set to the color of the robot in the previous round. In dynamic background interference, dynamic backgrounds were extracted from different videos to replace the original background in the image. During the training process, 2 dynamic backgrounds were used to train the robot's decision-making, and another 30 dynamic backgrounds were used for evaluation during the test phase.

[0138] The above is only a preferred specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any changes or substitutions that can be easily thought of by any technician familiar with the technical field within the technical scope disclosed in the present invention should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims. The information disclosed in the background technology section of this article is only intended to deepen the understanding of the overall background technology of the present invention, and should not be regarded as an admission or in any form that the information constitutes prior art known to those skilled in the art.

Claims

1. A vision-based robot control method, characterized in that: include: Step 1, using a camera communicating with the robot to obtain three images containing the robot and the environment at a preset time interval, and stacking the three images using a reinforcement learning frame stack to obtain input image data; Step 2, encoding the input image data using a trained convolutional feature encoder to obtain output representation data; Step 3: input the output representation data into a multi-layer perception decision model trained based on additional rewards, and the multi-layer perception decision model outputs an optimal action, so that the robot executes the optimal action to perform optimal control.

2. The vision-based robot control method according to claim 1, characterized in that: The training method of the multi-layer perception decision model based on additional rewards includes: Step 31, using a camera communicating with the robot to obtain three images including the robot and the environment and process them into an input image sample; Step 32, encoding the input image sample with a convolutional feature encoder to obtain output representation data, mapping the output representation data into decision action data for controlling the robot behavior by a multi-layer perception decision model, and calculating reward data corresponding to the input image sample and the decision action data using a universal reward function corresponding to the robot control task; Step 33, storing the input image samples, decision action data and reward data collected in the above steps into the experience replay pool in the order of data collection time; Step 34, take 256 input image samples, 256 decision action data and 256 reward data from the experience replay pool storage data as a batch data, input all image samples in a batch data into the convolutional feature encoder to obtain output representation data, input the output representation data and all reward data in a batch data into the additional reward feature extraction model, and calculate the additional reward feature loss based on the information bottleneck theory using the following formula for: Wherein, ψ and η represent two parallel three-layer perceptrons in the additional reward feature extraction model, and the input data of the two three-layer perceptrons are the reward data and the image representation data output by the convolutional feature encoder; represents the KL divergence used to calculate the difference in the distribution of two three-layer perceptrons ψ and η; z t represents the additional reward feature output by the three-layer perceptron ψ in the additional reward feature extraction model; represents the additional reward feature label output by the three-layer perceptron η in the additional reward feature extraction model; t represents the current time; Represents the feature z used to calculate the additional reward t and bonus feature tags The mutual information of; α is a hyperparameter, selected as 0.001; It is the experience replay pool; It means to seek the expectation of the data distribution of the experience replay pool; Step 35, updating the extra reward feature extraction model according to the loss calculated by the extra reward feature extraction model; Step 36, use the additional reward feature output by the updated additional reward feature extraction model as the label signal to calculate the loss function of the multi-layer perception evaluation model θ Updated multi-layer perception evaluation model; Step 37, use multi-layer perception to evaluate the floating point number Q output by the model θ , calculate the loss function of the multi-layer perception decision model φ for: Among them, φ is the multi-layer perception decision model, Q θ (o t ,a t ) is the output of the multi-layer perception evaluation model; It is the experience replay pool; It means to seek the expectation of the data distribution of the experience replay pool; Step 38, using the loss function of the multi-layer perception decision model to update the multi-layer perception decision model; Step 39, repeat steps 31 to 38 until the amount of data in the experience replay pool exceeds the preset training end data amount, then the training is completed.

3. The vision-based robot control method according to claim 2, characterized in that: In step 35, the loss function of the additional reward feature extraction model calculation loss for: Among them, θ is the parameter of the multi-layer perception evaluation model, and its input data is the state-action pair (o t ,a t ), the output data is a floating point number Q θ (o t ,a t );t represents the current time;Z θ To pay attention to the output of the second layer perceptron of the multi-layer perceptual evaluation model, that is, the value representation of the multi-layer perceptual evaluation model; is the output of the target multi-layer perceptual evaluation model, which is a copy of the multi-layer perceptual evaluation model before the parameter θ is updated; The output of the second layer perceptron for the target multi-layer perceptron evaluation model; is the experience replay pool; c represents the hyperparameter, which is selected as 1.0; It represents the expectation of the data distribution of the experience replay pool.

4. The vision-based robot control method according to claim 2 or 3, characterized in that: In step 31, three images including the robot and the environment are acquired at preset time intervals through a camera communicating with the robot, and the three images are stacked using a reinforcement learning frame stack to obtain an input image sample.

5. The vision-based robot control method according to claim 1 or 2, characterized in that: The convolution feature encoder adopts a four-layer convolutional neural network. The convolution kernel of each layer of the convolutional neural network uses a 32×32 matrix, and no pooling operation is added between every two layers of the convolutional neural network.

6. The vision-based robot control method according to claim 1 or 2, characterized in that: The multi-layer perception decision model adopts a multi-layer perceptron.

7. A vision-based robot control system, characterized in that: The method for implementing any one of claims 1 to 6 comprises: Input image processing unit, convolutional feature encoder and multi-layer perception decision model trained based on additional rewards; The input image processing unit is connected to the robot for communication, and can use a camera communicating with the robot to obtain three images including the robot and the environment at preset time intervals, and stack the three images using a reinforcement learning frame stack to obtain input image data; The convolution feature encoder is communicatively connected to the input image processing unit and can encode the input image data obtained by the input image processing unit to obtain output representation data; The multi-layer perception decision model trained based on additional rewards is respectively connected to the convolutional feature encoder and the robot for communication. It can take the output representation data obtained by the convolutional feature encoder as input, map the output representation data to an optimal action, and send it to the robot to execute the optimal action for optimal control.

8. The vision-based robot control system according to claim 7, characterized in that: The training method of the multi-layer perception decision model based on additional rewards includes: Step 31, using a camera communicating with the robot to obtain three images including the robot and the environment and process them into an input image sample; Step 32, encoding the input image sample with a convolutional feature encoder to obtain output representation data, mapping the output representation data into decision action data for controlling the robot behavior by a multi-layer perception decision model, and calculating reward data corresponding to the input image sample and the decision action data using a universal reward function corresponding to the robot control task; Step 33, storing the input image samples, decision action data and reward data collected in the above steps into the experience replay pool in the order of data collection time; Step 34, take 256 input image samples, 256 decision action data and 256 reward data from the experience replay pool storage data as a batch data, input all image samples in a batch data into the convolutional feature encoder to obtain output representation data, input the output representation data and all reward data in a batch data into the additional reward feature extraction model, and calculate the additional reward feature loss based on the information bottleneck theory using the following formula for: Among them, ψ and η represent two parallel three-layer perceptrons in the additional reward feature extraction model, and the input data of these two three-layer perceptrons are the reward data and the image representation data output by the convolutional feature encoder; represents the KL divergence used to calculate the difference in the distribution of parameters ψ and η of two three-layer perceptrons; z t represents the additional reward feature output by the three-layer perceptron ψ in the additional reward feature extraction model; represents the additional reward feature label output by the three-layer perceptron η in the additional reward feature extraction model; t represents the current time; Represents the feature z used to calculate the additional reward t and bonus feature tags The mutual information of; α is a hyperparameter, selected as 0.001; It is the experience replay pool; It means to seek the expectation of the data distribution of the experience replay pool; Step 35, updating the extra reward feature extraction model according to the loss calculated by the extra reward feature extraction model; Step 36, use the additional reward feature output by the updated additional reward feature extraction model as the label signal to calculate the loss function of the multi-layer perception evaluation model θ Updated multi-layer perception evaluation model; Step 37, use multi-layer perception to evaluate the floating point number Q output by the model θ , calculate the loss function of the multi-layer perception decision model φ for: Among them, φ is the multi-layer perception decision model, Q θ (o t ,a t ) is the output of the multi-layer perception evaluation model; It is the experience replay pool; It means to seek the expectation of the data distribution of the experience replay pool; Step 38, using the loss function of the multi-layer perception decision model to update the multi-layer perception decision model; Step 39, repeat steps 31 to 38 until the amount of data in the experience replay pool exceeds the preset training end data amount, then the training is completed.

9. A processing device, characterized in that: include: at least one memory for storing one or more programs; At least one processor is capable of executing one or more programs stored in the memory, and when the one or more programs are executed by the processor, the processor is capable of implementing the method described in any one of claims 1 to 6.

10. A readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 6 can be implemented.

Images