A vision-based robot control method, system, device, and medium.

Abstract

This invention discloses a vision-based robot control method, system, device, and medium, belonging to the field of robot control. The method includes: Step 1, acquiring three images containing the robot and the environment at preset time intervals using a camera communicating with the robot, and stacking the three images using a reinforcement learning frame stack to obtain input image data; Step 2, encoding the input image data using a trained convolutional feature encoder to obtain output representation data; Step 3, inputting the output representation data into a multilayer perception decision model trained based on extrinsic rewards to map it into an optimal action, enabling the robot to execute the optimal action for optimal control. This invention employs a multilayer perception decision model trained based on extrinsic rewards, which can encode task-related information from long-term accumulated rewards and use information bottleneck constraints to remove task-irrelevant features, thereby enhancing the robot's visual generalization ability and improving the control accuracy of vision-based robots.

Description

Technical Field

[0001] This invention relates to the field of vision-assisted robot control technology, and in particular to a robot control method that enhances the generalization of robot vision. Background Technology

[0002] Visual reinforcement learning (VRL) algorithms have achieved great success in fields such as robot control and game intelligence. It is a type of reinforcement learning algorithm that directly uses high-dimensional vision as input, aiming to handle complex control tasks with high-dimensional perceptual data. Taking the classic Deep Q-network (DQN) algorithm as an example, it uses raw images as input, combines them with deep neural networks to build an end-to-end training model, and achieved excellent performance on 49 Atari games. Furthermore, some existing VRL algorithms, such as the IMPALA and QT-Opt algorithms, have achieved good results in solving vision-based multi-task learning and vision-based robot grasping problems, respectively.

[0003] However, while these methods perform well in training environments, the policies they train are difficult to generalize to new environments. This is because input images often contain many visual factors irrelevant to the decision-making task, such as the background, the shooting angle, and the color of obstacles. When a robot is placed in a new environment and only these irrelevant visual factors are changed, although the actual meaning of the environment remains the same (or semantically equivalent) from a decision-making perspective, from a data perspective, the distribution of the environment observed by the robot will shift significantly due to these small changes in visual factors. This will cause the robot to fail to recognize the task-related information it previously learned, thus limiting its generalization ability. Existing visual reinforcement algorithms can train optimal policies in a single environment, but their performance is usually poor when these policies are applied to other similar environments.

[0004] To improve generalization performance, researchers have turned their attention to representation learning, hoping that the trained representations will encode only task-relevant features from the image input, while discarding task-irrelevant information caused by visual factors. This would enable robots optimized based on the learned representations to possess strong generalization capabilities. To achieve this, some researchers have proposed similarity metrics, which determine semantically equivalent images and assist representation learning by making their representations similar. Other researchers utilize the unique properties of Markov decision processes (such as reward and probability transition functions) to design auxiliary tasks, thereby learning invariant representations unaffected by irrelevant visual factors. All existing methods utilize reward functions and probability transition functions to obtain task-relevant features during the representation learning process.

[0005] However, these representation learning algorithms typically fail to consider the crucial downstream task of decision-making. This results in the learned representations failing to capture long-term task-related information, which is a key factor in the generalization ability of visual reinforcement learning agents. While the representations learned by these previous methods are robust to visual disturbances, they fail to effectively facilitate decision-making and thus improve generalization performance.

[0006] In view of this, the present invention is hereby proposed. Summary of the Invention

[0007] The purpose of this invention is to provide a vision-based robot control method, system, device, and medium that can improve the performance of vision-based controlled robots in resisting unknown visual interference, improve control accuracy, and thus solve the aforementioned technical problems existing in the prior art.

[0008] The objective of this invention is achieved through the following technical solution:

[0009] A vision-based robot control method, comprising:

[0010] Step 1: Use a camera that communicates with the robot to acquire three images containing the robot and the environment at preset time intervals, and stack the three images using a reinforcement learning frame stack to obtain input image data;

[0011] Step 2: Encode the input image data using the trained convolutional feature encoder to obtain output representation data;

[0012] Step 3: Input the output representation data into the multilayer perception decision model trained based on additional rewards. The multilayer perception decision model outputs an optimal action, which the robot then executes to achieve optimal control.

[0013] A vision-based robot control system for implementing the method described in this invention includes:

[0014] The input consists of an image processing unit, a convolutional feature encoder, and a multilayer perceptual decision model trained based on additional rewards.

[0015] The input image processing unit is connected to the robot and can use a camera that communicates with the robot to acquire three images containing the robot and the environment at preset time intervals. The three images are stacked using a reinforcement learning frame stack to obtain input image data.

[0016] The convolutional feature encoder is communicatively connected to the input image processing unit and can encode the input image data obtained by the input image processing unit to obtain output representation data.

[0017] The multilayer perception decision model trained based on additional rewards is connected to the convolutional feature encoder and the robot for communication. It can take the output representation data obtained by the convolutional feature encoder as input, map the output representation data into an optimal action, and send it to the robot to execute the optimal action for optimal control.

[0018] A processing apparatus, comprising:

[0019] At least one memory for storing one or more programs;

[0020] At least one processor is capable of executing one or more programs stored in the memory, such that when the processor executes one or more programs, the processor can implement the method of the present invention.

[0021] A readable storage medium storing a computer program that, when executed by a processor, enables the implementation of the methods described in this invention.

[0022] Compared with existing technologies, the vision-based robot control method, system, device, and medium provided by this invention have the following advantages:

[0023] By leveraging additional reward features based on information bottlenecks, this invention robustly and efficiently learns value representations in multilayer perceptual evaluation models, thereby training multilayer perceptual decision-making models. This means that robots can directly use reward signals to train robust models, significantly enhancing generalization ability and improving the resistance of the robot's controlling agent to visual interference. This invention encodes value information in multilayer perceptual evaluation models without including irrelevant features, providing a robust value representation that facilitates decision-making and improves generalization performance. It effectively addresses the problem that existing representation learning methods focus on learning robust representations that resist task-irrelevant visual interference, often neglecting long-term decision-related information, which may reduce generalization performance. Attached Figure Description

[0024] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the following description of the embodiments will be briefly introduced. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0025] Figure 1 A flowchart illustrating a vision-based robot control method provided in an embodiment of the present invention.

[0026] Figure 2 The following is a detailed control flowchart of the vision-based robot control method provided in an embodiment of the present invention.

[0027] Figure 3 A block diagram of a vision-based robot control system provided in an embodiment of the present invention. Detailed Implementation

[0028] The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the specific content of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments, which do not constitute a limitation of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the protection scope of the present invention.

[0029] First, the following explanations are provided for the terms that may be used in this article:

[0030] The term "and / or" means that either or both can be achieved simultaneously. For example, X and / or Y means that it includes both "X" or "Y" as well as the three cases of "X and Y".

[0031] The terms “including,” “comprising,” “containing,” “having,” or other similar semantic descriptions should be interpreted as non-exclusive inclusion. For example, “including a technical feature element (such as raw material, component, ingredient, carrier, dosage form, material, size, part, component, mechanism, device, step, process, method, reaction conditions, processing conditions, parameter, algorithm, signal, data, product or article of manufacture, etc.)” should be interpreted as including not only the expressly listed technical feature element, but also other technical feature elements that are not expressly listed and are well-known in the art.

[0032] The term "composed of" excludes any technical features not expressly listed. When used in a claim, it closes the claim to exclude all technical features other than those expressly listed, except for associated conventional impurities. If the term appears only in a clause of a claim, it limits the claim to the elements expressly listed in that clause; elements recited in other clauses are not excluded from the overall claim.

[0033] Unless otherwise explicitly specified or limited, the terms "installation," "connection," "linking," and "fixing," etc., should be interpreted broadly. For example, they can refer to fixed connections, detachable connections, or integral connections; they can refer to mechanical connections or electrical connections; they can refer to direct connections or indirect connections through an intermediate medium; and they can refer to the internal connection between two components. Those skilled in the art can understand the specific meaning of the above terms in this document according to the specific circumstances.

[0034] The terms “center,” “longitudinal,” “lateral,” “length,” “width,” “thickness,” “upper,” “lower,” “front,” “back,” “left,” “right,” “vertical,” “horizontal,” “top,” “bottom,” “inner,” “outer,” “clockwise,” and “counterclockwise” indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are used only for the convenience and simplification of description and do not imply that the device or component referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on this document.

[0035] The solution provided by this invention will be described in detail below. Contents not described in detail in the embodiments of this invention are prior art known to those skilled in the art. Where specific conditions are not specified in the embodiments of this invention, they shall be performed according to conventional conditions in the art or conditions recommended by the manufacturer. Reagents or instruments used in the embodiments of this invention whose manufacturers are not specified are all conventional products that can be purchased commercially.

[0036] like Figure 1 , Figure 2 As shown, an embodiment of the present invention provides a vision-based robot control method, including:

[0037] Step 1: Use a camera that communicates with the robot to acquire three images containing the robot and the environment at preset time intervals, and stack the three images using a reinforcement learning frame stack to obtain input image data;

[0038] Step 2: Encode the input image data using the trained convolutional feature encoder to obtain output representation data;

[0039] Step 3: Input the output representation data into the multilayer perception decision model trained based on additional rewards. The multilayer perception decision model outputs an optimal action, which the robot then executes to achieve optimal control.

[0040] Preferably, in the above control method, the training method of the multi-layer perception decision model based on additional rewards includes:

[0041] Step 31: Use the camera that communicates with the robot to acquire three images containing the robot and the environment and process them into an input image sample;

[0042] Step 32: Encode the input image sample with a convolutional feature encoder to obtain output representation data. The multilayer perception decision model maps the output representation data to decision action data for controlling the robot's behavior. The general reward function corresponding to the robot control task is used to calculate the reward data corresponding to the input image sample and the decision action data.

[0043] Step 33: Store the input image samples, decision action data, and reward data collected in the above steps into the experience playback pool in the order of data collection time;

[0044] Step 34: Take 256 input image samples, 256 decision action data, and 256 reward data from the experience replay pool as a batch. Input all image samples in this batch into a convolutional feature encoder to obtain output representation data. Input the output representation data and all reward data from the batch into an additional reward feature extraction model. Calculate the additional reward feature loss based on the information bottleneck theory using the following formula. for:

[0045]

[0046] Where ψ and η represent two parallel three-layer perceptrons in the extra reward feature extraction model, and the input data of these two three-layer perceptrons are reward data and image representation data output by the convolutional feature encoder; This represents the KL divergence used to calculate the difference in the distributions of parameters ψ and η between two three-layer perceptrons; z t This represents the additional reward feature output by the three-layer perceptron ψ in the additional reward feature extraction model; This represents the additional reward feature label output by the three-layer perceptron η in the additional reward feature extraction model; This represents the feature z used to calculate the additional reward. t and additional reward feature tags Mutual information; α is a hyperparameter, chosen as 0.001; For experience replay pool; This represents the expectation of the data distribution over the experience replay pool;

[0047] Step 35: Update the extra reward feature extraction model based on the loss calculated by the extra reward feature extraction model;

[0048] Step 36: Use the updated additional reward features to extract the additional reward features output by the model as the label signal, and calculate the loss function of the multilayer perceptron evaluation model. Update the multi-layer perception evaluation model;

[0049] Step 37: Use the floating-point number Q output by the multilayer perceptron evaluation model. θ Calculate the loss function of the multilayer perception decision model. for:

[0050]

[0051] Among them, Q θ (o t ,a t () represents the output of the multi-layer perception evaluation model; For experience replay pool; This represents the expectation of the data distribution over the experience replay pool;

[0052] Step 38: Update the multi-layer perception decision model using the loss function of the multi-layer perception decision model;

[0053] Step 39: Repeat steps 31 to 38 until the amount of data in the experience replay pool exceeds the preset training end data amount, then the training is complete.

[0054] Preferably, in the above control method, in step 35, the loss function of the additional reward feature extraction model calculates the loss. for:

[0055]

[0056] Where θ represents the parameters of the multilayer sensing evaluation model, and its input data consists of state-action pairs (o... t ,a t The output data is a floating-point number Q. θ (o t ,a t );Z θ To pay attention to the output of the second-layer perceptron of the multilayer perceptron evaluation model, i.e. the value representation of the multilayer perceptron evaluation model; The output of the target multi-layer perception evaluation model is a copy of the multi-layer perception evaluation model before the parameter θ is updated. The output of the second-layer perceptron of the multi-layer perceptron evaluation model for the target; This is for the experience replay pool; c represents the hyperparameter, which is set to 1.0. This represents the expectation of the data distribution in the experience replay pool.

[0057] Preferably, in the above control method, in step 31, three images containing the robot and the environment are acquired at preset time intervals through a camera that communicates with the robot, and the three images are stacked using a reinforcement learning frame stack to obtain an input image sample.

[0058] Preferably, in the above control method, the convolutional feature encoder uses a four-layer convolutional neural network, the convolution kernel of each layer uses a 32×32 matrix, and no pooling operation is added between every two layers of the convolutional neural network.

[0059] Preferably, in the above control method, the multi-layer perception decision model adopts a multi-layer perceptron.

[0060] like Figure 3 As shown, embodiments of the present invention also provide a vision-based robot control system for implementing the above-described method, comprising:

[0061] The input consists of an image processing unit, a convolutional feature encoder, and a multilayer perceptual decision model trained based on additional rewards.

[0062] The input image processing unit is connected to the robot and can use a camera that communicates with the robot to acquire three images containing the robot and the environment at preset time intervals. The three images are stacked using a reinforcement learning frame stack to obtain input image data.

[0063] The convolutional feature encoder is communicatively connected to the input image processing unit and can encode the input image data obtained by the input image processing unit to obtain output representation data.

[0064] The multilayer perception decision model trained based on additional rewards is connected to the convolutional feature encoder and the robot for communication. It can map the output representation data obtained by the convolutional feature encoder into an optimal action and send it to the robot to execute the optimal action for optimal control.

[0065] Preferably, in the above control system, the training method of the multi-layer perception decision model based on additional rewards includes:

[0066] Step 31: Use the camera that communicates with the robot to acquire three images containing the robot and the environment and process them into an input image sample;

[0067] Step 32: Encode the input image sample with a convolutional feature encoder to obtain output representation data. The multilayer perception decision model maps the output representation data to decision action data for controlling the robot's behavior. The general reward function corresponding to the robot control task is used to calculate the reward data corresponding to the input image sample and the decision action data.

[0068] Step 33: Store the input image samples, decision action data, and reward data collected in the above steps into the experience playback pool in the order of data collection time;

[0069] Step 34: Take 256 input image samples, 256 decision action data, and 256 reward data from the experience replay pool as a batch. Input all image samples in this batch into a convolutional feature encoder to obtain output representation data. Input the output representation data and all reward data from the batch into an additional reward feature extraction model. Calculate the additional reward feature loss based on the information bottleneck theory using the following formula. for:

[0070]

[0071] Where ψ and η represent two parallel three-layer perceptrons in the extra reward feature extraction model, and the input data of these two three-layer perceptrons are reward data and image representation data output by the convolutional feature encoder; This represents the KL divergence used to calculate the difference between the distributions of ψ and η in two three-layer perceptrons; z t This represents the additional reward feature output by the three-layer perceptron ψ in the additional reward feature extraction model; This represents the additional reward feature label output by the three-layer perceptron η in the additional reward feature extraction model; This represents the feature z used to calculate the additional reward. t and additional reward feature tags Mutual information; α is a hyperparameter, chosen as 0.001; For experience replay pool; This represents the expectation of the data distribution over the experience replay pool;

[0072] Step 35: Update the extra reward feature extraction model based on the loss calculated by the extra reward feature extraction model;

[0073] Step 36: Use the updated additional reward features to extract the additional reward features output by the model as the label signal, and calculate the loss function of the multilayer perceptron evaluation model. Update the multi-layer perception evaluation model;

[0074] Step 37: Use the floating-point number Q output by the multilayer perceptron evaluation model. θCalculate the loss function of the multilayer perception decision model. for:

[0075]

[0076] Among them, Q θ (o t ,a t () represents the output of the multi-layer perception evaluation model; For experience replay pool; This represents the expectation of the data distribution over the experience replay pool;

[0077] Step 38: Update the multi-layer perception decision model using the loss function of the multi-layer perception decision model;

[0078] Step 39: Repeat steps 31 to 38 until the amount of data in the experience replay pool exceeds the preset training end data amount, then the training is complete.

[0079] This invention also provides a processing apparatus, comprising:

[0080] At least one memory for storing one or more programs;

[0081] At least one processor is capable of executing one or more programs stored in the memory, such that when the processor executes one or more programs, the processor can implement the methods described above.

[0082] The present invention further provides a readable storage medium storing a computer program, characterized in that the above-described method can be implemented when the computer program is executed by a processor.

[0083] In summary, the method and system of this invention, by employing a multilayer perception decision model trained based on external rewards, can encode task-related information from long-term cumulative rewards (i.e., value), and use information bottleneck constraints to remove task-irrelevant features, thereby enhancing the robot's visual generalization ability and improving the control accuracy of vision-based robots. This method and system, by integrating the information bottleneck into the representation learning process under the temporal difference learning paradigm, can directly use one-step rewards to extract robust value representations, significantly enhancing generalization ability and improving the anti-visual interference ability of the robot's agent. This invention encodes value information without including irrelevant features, providing a robust value representation that promotes decision-making and improves generalization performance. It effectively solves the problem that existing representation learning methods focus on learning robust representations that can resist task-irrelevant visual interference, but often do not consider long-term decision-related information, which may reduce generalization performance.

[0084] To more clearly demonstrate the technical solution and its effects provided by the present invention, the following detailed description of the solution provided by the embodiments of the present invention is provided with reference to specific examples.

[0085] Example 1

[0086] This embodiment provides a vision-based robot control method, which is a method to improve the visual generalization performance of vision-based control robots.

[0087] (a) In general reinforcement learning, the agent controlling the robot can capture observations o, actions a, and perform value assessments to obtain Q. π (o t ,a t The value function is defined as H. π (o t ,a t ), where Φ is a mapping relationship, and this representation is related to the general value function as follows:

[0088]

[0089] The robustness value is represented by Z. π (o t ,a t This indicates that the following constraints must be met:

[0090]

[0091] in, This represents the mutual information constraint between two variables. Based on temporal difference learning in traditional reinforcement learning, the following iterative learning form can be deduced:

[0092]

[0093] This iterative approach allows the agent to learn value representations and make decisions accordingly, thereby improving the generalization performance of the agent.

[0094] The method for learning such representations proposed in this invention achieves robust value representation learning by overcoming information bottlenecks. It can encode long-term task-related information from value assessment, thereby effectively promoting decision-making that resists visual interference in unseen environments, improving the visual generalization performance of vision-based control robots, and enhancing control accuracy.

[0095] To learn robust value representations without affecting the existing iterative framework, this invention introduces an additional reward feature extraction model, which is updated under constraints imposed by the information bottleneck. This additional reward feature extraction model can extract reward representations based on a temporal difference learning paradigm, thereby achieving the learning of value (cumulative reward) representations.

[0096] This invention proposes a learning loss function. for:

[0097]

[0098] Where θ is a multilayer perception evaluation model, and its input data are state-action pairs (o t ,a t The output data is a floating-point number Q. θ (o t ,a t Note that the output of the second-layer perceptron in the multilayer sensing (using a three-layer perceptron) evaluation model is Z. θ That is, the value representation of the multi-layer perception evaluation model; This is the output of the target multi-layer perception evaluation model, which is a copy of the multi-layer perception evaluation model before parameter θ was updated; the output of the second-layer perceptron of the target multi-layer perception evaluation model is... This is the experience replay pool; c represents the hyperparameter, which is set to 1.0. This represents the expectation of the data distribution in the experience replay pool. Through iterative updates, the agent controlling the robot can learn valuable representations by removing long-term interference information, and make decisions based on these representations to resist the influence of environmental interference and improve the accuracy of robot control.

[0099] (II) Forecasting Phase:

[0100] Step 1: Acquire three images containing the robot and the environment at preset time intervals using cameras (including internal and external cameras connected to the robot) that are in communication with the robot. Stack the three acquired images using reinforcement learning frame stacks to obtain input image data for subsequent control.

[0101] Step 2: Encode the input image data obtained in Step 1 using the trained convolutional feature encoder;

[0102] Step 3: Input the image data into the multilayer perception decision model trained based on additional rewards, map it into an optimal action, and make the robot execute the optimal action to achieve optimal control of the robot.

[0103] (III) Training Phase:

[0104] Step 31: Acquire three images containing the robot and the environment through the internal camera connected to the robot and the surrounding external cameras. Process the three acquired images into a single input image sample by stacking the frame stacks through reinforcement learning.

[0105] Step 32: The input image samples obtained in Step 1 are encoded by a convolutional neural network, which acts as a convolutional feature encoder, to obtain corresponding output representation data. Then, a multilayer perceptron, which acts as a multilayer perceptron decision model, maps the output representation data of the convolutional feature encoder to decision action data for specific robot behavior. Finally, the reward data corresponding to the input image samples and decision action data is calculated using the general reward function corresponding to the robot according to the specific robot control task.

[0106] Step 33: The input image samples, decision action data, and reward data collected in steps 1 and 2 are stored in the experience playback pool in the order of data collection time;

[0107] Step 34: Take 256 input image samples, 256 decision action data, and 256 reward data from the data stored in the experience replay pool as a batch data. Input all image samples in a batch data into the convolutional feature encoder to obtain output representation data. Input the image representation data and all reward data into the additional reward feature extraction model to calculate the loss.

[0108] Step 35: Based on the loss calculated by the additional reward feature extraction model, update the additional reward feature extraction model, the multilayer perception evaluation model, and the multilayer perception decision model in sequence;

[0109] Step 36: Repeat steps 31 to 35 until the amount of data in the experience replay pool exceeds the preset training end data amount, then end the training.

[0110] like Figure 1 As shown, the additional reward feature extraction model can extract single-step reward representations to learn value representations. The constraints introduced in this invention enable the model to learn value representations that remove long-term interference information, thereby improving the generalization ability of the controlled robot agent.

[0111] Table 1 shows the performance improvement results of the robust value representation learning method of the present invention.

[0112]

[0113] Based on the method of this invention, two advanced baseline methods were improved and evaluated in multiple task environments. In all the evaluation environments, the same level of interference from visually irrelevant information was introduced, and the method of this invention achieved significant performance improvements.

[0114] Example 2

[0115] This embodiment provides a vision-based robot control method, as detailed below:

[0116] Task environment settings:

[0117] The control method of this invention was tested in the simulation environment of MuJoCo (MuJoCo stands for Multi-Joint dynamics with Contact, mainly developed by Professor Emo Todorov of the University of Washington, and applied in fields such as optimal control, state estimation, and system identification, with significant advantages in applications involving dynamic multi-point contact in robots (such as multi-finger dexterity hand operation).

[0118] This embodiment is based on MuJoCo's robot modeling platform. Google's Robotics Lab has extended DeepMind's open-source DM Control benchmark and provided the Distracting Control Suite (DCS) platform, which encapsulates three types of visual disturbances (changes in background, color, and camera pose), providing a new benchmark for vision-based control tasks.

[0119] The control method of this invention was implemented and tested in six different simulated robot control environments. In these task environments, the observable states are the different physical quantities (such as position, angle, velocity, etc.) of various parts of the simulated robot (such as legs, joints, etc.), and the controllable actions are the magnitude of the force applied to specific parts (such as legs, head). Specifically, the six simulated robot control environments are as follows:

[0120] (1) Ball In Cup Catch: Control the planar container to translate in a vertical plane so that it swings and catches a ball attached to its bottom. The catch task has a sparse reward: 1 point when the ball is in the cup, and 0 points otherwise. The observation space is 8-dimensional and the action space is 2-dimensional.

[0121] (2) Cartpole Swingup: Controls the cart to swing and balance a lever without external force. In the swingup task, the lever initially points directly downwards. The observation space is 5-dimensional, and the action space is 1-dimensional.

[0122] (3) Cheetah Run: Controls a two-dimensional bipedal robot to run forward. The observation space is 17-dimensional and the action space is 6-dimensional.

[0123] (4) Finger Spin: Controls a two-dimensional finger to rotate a body on a hinge without external force. The state space is 12-dimensional and the action space is 2-dimensional. In the spin task, the body must rotate continuously without interruption.

[0124] (5) Reacher Easy: Controls a two-dimensional double-link reach, causing the end effector on the reach to pass through the target sphere. In the easy task, the target sphere is relatively large. The observation space is 7-dimensional, and the action space is 2-dimensional.

[0125] (6) Walker Walk: Control a 2D humanoid robot to run forward. In the walking task, the reward includes the robot's forward speed. The observation space is 24-dimensional and the action space is 6-dimensional.

[0126] The input images for the above six tasks are 3×84×84 pixels. In the experiment, the input was a superposition of three consecutive images, i.e., 9×84×84 pixels. Furthermore, the experiment was designed so that the robot would repeatedly execute the action selected by the current strategy every few frames. Table 2 below shows the number of frames in which the robot repeatedly executed the action in different environments.

[0127] Table 2 shows the number of frames for the robot to repeatedly perform actions in different environments.

[0128]

[0129] In this embodiment, the generalization performance of the visual input-based robot control method is improved in unseen test environments with visual interference. The robot is trained in multiple environments without any prior environmental knowledge (such as data augmentation, downstream adjustments, or environmental labeling), and the performance of robots trained with different control algorithms in these unseen test environments is investigated.

[0130] For each environment, the control method of this invention is compared with the most advanced methods previously used:

[0131] (1) CURL: A RL algorithm that uses contrastive learning as an auxiliary task;

[0132] (2) DrQv2: An efficient method with state-of-the-art performance on DeepMind Control (DMControl) tasks;

[0133] (3) SRM: Data augmentation is used in the frequency domain to facilitate the learning of robust policies;

[0134] (4) RAP: Effectively enhances the robustness of representation by leveraging behavioral similarity;

[0135] (5) CRESP: Learns task-related representations by predicting feature functions of reward sequences to achieve generalization;

[0136] (6)TACO: Learn state and action representations containing sufficient control information to improve sample efficiency.

[0137] The generalization ability of the control method of this invention will be tested under two visual interference scenarios. In dynamic color interference, the robot's color will be continuously changed in each round. The color change is modeled as a Gaussian distribution, with its mean set to the robot's color in the previous round. In dynamic background interference, dynamic backgrounds are extracted from different videos to replace the original background in the image. During training, two dynamic backgrounds are used to train the robot's decisions, and in the testing phase, another 30 dynamic backgrounds are used for evaluation.

[0138] The above description is merely a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims. The information disclosed in the background section is intended only to enhance the understanding of the overall background technology of the present invention and should not be construed as an admission or implication in any way that such information constitutes prior art known to those skilled in the art.

Claims

1. A vision-based robot control method, characterized in that, include: Step 1: Use a camera that communicates with the robot to acquire three images containing the robot and the environment at preset time intervals, and stack the three images using a reinforcement learning frame stack to obtain input image data; Step 2: Encode the input image data using the trained convolutional feature encoder to obtain output representation data; Step 3: Input the output representation data into the multilayer perception decision model trained based on additional rewards. The multilayer perception decision model outputs an optimal action, which the robot executes to achieve optimal control. The training methods for the multilayer perception decision-making model based on additional rewards include: Step 31: Use the camera that communicates with the robot to acquire three images containing the robot and the environment and process them into an input image sample; Step 32: Encode the input image sample with a convolutional feature encoder to obtain output representation data. The multilayer perceptual decision model maps the output representation data to decision action data for controlling the robot's behavior. The general reward function corresponding to the robot control task is used to calculate the reward data corresponding to the input image sample and the decision action data. Step 33: Store the input image samples, decision action data, and reward data collected in the above steps into the experience playback pool in the order of data collection time; Step 34: Take 256 input image samples, 256 decision action data, and 256 reward data from the experience replay pool as a batch. Input all image samples in this batch into a convolutional feature encoder to obtain output representation data. Input the output representation data and all reward data from the batch into an additional reward feature extraction model. Calculate the additional reward feature loss based on the information bottleneck theory using the following formula. for: ； in, and These represent two parallel three-layer perceptrons in the extra reward feature extraction model. The input data for both three-layer perceptrons are reward data and image representation data output by a convolutional feature encoder. This indicates the computation of two three-layer perceptrons. and KL divergence of distribution differences; This indicates that the additional reward feature extraction model consists of a three-layer perceptron. Additional reward features in the output; This indicates that the additional reward feature extraction model consists of a three-layer perceptron. Output additional reward feature labels; Indicates the current moment; Indicates the feature used to calculate additional rewards. and additional reward feature tags Mutual information; For hyperparameters, select 0.001; For experience replay pool; This represents the expectation of the data distribution over the experience replay pool; Step 35: Update the extra reward feature extraction model based on the loss calculated by the extra reward feature extraction model; Step 36: Use the updated additional reward features to extract the additional reward features output by the model as the label signal, and calculate the multilayer perception evaluation model. loss function Update the multi-layer perception evaluation model; Step 37: Use the multi-layer perceptron evaluation model to evaluate the output floating-point number. Calculate the multi-layer perception decision model loss function for: ； in, It is a multi-layer perception decision model. This is the output of the multi-layer perception evaluation model; For experience replay pool; This represents the expectation of the data distribution in the experience replay pool; Step 38: Update the multi-layer perception decision model using the loss function of the multi-layer perception decision model; Step 39: Repeat steps 31 to 38 until the amount of data in the experience replay pool exceeds the preset training end data amount, then the training is complete.

2. The vision-based robot control method according to claim 1, characterized in that, In step 35, the additional reward feature extraction model calculates the loss function. for: ； in, The parameters of the multi-layer perception evaluation model are state-action pairs. The output data is a floating-point number. ; Indicates the current moment; To pay attention to the output of the second-layer perceptron of the multilayer perceptron evaluation model, i.e. the value representation of the multilayer perceptron evaluation model; The output of the target multi-layer perception evaluation model is the parameter. A copy of the previous multi-layer perception evaluation model; The output of the second-layer perceptron of the multi-layer perceptron evaluation model for the target; For experience replay pool; This indicates a hyperparameter; 1.0 is selected. This represents the expectation of the data distribution in the experience replay pool.

3. The vision-based robot control method according to claim 1 or 2, characterized in that, In step 31, three images containing the robot and the environment are acquired at preset time intervals through a camera that communicates with the robot, and the three images are stacked using a reinforcement learning frame stack to obtain an input image sample.

4. The vision-based robot control method according to claim 1, characterized in that, The convolutional feature encoder uses a four-layer convolutional neural network. The convolutional kernel of each layer uses a 32×32 matrix, and no pooling operation is added between every two layers of the convolutional neural network.

5. The vision-based robot control method according to claim 1, characterized in that, The multi-layer perception decision model employs a multi-layer perceptron.

6. A vision-based robot control system, characterized in that, To implement the method according to any one of claims 1-5, comprising: The input consists of an image processing unit, a convolutional feature encoder, and a multilayer perceptual decision model trained based on additional rewards. The input image processing unit is connected to the robot and can use a camera that communicates with the robot to acquire three images containing the robot and the environment at preset time intervals. The three images are stacked using a reinforcement learning frame stack to obtain input image data. The convolutional feature encoder is communicatively connected to the input image processing unit and can encode the input image data obtained by the input image processing unit to obtain output representation data. The multilayer perception decision model trained based on additional rewards is connected to the convolutional feature encoder and the robot for communication. It can take the output representation data obtained by the convolutional feature encoder as input, map the output representation data into an optimal action, and send it to the robot to execute the optimal action for optimal control.

7. The vision-based robot control system according to claim 6, characterized in that, The training methods for the multilayer perception decision-making model based on additional rewards include: Step 31: Use the camera that communicates with the robot to acquire three images containing the robot and the environment and process them into an input image sample; Step 32: Encode the input image sample with a convolutional feature encoder to obtain output representation data. The multilayer perceptual decision model maps the output representation data to decision action data for controlling the robot's behavior. The general reward function corresponding to the robot control task is used to calculate the reward data corresponding to the input image sample and the decision action data. Step 33: Store the input image samples, decision action data, and reward data collected in the above steps into the experience playback pool in the order of data collection time; Step 34: Take 256 input image samples, 256 decision action data, and 256 reward data from the experience replay pool as a batch. Input all image samples in this batch into a convolutional feature encoder to obtain output representation data. Input the output representation data and all reward data from the batch into an additional reward feature extraction model. Calculate the additional reward feature loss based on the information bottleneck theory using the following formula. for: ； in, and These represent two parallel three-layer perceptrons in the extra reward feature extraction model. The input data for both three-layer perceptrons are reward data and image representation data output by a convolutional feature encoder. This represents the parameters used to calculate the two three-layer perceptrons. and KL divergence of distribution differences; This indicates that the additional reward feature extraction model consists of a three-layer perceptron. Additional reward features in the output; This indicates that the additional reward feature extraction model consists of a three-layer perceptron. Output additional reward feature labels; Indicates the current moment; Indicates the feature used to calculate additional rewards. and additional reward feature tags Mutual information; For hyperparameters, select 0.001; For experience replay pool; This represents the expectation of the data distribution in the experience replay pool; Step 35: Update the extra reward feature extraction model based on the loss calculated by the extra reward feature extraction model; Step 36: Use the updated additional reward features to extract the additional reward features output by the model as the label signal, and calculate the multilayer perception evaluation model. loss function Update the multi-layer perception evaluation model; Step 37: Use the multi-layer perceptron evaluation model to evaluate the output floating-point number. Calculate the multi-layer perception decision model loss function for: ； in, It is a multi-layer perception decision model. This is the output of the multi-layer perception evaluation model; For experience replay pool; This represents the expectation of the data distribution in the experience replay pool; Step 38: Update the multi-layer perception decision model using the loss function of the multi-layer perception decision model; Step 39: Repeat steps 31 to 38 until the amount of data in the experience replay pool exceeds the preset training end data amount, then the training is complete.

8. A processing apparatus, characterized in that, include: At least one memory for storing one or more programs; At least one processor is capable of executing one or more programs stored in the memory, such that when the one or more programs are executed by the processor, the processor is able to implement the method of any one of claims 1-5.

9. A readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it can implement the method described in any one of claims 1-5.

Images