Strategy learning method based on optimizable image transformation for mechanical arm grabbing

Abstract

The application discloses a strategy learning method based on an optimizable image conversion for mechanical arm grabbing, comprising the following steps: designing a task environment, setting parameters of a mechanical arm and a target object, and setting hyperparameters of a reinforcement learning algorithm; building a virtual environment consistent with the task environment; manipulating the mechanical arm to interact in the virtual environment, collecting training data, and calculating a task reward function according to the distance between a grabbed object and a target position, and storing the training data and the task reward function in an experience replay pool; determining a calculation mode of an optimal invariant metric under feature learning by using the optimizable image conversion; collecting a batch of data from the experience replay pool, and training an optimal strategy of the mechanical arm for moving a grabbed object to a target position in a dynamic environment by using a reinforcement learning algorithm combined with the optimizable image conversion. The application can improve the training sample efficiency of an intelligent agent based on visual perception under the condition of ensuring the migration of the strategy, thereby improving the learning efficiency and the convergence rate of the visual perception reinforcement learning algorithm.

Description

Technical Field

[0001] This invention belongs to the field of robot control technology, and in particular to a strategy learning method based on optimizable image transformation for reinforcement learning robotic arm grasping control. Background Technology

[0002] Traditional robot control algorithms, which typically require modeling the manipulated object based on kinematic and dynamic equations and solving for the end-effector pose and angle values ​​of each joint, lack the ability to autonomously learn from and generalize to the environment, making them unsuitable for increasingly complex industrial applications. In recent years, intelligent control algorithms have become a highly focused area in robot control, and reinforcement learning, as a branch of artificial intelligence, has been successfully applied to many aspects of robot control, enabling robots to acquire autonomous learning capabilities that traditional algorithms cannot provide. Visual perception-based reinforcement learning refers to an agent directly recognizing task-related information in image data and taking actions to maximize the accumulated reward, thereby achieving a predetermined goal. This method has attracted significant attention because it does not require expert-designed handcrafted features, and its applications are broad, including autonomous driving, robotic arms, drones, and other imitation and reinforcement learning scenarios. Among these, the research on reinforcement learning algorithms for robotic arms, as they are crucial actuators in robots, has also received considerable attention.

[0003] While reinforcement learning possesses autonomous learning capabilities and has demonstrated human-like decision-making and control abilities in numerous games and strategic scenarios, its application in robotic arm control still faces significant challenges due to the complexity of the robotic arm's operational model and the volatile dynamic environment. Firstly, direct control via visual perception requires frequent interaction with the environment to train the optimal policy, resulting in low sample efficiency and hindering its application in scenarios with high interaction costs. Secondly, training the optimal policy in a low-cost simulated environment and then transferring it can lead to arbitrarily poor performance in the testing environment due to differences in data distribution between the two environments. This necessitates designing a transferable reinforcement learning algorithm robust to environment-independent information to advance the application of related artificial intelligence methods in the real world. Summary of the Invention

[0004] The purpose of this invention is to address the problem of low sample efficiency during end-to-end training in existing visual perception reinforcement learning robotic arm control methods. It proposes a policy learning method based on optimizable image transformation for reinforcement learning robotic arm grasping control, which can improve the training sample efficiency of visual perception-based agents while ensuring policy transferability, thereby improving the learning efficiency and convergence rate of visual perception reinforcement learning algorithms.

[0005] The technical solution for achieving the objective of this invention is: a strategy learning method based on optimizable image transformation for robotic arm grasping, the method comprising the following steps:

[0006] Step 1: Design the task environment, set the relevant parameters of the robotic arm and the target object, and set the various hyperparameters of the reinforcement learning algorithm;

[0007] Step 2: Use the Mujoco physics engine to build a virtual environment that is consistent with the task environment;

[0008] Step 3: The intelligent agent manipulates the robotic arm to interact in the virtual environment, collects training data, calculates the task reward function based on the distance between the grasped object and the target position, and stores it together with the training data in the experience replay pool.

[0009] Step 4: Determine the calculation method of the optimal invariant metric under representation learning using optimizable image transformation;

[0010] Step 5: Collect a batch of data from the experience revisit pool and use a reinforcement learning algorithm that combines optimized image transformation to train the robotic arm to the optimal strategy for grasping objects and moving them to the target position in a dynamic environment.

[0011] Step 6: Transfer the optimal policy trained in the virtual environment to the real environment for testing.

[0012] Furthermore, step 2, which involves building a virtual environment consistent with the task environment using the Mujoco physics engine, specifically includes the following process:

[0013] Step 2.1: Based on the actual robotic arm being used, establish a physical model of the robotic arm on the SolidWorks platform and define the joint information;

[0014] Step 2.2: Use the SW2URDF plugin provided by ROS to convert the robot arm model into an URDF file that can be compiled and recognized by the Mujoco engine;

[0015] Step 2.3, Mujoco configures the relevant physical environment by generating XML files;

[0016] Step 2.4: At the beginning of each interaction segment, the virtual environment is randomized, that is, the texture and color of each component in the environment are randomized to broaden the distribution of the agent training set.

[0017] Furthermore, the specific process of step 3 includes:

[0018] Step 3.1: Randomly initialize each joint of the robotic arm in the physical environment; randomly set the positions of the grasped object and the task target; and obtain the RGB image of the control panel as the state observation value through the camera position above the robotic arm control panel.

[0019] Step 3.2: Process the RGB image obtained from the camera position using an optimizable image transformation. The specific process is as follows:

[0020] The environment returns an RGB image of size W1*H1. Pooling is used to copy the pixels at the image edges, thus increasing the RGB image size by 10%.

[0021] Restore the RGB image dimensions to W1*H1 and transform the parameters from the image. The mean μ of the Gaussian distribution is obtained and added as a pixel offset factor to the coordinate grid constructed during the reconstruction process, so that the pixels in the RGB image are affected by the image transformation parameters. Control, where σ 2 The variance is 0; the original image returned by the environment here is 0. t The converted image is o' t ;

[0022] Step 3.3, the robotic arm uses the converted current state observation value o' t And strategy, output action and add noise to it to get a t After interacting with the environment, the next state is obtained. t+1 and reward r t Reward r t The cost function c is the distance d between the target and the mission objective and the total time spent in this round of interaction. t composition;

[0023] Step 3.4, (o t ,a t ,r t ,o t+1 The data is stored as a set of training data in the experience replay pool for use in training subsequent reinforcement learning algorithms.

[0024] Furthermore, the calculation method for determining the optimal invariant metric under representation learning using optimizable image transformation in step 4 specifically includes:

[0025] Step 4.1: Given an observation x in an environment, assume that the transformed observation x' satisfies x'~p(·|e), where e represents the experience replay pool. For any random variable in the array, p(·|e) represents a probability distribution based on e;

[0026] Constraining the distance between observations x and x' is equivalent to the distribution distance of high-dimensional observations before and after the smoothing transformation, specifically represented by a divergence d(·||·):

[0027]

[0028] In the formula, x and x' represent the original sample and the enhanced sample, respectively; p represents the probability distribution; e represents the dynamic transition tuple; e is the overall distribution of the dynamic transition tuple; and d KL Denotes KL divergence, Indicates from the experience replay pool Let p(x|e=e) and p(x'|e=e) represent the distributions of the original sample and the augmented sample in e, respectively.

[0029] Step 4.2, rewrite the divergence expression from Step 4.1 using Bayesian conditional probability, as follows:

[0030]

[0031] In the formula, s represents the objective and real state of the agent at present; It represents the mathematical expectation of a certain dynamic transition tuple e; Let f(x) represent a mathematical expectation under the distribution of a tuple e given the current true state s; p(x|e) and p(x'|e) represent the data distribution of the original sample or the augmented sample based on a certain dynamic transition tuple e, respectively; p(s|x) and p(s|x') represent the probability distribution of the current state of the original sample or the augmented sample being the true state s, respectively, which is achieved by introducing an encoder f(·); p(x) and p(x') represent the true prior distribution of the original sample or the augmented sample, respectively.

[0032] Step 4.3 introduces an additional target encoder. in This refers to the aforementioned optimizable image conversion. To obtain the probability distribution of the true state s after image transformation of the original sample x;

[0033] Step 4.4, the distance d(f(x), f'(x')) between encoders f(x) and f'(x) is given by the following triangle inequality:

[0034] d(f(x),f'(x'))≤d(f(x),f'(x))+d(f'(x),f'(x'))

[0035] In the formula, d(f(x),f'(x)) represents the functional similarity of the encoder; d(f'(x),f'(x')) represents the image transformation operation using β-similarity constraints;

[0036] Step 4.5 introduces the learning process of online features and momentum features in self-supervised contrastive learning to constrain the functional similarity within the inequality of step 4.4; specifically:

[0037] Assuming that the two encoders f(x) and f'(x) have the same structure, at time step t, based on the online encoder f... ψ Using the average exponential moving average ψ t =(1-τ) m )ψ t-1 +τ m ψ Update Momentum Encoder Where τ m ∈[0,1] represents the momentum update rate, and ψ represents the online encoder f. ψ The parameter, ψ t Indicates momentum encoder The parameter ψ at time t t-1 Indicates momentum encoder The parameters at time t-1; a mapping network g is also added: This means mapping the low-dimensional state S obtained by encoder f to another low-dimensional space. In low-dimensional space Minimize the distance within the metric;

[0038] Through the above analysis of the convexity of the function and structural design, minimization Equivalent to minimizing and These represent the sequence encoding g of sample x. ψ (f ψ (x) and This represents a momentum mapping network, with parameters also determined by ψ. t Given;

[0039] Step 4.6 introduces a data mixture, namely, from the learned distribution Multiple augmented samples are sampled, and then the encoded latent representations s' are blended; specifically:

[0040] Assume f ψ and If the convergence assumption holds, then for any input... With image transformation The generated enhanced sample x', optimizing the aforementioned divergence is equivalent to minimizing the upper bound of the following formula:

[0041]

[0042] In the formula, ρ=L f (CL g +||ψ g ||, τ=1-τ m Both are constants, L g and L f They represent network g respectively ψ and f ψ Lipschitz constant, ψ g The parameters represent the mapping network g. This represents the empirical replay pool obtained after introducing augmented samples. All samples x within.

[0043] Furthermore, step 5 involves collecting a batch of data from the experience revisit pool and using a reinforcement learning algorithm that combines image transformation optimization to train the robotic arm's optimal strategy for grasping objects and moving them to a target position in a dynamic environment. The specific process includes:

[0044] Step 5.1: Randomly collect a batch of training data from the experience replay pool. Each data point consists of a quadruple (o t ,a t ,r t ,o t+1 Composed of ) where o t with o t+1 Let a represent the environmental observations generated at times t and t+1, respectively. t r represents the action taken by the agent at time t. t Indicates that the agent takes action a at time t. t The rewards obtained later will be returned;

[0045] Step 5.2, for the observed value o t with o t+1 Perform the same image enlargement and restoration operations as in step 3.2, and transform the image parameters. A pixel offset factor is randomly sampled from a Gaussian distribution and added to the coordinate grid constructed during the reconstruction process, introducing randomness into the transformation of the observed image;

[0046] Step 5.3 involves fitting the statistical distribution of the transformed observations to the population distribution of the sample, specifically including:

[0047] Calculate the transformed observation o' within a batch t The distribution of RGB colors, i.e., the mean and variance, is obtained. Simultaneously, the built-in mean and variance of all batch normalization layers in the image encoder are acquired, and then the Wasserstein distance between their distributions is calculated.

[0048]

[0049] In the formula, o' t This represents the augmented samples generated within the current stride, and l represents a convolutional layer in the encoder. and Let represent the mean and variance of the observations after the current batch transformation, and the mean and variance stored in the normalized layers of each batch after each convolutional layer, respectively. Let o be any observation. Let t be the observation space, and t represent the current training step size. This represents the Gaussian distribution used for parameterized image transformation;

[0050] Step 5.4, using image encoder f ψ (·) and mapping network g ψ (·) Encodes the augmented samples o' generated within the current training step. t Get z' t Using the target network and Encode the raw observations within the current training step. t Get z t And optimize z' t With z t L2 norm between

[0051]

[0052] In the formula, g ψ (f ψ (o' t ))and These represent the augmentation of sample o' t and the original sample o t Sequence encoding is performed, where ||·||2 represents the magnitude of the vector, and this process is carried out in batches for training;

[0053] Step 5.5, use the transformed observations (o') t ,o' t+1 Update strategy network π θ (s) and value function network Q φ (s,a), specifically including:

[0054] Update the value function network Q using Q-learning. φ (s, a):

[0055]

[0056] In the formula, o' t with o' t+n Let φ represent the augmented samples generated at time t and t+n, respectively, and let φ represent the value function network Q.φ The parameters, Representation of the value function network Q φ The momentum update version, γ∈[0,1] represents the discount factor, Q φ (f ψ (o' t ), a t ) indicates based on augmented sample o' t With action a t The predicted reward value, This indicates that the predicted values ​​after n steps are used for fitting. This represents the augmented sample o' after n steps of discounting. t+n With policy network π θ The target reward prediction value;

[0057] Combining steps 5.3 and 5.4, we obtain encoder f. ψ (o) and the value function network Q φ The overall objective function of (s, a) is:

[0058]

[0059] In the formula, α and λ are weight parameters manually set according to the task. The larger the weight, the stronger the regularization effect of the term.

[0060] Policy Network π θ The update process of (s) is the DDPG algorithm, which uses a greedy strategy to directly optimize the value function network Q. φ (s,a) Predicted policy gradient

[0061]

[0062] In the formula, π θ Let π, a ~ π be the policy function parameterized by parameter θ. θ (f ψ (o' t )) indicates that from the augmented sample o' t Choose one action from the obtained action distribution, Q φ (f ψ (o' t ),a) indicates based on augmented sample o' t Predict the reward based on the selected action 'a';

[0063] Repeat steps 3.2 to 5.4 until the robotic arm grasps the target object and reaches the task destination, or the robotic arm encounters an illegal area, or the set maximum time step is reached, which completes this training section.

[0064] After initializing the environment, the training subsections are repeated K times until the algorithm fully converges, resulting in a transferable optimal policy network for the robotic arm to grasp target objects and reach designated positions in a dynamic environment.

[0065] Furthermore, step 5.2 also includes: parameterizing the Gaussian distribution, specifically including:

[0066] When sampling the pixel offset factor, first generate a random number ∈ within (0,1), then parameterize the required offset factor as μ+∈·σ, and truncate the factor within the set range, thus completing the sampling of image transformation based on a certain Gaussian distribution.

[0067] Compared with the prior art, the significant advantages of this invention are:

[0068] 1) This invention only requires a small number of matrix operations on high-dimensional image observations to improve performance, without the need for additional modeling design, resulting in low computational overhead and fast training speed.

[0069] 2) Compared with the existing technology, the present invention introduces data mixing operation, stabilizes the reward function, and improves the stability of algorithm training.

[0070] 3) Data augmentation algorithms improve sample utilization efficiency while expanding the distribution of training data, achieving regularization of the neural network model, which is beneficial for feature representation learning.

[0071] 4) This invention does not require any modification to the backbone algorithm of reinforcement learning, and has strong compatibility.

[0072] The present invention will now be described in further detail with reference to the accompanying drawings. Attached Figure Description

[0073] Figure 1 This is a flowchart of the network framework of the present invention.

[0074] Figure 2 This is a schematic diagram of the connection setup for the robotic arm experimental platform. Detailed Implementation

[0075] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0076] It should be noted that if the embodiments of the present invention involve descriptions such as "first" and "second," these descriptions are for descriptive purposes only and should not be construed as indicating or implying their relative importance or implicitly specifying the number of technical features indicated. Therefore, a feature defined with "first" and "second" may explicitly or implicitly include at least one of those features. Furthermore, the technical solutions of the various embodiments can be combined with each other, but this must be based on the ability of those skilled in the art to implement them. When the combination of technical solutions is contradictory or impossible to implement, it should be considered that such a combination of technical solutions does not exist and is not within the scope of protection claimed by the present invention.

[0077] In one embodiment, a policy learning method based on optimizable image transformation is provided for robotic arm grasping. This embodiment uses a 6-DOF research ArmPi robotic arm as an example. The designed task scenario is to use a reinforcement learning algorithm in a dynamic environment to control the robotic arm to grasp and move an object from an arbitrary position to a designated destination. The target object is a cuboid of different sizes, and its position changes randomly in each test segment. The framework of the policy learning method based on optimizable image transformation for reinforcement learning robotic arm grasping control described in this embodiment is as follows: Figure 1 As shown, it includes at least the following steps:

[0078] Step S1: Design the task environment, set the relevant parameters of the robotic arm and the target object, and set the hyperparameters of the reinforcement learning algorithm. This includes the following steps:

[0079] Step 1.1: The state observation value of the design task environment is the RGB image information captured by a fixed camera;

[0080] Step 1.2: Set the basic hyperparameters of the reinforcement learning algorithm, including at least: exploration noise and experience revisit pool. Size; number of updates K per training session, and batch size N of data used in each update; structure and number of layers of the neural network, number of nodes, activation function, and normalization function in each layer; discount factor γ; image encoder f. ψ (o); Policy Network Π θ (s) and value function network Q φ (s,a) Optimizer for parameter updates, learning rate, target network The soft update step size τ can optimize the image transformation parameters required for image transformation. With mapping network g ψ (s) Optimizer for parameter updates, learning rate, target network soft update step size τ m .

[0081] Step S2 involves using the Mujoco physics engine to build a virtual environment consistent with the task environment, specifically including the following steps:

[0082] Step 2.1: Establish an accurate physical model of the robotic arm on the SolidWorks platform based on the actual robotic arm being used, and define the joint information;

[0083] Step 2.2: Use the SW2URDF plugin provided by ROS to convert the robot arm model into an URDF file that can be compiled and recognized by the Mujoco engine;

[0084] Step 2.3, Mujoco configures the relevant physical environment (such as lighting, desktop, camera position, joint control, etc.) by generating XML files;

[0085] Step 2.4: At the beginning of each interaction section, the virtual environment needs to be randomized, that is, the texture and color of each component in the environment are randomized to broaden the distribution of the agent training set.

[0086] Step S3: The intelligent agent manipulates the robotic arm to interact in the virtual environment, collects training data, calculates the task reward function based on the distance between the grasped object and the target position, and stores it in the experience replay pool along with the training data. This specifically includes the following steps:

[0087] Step 3.1: Randomly initialize each joint of the robotic arm in the physical environment; randomly set the positions of the grasped object and the task target; and obtain RGB images as state observation values ​​through the camera position above the robotic arm's operating table.

[0088] Step 3.2: Process the RGB image obtained from the camera position using an optimizable image transformation. The specific steps are as follows:

[0089] The environment returns an RGB image of size 165*165. Pooling is used to copy pixels from the image edges, enlarging the RGB image size to 181*181. The `grid_sample` function from the PyTorch library is then used to restore the image size to 165*165, and parameters are transformed from the image. The mean μ of the Gaussian distribution is obtained and added as a pixel offset factor to the coordinate grid constructed during the reconstruction process, so that the pixels in the RGB image are affected by the image transformation parameters. Control. The original image returned by the environment here is o. t The converted image is o' t .

[0090] Step 3.3, the robotic arm uses the converted current state observation value o' t And strategy, output action and add noise to it to get at After interacting with the environment, the next state is obtained. t+1 and reward r t Reward r t The cost function c is the distance d between the target and the mission objective and the total time spent in this round of interaction. t Composition. The robot arm is controlled to move to the next state while ensuring that all joints are at legal angles.

[0091] Step 3.4, (o t a t r t o t+1 The data is stored as a set of training data in the experience replay pool for use in training subsequent reinforcement learning algorithms.

[0092] Step S4 involves determining the computation method for the optimal invariant metric under representation learning using optimizable image transformation. The specific process includes...

[0093] Step 4.1: Given an observation x in an environment, assume that the transformed observation x' satisfies x'~p(·|e), where e represents the experience replay pool. Any random variable in the array. Constraining the distance between them is equivalent to the distribution distance of the high-dimensional observations before and after the smoothing transformation, specifically represented by a divergence d(·||·):

[0094]

[0095] In the formula, x and x' represent the original sample and the augmented sample, respectively, and e represents the dynamic transition tuple obtained during the current training. Indicates from the experience replay pool Let p(x|e=e) and p(x'|e=e) be a mathematical expectation of e obtained from sampling, and let p(x|e=e) and p(x'|e=e) represent the distributions of the original sample and the augmented sample in e, respectively.

[0096] Step 4.2, rewrite the divergence expression using Bayesian conditional probability, specifically as follows:

[0097]

[0098] In the formula, s represents the objective and real state of the agent at present; It represents the mathematical expectation of a certain dynamic transition tuple e; Let p(x|e) represent a mathematical expectation under the distribution of a tuple e given the current true state s; p(x|e) and p(x'|e) represent the data distribution of the original sample or the augmented sample based on a certain dynamic transition tuple e, respectively; p(s|x) and p(s|x') represent the probability distribution of the current state of the original sample or the augmented sample being the true state s, respectively. This process can be achieved by introducing an encoder f(·); p(x) and p(x') represent the true prior distribution of the original sample or the augmented sample, respectively.

[0099] Step 4.3: To stabilize the convergence process of the aforementioned conditional divergence, an additional target encoder is introduced here. in This refers to the aforementioned optimizable image conversion. This is to obtain the probability distribution of the true state s after image transformation of the original sample x.

[0100] Step 4.4, the distance between the encoded latent states f(x) and f'(x) can be given by the following triangle inequality:

[0101] d(f(x),f'(x'))≤d(f(x),f'(x))+d(f'(x),f'(x'))

[0102] Minimizing the right-hand side of the inequality can raise the upper bound of the problem that the algorithm needs to solve. In the right-hand side of the inequality, the first term represents the functional similarity of the encoder; the second term represents the image transformation operation using β-similarity constraints.

[0103] Step 4.5 introduces the learning process of online features and momentum features from self-supervised contrastive learning to constrain the functional similarity within the aforementioned inequalities. Specifically, under the premise that the two encoders f(x) and f'(x) have the same structure, at time step t, based on the online encoder f... ψ Using the average exponential moving average ψ t =(1-τ) m )ψ t-1 +τ m ψ Update Momentum Encoder Where τ m ∈[0,1] represents the momentum update rate, and ψ represents the online encoder f. ψ The parameter, ψ t Indicates momentum encoder The parameters at time t. The method also incorporates a mapping network g: This means mapping the low-dimensional state S obtained by encoder f to another low-dimensional space. In low-dimensional space Minimize the distance within the metric.

[0104] By analyzing the convexity of the function and designing the structure, minimizing Equivalent to minimizing In the formula and This indicates that the sequence is encoded by g for sample x. ψ (f ψ (x) and This represents a momentum mapping network, with parameters also determined by ψ. t Given.

[0105] Step 4.6: By ensuring the Lipschitz property of the network model, the minimization can be achieved. The problem is related to the distance between constrained high-dimensional samples.

[0106] Specifically, assume f ψ and If the convergence assumption holds, then for any input... With image transformation The generated enhanced sample x', optimizing the conditional divergence given above is equivalent to minimizing the following upper bound:

[0107]

[0108] Where ρ=L f (CL g +||ψ g ||), τ=1-τ m Both are constants, L g and L f They represent network g respectively ψ and f ψ The Lipschitz constant, ψ represents the parameters of the mapping network g. This represents the empirical replay pool obtained after introducing augmented samples. All samples x within.

[0109] Step 4.7: To further stabilize the reward function, this invention introduces a data mixing method, namely, utilizing image transformation. Multiple augmented samples are sampled, and then the encoded latent representations s' are blended.

[0110] Data Mixing: Assuming g ψ and If the convergence assumption holds, then for any input... With image transformation The generated hybrid augmented sample x', optimization theorem 1, gives the conditional divergence as equivalent to minimizing the following upper bound:

[0111]

[0112] Where ρ=L f (CL g +||ψ g ||), τ=1-τ m Both are constants. This represents the expected distribution of the enhanced sample x' obtained by mixing multiple samples. This completes the theoretical analysis of the invariant transformation.

[0113] Step S5: Collect a batch of data from the experience revisit pool, and use a reinforcement learning algorithm that combines optimized image transformation to train the optimal strategy for the robotic arm to grasp objects and move them to the target position in a dynamic environment. The specific process includes:

[0114] Step 5.1: Randomly collect a batch of training data from the experience replay pool. Each data point consists of a quadruple (o) stored between the two pools. t ,a t ,r t ,o t+1 Composed of ) where o t with o t+1 Let a represent the environmental observations generated at times t and t+1. t r represents the action taken by the agent at time t. t Indicates that the agent takes action a at time t. t The rewards obtained will be returned.

[0115] Step 5.2, for the observed value o t with o t+1 Perform the same image enlargement and restoration operations as in step 3.2, and transform the image parameters. A pixel offset factor is randomly sampled from a Gaussian distribution and added to the coordinate grid constructed during the reconstruction process, introducing randomness into the transformation of the observed image. To allow the pixel offset factor to receive gradients for optimization learning, the Gaussian distribution is parameterized. The specific steps are as follows:

[0116] Image conversion parameters In this context, μ and σ are the mean and standard deviation of the Gaussian distribution, respectively. When sampling the pixel offset factor, a random number ∈ (0,1) is first generated. Then, the offset factor to be sampled can be parameterized as μ+∈·σ. By truncating this factor within the legal range, the sampling of image transformation based on a certain Gaussian distribution can be completed.

[0117] Step 5.3 involves fitting the statistical distribution of the transformed observations to the population distribution of the sample. The specific steps are as follows:

[0118] Calculate the image-transformed observation o' within a batch.t The distribution (mean and variance) of RGB colors is obtained, and the built-in mean and variance of all batch normalization layers in the image encoder are also obtained. Then, the Wasserstein distance between the two distributions is calculated:

[0119]

[0120] Among them o' t This represents the augmented samples generated within the current stride, and l represents a convolutional layer in the encoder. and Let represent the mean and variance of the observations after transformation in the current batch, and the mean and variance stored in the normalized layers of each batch after each convolutional layer, respectively. Let o be any observation. Let t be the observation space, and t represent the current training step size. This represents the Gaussian distribution used for parameterized image transformation.

[0121] Step 5.4, using image encoder f ψ (·) and mapping network g ψ (·) Encodes the augmented samples o' generated within the current training step. t Get z' t Using the target network and Encode the raw observations within the current training step. t Get z t And optimize z' t With z t L2 norm between:

[0122]

[0123] Here G represents the experience replay pool. ψ (f ψ (o' t ))and These represent the augmentation of sample o' t and the original sample o t Sequence encoding is performed, where ||·||2 represents the magnitude of the vector. This process is carried out in batches for training.

[0124] Step 5.5, use the transformed observations (o') t ,o' t+1 Update strategy network π θ (s) and value function network Q φ (s,a), the specific steps are as follows:

[0125] Update the value function network Q using Q-learning. φ (s,a):

[0126]

[0127] In the formula, o' t with o' t+n Let φ represent the augmented samples generated at time t and t+n, respectively, and let φ represent the value function network Q. φ The parameters, Representation of the value function network Q φ The momentum update version, γ∈[0,1] represents the discount factor, Q φ (f ψ (o' t ),a t ) indicates based on augmented sample o' t With action a t The predicted reward value, This indicates that the predicted values ​​after n steps are used for fitting. This represents the augmented sample o' after n steps of discounting. t+n With policy network π θ The target reward prediction value is used to improve training stability and reduce overestimation. This formula represents the optimizable image transformation parameters. This will optimize the Bellman error along with the value function network, improving the stability of representation learning. Combining steps 5.3 and 5.4, the encoder f can be obtained. ψ (o) and the value function network Q φ The overall objective function of (s,a) is:

[0128]

[0129] Where α and λ are weight parameters manually set according to the task, and the weight terms are positively correlated with the strength of the regularization effect.

[0130] Policy Network π θ The update process of (s) is the DDPG algorithm, which uses a greedy strategy to directly optimize the value function network Q. φ The policy gradient predicted by (s,a) is:

[0131]

[0132] π θ Let π, a ~ π be the policy function parameterized by parameter θ. θ (f ψ (o' t )) indicates that from the augmented sample o' t Choose one action from the obtained action distribution, Q φ (f ψ (o' t),a) indicates based on augmented sample o' t Reward prediction is performed based on the selected action 'a'.

[0133] Repeat steps 3.2 to 5.4 until the robotic arm grasps the target object and reaches the task destination, or encounters an illegal area (such as the ground), or reaches the set maximum time step; this completes the training sub-section. After initializing the environment, repeat the training sub-section K times until the algorithm fully converges, obtaining a transferable optimal policy network for the robotic arm to grasp the target object and reach the designated position in a dynamic environment.

[0134] Step S6 involves transferring the optimal policy trained in the virtual environment to the real environment for testing, specifically including the following steps:

[0135] After the agent converges in the Mujoco environment, the model weights are saved and then transferred to a robotic arm in a real-world experimental environment for final fine-tuning. For example... Figure 2 As shown, the robotic arm experimental platform used in this invention is a six-DOF research ArmPi robotic arm manufactured by Huaner Technology, equipped with a high-definition camera with a USB interface, and controlled by a Raspberry Pi 4B. The high-definition camera is model hv3808, based on a CMOS sensor with up to 300,000 pixels and a maximum resolution of 640×480. The Raspberry Pi 4B is equipped with a Raspberry Pi extension version, with a quad-core CPU and a clock speed of 1.5GHz. The GPU is Broadcom VideoCore VI, and wireless communication between the Raspberry Pi and the host is achieved through its built-in wireless network card. Considering that the performance of the Raspberry Pi itself is insufficient to complete the algorithm training, target localization and strategy execution are performed through wireless communication between the host and the Raspberry Pi. As the lower-level machine, the Raspberry Pi has an 8-channel overcurrent-protected PWM servo interface and a built-in single-bus circuit, which can directly control the serial port servos.

[0136] When the robotic arm performs an operation in the working environment, a high-definition camera captures image information including the target position and the robotic arm's state. This image information is transmitted to the Raspberry Pi controller via a USB interface, and then to the host computer via a wireless network for image feature extraction. The extracted features are then transmitted to a policy network trained in a simulated environment. The policy network outputs control actions based on these features. The output actions are transmitted to the Raspberry Pi controller via Wi-Fi, generating control current through the PWM servo interface to drive the robotic arm's joint angles, causing the end effector to reach the target position. The vision module then feeds back the grasping results to the controller and the host computer to complete the cycle, ultimately completing the target grasping task.

[0137] In one embodiment, a policy learning system based on optimizable image transformation is provided for robotic arm grasping, the system comprising:

[0138] The first module is used to design the task environment, set the relevant parameters of the robotic arm and the target object, and set the various hyperparameters of the reinforcement learning algorithm.

[0139] The second module is used to build a virtual environment that is consistent with the task environment using the Mujoco physics engine;

[0140] The third module is used to enable the intelligent agent to manipulate the robotic arm to interact in the virtual environment, collect training data, calculate the task reward function based on the distance between the grasped object and the target position, and store it together with the training data in the experience replay pool.

[0141] The fourth module is used to determine the computation method of the optimal invariant metric under representation learning by utilizing optimizable image transformation;

[0142] The fifth module is used to collect a batch of data from the experience revisit pool and use a reinforcement learning algorithm that combines optimized image transformation to train the robotic arm to the optimal strategy for grasping objects and moving them to the target position in a dynamic environment.

[0143] The sixth module is used to transfer the optimal policy trained in the virtual environment to the real environment for testing.

[0144] Specific limitations regarding the policy learning system based on optimizable image transformation for robotic arm grasping can be found in the limitations of the policy learning method based on optimizable image transformation for robotic arm grasping mentioned above, and will not be repeated here. Each module in the aforementioned policy learning system based on optimizable image transformation for robotic arm grasping can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in hardware or independently of the processor in a computer device, or stored in software in the memory of a computer device, so that the processor can call and execute the operations corresponding to each module.

[0145] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to perform the following steps:

[0146] Step 1: Design the task environment, set the relevant parameters of the robotic arm and the target object, and set the various hyperparameters of the reinforcement learning algorithm;

[0147] Step 2: Use the Mujoco physics engine to build a virtual environment that is consistent with the task environment;

[0148] Step 3: The intelligent agent manipulates the robotic arm to interact in the virtual environment, collects training data, calculates the task reward function based on the distance between the grasped object and the target position, and stores it together with the training data in the experience replay pool.

[0149] Step 4: Determine the calculation method of the optimal invariant metric under representation learning using optimizable image transformation;

[0150] Step 5: Collect a batch of data from the experience revisit pool and use a reinforcement learning algorithm that combines optimized image transformation to train the robotic arm to the optimal strategy for grasping objects and moving them to the target position in a dynamic environment.

[0151] Step 6: Transfer the optimal policy trained in the virtual environment to the real environment for testing.

[0152] For specific limitations on each step, please refer to the limitations on policy learning methods based on optimizable image transformation used in robotic arm grasping above, which will not be repeated here.

[0153] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, the computer program performing the following steps when executed by a processor:

[0154] Step 1: Design the task environment, set the relevant parameters of the robotic arm and the target object, and set the various hyperparameters of the reinforcement learning algorithm;

[0155] Step 2: Use the Mujoco physics engine to build a virtual environment that is consistent with the task environment;

[0156] Step 3: The intelligent agent manipulates the robotic arm to interact in the virtual environment, collects training data, calculates the task reward function based on the distance between the grasped object and the target position, and stores it together with the training data in the experience replay pool.

[0157] Step 4: Determine the calculation method of the optimal invariant metric under representation learning using optimizable image transformation;

[0158] Step 5: Collect a batch of data from the experience revisit pool and use a reinforcement learning algorithm that combines optimized image transformation to train the robotic arm to the optimal strategy for grasping objects and moving them to the target position in a dynamic environment.

[0159] Step 6: Transfer the optimal policy trained in the virtual environment to the real environment for testing.

[0160] For specific limitations on each step, please refer to the limitations on policy learning methods based on optimizable image transformation used in robotic arm grasping above, which will not be repeated here.

[0161] The foregoing has shown and described the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The embodiments and descriptions in the specification are merely illustrative of the principles of the invention. Any modifications, equivalent substitutions, or improvements made within the spirit and principles of the present invention without departing from its spirit and scope should be included within the protection scope of the present invention.

Claims

1. A strategy learning method based on optimizable image transformation for robotic arm grasping, characterized in that, The method includes the following steps: Step 1: Design the task environment, set the relevant parameters of the robotic arm and the target object, and set the various hyperparameters of the reinforcement learning algorithm; Step 2: Use the Mujoco physics engine to build a virtual environment that is consistent with the task environment; Step 3: The intelligent agent manipulates the robotic arm to interact in the virtual environment, collects training data, calculates the task reward function based on the distance between the grasped object and the target position, and stores it together with the training data in the experience replay pool. Step 4: Determine the calculation method of the optimal invariant metric under representation learning using optimizable image transformation; Step 5: Collect a batch of data from the experience revisit pool and use a reinforcement learning algorithm that combines optimized image transformation to train the robotic arm to the optimal strategy for grasping objects and moving them to the target position in a dynamic environment. Step 6: Transfer the optimal policy trained in the virtual environment to the real environment for testing; Step 4 describes the calculation method for determining the optimal invariant metric under representation learning using optimizable image transformation. The specific process includes: Step 4.1: Given an observation x in an environment, assume that the transformed observation x' satisfies x'~p(·|e), where e represents the experience replay pool. For any random variable in the array, p(·|e) represents a probability distribution based on e; Constraining the distance between observations x and x' is equivalent to the distribution distance of high-dimensional observations before and after the smoothing transformation, specifically represented by a divergence d(·||·): In the formula, x and x' represent the original sample and the enhanced sample, respectively; p represents the probability distribution; e represents the dynamic transition tuple; e is the overall distribution of the dynamic transition tuple; and d KL Denotes KL divergence, Indicates from the experience replay pool Let p(x|e=e) and p(x'|e=e) represent the distributions of the original sample and the augmented sample in e, respectively. Step 4.2, rewrite the divergence expression from Step 4.1 using Bayesian conditional probability, as follows: In the formula, s represents the objective and real state of the agent at present; It represents the mathematical expectation of a certain dynamic transition tuple e; Let f(x) represent a mathematical expectation under the distribution of a tuple e given the current true state s; p(x|e) and p(x'|e) represent the data distribution of the original sample or the augmented sample based on a certain dynamic transition tuple e, respectively; p(s|x) and p(s|x') represent the probability distribution of the current state of the original sample or the augmented sample being the true state s, respectively, which is achieved by introducing an encoder f(·); p(x) and p(x') represent the true prior distribution of the original sample or the augmented sample, respectively. Step 4.3 introduces an additional target encoder. in This refers to the aforementioned optimizable image conversion. To obtain the probability distribution of the true state s after image transformation of the original sample x; Step 4.4, the distance d(f(x), f'(x')) between encoders f(x) and f'(x) is given by the following triangle inequality: d(f(x),f'(x'))≤d(f(x),f'(x))+d(f'(x),f'(x')) In the formula, d(f(x),f'(x)) represents the functional similarity of the encoder; d(f'(x),f'(x')) represents the image transformation operation using β-similarity constraints; Step 4.5 introduces the learning process of online features and momentum features in self-supervised contrastive learning to constrain the functional similarity within the inequality of step 4.4; specifically: Assuming that the two encoders f(x) and f'(x) have the same structure, at time step t, based on the online encoder f ψ Using the average exponential moving average ψ t =(1-τ) m )ψ t-1 +τ m ψ Update Momentum Encoder Where τ m ∈[0,1] represents the momentum update rate, and ψ represents the online encoder f. ψ The parameter, ψ t Indicates momentum encoder The parameter ψ at time t t-1 Indicates momentum encoder The parameters at time t-1; a mapping network is also added. This means mapping the low-dimensional state S obtained by encoder f to another low-dimensional space. In low-dimensional space Minimize the distance within the metric; By analyzing the convexity of the function and designing the structure, minimizing Equivalent to minimizing and These represent the sequence encoding g of sample x. ψ (f ψ (x) and This represents a momentum mapping network, with parameters also determined by ψ. t Given; Step 4.6 introduces a data mixture, namely, from the learned distribution Multiple augmented samples are sampled, and then the encoded latent representations s' are blended; specifically: Assume f ψ and If the convergence assumption holds, then for any input... With image transformation The generated enhanced sample x', optimizing the aforementioned divergence is equivalent to minimizing the upper bound of the following formula: In the formula, ρ=L f (CL g +||ψ g ||), τ=1-τ m Both are constants, L g and L f They represent network g respectively ψ and f ψ Lipschitz constant, ψ g The parameters represent the mapping network g. This represents the empirical replay pool obtained after introducing augmented samples. All samples x within.

2. The strategy learning method based on optimizable image transformation for robotic arm grasping according to claim 1, characterized in that, Step 2 describes building a virtual environment consistent with the task environment using the Mujoco physics engine. The specific process includes: Step 2.1: Based on the actual robotic arm being used, establish a physical model of the robotic arm on the SolidWorks platform and define the joint information; Step 2.2: Use the SW2URDF plugin provided by ROS to convert the robot arm model into an URDF file that can be compiled and recognized by the Mujoco engine; Step 2.3, Mujoco configures the relevant physical environment by generating XML files; Step 2.4: At the beginning of each interaction segment, the virtual environment is randomized, that is, the texture and color of each component in the environment are randomized to broaden the distribution of the agent training set.

3. The strategy learning method based on optimizable image transformation for robotic arm grasping according to claim 1, characterized in that, Step 3 includes the following specific steps: Step 3.1: Randomly initialize each joint of the robotic arm in the physical environment; randomly set the positions of the grasped object and the task target; and obtain the RGB image of the control panel as the state observation value through the camera position above the robotic arm control panel. Step 3.2: Process the RGB image obtained from the camera position using an optimizable image transformation. The specific process is as follows: The environment returns an RGB image of size W1*H1. Pooling is used to copy the pixels at the image edges, thus increasing the RGB image size by 10%. Restore the RGB image dimensions to W1*H1 and transform the parameters from the image. The mean μ of the Gaussian distribution is obtained and added as a pixel offset factor to the coordinate grid constructed during the reconstruction process, so that the pixels in the RGB image are affected by the image transformation parameters. Control, where σ 2 The variance is 0; the original image returned by the environment here is 0. t The converted image is o' t ; Step 3.3, the robotic arm uses the converted current state observation value o' t And strategy, output action and add noise to it to get a t After interacting with the environment, the next state is obtained. t+1 and reward r t Reward r t The cost function c is the distance d between the target and the mission objective and the total time spent in this round of interaction. t Composition; controlling the movement of the robotic arm to the next state while ensuring that all joints of the robotic arm are at legal angles in the next state; Step 3.4, (o t ,a t ,r t ,o t+1 The data is stored as a set of training data in the experience replay pool for use in training subsequent reinforcement learning algorithms.

4. The strategy learning method based on optimizable image transformation for robotic arm grasping according to claim 3, characterized in that, Step 5 involves collecting a batch of data from the experience revisit pool and using a reinforcement learning algorithm that combines optimized image transformation to train the robotic arm's optimal strategy for grasping objects and moving them to a target position in a dynamic environment. The specific process includes: Step 5.1: Randomly collect a batch of training data from the experience replay pool. Each data point consists of a quadruple (o t ,a t ,r t ,o t+1 Composed of ) where o t With o t+1 Let a represent the environmental observations generated at times t and t+1, respectively. t Let r represent the action taken by the agent at time t. t Indicates that the agent takes action a at time t. t The rewards obtained later will be returned; Step 5.2, for the observed value o t with o t+1 Perform the same image enlargement and restoration operations as in step 3.2, and transform the image parameters. A pixel offset factor is randomly sampled from a Gaussian distribution and added to the coordinate grid constructed during the reconstruction process, introducing randomness into the transformation of the observed image; Step 5.3 involves fitting the statistical distribution of the transformed observations to the population distribution of the sample, specifically including: Calculate the transformed observation o' within a batch t The distribution of RGB colors, i.e., the mean and variance, is obtained. Simultaneously, the built-in mean and variance of all batch normalization layers in the image encoder are acquired, and then the Wasserstein distance between their distributions is calculated. In the formula, o' t This represents the augmented samples generated within the current stride, and l represents a convolutional layer in the encoder. and Let represent the mean and variance of the observations after the current batch transformation, and the mean and variance stored in the normalized layers of each batch after each convolutional layer, respectively. Let o be any observation. Let t be the observation space, and t represent the current training step size. This represents the Gaussian distribution used for parameterized image transformation; Step 5.4, using image encoder f ψ (·) and mapping network g ψ (·) Encodes the augmented samples o' generated within the current training step. t Get z' t Using the target network and Encode the raw observations within the current training step. t Get z t And optimize z' t With z t L2 norm between In the formula, g ψ (f ψ (o' t ))and These represent the augmentation of sample o' t and the original sample o t Sequence encoding is performed, where ||·||2 represents the magnitude of the vector, and this process is carried out in batches for training; Step 5.5, use the transformed observations (o') t ,o' t+1 Update strategy network π θ (s) and value function network Q φ (s,a), specifically including: Update the value function network Q using Q-learning. φ (s,a): In the formula, o' t with o' t+n Let φ represent the augmented samples generated at time t and t+n, respectively, and let φ represent the value function network Q. φ The parameters, Representation of the value function network Q φ The momentum update version, γ∈[0,1] represents the discount factor, Q φ (f ψ (o' t ),a t ) indicates based on augmented sample o' t With action a t The predicted reward value, This indicates that the predicted values ​​after n steps are used for fitting. This represents the augmented sample o' after n steps of discounting. t+n With policy network π θ The target reward prediction value is used to improve training stability and reduce overestimation; the formula represents the optimizable image transformation parameters. It will optimize the Bellman error along with the value function network, thereby improving the stability of representation learning; Combining steps 5.3 and 5.4, we obtain encoder f. ψ (o) and the value function network Q φ The overall objective function of (s,a) is: In the formula, α and λ are weight parameters manually set according to the task. The larger the weight, the stronger the regularization effect of the term. Policy Network π θ The update process of (s) is the DDPG algorithm, which uses a greedy strategy to directly optimize the value function network Q. φ (s,a) Predicted policy gradient In the formula, π θ Let π, a ~ π be the policy function parameterized by parameter θ. θ (f ψ (o' t )) indicates that from the augmented sample o' t Choose one action from the obtained action distribution, Q φ (f ψ (o' t ),a) indicates based on augmented sample o' t Predict the reward based on the selected action 'a'; Repeat steps 3.2 to 5.4 until the robotic arm grasps the target object and reaches the task destination, or the robotic arm encounters an illegal area, or the set maximum time step is reached, which completes this training section. After initializing the environment, the training subsections are repeated K times until the algorithm fully converges, resulting in a transferable optimal policy network for the robotic arm to grasp target objects and reach designated positions in a dynamic environment.

5. The strategy learning method based on optimizable image transformation for robotic arm grasping according to claim 4, characterized in that, Step 5.2 also includes: parameterizing the Gaussian distribution, specifically including: When sampling the pixel offset factor, first generate a random number ∈ within (0,1), then parameterize the required offset factor as μ+∈·σ, and truncate the factor within the set range, thus completing the sampling of image transformation based on a certain Gaussian distribution.

6. A strategy learning system based on optimizable image transformation for robotic arm grasping, based on the method of any one of claims 1 to 5, characterized in that, The system includes: The first module is used to design the task environment, set the relevant parameters of the robotic arm and the target object, and set the various hyperparameters of the reinforcement learning algorithm. The second module is used to build a virtual environment that is consistent with the task environment using the Mujoco physics engine; The third module is used to enable the intelligent agent to manipulate the robotic arm to interact in the virtual environment, collect training data, calculate the task reward function based on the distance between the grasped object and the target position, and store it together with the training data in the experience replay pool. The fourth module is used to determine the computation method of the optimal invariant metric under representation learning by utilizing optimizable image transformation; The fifth module is used to collect a batch of data from the experience revisit pool and use a reinforcement learning algorithm that combines optimized image transformation to train the robotic arm to the optimal strategy for grasping objects and moving them to the target position in a dynamic environment. The sixth module is used to transfer the optimal policy trained in the virtual environment to the real environment for testing.

7. A computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 5.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 5.

Images