A multi-objective reinforcement learning method for unsupervised image editing

Through unsupervised image editing technology, the task-related and irrelevant factors in the robot operation scenario are decoupled, and the editable direction is obtained, which improves the sample efficiency and generalization performance of multi-objective reinforcement learning, and solves the training complexity and data in existing methods. The problem of low utilization allows the robot to complete multiple target tasks efficiently.

CN115070753BActive Publication Date: 2024-11-08TONGJI UNIV
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202210469373.8
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2022-04-28
Publication Date
2024-11-08
Estimated Expiration
2042-04-28

AI Technical Summary

Technical Problem

The existing multi-objective reinforcement learning method has complex training process and is difficult to converge under image input, and the data utilization rate is low, so it cannot generalize to tasks with similar environment and task structure but different target positions. The traditional method relies on manual design of reward functions. Inefficient sample.

Method used

Using a multi-objective reinforcement learning method based on unsupervised image editing, the image hidden variables are divided into task-independent factors, robots and subspaces of operating objects through adversarial generation networks and feature space encoders, and singular value decomposition is performed to obtain editable directions, and train them. Editable direction encoder is used to generate editable representation space as input and reward function calculation of controllable editing of images and task generalization.

Benefits of technology

The sample efficiency and generalization performance of multi-objective reinforcement learning algorithms are improved, allowing robots to efficiently complete multiple target tasks, reducing training time and data requirements.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN115070753B_ABST
    Figure CN115070753B_ABST
Patent Text Reader

Abstract

The present invention relates to a multi-objective reinforcement learning method based on unsupervised image editing, comprising: obtaining a multi-objective task data set about a robot control scene; training a generative adversarial network and a feature space encoder to decouple factors in the image that are highly relevant and irrelevant to the task; performing singular value decomposition on the weights of the fully connected layer corresponding to each subspace, obtaining several feature vectors with the largest contribution as editable directions with semantic information, and training the editable direction encoder to identify the category and scale of the editable direction; obtaining an editable representation space of the image based on the output of the editable direction encoder as the input of the control strategy network and the calculation of the reward function, and training the robot by controllably sampling various target tasks in the editable representation space, and finally obtaining a control strategy that can complete multiple goals. Compared with the prior art, the present invention has the advantages of being able to unsupervisedly decouple task-related factors, improve sample efficiency and generalization performance, and the like.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of intelligent agent autonomous action learning, and in particular to a multi-objective reinforcement learning method based on unsupervised image editing. Background Art

[0002] With the rise of artificial intelligence algorithms and the rapid development of various hardware devices, robotics has played an important role in many fields such as medical care, service, assembly, security, rescue, and transportation. However, customized deployment of robots for a single task is time-consuming and laborious. Therefore, how to let robots learn a control strategy that can achieve multiple goals has always been our goal.

[0003] Traditional robot control methods are extremely dependent on the professional knowledge and software programming level of technicians, and different robot trajectory distributions need to be designed for specific task objectives. When the application scenario changes, the previously deployed robot skills cannot be reused. Deep reinforcement learning methods can enable robots to autonomously learn task-related skills through exploration and utilization. However, this learning method requires manual design of corresponding reward functions for different target tasks, and the learned control strategies can only complete specific target tasks and cannot be generalized to tasks with similar environment and task structures but different target positions. Multi-objective reinforcement learning methods give robots the ability to complete various different target tasks by simultaneously inputting current observations and task objectives into the control strategy. However, this type of algorithm uses images as input, the training process is complex and difficult to converge, and the data utilization rate is low. It may take hundreds or thousands of hours to collect data and train strategies in a simulation environment, which is impractical in real scenarios.

[0004] In daily life and work, humans are often able to decouple the various components of the environment and complete various corresponding target actions based on the state changes of key parts. How to decouple and represent the key objects in the environment image in an unsupervised way remains a challenge. Summary of the invention

[0005] The purpose of the present invention is to overcome the defects of the above-mentioned prior art and to provide a multi-objective reinforcement learning method based on unsupervised image editing that can improve sample efficiency and generalization performance.

[0006] The purpose of the present invention can be achieved by the following technical solutions:

[0007] A multi-objective reinforcement learning method based on unsupervised image editing, the multi-objective reinforcement learning method comprising:

[0008] Step 1: Obtain a multi-objective task dataset for robot control scenarios;

[0009] Step 2: Train the generative adversarial network and feature space encoder to divide the latent variable space into three subspaces, corresponding to task-irrelevant factors, the robot, and the manipulated object, to decouple the various parts of the image;

[0010] Step 3: Perform singular value decomposition on the weights of the fully connected layer corresponding to each subspace to obtain several eigenvectors with the largest contribution as editable directions with semantic information. The latent variables can be increased or decreased along the editable directions to perform image editing, and an editable direction encoder is trained to identify the category and scale of the editable directions.

[0011] Step 4: Based on the output of the editable direction encoder, an editable representation space of the image is obtained as the input of the control strategy network and the calculation of the reward function. At the same time, the robot is trained by controllably sampling various target tasks in the editable representation space, and finally a control strategy that can achieve multiple goals is obtained.

[0012] Preferably, the step 1 is specifically:

[0013] Build a robot virtual simulation environment, and control the robot to complete multiple target tasks through sampling control strategies, and create a multi-target task dataset for robot control scenarios.

[0014] More preferably, the multi-objective task includes two tasks: grabbing wooden blocks and assembling. The robot virtual simulation environment includes a UR5 robotic arm, an operating table and a number of operating objects. The robot is controlled to interact with the environment by sampling action instructions from a random exploration strategy to obtain various image sequence data for each task, and the target position of each sequence is obtained by sampling from a uniform distribution.

[0015] Preferably, the adversarial generative network comprises:

[0016] The generative network is expected to generate sufficiently realistic images that the discriminative network cannot distinguish between the real and the fake;

[0017] The discriminative network is expected to distinguish whether the image is real or generated by the generative network;

[0018] The latent variables randomly sampled from the Gaussian distribution are mapped to 3 W feature spaces through 8 fully connected layers, where W 1 corresponds to task-independent factors, and the other two subspaces correspond to task-related factors, where W 2 Corresponding to the robot, W 3 Corresponding to the object that needs to be operated;

[0019] The feature space encoder is used to encode the image generated by the generating network back to the features in the W space;

[0020] In step 2, the image generated by the generative network is used as the input of the feature space encoder, and the generative network input corresponding to each image is used as the output label of the feature space encoder, so as to supervise the training of the feature space encoder.

[0021] Preferably, the method for decoupling the various parts of the image in step 2 is:

[0022] The various parts of the image are decoupled by exchanging the latent variables of one subspace and keeping the latent variables of other subspaces unchanged.

[0023] More preferably, the method of decoupling the various parts of the image in step 2 is specifically as follows:

[0024] The two images in the multi-objective task dataset are taken as a set of data, and the corresponding W spatial features are recorded as W 1,2,3 and W a,b,c ;

[0025] The features of a certain W subspace are exchanged with each other, and the other features remain unchanged, and two new features W are obtained. 1,b,3 and W a,2,c ;

[0026] Input the new features into the generative network to obtain two new images;

[0027] The two generated images are encoded through the feature space encoder, and the feature exchange operation is performed again to obtain the new feature W′ 1,2,3 and W′ a,b,c , and supervise the feature through the root mean square error, specifically:

[0028]

[0029] Preferably, the fully connected layer parameters corresponding to each subspace in step 3 are set to is a non-square matrix;

[0030] Then the affine transformation F of the fully connected layer for the hidden variable z obtained by random sampling is:

[0031] W=F(z)=Az+b

[0032] By adding a direction N with semantic information to the corresponding W spatial feature, the generated image can be controllably edited so that the generated image changes in a certain content factor.

[0033] Preferably, the singular value decomposition in step 3 is specifically:

[0034] Singular value decomposition is used to decompose the fully connected layer parameters A i Perform matrix decomposition:

[0035] A i =U∑V T

[0036] Among them, U is an m*m square matrix; ∑ is an m*n matrix, all elements except the elements on the main diagonal are 0, and each element on the main diagonal is called a singular value; V is an n*n square matrix;

[0037] Then, the singular values ​​in ∑ are sorted according to their sizes to obtain the first several singular values ​​with the largest contribution values. The corresponding feature vector is the direction N with semantic information, that is, the editable direction N of the image is learned in an unsupervised way.

[0038] Preferably, the editable direction encoder is specifically:

[0039] The editable direction encoder is used to encode the changes of the two images in the W space of the adversarial generation network, including the category N of the editable direction i and scale α, step 3 performs controllable editing on the generated image based on the editable direction N to obtain image pairs and corresponding editing categories and scales, thereby performing supervised training on the editable direction encoder.

[0040] Preferably, the step 4 is specifically:

[0041] The control policy network takes the output of the editable direction encoder as the embedding space input of the observation image and task goal, and outputs the joint angles of the robot to control the robot to complete the task.

[0042] Compared with the prior art, the present invention has the following beneficial effects:

[0043] Improved sample efficiency and generalization performance of multi-objective reinforcement learning algorithm: The multi-objective reinforcement learning method based on unsupervised image editing in the present invention uses the editable direction category and scale of the image as the encoding input control strategy of the image, which more clearly and efficiently reflects the state changes of each element in the image. The control strategy network is trained through the reinforcement learning algorithm PPO to output appropriate robot actions, so that the robot can complete tasks with various different goals, and greatly improves the sample efficiency and generalization performance of the multi-objective reinforcement learning algorithm. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Figure 1 Schematic diagram of the process of the multi-objective reinforcement learning method for unsupervised image editing in an embodiment of the present invention;

[0045] Figure 2 A schematic diagram of a robot virtual simulation environment in an embodiment of the present invention;

[0046] Figure 3A schematic diagram of the structure of a generative adversarial network in an embodiment of the present invention;

[0047] Figure 4 Schematic diagram of the network structure of the editable direction encoder in an embodiment of the present invention. DETAILED DESCRIPTION

[0048] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0049] This embodiment takes the Universal Robots UR5 robotic arm as an example. Its control workstation is installed with Ubuntu 16.04 system, equipped with Intel Core i7-10700K, 8 cores and 16 threads, turbo frequency 5.1GHz, GPU is NVIDIA GTX1080*2, memory is 32GDDR4 memory, and there is also a general RGB camera to observe the overall scene of the operating console.

[0050] A multi-objective reinforcement learning method based on unsupervised image editing, the process is as follows Figure 1 As shown, including:

[0051] Step 1: Build a robot simulation environment based on OpenAI Gym and Mujoco simulation platform. The simulation environment diagram is as follows: Figure 2 As shown in the figure. In the constructed virtual environment, a multi-objective task dataset for robot control scenarios is collected, including two tasks: grabbing wooden blocks and assembling. The environment includes a UR5 robot arm, an operating table, and several operating objects. By sampling action instructions from the random exploration strategy to control the robot to interact with the environment, various image sequence data for each task are obtained, and the target position of each sequence is sampled from a uniform distribution.

[0052] Step 2: Based on the existing dataset, train a generative adversarial network and a feature space encoder. The generative adversarial network consists of a generative network and a discriminative network. The discriminative network is expected to be able to distinguish whether the image is real or generated by the generative network. The generative network is expected to generate sufficiently real images that the discriminative network cannot distinguish between true and false. The feature space encoder aims to encode the images generated by the generative network back to features in the W space, and can be supervised by randomly generated samples from the generative network.

[0053] The structure of the adversarial generation network is as follows Figure 3As shown in the figure, the generator network consists of 3 convolutional layers with a convolution size of 5, a stride of 0.5, and an activation function of ReLU. The discriminator network consists of 3 convolutional layers with a convolution size of 5, a stride of 0.5, and an activation function of ReLU, 2 fully connected layers, and an activation function layer. The latent variables randomly sampled from the Gaussian distribution are mapped to 3 W feature spaces through 8 fully connected layers, where W 1 Corresponds to task-independent factors, including the lighting of the simulation environment, desktop color, and wall color. The other two subspaces correspond to task-related factors, where W 2 Corresponding to the robot, W 3 Corresponding to the object that needs to be operated.

[0054] The two images in the multi-objective task dataset are taken as a set of data, and the corresponding W spatial features are recorded as W 1,2,3 and W a,b,c ;

[0055] The features of a certain W subspace are exchanged with each other, and the other features remain unchanged, and two new features W are obtained. 1,b,3 and W a,2,c ;

[0056] Input the new features into the generative network to obtain two new images;

[0057] The two generated images are encoded through the feature space encoder, and the feature exchange operation is performed again to obtain the new feature W′ 1,2,3 and W′ a,b,c , and supervise the feature through the root mean square error, specifically:

[0058]

[0059] This ensures that the three W subspaces can be decoupled, and changes in the characteristics of one subspace will not affect the image content corresponding to other subspaces.

[0060] Step 3: Weights of the fully connected layer corresponding to each subspace Perform singular value decomposition and obtain the eigenvectors with the largest contribution as the editable directions N with semantic information. Set the fully connected layer parameters corresponding to each subspace to is a non-square matrix. Then the affine transformation F of the fully connected layer for the hidden variable z obtained by random sampling can be expressed as follows:

[0061] W=F(z)=Az+b

[0062] By adding the direction N with semantic information to the corresponding W spatial feature, the generated image can be edited in a controllable manner so that the generated image changes in a certain content factor, such as changing the position, shape, ambient lighting and color of the operating object.

[0063] Non-square The singular value decomposition can be expressed as:

[0064] A i =U∑V T

[0065] Among them, U is an m*m square matrix; ∑ is an m*n matrix, all elements except the elements on the main diagonal are 0, and each element on the main diagonal is called a singular value; V is an n*n square matrix;

[0066] Then, the singular values ​​in ∑ are sorted according to their sizes to obtain the top 10 singular values ​​with the largest contribution values. The corresponding eigenvector is the direction N with semantic information, that is, the editable direction N of the image is learned in an unsupervised way.

[0067] In addition, according to the learned editable direction N, an editable direction encoder Enc is learned, such as Figure 4 As shown. First, randomly sample the latent variable z i , after two layers of fully connected layers FC, the feature W in the corresponding W space is obtained, and the corresponding image I is generated through the generative network Gen i Then uniformly sample an editable direction N from the K editable directions i , and randomly sample the editing scale α~U{-ε,ε}. Multiply the two and add them to w, and generate the image I through the generative network j . Image I i and I j Jointly input the editable direction encoder Enc and predict the corresponding editing category N i And scale α. A large number of samples and true labels are generated by the generator Gen, so as to perform supervised training on Enc.

[0068] S4: Get the robot's initial observation image obs through the editable direction encoder t=0 With the current frame image obs t Edit category N t and scale α t , and obs t=0 With the task target image obs goal Edit category N goal and scale α goal , will (N t , α t , N goal, α goal ) is used as the input of the control strategy network, and finally outputs the joint angles of the robot to control the robot to complete the task. The specific expression of the control strategy network is as follows:

[0069] π=(A|·)=π(A|N t ,α t ,N goal ,α goal )

[0070] Based on the output of the editable direction encoder, the editable representation space of the image is obtained as the input of the control strategy and the calculation of the reward function. At the same time, the robot is trained by controllably sampling various target tasks in the editable representation space, and finally a control strategy that can complete multiple goals is obtained. The present invention uses the editable direction category and scale of the image as the encoding input control strategy of the image, which more clearly and efficiently reflects the state changes of each element in the image. The control strategy network is trained to output appropriate robot actions through the reinforcement learning algorithm PPO, so that the robot can complete tasks with various different goals, and greatly improves the sample efficiency and generalization performance of the multi-objective reinforcement learning algorithm.

[0071] The innovation of the present invention is:

[0072] (I) A cyclic exchange operation method based on hidden variables in a generative adversarial network is proposed to decouple task-related factors from task-irrelevant factors in the robot operation scenario, so that the control strategy can focus on the state changes of task-related factors and reduce the redundancy of the robot's observation representation.

[0073] (II) An unsupervised image editing algorithm is proposed to perform singular value decomposition on the parameters of different factors in the control scene in latent space, and obtain a series of semantically editable directions, thereby achieving controllable generation of each object in the image. Based on the learned series of editable directions, an editable direction encoder is trained to extract the structured representation of each object.

[0074] (III) A multi-objective control strategy based on image editing is proposed. The output of the encoder is used to represent the robot observation and target task images, which can greatly shorten the training time of the algorithm. The structured representation with semantic information is used to improve the sample efficiency and generalization performance of the multi-objective reinforcement learning algorithm.

[0075] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed by the present invention, and these modifications or replacements should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention shall be based on the protection scope of the claims.

Claims

1. A multi-objective reinforcement learning method based on unsupervised image editing, characterized in that: The multi-objective reinforcement learning method comprises: Step 1: Obtain a multi-objective task dataset for robot control scenarios; Step 2: Train the generative adversarial network and feature space encoder to divide the latent variable space into three subspaces, corresponding to task-irrelevant factors, the robot, and the manipulated object, to decouple the various parts of the image; Step 3: Perform singular value decomposition on the weights of the fully connected layer corresponding to each subspace to obtain several feature vectors with the largest contribution as editable directions with semantic information, and train an editable direction encoder to identify the category and scale of the editable direction; Step 4: Based on the output of the editable direction encoder, an editable representation space of the image is obtained as the input of the control strategy network and the calculation of the reward function. At the same time, the robot is trained by controllably sampling various target tasks in the editable representation space, and finally a control strategy that can achieve multiple goals is obtained; The method for decoupling the various parts of the image in step 2 is: Decouple the various parts of the image by exchanging one subspace latent variable and keeping the other subspace latent variables unchanged; The method for decoupling the various parts of the image in step 2 is specifically as follows: The two images in the multi-objective task dataset are taken as a set of data, and the corresponding W spatial features are recorded as W 1,2,3 and W a,b,c ; The features of a certain W subspace are exchanged with each other, and the other features remain unchanged, and two new features W are obtained. 1,b,3 and W a,2,c ; Input the new features into the generative network to obtain two new images; The two generated images are encoded by the feature space encoder, and the feature exchange operation is performed again to obtain the new feature W1' ,2,3 and W a ' ,b,c , and supervise the feature through the root mean square error, specifically: The singular value decomposition in step 3 is specifically as follows: Singular value decomposition is used to decompose the fully connected layer parameters A i Perform matrix decomposition: A i =UΣV T Among them, U is an m*m square matrix; Σ is an m*n matrix, all elements except the elements on the main diagonal are 0, and each element on the main diagonal is called a singular value; V is an n*n square matrix; Then, the singular values ​​in Σ are sorted according to their sizes to obtain the first several singular values ​​with the largest contribution values. The corresponding feature vectors are the directions N with semantic information, that is, the editable directions N of the image are learned in an unsupervised way. The editable direction encoder is specifically: The editable direction encoder is used to encode the changes of the two images in the W space of the adversarial generation network, including the category N of the editable direction i And scale α, based on the linear weighted operation of the editable direction in the W space, the generated image is controllably edited to obtain image pairs and the corresponding editing categories and scales, thereby supervising the training of the editable direction encoder.

2. A multi-objective reinforcement learning method based on unsupervised image editing according to claim 1, characterized in that: The step 1 is specifically as follows: Build a robot virtual simulation environment, and control the robot to complete multiple target tasks through sampling control strategies, and create a multi-target task dataset for robot control scenarios.

3. A multi-objective reinforcement learning method based on unsupervised image editing according to claim 2, characterized in that: The multi-objective tasks include two tasks: grasping wooden blocks and assembling. The robot virtual simulation environment includes a UR5 robotic arm, an operating table and several operating objects. The robot is controlled to interact with the environment by sampling action instructions from a random exploration strategy to obtain various image sequence data for each task, and the target position of each sequence is obtained by sampling from a uniform distribution.

4. The multi-objective reinforcement learning method based on unsupervised image editing according to claim 1, characterized in that: The adversarial generation network includes: The generative network is expected to generate sufficiently realistic images that the discriminative network cannot distinguish between the real and the fake; The discriminative network is expected to distinguish whether the image is real or generated by the generative network; The latent variables randomly sampled from the Gaussian distribution are mapped to three W feature spaces through 8 fully connected layers, where W1 corresponds to task-independent factors and the other two subspaces correspond to task-related factors, where W2 corresponds to the robot and W3 corresponds to the object to be operated. The feature space encoder is used to encode the image generated by the generating network back to the features in the W space; The image generated by the generative network is used as the input of the feature space encoder, and the generative network input corresponding to each image is used as the output label of the feature space encoder, so as to supervise the training of the feature space encoder; 5. The multi-objective reinforcement learning method based on unsupervised image editing according to claim 1, characterized in that: In step 3, the parameters of the fully connected layer corresponding to each subspace are set to is a non-square matrix; Then the affine transformation F of the fully connected layer for the hidden variable z obtained by random sampling is: W=F(z)=Az+b By adding a direction N with semantic information to the corresponding W spatial feature, the generated image can be controllably edited so that the generated image changes in a certain content factor.

6. The multi-objective reinforcement learning method based on unsupervised image editing according to claim 1, characterized in that: The step 4 is specifically as follows: The control policy network takes the output of the editable direction encoder as the embedding space input of the observation image and task goal, and outputs the joint angles of the robot to control the robot to complete the task.