Robot skill virtual-real migration learning method and system based on semantic domain adaptation

Abstract

The application provides a robot skill virtual-real migration learning method and system based on semantic domain self-adaption, which comprises the following steps: S1. Constructing a simulation environment, generating a perception semantic label corresponding to the simulation environment, and constituting a source domain simulation image and label pair sample set; S2. Inputting the simulation image and the real image, and training a semantic abstraction network model using a semantic domain self-adaption method based on adversarial learning; S3. Training a robot skill policy neural network using a reinforcement learning method based on the simulation semantic, and obtaining a robot system skill policy model; and S4. Converting a real environment image into semantic information using the semantic abstraction neural network model, inputting the real semantic information into the robot skill policy neural network model, and obtaining a real-time robot control instruction. The application can realize efficient and safe virtual-real migration of the robot skill policy, and has the advantages of simple implementation, strong migration performance, safety and reliability, high efficiency and the like.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of robots, in particular to a robot skill virtual-real migration learning method and system based on semantic domain adaptation. BACKGROUND

[0002] With the development of machine learning technology, especially reinforcement learning technology, its ability to adapt to learning and perform complex tasks is significantly enhanced. Based on reinforcement learning, robots can acquire adaptive dexterous skill operation ability through autonomous learning, and then assist or replace humans in performing tasks in complex scenarios to improve task execution efficiency. When implementing robot operation skill training based on reinforcement learning technology, the skill strategy needs to be optimized through interaction with the environment. However, due to the high cost of sampling and the difficulty in ensuring training safety, directly using deep reinforcement learning to interact in a real system is not only complex to implement but also very costly.

[0003] To solve the above problems, the existing technology usually trains an agent in a similar robot skill operation simulation environment through reinforcement learning, and then deploys the agent to a real robot system. However, this type of solution still has the following defects:

[0004] 1. The simulation graphics engine is difficult to generate a consistent visual environment with the real system, which will cause semantic deviation between the simulation environment and the real physical system, making it difficult to effectively extract consistent semantic information from the simulation environment and the real physical system, resulting in the trained strategy in the simulation not being ideal when used in the actual physical world.

[0005] 2. The simulation environment and the real physical system not only have semantic deviation but also have dynamic differences, which can cause safety problems when the trained strategy in the simulation is used in reality. SUMMARY

[0006] The technical problem to be solved by the present application is to provide a robot skill virtual-real migration learning method and system based on semantic domain adaptation, which is simple to implement, has strong migration performance, is safe and reliable, and is efficient, can effectively align semantic information between virtual and real environments, and ensure efficient and safe virtual-real migration of robot skill strategies.

[0007] To solve the above technical problems, the technical solution adopted by the present application is as follows:

[0008] A robot skill virtual-real migration learning method based on semantic domain adaptation, comprising the following steps:

[0009] Step S1: Simulation environment construction: A simulation environment for robot skill learning is constructed based on the real environment in which the robot skill is operated, and semantic labels corresponding to the simulation environment are generated. The source domain simulation image and label pair sample set is composed of the corresponding simulation images in the simulation environment and the semantic labels.

[0010] Step S2: semantic abstraction neural network training: real-object images are collected in the target domain physical system, and the source domain simulation images and label pair sample sets and the target domain physical images are used as input samples. The semantic abstraction neural network model is trained based on the semantic domain adaptation method of adversarial learning. The semantic abstraction neural network model converts the input source domain simulation images into the semantic space to obtain simulation semantics and converts the target domain physical images into the semantic space to obtain physical semantics, so as to achieve alignment of semantic features between the simulation domain and the physical domain.

[0011] During the training process of the semantic domain adaptation method based on adversarial learning, a semantic abstraction neural network model is first trained using a sample set of source domain simulated images and label pairs, and then a discriminator model is constructed and the discriminator model is jointly trained using two sets of images, the source domain simulated images and the target domain real-object images. When training the discriminator model, an adversarial loss function is used to narrow the distance between the target domain and the source domain to achieve adaptive semantic abstraction learning. The adversarial loss function is used to calculate the adversarial loss by using the target domain real-object images input into the semantic abstraction neural network model to obtain target scene features;

[0012] Step S3: Reinforcement learning training: The robot skill strategy neural network is trained based on the reinforcement learning method using the simulation semantics converted from the semantic abstract neural network model to obtain a trained robot system skill strategy model. The input of the robot skill strategy neural network is the semantic information output by the semantic abstract neural network model, and the output is the control instructions for each joint of the robot.

[0013] Step S4: Strategy deployment migration: When a real-time real environment image is collected, the semantic abstract neural network model is used to convert the real environment image into a semantic space to obtain physical semantic information, and the physical semantic information obtained by the real-time conversion is input into the robot skill strategy neural network model to obtain real-time robot control instructions, thereby realizing the migration and deployment of the skill strategy trained in the source domain virtual environment to the target domain physical robot.

[0014] Furthermore, in step S2, during the training of the semantic domain adaptation method based on adversarial learning, using the source domain simulation image and the label pair sample set to train the semantic abstract neural network model includes: Input to the semantic abstraction neural network model In the source domain, we get the abstract result , by minimizing the abstraction result of the source domain and source domain semantic labels Distance training abstract model .

[0015] Furthermore, in step S2, during the training of the semantic domain adaptation method based on adversarial learning, training the discriminator model includes:

[0016] The source domain simulated image and the target domain object image Input to the semantic abstraction neural network model In the example, we get the source scene features output by the middle layer of the model and target scene features ;

[0017] Source scene features and target scene features Input to the discriminator model In the example, the discriminator model is trained by minimizing the cross entropy loss function. :

[0018]

[0019] Freeze the discriminator model , the target domain physical image Input to the semantic abstraction neural network model The target scene features output by the intermediate layer are obtained , and in the discriminator model Use target scene features in Calculate adversarial loss to bring the target scene features closer Source scene characteristics distance.

[0020] Furthermore, in step S2, the loss function during the training of the semantic domain adaptation method based on adversarial learning is:

[0021]

[0022]

[0023] in, is the balance coefficient used to balance the semantic abstraction loss and the adversarial loss, The number of categories representing semantics, Indicates the size of the image, , The loss function representing the semantic abstraction process, Denotes the adversarial loss.

[0024] Furthermore, in step S3, a reinforcement learning method based on a safe Bellman update operator is used to train the robot skill strategy neural network. In the reinforcement learning method based on a safe Bellman update operator, a safe action value function is learned using a safe Bellman update operator:

[0025]

[0026] in For robot skill strategy, is the reward function, Status and action of Step potential penalty cost function, For the secure Bellman update operator, is the safety action value function, represents the expectation under the strategy π and state transition distribution P, Represents the discount factor.

[0027] Furthermore, in step S3, the training steps of the safety action value function and the strategy function in the process of training the robot skill strategy neural network are as follows:

[0028] Update the safe action value function by minimizing the safe soft Bellman residual :

[0029]

[0030] in,

[0031]

[0032] In the above formula, represents the expectation calculated by sampling data from the dataset D, represents the expectation under the action of strategy π, represents the learning rate, is the network parameter of the security action value function, are the parameters of the strategy;

[0033] The policy function minimizes the following objective function when updating :

[0034]

[0035] Furthermore, during the training of the robot skill policy neural network, the safe action value function accepts both state and action inputs, where the action input is sampled from the Gaussian distribution of the policy function output. When sampling the action, the reparameterized network shown below is used for resampling:

[0036]

[0037] in, is the noise sampled from a specified fixed distribution to simulate the sampling process, represents a reparameterized network;

[0038] Actions after resampling using a reparameterized network for:

[0039] .

[0040] in, and are the mean and variance distribution of the Gaussian distribution output by the strategy respectively.

[0041] Furthermore, it also includes the use of the reversible hyperbolic tangent function Transform the Gaussian distribution resampling action and constrain it to the normalized interval inside, among them The function is calculated as follows:

[0042]

[0043] The action entropy term after interval transformation is calculated by the following formula:

[0044]

[0045]

[0046] in is the dimension of the action space, for The elements, the Gaussian distribution resampling action is ,pass The action after function transformation is , Right now , x Indicates the input signal.

[0047] Furthermore, step S3 also includes using semantic information to model the risk penalty function , among which Indicates status In a safe range, when Indicates status Violation of safety constraints is subject to penalty costs , state transfer function Guide the update of strategy, for strategy ,state and action of The potential penalty cost function of the step is calculated as follows:

[0048]

[0049] in, for Perturbation function under standard Gaussian noise.

[0050] A robot skill virtual-reality transfer learning system based on semantic domain adaptation includes a microprocessor and a memory connected to each other, wherein the microprocessor is programmed or configured to execute a robot skill virtual-reality transfer learning method based on semantic domain adaptation.

[0051] Compared with the prior art, the advantages of the present invention are:

[0052] 1. The present invention converts the input source domain simulation image into the semantic space to obtain simulation semantics and converts the target domain physical image into the semantic space to obtain physical semantics through the semantic abstract neural network model. By simultaneously learning the semantic abstract representations of the simulation visual domain and the actual visual domain, it can effectively extract the semantic information consistently contained in the simulation environment and the physical system, so that the strategies trained in the simulation environment can be effectively transferred to the actual physical world, realizing fast and accurate migration from the virtual environment to the actual physical world, while also ensuring the safety and reliability of the migration.

[0053] 2. By converting the original image space into semantic space, the present invention can significantly reduce the complexity of the original state space, thereby improving the efficiency of solving Markov process problems in large-scale continuous state-action space.

[0054] 3. The present invention further ensures that the robot explores and takes actions within a preset safety range by adopting a soft policy learning method under the constraints of a safe action neighborhood, significantly improving the migration security of the learned policy when the actual system is noisy compared to the simulation system. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] Figure 1 This is a flow chart of a method for learning the virtual-real transfer of robot skills based on semantic domain adaptation according to an embodiment of the present invention.

[0056] Figure 2 This is a structural diagram of the robot skill virtual-to-real transfer learning method based on semantic domain adaptation implemented in this embodiment.

[0057] Figure 3 It is a schematic diagram of the physical demonstration effect of the present invention in a specific application embodiment using a robot skill strategy virtual-real transfer learning method based on semantic domain adaptation to achieve robotic arm grasping control. DETAILED DESCRIPTION

[0058] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts shall fall within the scope of protection of the present invention.

[0059] like Figure 1 As shown, the robot skill virtual-real transfer learning method based on semantic domain adaptation in this embodiment includes the following steps:

[0060] Step S1: Simulation environment construction: A simulation environment for robot skill learning is constructed based on the real environment in which the robot skill is operated, and semantic labels corresponding to the simulation environment are generated. The source domain simulation image and label pair sample set is composed of the corresponding simulation images in the simulation environment and the semantic labels.

[0061] Step S2: Semantic abstraction neural network training: Real-object images are collected in the target domain physical system, and the source domain simulation images and label pair sample sets and the target domain physical images are used as input samples. The semantic abstraction neural network model is trained based on the semantic domain adaptation method of adversarial learning. The semantic abstraction neural network model converts the input source domain simulation images into the semantic space to obtain simulation semantics, and converts the target domain physical images into the semantic space to obtain physical semantics, so as to achieve alignment of semantic features between the simulation domain and the physical domain.

[0062] In the training process of the semantic domain adaptation method based on adversarial learning, the semantic abstraction neural network model is first trained using the sample set of source domain simulated images and labeled pairs. Then, a discriminator model is constructed and trained using two sets of images: the source domain simulated images and the target domain real-life images. When training the discriminator model, the distance between the target domain and the source domain is shortened by using an adversarial loss function to achieve adaptive semantic abstraction learning. The adversarial loss function is used to calculate the adversarial loss by using the target domain real-life images input into the semantic abstraction neural network model to obtain the target scene features.

[0063] Step S3: Reinforcement learning training: The robot skill strategy neural network is trained based on the reinforcement learning method using the simulation semantics converted from the semantic abstract neural network model to obtain a trained robot system skill strategy model. The input of the robot skill strategy neural network is the semantic information output by the semantic abstract neural network model, and the output is the control instructions for each joint of the robot.

[0064] Step S4: Strategy deployment migration: When real-time real-world environment images are collected, the semantic abstract neural network model is used to convert the real-world environment images into semantic space to obtain physical semantic information. The physical semantic information obtained by real-time conversion is input into the robot skill strategy neural network model to obtain real-time robot control instructions, thereby realizing the migration and deployment of the skill strategy trained in the source domain virtual environment to the target domain physical robot.

[0065] This embodiment uses two types of images, source domain simulation images and target domain real-object images, to train a semantic abstraction neural network model, converts the source domain simulation images and the target domain real-object images into semantic spaces respectively, and trains them based on a semantic domain adaptation method of adversarial learning to align the simulation domain and the real-object domain in terms of semantic features, thereby achieving adaptive semantic abstraction learning. This allows the semantic abstraction representations of the simulation visual domain and the actual visual domain to be learned simultaneously, ensuring that the semantic information contained in the simulation environment and the real-object system is consistent. The simulation semantics obtained by the semantic abstraction neural network model is then used to train the robot skill policy neural network based on reinforcement learning. The trained robot skill policy neural network can quickly and accurately achieve migration from the virtual environment to the actual physical world, while ensuring the safety and reliability of the migration. Moreover, since the original image space is converted into the semantic space, the complexity of the original state space can be significantly reduced, and the efficiency of solving the Markov process problem in the large-scale continuous state-action space can be effectively improved, thereby further improving the efficiency of migration.

[0066] Specifically, when constructing the simulation environment in step S01 of this embodiment, the size and shape of each object in the simulation environment are the same as those in the actual environment, the dynamic properties of the robot in the simulation are consistent with the physical system, the simulation environment can generate different visual backgrounds, and the perceptual semantic labels corresponding to the simulation environment can be directly generated by the simulation environment or obtained through manual annotation.

[0067] In step S02 of this embodiment, the constructed semantic abstract neural network specifically includes a convolutional neural network and a fully connected network module. In the process of training the semantic domain adaptation method based on adversarial learning, the semantic abstract neural network model is trained using the source domain simulation image and the label pair sample set, including: Input to the semantic abstraction neural network model In the source domain, we get the abstract result , by minimizing the abstraction result of the source domain and source domain semantic labels Distance training abstract model .

[0068] The two sets of images from the source scene and the target scene are defined as , training semantic abstraction neural network model When Input to the abstract model In the example above, we get the abstract result of the source scene. , by minimizing Source scene tags distance to train the abstract model .

[0069] In this embodiment, the loss function of the semantic abstraction process can be specifically expressed as:

[0070] (1)

[0071] in The number of categories representing semantics.

[0072] Further discriminator model , train the discriminator model The steps include:

[0073] The source domain simulated image and the target domain object image Input to the semantic abstraction neural network model In the example, we get the source scene features output by the middle layer of the model and target scene features ;

[0074] Source scene features and target scene features Input to the discriminator model In the example, the discriminator model is trained by minimizing the cross entropy loss function. :

[0075] (2)

[0076] Freeze the discriminator model , the target domain physical image Input to the semantic abstraction neural network model The target scene features output by the intermediate layer are obtained , and in the discriminator model Use target scene features in Calculate adversarial loss to bring the target scene features closer Source scene characteristics distance, and realize adaptive semantic abstract learning.

[0077] Specifically, the discriminator model Utilize target scene features The adversarial loss is calculated as follows:

[0078] (3)

[0079] By using the above adversarial loss function to train the discriminator model , which can shorten the distance between the target scene and the source scene and realize adaptive semantic abstract learning.

[0080] Then, the final loss function can be obtained from the semantic abstraction loss function and the adversarial loss function. That is, the final loss function in the training process of the semantic domain adaptation method based on adversarial learning is:

[0081] (4)

[0082] in, is the balance coefficient used to balance the semantic abstraction loss and the adversarial loss, The number of categories representing semantics, Indicates the size of the image, , The loss function representing the semantic abstraction process, Denotes the adversarial loss.

[0083] The ultimate goal of further model optimization can be expressed as:

[0084] (5)

[0085] The above optimal goal is to minimize the source scene input in the semantic abstract model While maximizing the target scene in the discriminator model The probability that the scene is considered as the source scene.

[0086] In this embodiment, when performing reinforcement learning training in step S03, the input state The semantic information obtained by the original observation image through the semantic abstract neural network model outputs the action These are the control instructions for each joint of the robot.

[0087] Taking the specific robotic arm grasping task as an example, the state space, action space, and reward function are obtained as follows:

[0088] (1) State space. The strategy is trained on the semantic abstract state corresponding to the RGB image. Assume that the target objects in the scene have Considering the background as a separate semantic category, the semantic information corresponding to the task is assumed to have 11 categories. The state space after semantic abstraction using the policy learning module of the SASRL algorithm is dimensional discrete space.

[0089] (2) Action space. In the corresponding task, the action Used to describe the rotation angle (Euler angle) and translation of the end-arm, thereby describing the relative pose transformation of the end-arm. The low-level control of the manipulator is performed using the UR5's built-in inverse kinematics solver. The closure of the manipulator's gripper is determined by the depth map provided by the RGB-D camera. When the average distance between the 70 nearest pixels in the depth map and the gripper's closure range is less than 18 cm, the gripper is considered to contain an object and the strategy has completed end-arm planning, at which point the gripper is closed.

[0090] This embodiment specifically adopts a reinforcement learning method based on a safe Bellman update operator to train the robot skill strategy neural network. In the reinforcement learning method based on a safe Bellman update operator, a safe action value function is learned using a safe Bellman update operator. , so that the potential penalty cost is minimized while maximizing the cumulative reward. Use the secure Bellman update operator to learn the secure action value function Specifically, it can be expressed as:

[0091] (6)

[0092] in, For robot skill strategy, is the reward function, Status and action of Step potential penalty cost function, is the secure Bellman update operator, is the safety action value function, represents the expectation under the strategy π and state transition distribution P, Represents the discount factor.

[0093] In this embodiment, the training steps of the safe action value function and the strategy function in the process of training the robot skill strategy neural network are as follows:

[0094] Update the safe action value function by minimizing the safe soft Bellman residual :

[0095] (7)

[0096] (8)

[0097] In the above formula, represents the expectation calculated by sampling data from the dataset D, represents the expectation under the action of strategy π, Represents the learning rate, safe action value function network parameters Can be achieved through Perform exponential sliding average to obtain, Parameters of the strategy.

[0098] The policy function minimizes the following objective function during update :

[0099] (9)

[0100] Considering that the safe action value function needs to accept both state and action inputs, in this embodiment, the action input is sampled from the Gaussian distribution of the policy function's output. When using gradient descent to derive formula (9), the safe action value function needs to be derived with respect to itself first, and then with respect to the policy function. Furthermore, to avoid gradient interruption caused by the action sampling process, this embodiment uses a reparameterized sampling method when sampling actions. The reparameterized network used can be expressed as:

[0101] (10)

[0102] in, is noise sampled from some fixed distribution, used to simulate the sampling process, represents a reparameterized network.

[0103] Let the Gaussian distribution mean and variance distribution of the strategy output be and Then, the action after resampling It can be expressed as:

[0104] (11)

[0105] Furthermore, considering that the actions generated by directly sampling the Gaussian distribution have no numerical constraints, in practical applications, the action space is usually defined in a certain interval (such as ), so it is necessary to transform the action after sampling to meet the requirements. In this embodiment, the reversible hyperbolic tangent function is also used Transform the Gaussian distribution resampling action and constrain it to the normalized interval within, among them The function is calculated as follows:

[0106] (12)

[0107] in, x Indicates the input signal.

[0108] Let the Gaussian distribution resampling action be ,pass The action after function transformation is ,Right now Therefore, the following relationship holds:

[0109] (13)

[0110] And because the Jacobian matrix is a diagonal matrix, then the action entropy term after interval transformation is calculated by the following formula:

[0111] (14)

[0112] in is the dimension of the action space, for The elements, the Gaussian distribution resampling action is ,pass The action after function transformation is ,Right now .

[0113] Furthermore, in this embodiment, the unsafe action is defined as follows: If in a safe state Take action , , like If the state is unsafe, that is, if the safety limit is violated or the state is inevitably entered into unsafe state in the future, then it is called action. Is in state Based on the above definition, this embodiment also includes using semantic information to model the risk penalty function , among which Indicates status In a safe range, when Indicates status Violation of safety constraints is subject to penalty costs , state transfer function Guide the update of the strategy. Specifically, for the strategy ,state and action of The potential penalty cost function of the step is calculated as follows:

[0114] (13)

[0115] in, is the neighborhood, for The perturbation function under standard Gaussian noise, the output of the non-compliant En is the original action of Actions within the neighborhood .

[0116] This embodiment calculates the penalty cost function according to the above formula, and uses the perturbation of the action to explore the unsafe state, so that the action output by the strategy is and its neighborhood can remain safe. The term simulates the output action deviation caused by the noise in the actual system. For dangerous actions, In addition, the state transfer function can be constructed using an autoregressive attention network (Transformer). During model training, the state transfer function is optimized by minimizing the distance between the predicted abstract semantic state and the true abstract semantic state.

[0117] In order to enable the strategy to better consider potential dangerous situations in the action neighborhood during training, this embodiment further constructs a danger discrimination model to accurately estimate whether the current state violates the safety constraint. The danger discrimination model constructs a positive sample training set by collecting semantic abstract information of the state that violates the safety constraint in the task. , and randomly sample a considerable number of semantic abstractions of safe states as negative samples to estimate the hazard discrimination model To better estimate , capturing as many states as possible that violate safety constraints in the simulator as possible for use in training the hazard discrimination model. Furthermore, in addition to collecting states that violate safety constraints during normal policy interaction, additional hazard exploration is performed. The policy being trained interacts with the environment. After a certain number of steps (randomly generated within a maximum number of steps), the policy is switched to random policy interaction. Using a random policy makes it easier to cause the agent to violate safety constraints, making it easier to obtain states that violate safety constraints.

[0118] like Figure 2 As shown, in this embodiment, when performing the robot skill virtual-real transfer learning, the semantic abstract neural network model is first trained , training semantic abstraction neural network model When the simulation image in the simulation environment and the real-object image collected in the target domain are input respectively, they are input into the semantic abstract neural network respectively, and the semantic domain adaptive method based on adversarial learning is trained to form a semantic abstract neural network model. First, use the simulated images and labels to train the semantic abstract neural network model. , by semantic abstract neural network model The corresponding semantic abstract neural network models for converting simulated images into semantic space to obtain simulated semantic output and converting real images into semantic space to obtain real semantic output are respectively Parameters are shared between them, ultimately achieving alignment of semantic features between the simulation domain and the physical domain. The discriminator model is then trained using two sets of images, the simulation image and the physical image, using the adversarial loss function. Calculate the adversarial loss to narrow the distance between the target domain and the source domain, realize adaptive semantic abstraction learning, and finally obtain the trained semantic abstraction neural network model after adversarial learning. , using this semantic abstract neural network model The input image can be converted into corresponding semantic information output. Then, using the simulated semantics converted from the semantic abstract neural network model, a soft policy learning method under the safe action neighborhood constraint based on reinforcement learning is used to train the robot skill policy neural network, resulting in a robot system skill policy model. This robot system skill policy model is then used to obtain a safety policy applicable to the target scenario to control the robot based on the semantic information of the physical object corresponding to the real-world environment image, thus enabling the transfer and deployment of the skill policy trained in the source domain virtual environment to the physical robot in the target domain.

[0119] In this embodiment, when the strategy is deployed from simulation to physical objects, at each moment, the input image received by the robot system is converted into a semantic state through the semantic abstract neural network model, and then the control amount of each joint of the robot system is obtained through the output of the robot skill strategy network. Figure 3 As shown in the figure, in a specific application embodiment, when the robot arm grasping control is performed based on kernel least squares soft Bellman residual reinforcement learning on the UR5 physical robot arm, the physical grasping system consists of four parts: a Linux workstation, a Realsense d435i depth camera, a UR5 robot arm, and an RG2 gripper. Figure 3 Each row in the table shows the process of completing object grasping in a typical scene after migration. From top to bottom, they correspond to the grasping scenes with increasing difficulty of stacked objects, showing the grasping process of the embodiment of the present invention on the actual UR5. Figure 3 It can be shown that the present invention can well realize the virtual-real transfer of robot grasping skill strategies in scenarios with different object stacking difficulties.

[0120] This embodiment also provides a robot skill virtual-to-real transfer learning system based on semantic domain adaptation, including an interconnected microprocessor and a memory, and the microprocessor is programmed or configured to execute a robot skill virtual-to-real transfer learning method based on semantic domain adaptation.

[0121] It is understandable that the above method of this embodiment can be executed by a single device, such as a computer or server, etc., and can also be applied to a distributed scenario and completed by multiple devices cooperating with each other. In the case of a distributed scenario, one of the multiple devices can only execute one or more steps in the above method of this embodiment, and multiple devices interact to complete the above method. The processor can be implemented in the form of a general-purpose CPU, a microprocessor, an application-specific integrated circuit, or one or more integrated circuits, etc., for executing relevant programs to implement the above method of this embodiment. The memory can be implemented in the form of a read-only memory ROM, a random access memory RAM, a static storage device, and a dynamic storage device. The memory can store an operating system and other application programs. When the above method of this embodiment is implemented by software or firmware, the relevant program code is stored in the memory and called and executed by the processor.

[0122] The above description is merely a preferred embodiment of the present invention. The scope of protection of the present invention is not limited to the above embodiment. All technical solutions based on the concept of the present invention are within the scope of protection of the present invention. It should be noted that for those skilled in the art, various improvements and modifications that do not depart from the principles of the present invention should also be considered within the scope of protection of the present invention.

Claims

1. A method for transferring robot skills from virtual to real based on semantic domain adaptation, characterized in that: The following steps are involved: Step S1: Simulation environment construction: A simulation environment for robot skill learning is constructed based on the real environment in which the robot skill is operated, and semantic labels corresponding to the simulation environment are generated. The source domain simulation image and label pair sample set is composed of the corresponding simulation images in the simulation environment and the semantic labels. Step S2: semantic abstraction neural network training: real-object images are collected in the target domain physical system, and the source domain simulation images and label pair sample sets and the target domain physical images are used as input samples. The semantic abstraction neural network model is trained based on the semantic domain adaptation method of adversarial learning. The semantic abstraction neural network model converts the input source domain simulation images into the semantic space to obtain simulation semantics and converts the target domain physical images into the semantic space to obtain physical semantics, so as to achieve alignment of semantic features between the simulation domain and the physical domain. During the training process of the semantic domain adaptation method based on adversarial learning, a semantic abstraction neural network model is first trained using a sample set of source domain simulated images and label pairs. Then, a discriminator model is constructed and the discriminator model is jointly trained using two sets of images, the source domain simulated images and the target domain real-object images. When training the discriminator model, an adversarial loss function is used to narrow the distance between the target domain and the source domain to achieve adaptive semantic abstraction learning. The adversarial loss function is used to calculate the adversarial loss by using the target domain real-object images input into the semantic abstraction neural network model to obtain target scene features. Step S3: Reinforcement learning training: The robot skill strategy neural network is trained based on the reinforcement learning method using the simulation semantics converted from the semantic abstract neural network model to obtain a trained robot system skill strategy model. The input of the robot skill strategy neural network is the semantic information output by the semantic abstract neural network model, and the output is the control instructions for each joint of the robot. Step S4: Strategy deployment migration: When a real-time real environment image is collected, the semantic abstract neural network model is used to convert the real environment image into a semantic space to obtain physical semantic information, and the physical semantic information obtained by the real-time conversion is input into the robot skill strategy neural network model to obtain real-time robot control instructions, thereby realizing the migration and deployment of the skill strategy trained in the source domain virtual environment to the target domain physical robot.

2. The method for learning the virtual-real transfer of robot skills based on semantic domain adaptation according to claim 1 is characterized in that: In step S2, during the training of the semantic domain adaptation method based on adversarial learning, the semantic abstraction neural network model is trained using the source domain simulation image and the label pair sample set, including: Input to the semantic abstraction neural network model In the source domain, we get the abstract result , by minimizing the abstraction result of the source domain and source domain semantic labels Distance training abstract model .

3. The method for learning robot skills from virtual to real based on semantic domain adaptation according to claim 1 is characterized in that: In step S2, during the training of the semantic domain adaptation method based on adversarial learning, training the discriminator model includes: The source domain simulated image and the target domain object image Input to the semantic abstraction neural network model In the example, we get the source scene features output by the middle layer of the model and target scene features ; Source scene features and target scene features Input to the discriminator model In the example, the discriminator model is trained by minimizing the cross entropy loss function. : Freeze the discriminator model , the target domain object image Input to the semantic abstraction neural network model The target scene features output by the intermediate layer are obtained , and in the discriminator model Use target scene features in Calculate adversarial loss to bring the target scene features closer Source scene characteristics distance.

4. The method for learning the virtual-real transfer of robot skills based on semantic domain adaptation according to claim 3 is characterized in that: In step S2, the loss function during the training process of the semantic domain adaptation method based on adversarial learning is: in, is the balance coefficient used to balance the semantic abstraction loss and the adversarial loss, The number of categories representing semantics, Indicates the size of the image, , The loss function representing the semantic abstraction process, Denotes the adversarial loss.

5. The method for learning robot skills from virtual to real based on semantic domain adaptation according to claim 1, characterized in that: In step S3, a reinforcement learning method based on a safe Bellman update operator is used to train the robot skill policy neural network. In the reinforcement learning method based on a safe Bellman update operator, a safe action value function is learned using a safe Bellman update operator: in, For robot skill strategy, is the reward function, Status and action of Step potential penalty cost function, For the secure Bellman update operator, is the safety action value function, represents the expectation under the strategy π and state transition distribution P, Represents the discount factor.

6. The method for learning the virtual-real transfer of robot skills based on semantic domain adaptation according to claim 5 is characterized in that: In step S3, the training steps of the safety action value function and the strategy function in the process of training the robot skill strategy neural network are as follows: Update the safe action value function by minimizing the safe soft Bellman residual : in, In the above formula, represents the expectation calculated by sampling data from the dataset D, represents the expectation under the action of strategy π, represents the learning rate, is the network parameter of the security action value function, are the parameters of the strategy; The policy function minimizes the following objective function during update : 。 7. The method for learning the virtual-real transfer of robot skills based on semantic domain adaptation according to claim 6 is characterized in that: During the training of the robot skill policy neural network, the safe action value function accepts both state and action inputs. The action input is sampled from the Gaussian distribution of the policy function output. The action is resampled using the following reparameterized network: in, is the noise sampled from a specified fixed distribution to simulate the sampling process, represents a reparameterized network; Actions after resampling using a reparameterized network for: in, and are the mean and variance distribution of the Gaussian distribution output by the strategy respectively.

8. The method for learning the transfer of robot skills from virtual to real based on semantic domain adaptation according to claim 7 is characterized in that: Also includes the use of the reversible hyperbolic tangent function Transform the Gaussian distribution resampling action and constrain it to the normalized interval inside, among them The function is calculated as follows: The action entropy term after interval transformation is calculated by the following formula: in, is the dimension of the action space, for The elements, the Gaussian distribution resampling action is ,pass The action after function transformation is , Right now , x Indicates the input signal.

9. The method for learning the virtual-real transfer of robot skills based on semantic domain adaptation according to any one of claims 1 to 5, characterized in that: Step S3 also includes using semantic information to model the risk penalty function , among which Indicates status In a safe range, when Indicates status Violation of safety constraints is subject to penalty costs , state transfer function Guide the update of strategy, for strategy ,state and action of The potential penalty cost function of the step is calculated as follows: in, for Perturbation function under standard Gaussian noise.

10. A robot skill virtual-real transfer learning system based on semantic domain adaptation, comprising an interconnected microprocessor and a memory, characterized in that: The microprocessor is programmed or configured to execute the robot skill virtual-to-real transfer learning method based on semantic domain adaptation as described in any one of claims 1 to 9.

Images