A robot assembly obstacle avoidance method and system based on intelligent cross-domain strategy migration

Abstract

The present invention discloses a robot assembly obstacle avoidance method and system based on intelligent cross-domain strategy migration, the method comprising: constructing a source environment scene and a target environment scene; based on the source environment scene, through a deep reinforcement learning strategy and a sparse reward mechanism, performing three-dimensional assembly operations on the robot, and constructing an approximate optimal source domain strategy; based on the target environment scene, migrating the approximate optimal source domain strategy, and combining the state splicing strategy to perform three-dimensional assembly obstacle avoidance operations on the robot, and constructing an optimal target domain strategy; deploying the optimal target domain strategy on the robot control platform to complete the robot's three-dimensional assembly obstacle avoidance operation. By using the present invention, the operational safety and production efficiency of the robot can be improved, and the risks of collision and failure can be reduced. As a robot assembly obstacle avoidance method and system based on intelligent cross-domain strategy migration, the present invention can be widely used in the field of robot obstacle avoidance technology.

Description

Technical Field

[0001] The present invention relates to the technical field of robot obstacle avoidance, and in particular to a robot assembly obstacle avoidance method and system based on intelligent cross-domain strategy migration. Background Art

[0002] Currently, the widely used robotic assembly and obstacle avoidance technologies rely on deep reinforcement learning (DRL). These technologies gradually learn strategies to complete tasks through a large amount of environmental interaction and data accumulation. For example, robots trained using reinforcement learning are able to optimize their path selection and object manipulation strategies through continuous trial and error, thereby achieving effective obstacle avoidance and precise assembly in an environment with obstacles. These methods mainly rely on a large amount of interaction data and a long training cycle to adapt to complex scenarios by gradually learning environmental features and task strategies. However, there are some shortcomings in related technologies, such as long training time. DRL usually requires a long time of training to achieve stability and efficiency for industrial use; strong data dependence. Effective DRL strategies require a large amount of environmental interaction data, which is often not feasible in practical applications; poor environmental adaptability. When a robot is transferred from a training environment to an actual production environment, the performance may drop significantly due to environmental differences.

[0003] In summary, the technical problems existing in the relevant technologies need to be improved. Summary of the invention

[0004] In order to solve the above technical problems, the purpose of the present invention is to provide a robot assembly obstacle avoidance method and system based on intelligent cross-domain strategy migration, which can improve the operational safety and production efficiency of the robot and reduce the risks of collision and failure.

[0005] The first technical solution adopted by the present invention is: a robot assembly obstacle avoidance method based on intelligent cross-domain strategy migration, comprising the following steps:

[0006] Constructing a source environment scene and a target environment scene, wherein the source environment scene represents an obstacle-free simulation environment, and the target environment scene represents an obstacle-containing simulation environment;

[0007] Based on the source environment scenario, the robot is trained to perform three-dimensional assembly operations through deep reinforcement learning strategies and sparse reward mechanisms, and an approximate optimal source domain strategy is constructed.

[0008] Based on the target environment scenario, the approximate optimal source domain strategy is migrated, and the robot is operated to avoid obstacles in three-dimensional assembly by combining the state splicing strategy to construct the optimal target domain strategy.

[0009] The optimal target domain strategy is deployed on the robot control platform to complete the robot's three-dimensional assembly obstacle avoidance operation.

[0010] Furthermore, the step of constructing an approximate optimal source domain strategy by performing three-dimensional assembly operations on the robot based on the source environment scenario through a deep reinforcement learning strategy and a sparse reward mechanism specifically includes:

[0011] Based on the deep reinforcement learning strategy, initialize the policy network, value function and experience replay buffer;

[0012] Based on the source environment scene, the robot's three-dimensional assembly operation data is collected and stored in the experience playback buffer;

[0013] Based on the actor-critic architecture algorithm, the maximum entropy target is set, and the robot 3D assembly operation data in the experience replay buffer is randomly updated by maximizing the expected value of the cumulative maximum entropy target to obtain the randomly updated robot 3D assembly operation data;

[0014] Set up a sparse reward mechanism to obtain the reward incentives for the randomly updated robot 3D assembly operation data and update the experience replay buffer;

[0015] The randomly updated robot three-dimensional assembly operation data is marked by using the HER algorithm to obtain the marked robot three-dimensional assembly operation data;

[0016] The policy network and value function are updated according to the labeled robot 3D assembly operation data until the policy network and value function meet the preset requirements, and an approximate optimal source domain strategy is constructed.

[0017] Furthermore, the expression of the expected value of maximizing the cumulative maximum entropy target is specifically as follows:

[0018] ;

[0019] In the above formula, represents the expected value of the maximum entropy objective, represents the immediate reward term, which quantifies the immediate feedback from the environment. represents the entropy return term, represents the temperature parameter, which is used to adjust the weight of the entropy term and control the trade-off between exploration and utilization. represents the expected value on the state-action distribution, According to the strategy The distribution of generated state-action pairs, Indicates that at time step The environmental status, Indicates that at time step The actions taken by the agent, represents the policy, i.e., the mapping from state to action distribution, which is used to define the behavior of the agent. represents the total time horizon, the number of time steps of the entire round trajectory considered, Indicates that between 0 and A specific time step within the range.

[0020] Furthermore, the step of migrating the approximately optimal source domain strategy based on the target environment scenario, and combining the state splicing strategy to perform three-dimensional assembly obstacle avoidance operations on the robot to construct the optimal target domain strategy specifically includes:

[0021] Considering the differences between the target environment scenario and the source environment scenario, the approximate optimal source domain strategy is migrated to construct the optimal decision strategy for the target environment scenario;

[0022] By using a dynamic strategy dependency method, the optimal decision strategy for the target environment scenario is updated to obtain an updated optimal decision strategy for the target environment scenario;

[0023] Based on the updated optimal decision-making strategy for the target environment scenario, the state splicing strategy is combined to perform three-dimensional assembly obstacle avoidance operations on the robot and construct the optimal target domain strategy.

[0024] Furthermore, the step of considering the difference between the target environment scenario and the source environment scenario, migrating the approximately optimal source domain strategy, and constructing the optimal decision strategy for the target environment scenario specifically includes:

[0025] Define the target domain of the target environment scenario and the source domain of the source environment scenario;

[0026] Acquire external knowledge of the source domain and internal knowledge of the target domain;

[0027] By fusing the external knowledge of the source domain with the internal knowledge of the target domain, the approximate optimal source domain strategy is migrated to obtain the optimal decision-making strategy for the target environment scenario.

[0028] Furthermore, the expression of the optimal decision strategy for the target environment scenario is specifically as follows:

[0029] ;

[0030] In the above formula, represents the optimal decision-making strategy for the target environment scenario, is the state-action value function or Function, in strategy Next, in the state Take Action The expected cumulative return at Indicates the distribution in state and strategies The expected value under From the initial state to the The state distribution of the step, Designed to maximize function The expected value of Indicates the current environment status. Indicates that the agent is in state Next, select the action.

[0031] Further, the step of updating the optimal decision-making strategy for the target environment scenario by the dynamic strategy dependency method to obtain the updated optimal decision-making strategy for the target environment scenario specifically includes:

[0032] Through the online critic network of the approximate optimal source domain strategy and the online critic network of the optimal decision strategy for the target environment scenario, the given robot action and state are evaluated to obtain the source domain Q value and the target domain Q value;

[0033] Integrate the source domain Q value with the target domain Q value to obtain the comprehensive evaluation network Q value;

[0034] The optimal robot action is defined according to the comprehensive evaluation network Q value, and the expected value Q value of the optimal robot action is obtained through the Bellman equation;

[0035] The optimal decision-making strategy for the target environment scenario is updated based on the expected value Q value to obtain an updated optimal decision-making strategy for the target environment scenario.

[0036] Furthermore, the step of constructing the optimal target domain strategy based on the updated optimal decision strategy for the target environment scene and combining the state splicing strategy to perform three-dimensional assembly obstacle avoidance operations on the robot specifically includes:

[0037] Constructing a target domain state vector, wherein the target domain state vector includes state information of the robot and state information of the obstacle;

[0038] Input the target domain state vector into the updated target environment scenario optimal decision strategy to evaluate the value of the current state and action;

[0039] Through the policy gradient method, the parameters of the optimal decision-making strategy for the target environment scenario are updated by combining the current state and the value of the action, maximizing the expected return and constructing the optimal target domain strategy.

[0040] The second technical solution adopted by the present invention is: a robot assembly obstacle avoidance system based on intelligent cross-domain strategy migration, comprising:

[0041] The first module is used to construct a source environment scene and a target environment scene, wherein the source environment scene represents a barrier-free simulation environment, and the target environment scene represents a barrier-free simulation environment;

[0042] The second module is used to perform three-dimensional assembly operations on the robot based on the source environment scenario through deep reinforcement learning strategy and sparse reward mechanism, and construct an approximate optimal source domain strategy;

[0043] The third module is used to migrate the approximate optimal source domain strategy based on the target environment scenario, and combine the state splicing strategy to perform three-dimensional assembly obstacle avoidance operations on the robot to construct the optimal target domain strategy;

[0044] The fourth module is used to deploy the optimal target domain strategy on the robot control platform to complete the robot's three-dimensional assembly obstacle avoidance operation.

[0045] The beneficial effects of the method and system of the present invention are as follows: the present invention constructs a source environment scene and a target environment scene, and based on the source environment scene, performs three-dimensional assembly operations on the robot through a deep reinforcement learning strategy and a sparse reward mechanism, constructs an approximate optimal source domain strategy, encourages the strategy to explore the environment more extensively through the deep reinforcement learning strategy, and prevents premature convergence to the local optimum, and combines with a sparse reward mechanism, wherein the reward is only granted when the assembly task is successfully completed. This reward setting motivates the agent to focus on achieving the key goals of the task, rather than pursuing irrelevant intermediate states in the process, and accelerates the effective convergence of the deep reinforcement learning strategy. Based on the target environment scene, the approximate optimal source domain strategy is migrated, and the robot is combined with a state splicing strategy to perform three-dimensional assembly obstacle avoidance operations, and construct an optimal target domain strategy. During the migration process, the characteristics of the strategy action space are shared, and the differences between the source environment and the target environment in the observation space are considered. By adjusting the observation space to capture more key information, the performance of the model in the new environment is improved, and then the reaction speed and accuracy of the robot in the face of environmental changes are improved, the operational safety and production efficiency of the robot are improved, and the risks of collision and failure are reduced. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] Figure 1 It is a flowchart of the steps of a robot assembly obstacle avoidance method based on intelligent cross-domain strategy migration of the present invention;

[0047] Figure 2 It is a structural block diagram of a robot assembly obstacle avoidance system based on intelligent cross-domain strategy migration of the present invention;

[0048] Figure 3 It is a schematic diagram of a framework of robot assembly obstacle avoidance with intelligent cross-domain strategy migration provided by a specific embodiment of the present invention;

[0049] Figure 4 is a schematic diagram of a source environment scenario provided by a specific embodiment of the present invention;

[0050] Figure 5 is a schematic diagram of a target environment scenario provided by a specific embodiment of the present invention;

[0051] Figure 6 It is a schematic diagram of a process of a nearly optimal source domain strategy provided by a specific embodiment of the present invention;

[0052] Figure 7 It is a flowchart of the optimal target domain strategy provided by a specific embodiment of the present invention. DETAILED DESCRIPTION

[0053] The present invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments. The step numbers in the following embodiments are only provided for the convenience of explanation and description, and the order between the steps is not limited in any way. The execution order of each step in the embodiment can be adaptively adjusted according to the understanding of those skilled in the art.

[0054] Reference Figure 1 The present invention provides a robot assembly obstacle avoidance method based on intelligent cross-domain strategy migration, the method comprising the following steps:

[0055] S100, constructing a source environment scene and a target environment scene, wherein the source environment scene represents a barrier-free simulation environment, and the target environment scene represents a barrier-free simulation environment;

[0056] Specifically, a simulation experiment environment scene is designed based on Gazebo + ROS Notice. In the simulation environment, the positions of the robot arm and obstacles can be collected, such as Figure 4 As shown in, it is a schematic diagram of the source environment scene, such as Figure 5 , which is a schematic diagram of the target environment scene.

[0057] S200, based on the source environment scenario, through deep reinforcement learning strategy and sparse reward mechanism, the robot performs three-dimensional assembly operations and constructs an approximate optimal source domain strategy;

[0058] Specifically, based on the deep reinforcement learning strategy, the policy network, value function and experience replay buffer are initialized; based on the source environment scene, the robot's three-dimensional assembly operation data is collected and stored in the experience replay buffer; based on the actor-critic architecture algorithm, the maximum entropy target is set, and the robot's three-dimensional assembly operation data in the experience replay buffer is randomly updated by maximizing the expected value of the cumulative maximum entropy target to obtain the randomly updated robot's three-dimensional assembly operation data; a sparse reward mechanism is set to obtain the reward incentive for the randomly updated robot's three-dimensional assembly operation data and update the experience replay buffer; the randomly updated robot's three-dimensional assembly operation data is labeled by the HER algorithm to obtain the labeled robot's three-dimensional assembly operation data; the policy network and value function are updated according to the labeled robot's three-dimensional assembly operation data until the policy network and the value function meet the preset requirements, and an approximate optimal source domain strategy is constructed.

[0059] In the embodiment of the present invention, the approximate optimal source domain strategy is learned, and the process is as follows: Figure 6 As shown. In a simulated environment, deep reinforcement learning technology is applied to successfully train the robot to perform precise assembly operations in a three-dimensional action space. An end-to-end deep reinforcement learning strategy is adopted, using the SAC algorithm of the actor-critic architecture with continuous control function. SAC adopts a random strategy update method. One of its innovations is that it introduces the concept of entropy regularization and sets a maximum entropy target. , by maximizing the expected value of the cumulative maximum entropy objective (Cumulative average return) to make the strategy The expression of randomization and maximizing the expected value of the cumulative maximum entropy objective is as follows:

[0060] ;

[0061] In the above formula, represents the expected value of the maximum entropy objective, represents the immediate reward term, which quantifies the immediate feedback from the environment. represents the entropy return term, represents the temperature parameter, which is used to adjust the weight of the entropy term and control the trade-off between exploration and utilization. represents the expected value on the state-action distribution, According to the strategy The distribution of generated state-action pairs, Indicates that at time step The environmental status, Indicates that at time step The actions taken by the agent, represents the policy, i.e., the mapping from state to action distribution, which is used to define the behavior of the agent. represents the total time horizon, the number of time steps of the entire round trajectory considered, Indicates that between 0 and A specific time step within the range.

[0062] Further:

[0063] ;

[0064] in, Represents a normal immediate reward item; represents the entropy return term; is the temperature parameter (weight) that determines the relative importance of the entropy term to the immediate reward.

[0065] Entropy is used here as a measure of the randomness of the policy. An increase in entropy indicates an increase in the randomness of the policy, which encourages the policy to explore the environment more extensively. This approach not only accelerates learning, but also prevents premature convergence to the local optimum. By maintaining the randomness of the policy, the SAC algorithm can better find the global optimal solution in the continuous action space. To encourage the agent to focus on achieving the key goals of the task, we chose a sparse reward setting. In order to improve the obstacle avoidance learning performance of the robot arm, we combined the HER (Hindsight Experience Replay) technology. Through experience replay, the robot can learn from previous experience and conduct more effective training by reusing historical data.

[0066] In summary, a suboptimal policy is pre-trained in an obstacle-free source environment, and the policy is transferred to an obstacle-ridden target environment through sharing of the policy action space.

[0067] S300, based on the target environment scenario, migrate the approximate optimal source domain strategy, and combine the state splicing strategy to perform three-dimensional assembly obstacle avoidance operations on the robot to construct the optimal target domain strategy;

[0068] S310, considering the difference between the target environment scenario and the source environment scenario, migrating the approximate optimal source domain strategy to construct the optimal decision strategy for the target environment scenario;

[0069] Specifically, the target domain of the target environment scenario and the source domain of the source environment scenario are defined; the external knowledge of the source domain and the internal knowledge of the target domain are obtained; and the approximate optimal source domain strategy is migrated by fusing the external knowledge of the source domain with the internal knowledge of the target domain to obtain the optimal decision strategy for the target environment scenario.

[0070] In the embodiment of the present invention, the suboptimal strategy obtained by training in the source environment is used to perform strategy migration in the target environment with obstacles, such as Figure 7 As shown. Define the source domains as: , the target domain is: Can be Providing prior knowledge enables the target agent to learn better (compared to not obtaining any prior knowledge).

[0071] In a simplified scenario, if two agents are in the same domain, knowledge can be transferred between them. and If transfer learning is not considered, conventional reinforcement learning can be regarded as a special case of the above definition. In this case, it is assumed that (empty set), so that the strategy Based entirely on the target area Provide feedback to learn, i.e. In the context of reinforcement learning, transfer learning involves transferring To the target domain The goal of this process is to transfer knowledge from the source domain to External knowledge provided Target Area Internal knowledge , to form the optimal decision strategy for the target domain , expressed as:

[0072] ;

[0073] Here, the strategy Targeted areas Through comprehensive and The knowledge learned is expressed as:

[0074] ;

[0075] use knowledge-based The learned policy, which is often approximated by a deep neural network in deep reinforcement learning.

[0076] S320, updating the optimal decision-making strategy for the target environment scenario through a dynamic strategy dependency method to obtain an updated optimal decision-making strategy for the target environment scenario;

[0077] Specifically, the given robot action and state are evaluated through the online critic network of the approximate optimal source domain strategy and the online critic network of the optimal decision-making strategy for the target environment scenario to obtain the source domain Q value and the target domain Q value; the source domain Q value and the target domain Q value are integrated to obtain the comprehensive evaluation network Q value; the optimal robot action is defined according to the comprehensive evaluation network Q value, and the value expected Q value of the optimal robot action is obtained through the Bellman equation; the optimal decision-making strategy for the target environment scenario is updated based on the value expected Q value to obtain the updated optimal decision-making strategy for the target environment scenario.

[0078] In the embodiments of the present invention, although the two environments may have different action spaces, there may be some similarities between them. For example, in the operation of a robotic arm, the basic features of the action may include the direction of the action (e.g., forward, backward, upward, downward), the amplitude of the action (e.g., the distance moved, the strength of the grasp), etc. Our method uses the suboptimal policy obtained from the source environment and the features of the shared policy action space to guide the training of the target policy. The core of our method is to use the source domain policy as a guide to accelerate and enhance the learning process in new scenarios.

[0079] In the embodiment of the present invention, the source strategy online critic network (marked as ) and the target strategy online critic network (labeled as ) evaluates the given action and status Then a dynamic strategy-dependent method is proposed, and the output of these networks is evaluated through a comprehensive To integrate, the calculation method is as follows:

[0080] ;

[0081] in, , is the current accumulated number of training steps, is a constant. In the early stage of training, since the source policy network has random parameters, the target policy provides The value plays a major role, and as the training progresses, it gradually turns to autonomous learning that relies on the target strategy.

[0082] Then, the optimal action Defined as:

[0083] ;

[0084] in, is the action generated by the original actor network, is the action generated by the prophet actor network.

[0085] In deep reinforcement learning, the Bellman equation is used to calculate expected value , which is defined as follows:

[0086]

[0087] in, is the discount factor, which represents the current value of future rewards; is the target actor network, is the target critic network, which uses parameters and .

[0088] S330. Based on the updated optimal decision-making strategy for the target environment scenario, the robot is subjected to three-dimensional assembly obstacle avoidance operations in combination with the state splicing strategy to construct the optimal target domain strategy.

[0089] Specifically, a target domain state vector is constructed, wherein the target domain state vector includes the state information of the robot and the state information of the obstacle; the target domain state vector is input into the updated target environment scenario optimal decision-making strategy to evaluate the value of the current state and action; and the parameters of the target environment scenario optimal decision-making strategy are updated by combining the value of the current state and action through the policy gradient method to maximize the expected return and construct the optimal target domain strategy.

[0090] In the embodiment of the present invention, in the field of robot assembly, the observation space is the set of all environmental information that the robot can perceive. The main difference between the source environment and the target environment is the existence of obstacles, and the state of obstacles (including position and speed) is different in the source environment and the target environment. This difference leads to different dimensions of the observation space. To solve this problem, we use a method called "state splicing strategy" to complete the state information of the target domain, so that the robot can consider more environmental information in its decision-making process.

[0091] Specifically, the state information of the target domain is represented as ,in is the status information of the robot, is the state information of the obstacle. By splicing these two pieces of information together, a complete target domain state vector is constructed, allowing the robot to perceive the environment more comprehensively.

[0092] The concatenated state vector is used as input to a policy or value function, such as the policy function or value function , at each time step, the policy or value function selects an action or evaluates the value of the current state and action based on the concatenated state vector.

[0093] Finally, the policy gradient method is used to update the policy parameters , to maximize the expected return:

[0094] ;

[0095] here, is the learning rate, About The gradient of the value function.

[0096] Considering the difference between the source environment and the target environment in the observation space, a method called "state splicing strategy" is adopted to complete the state information of the target domain, so that the robot can consider more environmental information in its decision-making process. A dynamic strategy dependency method is proposed. The output of these networks is evaluated through a comprehensive To integrate, the initial stage of training mainly relies on the experience of the source strategy, and gradually turns to autonomous learning that relies on the target strategy as the training progresses.

[0097] S400, deploy the optimal target domain strategy on the robot control platform to complete the robot's three-dimensional assembly obstacle avoidance operation.

[0098] In summary, in the field of robotic control, deep reinforcement learning has been used to develop autonomous robots capable of performing complex tasks. By using DRL, robots are able to learn how to autonomously complete tasks from grasping different objects to planning in complex environments without human intervention. These capabilities not only improve the operational efficiency of robots, but also expand their application range.

[0099] In industrial robotic arm assembly tasks, DRL can be used to train robots to perform various tasks, including obstacle avoidance and collision avoidance. In order to be closer to the actual assembly work scenario, such as Figure 3 As shown in FIG. 1 , a sparse reward structure is used in training in an embodiment of the present invention, in which rewards are granted only when the assembly task is successfully completed. This reward setting motivates the agent to focus on achieving the key goals of the task rather than pursuing irrelevant intermediate states in the process. However, due to the complexity and uncertainty of the robot assembly environment, DRL takes a long time to train and is prone to falling into local optimality, making it difficult to achieve effective convergence within a reasonable time.

[0100] The embodiment of the present invention proposes a robot assembly safety obstacle avoidance and anti-collision strategy for intelligent cross-domain policy migration, trains a suboptimal strategy in an obstacle-free source environment, and then deploys the suboptimal strategy to a scene with obstacles for training, learning to avoid obstacles and complete tasks. During the migration process, the characteristics of the strategy action space are shared, and the differences in the observation space between the source environment and the target environment are considered. The observation space is adjusted to capture more key information, thereby improving the performance of the model in the new environment. At the same time, we also propose a method of dynamic policy dependence, which mainly relies on the experience of the source policy in the early stage of training, and gradually turns to autonomous learning that relies on the target policy as the training progresses.

[0101] Therefore, compared with the prior art, the embodiments of the present invention have the following advantages:

[0102] 1) Through the transfer strategy, the robot can quickly adapt to the new environment, significantly reducing the data and time required for training from scratch.

[0103] 2) The strategy’s action space and observation space adjustments optimize the robot’s response speed and accuracy when facing environmental changes.

[0104] 3) Optimized obstacle avoidance and assembly strategies improve the robot’s operational safety and production efficiency, and reduce the risk of collision and failure.

[0105] Reference Figure 2 , a robot assembly obstacle avoidance system based on intelligent cross-domain strategy migration, comprising:

[0106] The first module 201 is used to construct a source environment scene and a target environment scene, wherein the source environment scene represents a barrier-free simulation environment, and the target environment scene represents a barrier-free simulation environment;

[0107] The second module 202 is used to perform three-dimensional assembly operations on the robot based on the source environment scene through deep reinforcement learning strategy and sparse reward mechanism to construct an approximate optimal source domain strategy;

[0108] The third module 203 is used to migrate the approximately optimal source domain strategy based on the target environment scene, and combine the state splicing strategy to perform three-dimensional assembly obstacle avoidance operations on the robot to construct the optimal target domain strategy;

[0109] The fourth module 204 is used to deploy the optimal target domain strategy on the robot control platform to complete the robot's three-dimensional assembly obstacle avoidance operation.

[0110] The contents of the above method embodiments are all applicable to the present system embodiments. The functions specifically implemented by the present system embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0111] The above is a specific description of the preferred implementation of the present invention, but the invention is not limited to the embodiments. Those skilled in the art may make various equivalent modifications or substitutions without violating the spirit of the present invention. These equivalent modifications or substitutions are all included in the scope defined by the claims of this application.

Claims

1. A robot assembly obstacle avoidance method based on intelligent cross-domain strategy migration, characterized in that: The following steps are involved: Constructing a source environment scene and a target environment scene, wherein the source environment scene represents an obstacle-free simulation environment, and the target environment scene represents an obstacle-containing simulation environment; Based on the source environment scenario, the robot is trained to perform three-dimensional assembly operations through deep reinforcement learning strategies and sparse reward mechanisms, and an approximate optimal source domain strategy is constructed. Based on the target environment scenario, the approximate optimal source domain strategy is migrated, and the robot is operated to avoid obstacles in three-dimensional assembly by combining the state splicing strategy to construct the optimal target domain strategy. The optimal target domain strategy is deployed on the robot control platform to complete the robot's three-dimensional assembly obstacle avoidance operation.

2. According to claim 1, a robot assembly obstacle avoidance method based on intelligent cross-domain strategy migration is characterized in that: The step of constructing an approximate optimal source domain strategy by performing three-dimensional assembly operations on the robot based on the source environment scenario through a deep reinforcement learning strategy and a sparse reward mechanism specifically includes: Based on the deep reinforcement learning strategy, initialize the policy network, value function and experience replay buffer; Based on the source environment scene, the robot's three-dimensional assembly operation data is collected and stored in the experience playback buffer; Based on the actor-critic architecture algorithm, the maximum entropy target is set, and the robot 3D assembly operation data in the experience replay buffer is randomly updated by maximizing the expected value of the cumulative maximum entropy target to obtain the randomly updated robot 3D assembly operation data; Set up a sparse reward mechanism to obtain the reward incentives for the randomly updated robot 3D assembly operation data and update the experience replay buffer; The randomly updated robot three-dimensional assembly operation data is labeled by using the HER algorithm to obtain the labeled robot three-dimensional assembly operation data; The policy network and value function are updated according to the labeled robot 3D assembly operation data until the policy network and value function meet the preset requirements, and an approximate optimal source domain strategy is constructed.

3. According to claim 2, a robot assembly obstacle avoidance method based on intelligent cross-domain strategy migration is characterized in that: The expression of the expected value of the maximized cumulative maximum entropy objective is specifically as follows: ； In the above formula, represents the expected value of the maximum entropy objective, represents the immediate reward term, which quantifies the immediate feedback from the environment. represents the entropy return term, represents the temperature parameter, which is used to adjust the weight of the entropy term and control the trade-off between exploration and utilization. represents the expected value on the state-action distribution, According to the strategy The distribution of generated state-action pairs, Indicates that at time step The environmental status, Indicates that at time step The actions taken by the agent, represents the policy, which is a mapping from state to action distribution, and is used to define the behavior of the agent. represents the total time horizon, the number of time steps of the entire round trajectory considered, Indicates that between 0 and A specific time step within the range.

4. According to claim 3, a robot assembly obstacle avoidance method based on intelligent cross-domain strategy migration is characterized in that: The step of migrating the approximately optimal source domain strategy based on the target environment scenario, and combining the state splicing strategy to perform three-dimensional assembly obstacle avoidance operations on the robot to construct the optimal target domain strategy specifically includes: Considering the differences between the target environment scenario and the source environment scenario, the approximate optimal source domain strategy is migrated to construct the optimal decision strategy for the target environment scenario; By using a dynamic strategy dependency method, the optimal decision strategy for the target environment scenario is updated to obtain an updated optimal decision strategy for the target environment scenario; Based on the updated optimal decision-making strategy for the target environment scenario, the state splicing strategy is combined to perform three-dimensional assembly obstacle avoidance operations on the robot and construct the optimal target domain strategy.

5. According to claim 4, a robot assembly obstacle avoidance method based on intelligent cross-domain strategy migration is characterized in that: The step of considering the difference between the target environment scenario and the source environment scenario, migrating the approximately optimal source domain strategy, and constructing the optimal decision strategy for the target environment scenario specifically includes: Define the target domain of the target environment scenario and the source domain of the source environment scenario; Acquire external knowledge of the source domain and internal knowledge of the target domain; By fusing the external knowledge of the source domain with the internal knowledge of the target domain, the approximate optimal source domain strategy is migrated to obtain the optimal decision-making strategy for the target environment scenario.

6. According to claim 5, a robot assembly obstacle avoidance method based on intelligent cross-domain strategy migration is characterized in that: The expression of the optimal decision-making strategy for the target environment scenario is specifically as follows: ； In the above formula, represents the optimal decision-making strategy for the target environment scenario, is the state-action value function or Function, in strategy Next, in the state Take Action The expected cumulative return at Indicates the distribution in state and strategies The expected value under From the initial state to the The state distribution of the step, Designed to maximize function The expected value of Indicates the current environment status. Indicates that the agent is in state Next, select the action.

7. The robot assembly obstacle avoidance method based on intelligent cross-domain strategy migration according to claim 6 is characterized in that: The step of updating the optimal decision-making strategy for the target environment scenario by the dynamic strategy dependency method to obtain the updated optimal decision-making strategy for the target environment scenario specifically includes: Through the online critic network of the approximate optimal source domain strategy and the online critic network of the optimal decision strategy for the target environment scenario, the given robot action and state are evaluated to obtain the source domain Q value and the target domain Q value; Integrate the source domain Q value with the target domain Q value to obtain the comprehensive evaluation network Q value; The optimal robot action is defined according to the comprehensive evaluation network Q value, and the expected value Q value of the optimal robot action is obtained through the Bellman equation; The optimal decision-making strategy for the target environment scenario is updated based on the expected value Q value to obtain an updated optimal decision-making strategy for the target environment scenario.

8. According to claim 7, a robot assembly obstacle avoidance method based on intelligent cross-domain strategy migration is characterized in that: The step of constructing the optimal target domain strategy based on the updated optimal decision strategy for the target environment scene and combining the state splicing strategy to perform three-dimensional assembly obstacle avoidance operations on the robot specifically includes: Constructing a target domain state vector, wherein the target domain state vector includes state information of the robot and state information of the obstacle; Input the target domain state vector into the updated target environment scenario optimal decision strategy to evaluate the value of the current state and action; Through the policy gradient method, the parameters of the optimal decision-making strategy for the target environment scenario are updated by combining the current state and the value of the action, maximizing the expected return and constructing the optimal target domain strategy.

9. A robot assembly obstacle avoidance system based on intelligent cross-domain strategy migration, characterized in that: Includes the following modules: The first module is used to construct a source environment scene and a target environment scene, wherein the source environment scene represents a barrier-free simulation environment, and the target environment scene represents a barrier-free simulation environment; The second module is used to perform three-dimensional assembly operations on the robot based on the source environment scenario through deep reinforcement learning strategy and sparse reward mechanism, and construct an approximate optimal source domain strategy; The third module is used to migrate the approximate optimal source domain strategy based on the target environment scenario, and combine the state splicing strategy to perform three-dimensional assembly obstacle avoidance operations on the robot to construct the optimal target domain strategy; The fourth module is used to deploy the optimal target domain strategy on the robot control platform to complete the robot's three-dimensional assembly obstacle avoidance operation.

Images