Robot autonomous operation optimization method and system based on virtual simulation and multi-hypothesis planning

Abstract

The invention relates to a robot autonomous operation optimization method and system based on virtual simulation and multi-hypothesis planning. The method comprises the steps that a virtual simulation platform scene with the same configuration as a real experiment scene is built; robot training is carried out in the virtual simulation platform scene, and an autonomous operation strategy is generated; constructing a plurality of hypothetical states based on the current state of the robot, and obtaining a virtual track action sequence of each hypothetical state based on an autonomous operation strategy; sequentially verifying whether the virtual track action sequence corresponding to each virtual state is feasible or not according to a time sequence, and if the virtual track action sequence is feasible, taking the virtual track action sequence as an optimal virtual track action sequence; and aligning the real robot state with the assumed state corresponding to the optimal virtual track action sequence, and generating a real robot action sequence based on the current state and the optimal virtual track action sequence by adopting a cooperative action integration mechanism. Compared with the prior art, the method has the advantages that the difference between virtuality and reality of the model is reduced, and the execution success rate is increased.

Description

Technical Field

[0001] This invention relates to the field of embodied intelligence technology, and in particular to a method and system for optimizing autonomous robot operation based on virtual simulation and multi-hypothesis planning. Background Technology

[0002] With the continuous advancement of robotics technology, especially its widespread application in fields such as automated production, intelligent logistics, and healthcare, the accuracy and efficiency of autonomous robot operation have become a key research focus. As environmental complexity increases, traditional path planning and control strategies often fall short of the challenges posed by dynamic environments and uncertainties. Therefore, improving the autonomous decision-making capabilities of robots in complex and changing environments has become an important research direction in the field of robotics.

[0003] Reinforcement learning training based on virtual simulation platforms provides a feasible technical route for training the autonomous operation skills of robots. By constructing scenarios similar to the real world in a virtual environment, robots can conduct extensive exploration attempts without being restricted by the actual environment. The autonomous operation strategy can be optimized by using reinforcement learning training methods. For example, Chinese patent application CN116339316A provides a path planning method for deep-sea mining robots based on deep reinforcement learning. First, the kinematic model of the robot is built and a path planning system is constructed based on reinforcement learning. Then, a virtual simulation environment is built, and the robot is trained and optimized in the virtual simulation environment. Although this method can greatly reduce training costs and accelerate the development process of robot systems by providing a highly controllable training environment, it still faces some problems: (1) Transferring the skill strategies trained in the virtual simulation environment to the real world inevitably faces the gap between the simulated environment and the real environment, such as physical characteristics, sensor signals, motor signals and environment, which leads to a decrease in robot performance and execution risks in the real environment. (2) The trained strategy is difficult to achieve a 100% success rate. The model will face local optima and is difficult to correct itself, which leads to performance far below expectations under some complex or unknown environmental conditions.

[0004] Therefore, how to seamlessly transfer the robot's simulation strategy to the display environment and achieve high success rate execution is a technical problem that needs to be solved. Summary of the Invention

[0005] The purpose of this invention is to overcome the shortcomings of the existing technology by providing a robot autonomous operation optimization method and system based on virtual simulation and multi-hypothesis planning. By using virtual-real fusion and multi-hypothesis planning technology, the gap between the model in the virtual and real worlds is reduced, and the success rate of training strategy execution is improved.

[0006] The objective of this invention can be achieved through the following technical solutions:

[0007] According to a first aspect of the present invention, a method for optimizing autonomous robot operation based on virtual simulation and multi-hypothesis planning is provided, the method comprising:

[0008] S1. Construct a virtual simulation platform scenario with the same configuration as the real experimental scenario. The scenario includes the scene environment, object state, and robot state.

[0009] S2. In the virtual simulation platform scenario, reinforcement learning based on Markov decision is used to train the robot and generate an autonomous operation strategy.

[0010] S3. In the virtual simulation platform scenario, multiple hypothetical states are constructed based on the robot's current state, and the multiple hypothetical states are deployed in the autonomous operation strategy to obtain the virtual trajectory action sequence of each hypothetical state.

[0011] S4. Verify the feasibility of the virtual trajectory action sequence corresponding to each virtual state in sequence according to the time sequence. If it is feasible, take the virtual trajectory action sequence as the optimal virtual trajectory action sequence. If the virtual trajectory action sequence is not feasible, return to execute S3 to regenerate multiple hypothetical states.

[0012] S5. Align the real robot state with the hypothetical state corresponding to the optimal virtual trajectory action sequence, and generate the robot's real action sequence in the real experimental scenario based on the current state and the optimal virtual trajectory action sequence using a cooperative action integration mechanism.

[0013] As a preferred technical solution, the method for generating the aforementioned autonomous operation strategy is as follows:

[0014] The reinforcement learning task of generating robot autonomous operation strategies is modeled as a Markov decision process. Where S represents the robot's state space, A represents the robot's action space, ρ is the initial state probability distribution, P is the transition probability function, r is the reward value, and γ is the discount factor;

[0015] The RLPD reinforcement learning algorithm is used to train the robot to generate autonomous operation strategies in the virtual simulation platform. The loss function of the RLPD reinforcement learning algorithm includes a Q-value loss function. With policy loss function for:

[0016]

[0017] Among them, Q φ (s,a) represents the Q value of taking action a in state s; π represents the target Q value of taking action a′ in state s′; θ (·|s) represents the strategy π taken in state s. θ The probability of choosing action 'a'; Let r(s,a) represent the entropy of the policy; r(s,a) represent the reward for taking action a in state s; and α represent the hyperparameter.

[0018] As a preferred technical solution, the method for constructing the hypothetical state is as follows:

[0019]

[0020] in, s represents the robot's first assumed state at time t; t Indicates the robot's state at time t; Represents the m-th hypothetical state at time t; ∈ m This represents a small range of random action values ​​for generating the m-th hypothetical state; M represents the total number of hypothetical states.

[0021] As a preferred technical solution, the collaborative action integration mechanism is as follows:

[0022] S51, Based on the aforementioned real robot state Using the aforementioned autonomous operation strategy, generate realistic predicted actions. for:

[0023] S52. Obtain the robot's virtual action at time t from the optimal virtual trajectory action sequence. Calculate the L2 norm of the difference between the virtual action and the actual predicted action;

[0024] S53. If the second norm value is less than or equal to the determination parameter, then the action a deployed on the real robot... t The predicted action is the actual action; if the L2 value is greater than the decision parameter, then the action a deployed on the real robot is... t The average of the actual predicted action and the virtual action;

[0025] S54. Obtain the real state of the robot at the next moment after it performs an action. Determine whether the maximum number of iterations is met. If not, align the real state at the next moment with the assumed state corresponding to the optimal virtual trajectory action sequence and then execute S51. Otherwise, end the iteration.

[0026] According to a second aspect of the present invention, a robot autonomous operation optimization system based on virtual simulation and multi-hypothesis planning is provided, the system comprising:

[0027] The environment setup module is used to build a robot self-service operation optimization platform. The robot self-service operation optimization platform includes a real experimental scenario and a virtual simulation platform scenario. The virtual simulation platform scenario is configured the same as the real experimental scenario. The scenario includes the scene environment, object state and robot state.

[0028] The virtual reinforcement learning training module is used to train the robot in the virtual simulation platform scenario using reinforcement learning based on Markov decision-making to generate autonomous operation strategies.

[0029] The hypothesis state generation and verification module is used to construct multiple hypothesis states based on the robot's current state, and deploy the multiple hypothesis states to the autonomous operation strategy to obtain the virtual trajectory action sequence for each hypothesis state; and to verify the feasibility of the virtual trajectory action sequence corresponding to each virtual state in sequence according to the time sequence. If it is feasible, the virtual trajectory action sequence is taken as the optimal virtual trajectory action sequence; if the virtual trajectory action sequence is not feasible, multiple hypothesis states are regenerated.

[0030] The real motion generation module is used to align the real robot state with the hypothetical state corresponding to the optimal virtual trajectory motion sequence, and to generate the robot's real motion sequence in the real experimental scenario based on the current state and the optimal virtual trajectory motion sequence using a cooperative motion integration mechanism.

[0031] According to a third aspect of the present invention, an electronic device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the program to implement the method described thereon.

[0032] According to a fourth aspect of the present invention, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the method described thereon.

[0033] Compared with the prior art, the present invention has the following beneficial effects:

[0034] 1) This invention provides a method for robot planning through virtual-real fusion. First, the robot strategy is initially trained in a virtual simulation platform, and multiple hypothetical states are generated to simulate the robot's virtual task trajectory under the initially generated strategy. After aligning the robot's real state with the hypothetical state, the multi-hypothesis robot planning in the virtual simulation environment is used to supervise and correct the robot's action execution in the real experimental environment, reducing the gap between the virtual world and the real world and improving the efficiency of deploying pre-trained reinforcement learning strategies in the real environment. In addition, the hypothesis planning provided by this invention can help the robot to self-correct when errors occur during autonomous execution or when it gets trapped in a local optimum of the model, thereby improving the success rate of autonomous robot operation.

[0035] 2) This invention proposes a collaborative action integration mechanism, which dynamically adjusts the judgment parameters according to the actual environmental requirements to ensure that the predicted actions of the robot in the virtual simulation and real environment are more in line with the actual scenario requirements, thereby enhancing the robustness of the robot's autonomous execution. Attached Figure Description

[0036] Figure 1 This is a flowchart of the method of the present invention;

[0037] Figure 2 This is a flowchart of the collaborative action integration mechanism of the present invention. Detailed Implementation

[0038] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0039] Example 1

[0040] To address the problems existing in the prior art, this invention provides a robot autonomous operation optimization method based on virtual simulation and multi-hypothesis planning. It aims to reduce the gap in transferring reinforcement learning strategies from virtual simulation to real-world environments, and is applicable to robot reinforcement learning training in various autonomous operation scenarios. The method trains reinforcement learning strategies on a virtual simulation platform and generates multiple hypothesis plans, constructs a collaborative action integration mechanism to deploy and execute actions for the robot in the real environment, and uses virtual-real fusion technology to achieve autonomous operation skill training for the robot. This provides an effective training scheme for robots to autonomously execute operational tasks and is expected to promote the application and promotion of robots in actual production.

[0041] In detail, the method flow provided by this invention is as follows: Figure 1 As shown, it includes:

[0042] S1. Build a virtual simulation platform scenario with the same configuration as the real experimental scenario. The scenario includes the scene environment, object states, and robot states.

[0043] The scene environment includes lighting, weather, background, etc. When building the virtual simulation platform scene, it is necessary to initialize the object state, import the model file that is the same as the real experimental object, and set the synchronous initial pose state; and initialize the robot state, import the model file that is the same as the real experimental robot, and set the synchronous initial joint position, observation space and workspace.

[0044] S2. In a virtual simulation platform scenario, reinforcement learning based on Markov decision-making is used to train the robot and generate autonomous operation strategies.

[0045] S21. Model the reinforcement learning task of generating robot autonomous operation strategies as a Markov decision process. Where S represents the robot's state space, A represents the robot's action space, ρ is the initial state probability distribution, P is the transition probability function, r is the reward value, and γ is the discount factor.

[0046] S22. Initialize the scene environment in S21, such as the pose, color, size, lighting, gravity, friction, etc. of objects.

[0047] S22. The RLPD reinforcement learning algorithm is used to train the robot to generate autonomous operation strategies in a virtual simulation platform.

[0048] When performing the RLPD reinforcement learning algorithm, the algorithm's experience buffer records the online and offline data of the current policy. The data in the experience buffer is uniformly sampled. In this invention, the loss function of the RLPD reinforcement learning algorithm includes a Q-value loss function. With policy loss function for:

[0049]

[0050] Among them, Q φ (s,a) represents the Q value of taking action a in state s; π represents the target Q value of taking action a′ in state s′; θ (·|s) represents the strategy π taken in state s. θ The probability of choosing action 'a'; Let r(s,a) represent the entropy of the policy; r(s,a) represent the reward for taking action a in state s; and α represent the hyperparameter.

[0051] S3. In the virtual simulation platform scenario, multiple hypothetical states are constructed based on the robot's current state, and these hypothetical states are deployed in the autonomous operation strategy to obtain the virtual trajectory action sequence of each hypothetical state.

[0052] S31. Construct multiple hypothetical states.

[0053]

[0054] in, s represents the robot's first assumed state at time t; t Indicates the robot's state at time t; Represents the m-th hypothetical state at time t; ∈m This represents a small range of random action values ​​for generating the m-th hypothetical state; M represents the total number of hypothetical states.

[0055] S32. Generate a virtual trajectory action sequence, the expression of which is:

[0056]

[0057] in, Let m be the assumed state at time t; For strategy П in Actions to be taken in a given state; Let m be the final state of the m-th hypothesis.

[0058] S4. Verify the feasibility of the virtual trajectory action sequence corresponding to each virtual state in sequence according to the time sequence. If it is feasible, take the virtual trajectory action sequence as the optimal virtual trajectory action sequence. If the virtual trajectory action sequence is not feasible, return to execute S3 to regenerate multiple hypothetical states.

[0059] The detailed steps include:

[0060] S41, Prioritize Assumed States That is, the original robot state s t If the strategy planning has achieved the target task, then the corresponding action sequence is selected directly. As the optimal virtual trajectory action sequence.

[0061] S42. Consider the remaining hypothetical state action sequences in turn. Whether the target task has been completed, if so, select the corresponding action sequence as the optimal virtual trajectory action sequence.

[0062] S43. For all assumed states If all planning fails, regenerate the multiple hypothesis states based on step S3.

[0063] S5. Align the real robot state with the hypothetical state corresponding to the optimal virtual trajectory action sequence, and use a cooperative action integration mechanism to generate the robot's real action sequence in the real experimental scenario based on the current state and the optimal virtual trajectory action sequence.

[0064] For the optimal virtual trajectory action sequence Compare the real robot state with Alignment, i.e., performing actions Then, follow these steps to generate a sequence of real robot movements.

[0065] S51, Based on real robot state Generate realistic predicted actions using autonomous operation strategies. for:

[0066] S52. Obtain the robot's virtual action at time t from the optimal virtual trajectory action sequence. Calculate the L2 norm of the difference between the virtual action and the actual predicted action.

[0067] S53. If the L2 norm value is less than or equal to the decision parameter θ, i.e. Then the action a deployed on the real robot t The predicted action is the actual action; if the L2 value is greater than the decision parameter, then... Then the action a deployed on the real robot t The average of the actual predicted action and the virtual action, i.e.

[0068] S54. Obtain the real state of the robot at the next moment after it performs an action. Determine whether the maximum number of iterations n is satisfied and n≤T. If not, align the real state at the next moment with the hypothetical state corresponding to the optimal virtual trajectory action sequence and execute S51. Otherwise, end the iteration, synchronize the robot state of the virtual simulation platform, and iteratively perform multi-hypothetical action generation and planning.

[0069] Example 2

[0070] To verify the feasibility of the method provided by this invention, let's take a robotic arm's charger plugging task as an example. This task requires the robotic arm to grasp the charger plug on the table and accurately insert it into the socket. Successful plugging requires the charger plug held by the robotic arm's gripper to be accurately aligned with the corresponding socket hole. The detailed steps include:

[0071] S1. Build a virtual simulation platform scenario with the same configuration as the real experimental scenario. The scenario includes the scene environment, object states, and robot states.

[0072] The scene environment includes indoor lighting, desktop environment, charger plugs and sockets, etc. When building the virtual simulation platform scene, it is necessary to initialize the object state, import the model files of the charger plugs and sockets in the real experimental scene, and set the synchronous initial pose state; and initialize the robot state, import the model file of the same model as the real experimental robotic arm, and set the synchronous initial joint position, observation space and workspace.

[0073] S2. In a virtual simulation platform scenario, reinforcement learning based on Markov decision-making is used to train the robot and generate autonomous operation strategies.

[0074] S21. Model the reinforcement learning task of generating robot autonomous operation strategies as a Markov decision process. Where S represents the robot's state space, A represents the robot's action space, ρ is the initial state probability distribution, P is the transition probability function, r is the reward value, and γ is the discount factor.

[0075] S22. Initialize the scene environment in S21, such as the pose, color, size, lighting, gravity, friction, etc. of objects.

[0076] S22. The RLPD reinforcement learning algorithm is used to train the robot to generate autonomous operation strategies in a virtual simulation platform.

[0077] When performing the RLPD reinforcement learning algorithm, the algorithm's experience buffer records the online and offline data of the current policy. The data in the experience buffer is uniformly sampled. In this invention, the loss function of the RLPD reinforcement learning algorithm includes a Q-value loss function. With policy loss function for:

[0078]

[0079] Among them, Q φ (s,a) represents the Q value of taking action a in state s; π represents the target Q value of taking action a′ in state s′; θ (·|s) represents the strategy π taken in state s. θ The probability of choosing action 'a'; Let r(s,a) represent the entropy of the policy; r(s,a) represent the reward for taking action a in state s; and α represent the hyperparameter.

[0080] S3. In the virtual simulation platform scenario, multiple hypothetical states are constructed based on the robot's current state, and these hypothetical states are deployed in the autonomous operation strategy to obtain the virtual trajectory action sequence of each hypothetical state.

[0081] S31. Construct multiple hypothetical states.

[0082]

[0083] in, s represents the robot's first assumed state at time t; t Indicates the robot's state at time t; Represents the m-th hypothetical state at time t; ∈ m This represents a small range of random action values ​​for generating the m-th hypothetical state; M represents the total number of hypothetical states.

[0084] S32. Generate a virtual trajectory action sequence, the expression of which is:

[0085]

[0086] in, Let m be the assumed state at time t; For strategy П in Actions to be taken in a given state; Let m be the final state of the m-th hypothesis.

[0087] S4. Verify the feasibility of the virtual trajectory action sequence corresponding to each virtual state in sequence according to the time sequence. If it is feasible, take the virtual trajectory action sequence as the optimal virtual trajectory action sequence. If the virtual trajectory action sequence is not feasible, return to execute S3 to regenerate multiple hypothetical states.

[0088] The detailed steps include:

[0089] S41, Prioritize Assumed States That is, the original robot state s t If the strategy planning has achieved the target task, then the corresponding action sequence is selected directly. As the optimal virtual trajectory action sequence.

[0090] S42. Consider the remaining hypothetical state action sequences in turn. Whether the target task has been completed, if so, select the corresponding action sequence as the optimal virtual trajectory action sequence.

[0091] S43. For all assumed states If all planning fails, regenerate the multiple hypothesis states based on step S3.

[0092] S5. Align the real robot state with the hypothetical state corresponding to the optimal virtual trajectory action sequence, and use a cooperative action integration mechanism to generate the robot's real action sequence in the real experimental scenario based on the current state and the optimal virtual trajectory action sequence, such as... Figure 2 As shown.

[0093] For the optimal virtual trajectory action sequence Compare the real robot state with Alignment, i.e., performing actions Then, follow these steps to generate a sequence of real robot movements.

[0094] S51, Based on real robot state Generate realistic predicted actions using autonomous operation strategies. for:

[0095] S52. Obtain the robot's virtual action at time t from the optimal virtual trajectory action sequence. Calculate the L2 norm of the difference between the virtual action and the actual predicted action.

[0096] S53. If the L2 norm value is less than or equal to the decision parameter θ, i.e. Then the action a deployed on the real robot t The predicted action is the actual action; if the L2 value is greater than the decision parameter, then... Then the action a deployed on the real robot t The average of the actual predicted action and the virtual action, i.e.

[0097] S54. Obtain the real state of the robot at the next moment after it performs an action. Determine whether the maximum number of iterations n is satisfied and n≤T. If not, align the real state at the next moment with the hypothetical state corresponding to the optimal virtual trajectory action sequence and execute S51. Otherwise, end the iteration, synchronize the robot state of the virtual simulation platform, and iteratively perform multi-hypothetical action generation and planning.

[0098] Example 3

[0099] This invention also provides a robot autonomous operation optimization system based on virtual simulation and multi-hypothesis planning, the system comprising:

[0100] The environment setup module is used to build a robot self-service operation optimization platform. The robot self-service operation optimization platform includes real experimental scenarios and virtual simulation platform scenarios. The virtual simulation platform scenarios are configured the same as the real experimental scenarios. The scenarios include the scene environment, object states, and robot states.

[0101] The virtual reinforcement learning training module is used to train robots in a virtual simulation platform scenario using reinforcement learning based on Markov decision-making to generate autonomous operation strategies.

[0102] The hypothesis state generation and verification module is used to construct multiple hypothesis states based on the robot's current state, and deploy these multiple hypothesis states into the autonomous operation strategy to obtain the virtual trajectory action sequence for each hypothesis state; and to verify the feasibility of the virtual trajectory action sequence corresponding to each virtual state in sequence according to the time sequence. If it is feasible, the virtual trajectory action sequence is taken as the optimal virtual trajectory action sequence; if the virtual trajectory action sequence is not feasible, multiple hypothesis states are regenerated.

[0103] The real motion generation module is used to align the real robot state with the hypothetical state corresponding to the optimal virtual trajectory motion sequence. It uses a cooperative motion integration mechanism to generate a real robot motion sequence in a real experimental scenario based on the current state and the optimal virtual trajectory motion sequence.

[0104] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working process of the described module can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0105] Furthermore, this invention also provides an electronic device for optimizing autonomous robot operation based on virtual simulation and multi-hypothesis planning, comprising a central processing unit (CPU), which can execute various appropriate actions and processes according to computer program instructions stored in read-only memory (ROM) or loaded from storage units into random access memory (RAM). The RAM can also store various programs and data required for device operation. The CPU, ROM, and RAM are interconnected via a bus. Input / output (I / O) interfaces are also connected to the bus.

[0106] Multiple components in the device are connected to the I / O interface, including: input units such as keyboards and mice; output units such as various types of displays and speakers; storage units such as disks and optical discs; and communication units such as network interface cards (NICs), modems, and wireless transceivers. The communication unit allows the device to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0107] The processing unit executes the various methods and processes described above, such as methods S1 to S5. For example, in some embodiments, methods S1 to S5 may be implemented as computer software programs tangibly contained in a machine-readable medium, such as a storage unit. In some embodiments, part or all of the computer program may be loaded and / or installed on the device via ROM and / or a communication unit. When the computer program is loaded into RAM and executed by the CPU, one or more steps of methods S1 to S5 described above may be performed. Alternatively, in other embodiments, the CPU may be configured to execute methods S1 to S5 by any other suitable means (e.g., by means of firmware).

[0108] The functions described above in this document can be performed, at least in part, by one or more hardware logic components. For example, exemplary types of hardware logic components that can be used, without limitation, include: Field Programmable Gate Arrays (FPGAs), Application-Specific Integrated Circuits (ASICs), Application Standard Products (ASSPs), System-on-Chip (SoCs), Complex Programmable Logic Devices (CPLDs), and so on.

[0109] The program code used to implement the methods of the present invention can be written in any combination of one or more programming languages. This program code can be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing device, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code can be executed entirely on the machine, partially on the machine, as a standalone software package partially on the machine and partially on a remote machine, or entirely on a remote machine or server.

[0110] In the context of this invention, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. Machine-readable media can include, but are not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0111] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in the present invention, and these modifications or substitutions should all be covered within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A method for optimizing autonomous robot operation based on virtual simulation and multi-hypothesis programming, characterized in that, The methods include: S1. Construct a virtual simulation platform scenario with the same configuration as the real experimental scenario. The scenario includes the scene environment, object state, and robot state. S2. In the virtual simulation platform scenario, reinforcement learning based on Markov decision is used to train the robot and generate an autonomous operation strategy. S3. In the virtual simulation platform scenario, multiple hypothetical states are constructed based on the robot's current state, and the multiple hypothetical states are deployed in the autonomous operation strategy to obtain the virtual trajectory action sequence of each hypothetical state. S4. Verify the feasibility of the virtual trajectory action sequence corresponding to each virtual state in sequence according to the time sequence. If it is feasible, take the virtual trajectory action sequence as the optimal virtual trajectory action sequence. If the virtual trajectory action sequence is not feasible, return to execute S3 to regenerate multiple hypothetical states. S5. Align the real robot state with the hypothetical state corresponding to the optimal virtual trajectory action sequence, and generate the robot's real action sequence in the real experimental scenario based on the current state and the optimal virtual trajectory action sequence using a cooperative action integration mechanism.

2. The robot autonomous operation optimization method based on virtual simulation and multi-hypothesis programming according to claim 1, characterized in that, The method for generating the aforementioned autonomous operation strategy is as follows: The reinforcement learning task of generating robot autonomous operation strategies is modeled as a Markov decision process. Where S represents the robot's state space, A represents the robot's action space, ρ is the initial state probability distribution, P is the transition probability function, r is the reward value, and γ is the discount factor; The RLPD reinforcement learning algorithm is used to train the robot to generate autonomous operation strategies in the virtual simulation platform. The loss function of the RLPD reinforcement learning algorithm includes a Q-value loss function. With policy loss function for: Among them, Q φ (s,a) represents the Q value of taking action a in state s; π represents the target Q value of taking action a′ in state s′; θ (·|s) represents the strategy π taken in state s. θ The probability of choosing action 'a'; Let r(s,a) represent the entropy of the policy; r(s,a) represent the reward for taking action a in state s; and α represent the hyperparameter.

3. The robot autonomous operation optimization method based on virtual simulation and multi-hypothesis programming according to claim 1, characterized in that, The method for constructing the hypothetical state is as follows: in, s represents the robot's first assumed state at time t; t Indicates the robot's state at time t; Represents the m-th hypothetical state at time t; ∈ m This represents a small range of random action values ​​for generating the m-th hypothetical state; M represents the total number of hypothetical states.

4. The robot autonomous operation optimization method based on virtual simulation and multi-hypothesis programming according to claim 1, characterized in that, The aforementioned collaborative action integration mechanism is as follows: S51, Based on the aforementioned real robot state Using the aforementioned autonomous operation strategy, generate realistic predicted actions. for: S52. Obtain the robot's virtual action at time t from the optimal virtual trajectory action sequence. Calculate the L2 norm of the difference between the virtual action and the actual predicted action; S53. If the second norm value is less than or equal to the determination parameter, then the action a deployed on the real robot... t The predicted action is the actual action; if the L2 value is greater than the decision parameter, then the action a deployed on the real robot is... t The average of the actual predicted action and the virtual action; S54. Obtain the real state of the robot at the next moment after it performs an action. Determine whether the maximum number of iterations is met. If not, align the real state at the next moment with the assumed state corresponding to the optimal virtual trajectory action sequence and then execute S51. Otherwise, end the iteration.

5. A robot autonomous operation optimization system based on virtual simulation and multi-hypothesis programming, characterized in that, The system includes: The environment setup module is used to build a robot self-service operation optimization platform. The robot self-service operation optimization platform includes a real experimental scenario and a virtual simulation platform scenario. The virtual simulation platform scenario is configured the same as the real experimental scenario. The scenario includes the scene environment, object state and robot state. The virtual reinforcement learning training module is used to train the robot in the virtual simulation platform scenario using reinforcement learning based on Markov decision-making to generate autonomous operation strategies. The hypothesis state generation and verification module is used to construct multiple hypothesis states based on the robot's current state, and deploy the multiple hypothesis states to the autonomous operation strategy to obtain the virtual trajectory action sequence for each hypothesis state; and to verify the feasibility of the virtual trajectory action sequence corresponding to each virtual state in sequence according to the time sequence. If it is feasible, the virtual trajectory action sequence is taken as the optimal virtual trajectory action sequence; if the virtual trajectory action sequence is not feasible, multiple hypothesis states are regenerated. The real motion generation module is used to align the real robot state with the hypothetical state corresponding to the optimal virtual trajectory motion sequence, and to generate the robot's real motion sequence in the real experimental scenario based on the current state and the optimal virtual trajectory motion sequence using a cooperative motion integration mechanism.

6. The robot autonomous operation optimization system based on virtual simulation and multi-hypothesis programming according to claim 5, characterized in that, The method for generating the aforementioned autonomous operation strategy is as follows: The reinforcement learning task of generating robot autonomous operation strategies is modeled as a Markov decision process. Where S represents the robot's state space, A represents the robot's action space, ρ is the initial state probability distribution, P is the transition probability function, r is the reward value, and γ is the discount factor; The RLPD reinforcement learning algorithm is used to train the robot to generate autonomous operation strategies in the virtual simulation platform. The loss function of the RLPD reinforcement learning algorithm includes a Q-value loss function. With policy loss function for: Among them, Q φ (s,a) represents the Q value of taking action a in state s; π represents the target Q value of taking action a′ in state s′; θ (·|s) represents the strategy π taken in state s. θ The probability of choosing action 'a'; Let r(s,a) represent the entropy of the policy; r(s,a) represent the reward for taking action a in state s; and α represent the hyperparameter.

7. The robot autonomous operation optimization system based on virtual simulation and multi-hypothesis programming according to claim 5, characterized in that, The method for constructing the hypothetical state is as follows: in, s represents the robot's first assumed state at time t; t Indicates the robot's state at time t; Represents the m-th hypothetical state at time t; ∈ m This represents a small range of random action values ​​for generating the m-th hypothetical state; M represents the total number of hypothetical states.

8. The robot autonomous operation optimization system based on virtual simulation and multi-hypothesis programming according to claim 5, characterized in that, The aforementioned collaborative action integration mechanism is as follows: S51, Based on the aforementioned real robot state Using the aforementioned autonomous operation strategy, generate realistic predicted actions. for: S52. Obtain the robot's virtual action at time t from the optimal virtual trajectory action sequence. Calculate the L2 norm of the difference between the virtual action and the actual predicted action; S53. If the second norm value is less than or equal to the determination parameter, then the action a deployed on the real robot... t The predicted action is the actual action; if the L2 value is greater than the decision parameter, then the action a deployed on the real robot is... t The average of the actual predicted action and the virtual action; S54. Obtain the real state of the robot at the next moment after it performs an action. Determine whether the maximum number of iterations is met. If not, align the real state at the next moment with the assumed state corresponding to the optimal virtual trajectory action sequence and then execute S51. Otherwise, end the iteration.

9. An electronic device for optimizing autonomous robot operation based on virtual simulation and multi-hypothesis planning, comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the program, it implements the method as described in any one of claims 1 to 4.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the method as described in any one of claims 1 to 4.

Images