Surgical robot control inverse reinforcement learning method and device based on body intelligence

Abstract

The invention provides a surgical robot control inverse reinforcement learning method and device based on body intelligence, and the method comprises the steps: obtaining the recorded data of any object operating a mechanical arm of a surgical robot, and the recorded data comprise historical track data; the historical track data is preprocessed; based on an inverse reinforcement learning algorithm, speculating a corresponding reward function from the preprocessed historical trajectory data; and verifying the reward function based on a reinforcement learning algorithm, and determining a final reward function after verification is passed. According to the method and the device, the historical record of the surgical robot is preprocessed, so that the corresponding reward function is speculated from the processed historical record based on the inverse reinforcement learning method.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and more specifically, to an inverse reinforcement learning method and device for surgical robot control based on embodied intelligence. Background Art

[0002] Embodied intelligence refers to intelligent entities that have bodies and support interaction with the physical world, such as robots and driverless cars. By processing multiple sensor data inputs, a control center such as a large model generates motion commands to drive the intelligent entity, replacing the traditional motion-driving method based on rules or mathematical formulas, and achieving a deep integration of virtual and reality.

[0003] The surgical robot can be used as a perfect carrier of embodied intelligence to realize the functions of the surgical robot based on embodied intelligence. In the specific implementation process, the past experience of operating the surgical robot can be summarized first, and then a new control center can be generated based on the summarized experience.

[0004] However, how to summarize past surgical robots is a difficult problem to solve at present. Summary of the invention

[0005] The problem that this application solves is that it is currently difficult to summarize past surgical robots.

[0006] To solve the above problems, the first aspect of the present application provides an inverse reinforcement learning method for surgical robot control based on embodied intelligence, comprising:

[0007] Acquire recorded data of any subject operating a surgical robot arm, wherein the recorded data includes historical trajectory data;

[0008] Preprocessing the historical trajectory data;

[0009] Based on the inverse reinforcement learning algorithm, inferring the corresponding reward function from the preprocessed historical trajectory data;

[0010] The reward function is verified based on a reinforcement learning algorithm, and a final reward function is determined after the verification passes.

[0011] The second aspect of the present application provides an inverse reinforcement learning device for surgical robot control based on embodied intelligence, which comprises:

[0012] A data acquisition module, which is used to acquire recorded data of any subject operating a surgical robot arm, wherein the recorded data includes historical trajectory data;

[0013] A data preprocessing module, which is used to preprocess the historical trajectory data;

[0014] A reward inference module, which is used to infer a corresponding reward function from the preprocessed historical trajectory data based on an inverse reinforcement learning algorithm;

[0015] The reward verification module is used to verify the reward function based on the reinforcement learning algorithm and determine the final reward function after the verification is passed.

[0016] A third aspect of the present application provides an electronic device, comprising: a memory and a processor;

[0017] The memory is used to store programs;

[0018] The processor, coupled to the memory, is configured to execute the program to:

[0019] Acquire recorded data of any subject operating a surgical robot arm, wherein the recorded data includes historical trajectory data;

[0020] Preprocessing the historical trajectory data;

[0021] Based on the inverse reinforcement learning algorithm, inferring the corresponding reward function from the preprocessed historical trajectory data;

[0022] The reward function is verified based on a reinforcement learning algorithm, and a final reward function is determined after the verification passes.

[0023] A fourth aspect of the present application provides a computer-readable storage medium having a computer program stored thereon, wherein the program is executed by a processor to implement the above-mentioned inverse reinforcement learning method for surgical robot control based on embodied intelligence.

[0024] In the present application, the historical records of the surgical robot are preprocessed, and then based on the inverse reinforcement learning method, the corresponding reward function is inferred from the processed historical records, and the reward function is a summary of past experience; thereby completing the data generation from data to reward function and achieving a perfect summary of the data. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Figure 1 A schematic diagram of the motion of the joints of the robotic arm of a surgical robot controlled by an inverse reinforcement learning method based on embodied intelligence according to an embodiment of the present application;

[0026] Figure 2 It is a flow chart of an inverse reinforcement learning method for controlling a surgical robot based on embodied intelligence according to an embodiment of the present application;

[0027] Figure 3 A flowchart of reward calculation of an inverse reinforcement learning method for surgical robot control based on embodied intelligence according to an embodiment of the present application;

[0028] Figure 4 It is a structural block diagram of an inverse reinforcement learning device for controlling a surgical robot based on embodied intelligence according to an embodiment of the present application;

[0029] Figure 5 is a structural block diagram of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0030] In order to make the above-mentioned purposes, features and advantages of the present application more obvious and easy to understand, the specific embodiments of the present application are described in detail below in conjunction with the accompanying drawings. Although the exemplary embodiments of the present application are shown in the accompanying drawings, it should be understood that the present application can be implemented in various forms and should not be limited by the embodiments described herein. On the contrary, these embodiments are provided in order to enable a more thorough understanding of the present application and to fully convey the scope of the present application to those skilled in the art.

[0031] It should be noted that, unless otherwise specified, the technical terms or scientific terms used in this application should have the common meanings understood by technicians in the field to which this application belongs.

[0032] In response to the above problems, the present application provides a new inverse reinforcement learning solution for surgical robot control, which solves the current problem that it is difficult to summarize past surgical robots through inverse reinforcement learning.

[0033] The present application embodiment provides an inverse reinforcement learning method for surgical robot control based on embodied intelligence. The specific scheme of the method is as follows: Figure 1-Figure 3 As shown, the method can be performed by an inverse reinforcement learning device for controlling a surgical robot based on embodied intelligence, and the inverse reinforcement learning device for controlling a surgical robot based on embodied intelligence can be integrated into electronic devices such as computers, servers, computers, server clusters, and data centers. Figure 1 , Figure 2 As shown, it is a flow chart of an inverse reinforcement learning method for controlling a surgical robot based on embodied intelligence according to an embodiment of the present application; wherein the inverse reinforcement learning method for controlling a surgical robot based on embodied intelligence comprises:

[0034] S101, obtaining recorded data of any subject operating a mechanical arm of a surgical robot, wherein the recorded data includes historical trajectory data;

[0035] In one implementation, the robotic arm is a six-axis robotic arm, and the historical trajectory data is angle change data of six motors corresponding to the six-axis robotic arm over time.

[0036] In one embodiment, the recorded data also includes operating environment data; and the historical trajectory data is the motion data of the robotic arm under compliant control.

[0037] Among them, compliance control is a control strategy mainly used in the operation of robots and manipulators to achieve soft interaction with the environment. It allows the robot to show "compliance" when it comes into contact with an uncertain or dynamically changing environment, thereby avoiding applying excessive force or causing damage. In this application, compliance control refers to the active control of the surgical robot arm under the control of an expert doctor.

[0038] In the present application, the operating environment data may include application scene data captured by a camera during the operation of the robotic arm.

[0039] Preferably, the operating environment data also includes preoperative planning data of the robotic arm.

[0040] In this way, based on the operating environment data, the experience of expert data processing in preoperative planning can be learned to achieve a high degree of consistency with the experts and improve the accuracy of robotic arm control.

[0041] S102, preprocessing the historical trajectory data;

[0042] S103, inferring a corresponding reward function from the preprocessed historical trajectory data based on an inverse reinforcement learning algorithm;

[0043] S104, verifying the reward function based on a reinforcement learning algorithm, and determining a final reward function after the verification passes.

[0044] In this application, the planning corresponding to the historical trajectory data is trained by reinforcement learning based on the reward function. If the trajectory data obtained by training is highly similar to the historical trajectory data, the verification is passed; or the trajectory data obtained by training is judged by an expert doctor. If it is judged to be feasible, the verification is passed. Otherwise, the inverse reinforcement learning iteration is repeated.

[0045] In the present application, the historical records of the surgical robot are preprocessed, and then based on the inverse reinforcement learning method, the corresponding reward function is inferred from the processed historical records, and the reward function is a summary of past experience; thereby completing the data generation from data to reward function and achieving a perfect summary of the data.

[0046] In one embodiment, after determining the final reward function, the method further includes: training an action execution strategy in an application scenario through reinforcement learning based on the final reward function.

[0047] In this application, the final reward function is obtained through inverse reinforcement learning to achieve a perfect summary of the data; then reinforcement learning training is performed through the final reward function, so as to apply the perfect summary of the data to the application scenarios of the robotic arm and achieve efficient iteration of the robotic arm.

[0048] Preferably, in the present application, the action sequence of the joint motor is determined according to the action execution strategy, and the corresponding robot arm end path is obtained through the action sequence; the end path is the robot arm path that applies the summarized experience.

[0049] It should be noted that the application of the final reward function can be carried out according to actual conditions and will not be repeated in this application.

[0050] In one implementation, the preprocessing of the historical trajectory data includes:

[0051] Determine a unit time point according to a maximum value of the angle change data;

[0052] Divide the angle change data based on unit time points;

[0053] Based on the segmented angle change data, the angle value of the corresponding motor at each unit time point is determined.

[0054] Among them, the historical trajectory data includes the angle change data of the motor over time; based on the angle change data, first select the measurement unit, and count the angle change data of all joint motors over time, and count the shortest time required for the joint motor to change by one unit (the unit can be selected according to actual conditions, or it can be set by itself to facilitate accuracy constraints). The shortest time is used as the unit time point. In this way, the angle change of all motors within the unit time point does not exceed 1.

[0055] Among them, after determining the unit time point, the angle change data is divided with the unit time point as the duration; each unit time corresponds to an angle change data and a current angle value (determined by the angle value at the starting point or end point of the time point); the current angle value corresponding to each time point is processed by approximation or rounding down to obtain an approximate angle value, which is an integer, and the angle values ​​of adjacent time points differ by ±1 or 0.

[0056] In this way, after preprocessing, it is easy to obtain the corresponding action and state of each joint motor.

[0057] In the present application, by preprocessing the historical trajectory data, the continuous joint motor movements are converted into discrete movements that are easy to identify and use, thereby greatly reducing the number of movements.

[0058] It should be noted that in this application, if the motor action of each joint of the six-axis robot is set to three actions of ±1 or 0, then all the action combinations of the six motors are three to the sixth power. This will result in an overly large action space in the entire reinforcement learning and inverse reinforcement learning, and require longer training time.

[0059] In one implementation, the unit time point is divided twice, and each unit time point is divided into six virtual time points, which respectively execute the motor actions from the first joint to the end joint.

[0060] That is to say, at the first virtual time point, the motor action of the first joint is executed (the motor action is the action at the corresponding unit time point), but the motors of other joints remain unchanged; other virtual time points remain unchanged. In this way, each virtual time point corresponds to three actions, and six virtual time points correspond to 18 actions, which greatly reduces the combination of action space.

[0061] Preferably, after each unit time point is divided into six virtual time points, the action corresponding to each virtual time point is determined, and the state corresponding to each virtual time point is generated through the action and the state.

[0062] In the present application, the state can be split into the operating environment state and the robotic arm state. After each unit time point is divided into six virtual time points, the operating environment state remains unchanged (it is set to unchanged), and the robotic arm state changes accordingly with the motor action, thereby combining the changed robotic arm state and the unchanged operating environment state into a new state corresponding to the virtual time point.

[0063] Preferably, a reward function is determined based on the preprocessed inverse reinforcement learning method. After a new action sequence is obtained by training through a new reinforcement learning method based on the reward function, the actions corresponding to six consecutive virtual time points need to be synthesized into an action corresponding to a unit time point, so as to make the actions of the robotic arm coherent.

[0064] In one embodiment, in combination Figure 3 As shown,

[0065] The step S103, inferring a corresponding reward function from the pre-processed historical trajectory data based on an inverse reinforcement learning algorithm, includes:

[0066] S301, constructing a simulation space based on the operating environment data;

[0067] In the present application, the simulation space may be a virtual three-dimensional space for performing inverse reinforcement learning in the simulation space.

[0068] S302, constructing a state space based on the operating environment data and historical trajectory data;

[0069] In this application, the state of the robot arm can be represented by the angles of each joint. If the robot arm has n joints, then the state space is n-dimensional. It can also include data such as the position of the end effector.

[0070] Preferably, the overall state of the robotic arm includes the operating environment state determined by the operating environment data and the robotic arm state determined by the historical trajectory data, thereby performing the most complete simulation of the state of the robotic arm and achieving maximum correlation between the state and the environment.

[0071] In the state space, the state of the robot can be represented as a vector.

[0072] S303, setting an action space, wherein the action space includes a plurality of joint axes, each of which has a forward rotation action and a reverse rotation action;

[0073] In the action space, the robot arm action represents the change in the joint angle of the robot arm at each time step, which can also be used as a vector.

[0074] S304, constructing a complete running trajectory based on the state space and the action space;

[0075] The running trajectory is composed of a series of states and action sequences.

[0076] S305, constructing a reward function model;

[0077] The reward function model is a nonlinear function, so the model is iterated by adjusting the weight parameters in subsequent iterations.

[0078] S306, modeling the maximum entropy of the trajectory probability to obtain a maximum entropy model;

[0079] In one embodiment, the maximum entropy model is:

[0080]

[0081] Among them, τ is the trajectory, is a possible trajectory, R(s t ,a t ) is to execute action a t Get status s t The reward value is the normalized function, To perform actions Get Status The reward value, are the possible states and possible actions corresponding to the tth state and action, and P(τ) is the probability of trajectory τ.

[0082] In this application, the calculation of maximum entropy involves the summation of all possible trajectories.

[0083] S307, iteratively optimizing the parameters of the reward function model by maximizing the log-likelihood of the expert trajectory to obtain the optimal reward function parameters.

[0084] In this application, the calculation of maximum entropy involves the sum of all possible trajectories. Therefore, the optimization is performed by maximizing the log-likelihood of the expert trajectory.

[0085] Among them, the log-likelihood of the expert trajectory is determined, and it is optimized towards the maximum value by gradient descent or gradient ascent optimization until the optimization is completed.

[0086] In this application, the specific optimization process of gradient descent or ascent will not be described in detail.

[0087] In this application, the expert trajectory is the constructed complete operation trajectory.

[0088] In this application, once the optimal reward function parameters are learned, a reinforcement learning algorithm (such as value iteration and policy iteration) can be used to learn the optimal strategy under the reward function.

[0089] Specifically, this reward function is used to optimize the robot’s trajectory to maximize the cumulative reward, thereby generating new trajectories that are similar to the expert’s demonstrated trajectories and can perform similar tasks.

[0090] In one implementation, the reward function model is:

[0091] R(s t ,a t )=α·exp(-‖p t -g‖ 2 )-β·‖Δq t ‖ 2 -γ·exp(‖u t ‖ 2 )

[0092] Among them, R is the reward value, a t is the t-th action, s t is the tth state, α, β, γ are weight coefficients, p t is the position of the end effector of the robot arm, g is the target position, Δq t is the tth comprehensive change value of the joint angle, u t is the joint angle vector.

[0093] In this application, the reward function model is used to consider the target proximity, motion smoothness and energy efficiency of the robot arm movement, and the robot arm is encouraged to get as close to the target object as possible while maintaining smooth motion and low energy consumption.

[0094] Target proximity: Encourages the end effector to get close to the target object.

[0095] Movement smoothness: Penalizes drastic joint angle changes to keep the robot's movements smooth.

[0096] Energy Efficiency: Penalizes high energy consumption.

[0097] By optimizing the reward function, a robot arm motion trajectory can be learned based on the expert trajectory, which can effectively approach the target object while maintaining smooth movements and saving energy.

[0098] An embodiment of the present application provides an inverse reinforcement learning device for surgical robot control based on embodied intelligence, which is used to execute the inverse reinforcement learning method for surgical robot control based on embodied intelligence described in the above content of the present application. The inverse reinforcement learning device for surgical robot control based on embodied intelligence is described in detail below.

[0099] like Figure 4 As shown, the surgical robot control inverse reinforcement learning device based on embodied intelligence includes:

[0100] A data acquisition module 101 is used to acquire recorded data of any subject operating a surgical robot arm, wherein the recorded data includes historical trajectory data;

[0101] A data preprocessing module 102, which is used to preprocess the historical trajectory data;

[0102] A reward inference module 103, which is used to infer a corresponding reward function from the pre-processed historical trajectory data based on an inverse reinforcement learning algorithm;

[0103] The reward verification module 104 is used to verify the reward function based on the reinforcement learning algorithm and determine the final reward function after the verification is passed.

[0104] In one implementation, the robotic arm is a six-axis robotic arm, and the historical trajectory data is angle change data of six motors corresponding to the six-axis robotic arm over time.

[0105] In one embodiment, the data preprocessing module 102 is further used to:

[0106] Determine a unit time point according to the maximum value of the angle change data; divide the angle change data based on the unit time point; and determine the angle value of the motor corresponding to each unit time point based on the divided angle change data.

[0107] In one embodiment, the recorded data also includes operating environment data; and the historical trajectory data is the motion data of the robotic arm under compliant control.

[0108] In one embodiment, the reward speculation module 103 is further used to:

[0109] Based on the operating environment data, a simulation space is constructed; based on the operating environment data and the historical trajectory data, a state space is constructed; an action space is set, wherein the action space includes multiple joint axes, each joint axis has a forward rotation action and a reverse rotation action; based on the state space and the action space, a complete operating trajectory is constructed; a reward function model is constructed; the maximum entropy of the trajectory probability is modeled to obtain a maximum entropy model; by maximizing the log-likelihood of the expert trajectory, the parameters of the reward function model are iteratively optimized to obtain the optimal reward function parameters.

[0110] In one implementation, the reward function model is:

[0111] R(s t ,a t )=α·exp(-‖p t -g‖ 2 )-β·‖Δq t ‖ 2 -γ·exp(‖u t ‖ 2 )

[0112] Among them, R is the reward value, a t is the t-th action, s t is the tth state, α, β, γ are weight coefficients, p t is the position of the end effector of the robot arm, g is the target position, Δq t is the tth comprehensive change value of the joint angle, u t is the joint angle vector.

[0113] In one embodiment, the maximum entropy model is:

[0114]

[0115] Among them, τ is the trajectory, is a possible trajectory, R(s t ,a t ) is to execute action a t Get status s t The reward value is the normalized function, To perform actions Get Status The reward value, are the possible states and possible actions corresponding to the tth state and action, and P(τ) is the probability of trajectory τ.

[0116] The inverse reinforcement learning device for surgical robot control based on embodied intelligence provided in the above-mentioned embodiments of the present application has a corresponding relationship with the inverse reinforcement learning method for surgical robot control based on embodied intelligence provided in the embodiments of the present application. Therefore, the specific content in the device has a corresponding relationship with the inverse reinforcement learning method for surgical robot control based on embodied intelligence. The specific content can be referred to the records in the inverse reinforcement learning method for surgical robot control based on embodied intelligence, and will not be repeated in this application.

[0117] The inverse reinforcement learning device for surgical robot control based on embodied intelligence provided in the above-mentioned embodiments of the present application and the inverse reinforcement learning method for surgical robot control based on embodied intelligence provided in the embodiments of the present application are based on the same inventive concept and have the same beneficial effects as the methods adopted, run or implemented by the application programs stored therein.

[0118] The above describes the internal functions and structure of the inverse reinforcement learning device for surgical robot control based on embodied intelligence, such as Figure 5 As shown, in practice, the surgical robot control inverse reinforcement learning device based on embodied intelligence can be implemented as an electronic device, including: a memory 301 and a processor 303.

[0119] The memory 301 may be configured to store programs.

[0120] In addition, the memory 301 may also be configured to store various other data to support operations on the electronic device. Examples of such data include instructions for any application or method operating on the electronic device, contact data, phone book data, messages, pictures, videos, etc.

[0121] The memory 301 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.

[0122] The processor 303 is coupled to the memory 301 and is configured to execute a program in the memory 301 to:

[0123] Acquire recorded data of any subject operating a surgical robot arm, wherein the recorded data includes historical trajectory data;

[0124] Preprocessing the historical trajectory data;

[0125] Based on the inverse reinforcement learning algorithm, inferring the corresponding reward function from the preprocessed historical trajectory data;

[0126] The reward function is verified based on a reinforcement learning algorithm, and a final reward function is determined after the verification passes.

[0127] In one implementation, the robotic arm is a six-axis robotic arm, and the historical trajectory data is angle change data of six motors corresponding to the six-axis robotic arm over time.

[0128] In one implementation, the processor 303 is further configured to:

[0129] Determine a unit time point according to the maximum value of the angle change data; divide the angle change data based on the unit time point; and determine the angle value of the motor corresponding to each unit time point based on the divided angle change data.

[0130] In one embodiment, the recorded data also includes operating environment data; and the historical trajectory data is the motion data of the robotic arm under compliant control.

[0131] In one implementation, the processor 303 is further configured to:

[0132] Based on the operating environment data, a simulation space is constructed; based on the operating environment data and the historical trajectory data, a state space is constructed; an action space is set, wherein the action space includes multiple joint axes, each joint axis has a forward rotation action and a reverse rotation action; based on the state space and the action space, a complete operating trajectory is constructed; a reward function model is constructed; the maximum entropy of the trajectory probability is modeled to obtain a maximum entropy model; by maximizing the log-likelihood of the expert trajectory, the parameters of the reward function model are iteratively optimized to obtain the optimal reward function parameters.

[0133] In one implementation, the reward function model is:

[0134] R(s t ,a t )=α·exp(-‖p t -g‖ 2 )-β·‖Δq t ‖ 2 -γ·exp(‖u t ‖ 2 )

[0135] Among them, R is the reward value, a t is the t-th action, s t is the tth state, α, β, γ are weight coefficients, p t is the position of the end effector of the robot arm, g is the target position, Δq t is the tth comprehensive change value of the joint angle, u t is the joint angle vector.

[0136] In one embodiment, the maximum entropy model is:

[0137]

[0138]

[0139] Among them, τ is the trajectory, is a possible trajectory, R(s t ,a t ) is to execute action a t Get status s t The reward value is the normalized function, To perform actions Get Status The reward value, are the possible states and possible actions corresponding to the tth state and action, and P(τ) is the probability of trajectory τ.

[0140] In the present application, the processor is also specifically used to execute all processes and steps of the above-mentioned surgical robot control inverse reinforcement learning method based on embodied intelligence. The specific content can be referred to the records in the surgical robot control inverse reinforcement learning method based on embodied intelligence, and will not be repeated in this application.

[0141] In this application, Figure 5 Only some components are shown schematically, which does not mean that the electronic device only includes Figure 5 Components shown.

[0142] The electronic device provided in this embodiment is based on the same inventive concept as the inverse reinforcement learning method for surgical robot control based on embodied intelligence provided in the embodiment of the present application, and has the same beneficial effects as the method adopted, run or implemented by the application stored in it.

[0143] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-readable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.

[0144] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0145] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0146] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0147] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0148] The memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash memory (Flash RAM). The memory is an example of a computer-readable medium.

[0149] The present application also provides a computer-readable storage medium corresponding to the inverse reinforcement learning method for surgical robot control based on embodied intelligence provided in the aforementioned embodiments, on which a computer program (i.e., a program product) is stored. When the computer program is run by a processor, it will execute the inverse reinforcement learning method for surgical robot control based on embodied intelligence provided in any of the aforementioned embodiments.

[0150] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0151] The computer-readable storage medium provided in the above-mentioned embodiments of the present application and the inverse reinforcement learning method for surgical robot control based on embodied intelligence provided in the embodiments of the present application are based on the same inventive concept and have the same beneficial effects as the methods adopted, run or implemented by the application programs stored therein.

[0152] It should be noted that in the description provided herein, a large number of specific details are described. However, it is understood that the embodiments of the present application can be practiced without these specific details. In some instances, well-known structures and technologies are not shown in detail so as not to obscure the understanding of this description.

[0153] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0154] The above is only an embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included in the scope of the claims of the present application.

Claims

1. A surgical robot control inverse reinforcement learning method based on embodied intelligence, characterized in that: include: Acquire recorded data of any subject operating a surgical robot arm, wherein the recorded data includes historical trajectory data; Preprocessing the historical trajectory data; Based on the inverse reinforcement learning algorithm, inferring the corresponding reward function from the preprocessed historical trajectory data; The reward function is verified based on a reinforcement learning algorithm, and a final reward function is determined after the verification passes.

2. The inverse reinforcement learning method for surgical robot control according to claim 1, characterized in that: The robotic arm is a six-axis robotic arm, and the historical trajectory data is angle change data of six motors corresponding to the six-axis robotic arm over time.

3. The inverse reinforcement learning method for surgical robot control according to claim 2, characterized in that: The preprocessing of the historical trajectory data includes: Determine a unit time point according to a maximum value of the angle change data; Divide the angle change data based on unit time points; Based on the segmented angle change data, the angle value of the corresponding motor at each unit time point is determined.

4. The inverse reinforcement learning method for surgical robot control according to any one of claims 1 to 3, characterized in that: The recorded data also includes operating environment data; the historical trajectory data is the motion data of the robot arm under compliant control; After determining the final reward function, it also includes: based on the final reward function, training the action execution strategy in the application scenario through reinforcement learning.

5. The inverse reinforcement learning method for surgical robot control according to claim 4, characterized in that: The inference of the corresponding reward function from the preprocessed historical trajectory data based on the inverse reinforcement learning algorithm includes: Build simulation space based on operating environment data; Construct state space based on operating environment data and historical trajectory data; Setting an action space, wherein the action space includes a plurality of joint axes, each joint axis having a forward rotation action and a reverse rotation action; Based on the state space and action space, a complete running trajectory is constructed; Construct a reward function model; Model the maximum entropy of trajectory probability to obtain the maximum entropy model; By maximizing the log-likelihood of the expert trajectory, the parameters of the reward function model are iteratively optimized to obtain the optimal reward function parameters.

6. The inverse reinforcement learning method for surgical robot control according to claim 5, characterized in that: The reward function model is: R(s t ,a t )=α·exp(-‖p t -g‖ 2 )-β·‖Δq t ‖ 2 -γ·exp(‖u t ‖ 2 ) Among them, R is the reward value, a t is the tth action, s t is the tth state, α, β, γ are weight coefficients, p t is the position of the end effector of the robot arm, g is the target position, Δq t is the tth comprehensive change value of the joint angle, u t is the joint angle vector.

7. The inverse reinforcement learning method for surgical robot control according to claim 5, characterized in that: The maximum entropy model is: Among them, τ is the trajectory, is a possible trajectory, R(s t ,a t ) is to execute action a t Get status s t The reward value is the normalized function, To perform actions Get Status The reward value, are the possible states and possible actions corresponding to the tth state and action, and P(τ) is the probability of trajectory τ.

8. A surgical robot control inverse reinforcement learning device based on embodied intelligence, characterized in that: include: A data acquisition module, which is used to acquire recorded data of any subject operating a surgical robot arm, wherein the recorded data includes historical trajectory data; A data preprocessing module, which is used to preprocess the historical trajectory data; A reward inference module, which is used to infer a corresponding reward function from the preprocessed historical trajectory data based on an inverse reinforcement learning algorithm; The reward verification module is used to verify the reward function based on the reinforcement learning algorithm and determine the final reward function after the verification is passed.

9. An electronic device, characterized in that: include: Memory and processor; The memory is used to store programs; The processor, coupled to the memory, is configured to execute the program to: Acquire recorded data of any subject operating a surgical robot arm, wherein the recorded data includes historical trajectory data; Preprocessing the historical trajectory data; Based on the inverse reinforcement learning algorithm, inferring the corresponding reward function from the preprocessed historical trajectory data; The reward function is verified based on a reinforcement learning algorithm, and a final reward function is determined after the verification passes.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: The program is executed by the processor to implement the inverse reinforcement learning method for controlling a surgical robot based on embodied intelligence as described in any one of claims 1 to 7.

Images