Multi-mode comprehensive control method and device for intelligent mechanical arm of surgical robot

Abstract

The invention provides an intelligent mechanical arm multi-mode comprehensive control method and device for a surgical robot. The intelligent mechanical arm multi-mode comprehensive control method comprises the steps that a first reward function based on inverse reinforcement learning processing is obtained; acquiring a target position and obstacle information; constructing a second reward function based on the target position and a third reward function based on the obstacle information; according to the first reward function, the second reward function and the third reward function, the operation strategy of the mechanical arm is trained through reinforcement learning; and selecting the moving trajectory with the highest accumulated reward, and controlling the mechanical arm to execute the moving trajectory. In the application, a first reward function is generated through historical operation experience, and a second reward function and a third reward function are generated through visual obstacle avoidance; and through a reinforcement learning mode, an overall operation strategy based on different reward functions is realized, so that comprehensive control of the mechanical arm based on historical operation experience and visual obstacle avoidance is realized.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and more specifically, to a multi-mode integrated control method and device for a robotic arm of an embodied intelligent surgical robot. Background Art

[0002] Embodied intelligence refers to intelligent entities that have bodies and support interaction with the physical world, such as robots and driverless cars. By processing multiple sensor data inputs, a control center such as a large model generates motion commands to drive the intelligent entity, replacing the traditional motion-driving method based on rules or mathematical formulas, and achieving a deep integration of virtual and reality.

[0003] The surgical robot can be used as a perfect carrier of embodied intelligence to realize the functions of the surgical robot based on embodied intelligence. In the specific implementation process, the past experience of operating surgical robots can be summarized first, and then the robotic arm of the surgical robot can be controlled in combination with visual obstacle avoidance methods.

[0004] However, historical operating experience and visual obstacle avoidance are different control modes, and it is difficult to achieve integrated control. Summary of the invention

[0005] The problem solved by this application is that it is currently difficult to achieve comprehensive control of a robotic arm.

[0006] To solve the above problems, the first aspect of the present application provides a multi-mode integrated control method for a robotic arm of a surgical robot with embodied intelligence, comprising:

[0007] Obtaining a first reward function based on inverse reinforcement learning processing;

[0008] Acquiring target position and obstacle information, wherein the obstacle information includes static information and dynamic information;

[0009] Constructing a second reward function based on the target position and a third reward function based on obstacle information;

[0010] According to the first reward function, the second reward function, and the third reward function, the operation strategy of the robot arm is trained through reinforcement learning;

[0011] After training is completed, the running trajectory with the highest cumulative reward is selected and the robot arm is controlled to execute the running trajectory.

[0012] The second aspect of the present application provides a multi-mode integrated control device for a robotic arm of a surgical robot with embodied intelligence, comprising:

[0013] An inverse reinforcement learning module, which is used to obtain a first reward function based on inverse reinforcement learning processing;

[0014] An obstacle acquisition module, which is used to acquire target position and obstacle information, wherein the obstacle information includes static information and dynamic information;

[0015] A reward building module, which is used to build a second reward function based on the target position and a third reward function based on obstacle information;

[0016] A reinforcement learning module, which is used to train the operation strategy of the robot arm through reinforcement learning according to the first reward function, the second reward function, and the third reward function;

[0017] The robotic arm control module is used to select the running trajectory with the highest cumulative reward after training is completed, and control the robotic arm to execute the running trajectory.

[0018] A third aspect of the present application provides an electronic device, comprising: a memory and a processor;

[0019] The memory is used to store programs;

[0020] The processor, coupled to the memory, is configured to execute the program to:

[0021] Obtaining a first reward function based on inverse reinforcement learning processing;

[0022] Acquiring target position and obstacle information, wherein the obstacle information includes static information and dynamic information;

[0023] Constructing a second reward function based on the target position and a third reward function based on obstacle information;

[0024] According to the first reward function, the second reward function, and the third reward function, the operation strategy of the robot arm is trained through reinforcement learning;

[0025] After training is completed, the running trajectory with the highest cumulative reward is selected and the robot arm is controlled to execute the running trajectory.

[0026] A fourth aspect of the present application provides a computer-readable storage medium having a computer program stored thereon, wherein the program is executed by a processor to implement the above-mentioned multi-mode integrated control method of a robotic arm for embodied intelligence of a surgical robot.

[0027] In this application, the first reward function is generated through historical operating experience, and the second and third reward functions are generated through visual obstacle avoidance. Through reinforcement learning, an overall operation strategy based on different reward functions is implemented, thereby realizing comprehensive control of the robotic arm based on historical operating experience and visual obstacle avoidance. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Figure 1is a flow chart of a multi-mode integrated control method for a robotic arm of a surgical robot with embodied intelligence according to an embodiment of the present application;

[0029] Figure 2 is a structural block diagram of a multi-mode integrated control device for a robotic arm of a surgical robot with embodied intelligence according to an embodiment of the present application;

[0030] Figure 3 is a structural block diagram of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0031] In order to make the above-mentioned purposes, features and advantages of the present application more obvious and easy to understand, the specific embodiments of the present application are described in detail below in conjunction with the accompanying drawings. Although the exemplary embodiments of the present application are shown in the accompanying drawings, it should be understood that the present application can be implemented in various forms and should not be limited by the embodiments described herein. On the contrary, these embodiments are provided in order to enable a more thorough understanding of the present application and to fully convey the scope of the present application to those skilled in the art.

[0032] It should be noted that, unless otherwise specified, the technical terms or scientific terms used in this application should have the common meanings understood by technicians in the field to which this application belongs.

[0033] The present application embodiment provides a multi-mode integrated control method for a robotic arm of a surgical robot with embodied intelligence. The specific scheme of the method is as follows: Figure 1 As shown, the method can be performed by a multi-mode integrated control device for a robotic arm with embodied intelligence for a surgical robot, and the multi-mode integrated control device for a robotic arm with embodied intelligence for a surgical robot can be integrated into electronic devices such as computers, servers, computers, server clusters, and data centers. Figure 1 As shown, it is a flow chart of a multi-mode integrated control method of a robotic arm for embodied intelligence of a surgical robot according to an embodiment of the present application; wherein the multi-mode integrated control method of a robotic arm for embodied intelligence of a surgical robot comprises:

[0034] S101, obtaining a first reward function based on inverse reinforcement learning processing;

[0035] S102, obtaining target position and obstacle information, wherein the obstacle information includes static information and dynamic information;

[0036] S103, constructing a second reward function based on the target position and a third reward function based on obstacle information;

[0037] S104, training the operation strategy of the robot arm through reinforcement learning according to the first reward function, the second reward function, and the third reward function;

[0038] S105, after the training is completed, select the running trajectory with the highest cumulative reward, and control the robotic arm to execute the running trajectory.

[0039] In this application, after the training is completed, the trained operation strategy is used to evaluate all possible operation trajectories, the trajectory with the highest cumulative reward is selected as the robot arm operation trajectory, and the robot arm is controlled to execute the trajectory.

[0040] In this application, the first reward function is generated through historical operating experience, and the second and third reward functions are generated through visual obstacle avoidance. Through reinforcement learning, an overall operation strategy based on different reward functions is implemented, thereby realizing comprehensive control of the robotic arm based on historical operating experience and visual obstacle avoidance.

[0041] In one embodiment, obtaining a first reward function based on inverse reinforcement learning processing includes:

[0042] Acquire recorded data of any object operating a robotic arm, wherein the recorded data includes historical trajectory data;

[0043] Preprocessing the historical trajectory data;

[0044] Based on the inverse reinforcement learning algorithm, inferring the corresponding reward function from the preprocessed historical trajectory data;

[0045] The reward function is verified based on a reinforcement learning algorithm, and a final reward function is determined after the verification passes.

[0046] In one implementation, the robotic arm is a six-axis robotic arm, and the historical trajectory data is angle change data of six motors corresponding to the six-axis robotic arm over time.

[0047] In one embodiment, the recorded data also includes operating environment data; and the historical trajectory data is the motion data of the robotic arm under compliant control.

[0048] In one embodiment, the inference of the corresponding reward function from the preprocessed historical trajectory data based on the inverse reinforcement learning algorithm includes:

[0049] Build simulation space based on operating environment data;

[0050] Construct state space based on operating environment data and historical trajectory data;

[0051] Setting an action space, wherein the action space includes a plurality of joint axes, each joint axis having a forward rotation action and a reverse rotation action;

[0052] Based on the state space and action space, a complete running trajectory is constructed;

[0053] Construct a reward function model;

[0054] Model the maximum entropy of trajectory probability to obtain the maximum entropy model;

[0055] By maximizing the log-likelihood of the expert trajectory, the parameters of the reward function model are iteratively optimized to obtain the optimal reward function parameters.

[0056] In one implementation, the reward function model is:

[0057] R(s t ,a t )=α·exp(-‖p t -g‖ 2 )-β·‖Δq t ‖ 2 -γ·exp(‖u t ‖ 2 )

[0058] Among them, R is the reward value, a t is the t-th action, s t is the tth state, α, β, γ are weight coefficients, p t is the position of the end effector of the robot arm, g is the target position, Δq t is the tth comprehensive change value of the joint angle, u t is the joint angle vector.

[0059] In one embodiment, the maximum entropy model is:

[0060]

[0061]

[0062] Among them, τ is the trajectory, is a possible trajectory, R(s t ,a t ) is to execute action a t Get status s t The reward value is the normalized function, To perform actions Get Status The reward value, are the possible states and possible actions corresponding to the tth state and action, and P(τ) is the probability of trajectory τ.

[0063] In one embodiment, the step of acquiring and parsing a real surgical scene video stream to obtain target position and obstacle information includes:

[0064] Acquire a surgical scene video stream and segment it into continuously set depth image frames;

[0065] Inputting the depth image frame into an obstacle recognition model to obtain an obstacle recognition result; the obstacle recognition result includes static obstacles, dynamic obstacles and target positions;

[0066] The continuous depth image frames with labeled obstacle recognition results are input into the spatiotemporal prediction model to obtain the spatial motion trajectory of the dynamic obstacle.

[0067] In one implementation, inputting the continuous depth image frames with the obstacle recognition results annotated into the spatiotemporal prediction model to obtain the obstacle prediction results includes:

[0068] Based on the obstacle recognition results, the key points of the dynamic obstacles are marked in the depth image frame;

[0069] Generate a continuous feature map according to the position key points of continuous depth image frames;

[0070] The continuous feature map is input into a spatiotemporal prediction model to obtain the spatial motion trajectory of the dynamic obstacle.

[0071] In one embodiment, the spatiotemporal prediction model is a gated dilated causal convolutional neural network.

[0072] In one implementation, generating a continuous feature map according to the position key points of continuous depth image frames includes:

[0073] Constructing a domain space graph structure according to the position key points;

[0074] Based on the correlation between the key points of the same dynamic obstacle, a weighted adjacency matrix of the domain space graph structure is constructed;

[0075] According to the domain space graph structure of continuous depth image frames and the corresponding weighted adjacency matrix, feature representations of the domain space graph structure at different times are generated, namely, the continuous feature graph.

[0076] In one embodiment, the operation strategy of the robotic arm is trained by reinforcement learning according to the first reward function, the second reward function, and the third reward function;

[0077] Generate a first control mode reward, a second control mode reward, and a third control mode reward according to the first reward function, the second reward function, and the third reward function;

[0078] Based on the reinforcement learning method, preliminary training is performed on the first control mode reward, the second control mode reward, and the third control mode reward;

[0079] Constructing switching thresholds between the first control mode reward, the second control mode reward, and the third control mode reward;

[0080] According to the switching threshold, the first control mode reward, the second control mode reward, and the third control mode reward are trained as a whole to obtain the operation strategy of the robot arm.

[0081] In the present application, the second reward function is constructed based on the target position, and its specific form can be the Euclidean distance between the coordinates of the tool at the end of the robot arm and the coordinates of the target position.

[0082] In the present application, the third reward function is constructed based on obstacle information, and its specific target form can be the minimum distance between the current end of the robotic arm and the obstacle; or the minimum distance between the robotic arm itself (including multiple joints) and the obstacle.

[0083] In the present application, the first reward function, the second reward function, and the third reward function are not directly used as the reward functions of the three control modes. Instead, the first control mode reward, the second control mode reward, and the third control mode reward are generated through the first reward function, the second reward function, and the third reward function. This is because the first reward function, the second reward function, and the third reward function are reward functions with a single target. Directly using them as reward functions of the control modes will result in large differences between the three control modes, making them difficult to integrate.

[0084] In the present application, the first control mode reward, the second control mode reward, and the third control mode reward are generated through the first reward function, the second reward function, and the third reward function, thereby integrating the first reward function, the second reward function, and the third reward function into the three control modes, greatly increasing the similarity between the three control modes and improving the connectivity of the comprehensive control.

[0085] In this application, switching thresholds are constructed between the first control mode reward, the second control mode reward, and the third control mode reward, so that smooth switching between the three control modes can be achieved through the switching thresholds, precise training and reward accumulation can be achieved, and the accuracy of comprehensive control can be greatly improved.

[0086] In one embodiment, the first control mode reward is determined by a second reward function and a third reward function, and the second control mode reward and the third control mode reward are both determined by the first reward function, the second reward function, and the third reward function as a whole.

[0087] In one embodiment, the generation formula of the first control mode reward is:

[0088] R1=αr1+r2

[0089] Among them, R1 is the first control mode reward, r1 is the first reward function, r2 is the second reward function, and α is the generation coefficient.

[0090] In the present application, by setting a generation coefficient before the first reward function, the relative proportion of the first reward function and the second reward function is adjusted to improve the generation effect.

[0091] In the present application, the generation coefficient α in the generation formula of the first control mode reward is preferably 0.3.

[0092] In one embodiment, the generation formula of the second control mode reward is:

[0093] R2=αr1+βr2+r3

[0094] Among them, R2 is the second control mode reward, r1 is the first reward function, r2 is the second reward function, r3 is the third reward function, and α and β are generation coefficients.

[0095] In one embodiment, the generation formula of the third control mode reward is:

[0096] R2=r1+βr2+γr3

[0097] Among them, R3 is the third control mode reward, r1 is the first reward function, r2 is the second reward function, r3 is the third reward function, and β and γ are generation coefficients.

[0098] It should be noted here that the generation formulas for different control mode rewards all have generation coefficients, but their specific values ​​are not the same. For example, the generation coefficient α in the generation formula for the first control mode reward is preferably 0.3, and the generation coefficient α in the generation formula for the second control mode reward is preferably 0.5.

[0099] It should be noted here that the value range of the generation coefficients α, β, and γ is 0-1.

[0100] In this application, the second control mode reward is generated by maximizing the third reward function, and the third control mode reward is generated by maximizing the first reward function, thereby greatly improving the proportional difference between the three control mode rewards and maintaining the similarities and differences of the three controls.

[0101] In one embodiment, the preliminary training of the first control mode reward based on the reinforcement learning method includes:

[0102] Construct simulation space based on target location and obstacle information;

[0103] According to the parameters of the robot arm, a state space and an action space are constructed; the action space includes a plurality of joint axes, each of which has a forward rotation action and a reverse rotation action;

[0104] Constructing a deep Q network model and a first experience pool; the first control mode reward, the second control mode reward, and the third control mode reward have corresponding first experience pools, second experience pools, and third experience pools;

[0105] Taking the first control mode reward as the reward function, sampling from the experience replay pool, and calculating the Q value based on the deep Q network model;

[0106] By minimizing the loss function, the weights of the deep Q network model are updated until the predetermined number of iterations is reached.

[0107] In this application, the specific process of deep Q reinforcement learning and the setting of the corresponding deep Q network model can be carried out with reference to the prior art and will not be repeated in this application.

[0108] In this application, a simulation space is constructed: based on the target location and obstacle information, a simulation environment is created for training and testing the robotic arm. This can be a 3D simulation environment that reflects the mission scenario in the real world.

[0109] In this application, the state space and action space are constructed:

[0110] State space: The state space is constructed based on the current state of the robot (e.g., joint angles, speeds, positions) and environmental information (target position, obstacle positions).

[0111] Action space: defines the action space of the robot, which contains multiple joint axes. Each joint axis has two actions: forward rotation and reverse rotation, forming a discrete action set.

[0112] In this application, a deep Q network model and experience pool are constructed:

[0113] Deep Q Network Model: Design a deep Q network (DQN) to estimate the Q value of each action taken in a given state.

[0114] Experience pool: Create experience pools for the first control mode reward, the second control mode reward, and the third control mode reward. These experience pools are used to store the agent's experience for subsequent sampling and training.

[0115] In this application, sampling and calculation of Q value:

[0116] Use the first control mode reward as the reward function and sample experience (state, action, reward, next state) from the first experience pool.

[0117] The Q value is calculated based on the deep Q network model to evaluate the expected return of taking a certain action in the current state.

[0118] In this application, the deep Q network model is updated:

[0119] Define the loss function, usually the mean square error (MSE) of the Q value, and calculate the gap between the network's predicted Q value and the target Q value.

[0120] The loss function is minimized through the back-propagation algorithm and the weights of the deep Q-network model are updated.

[0121] Repeat the above steps until the predetermined number of iterations is reached.

[0122] It should be noted that in the present application, the preliminary training of the first control mode reward based on the reinforcement learning method is similar to the preliminary training of the second control mode and the third control mode. The difference is that the corresponding second experience pool and the third experience pool are selected for training; and the corresponding loss functions are different.

[0123] Preferably, in the present application, a task mode is reserved as a buffer between obstacle avoidance (third reward function, second control mode) and other tasks, and in addition to switching modes through thresholds, switching can also be performed through reward accumulation thresholds (that is, the switching threshold is a cumulative threshold, and the threshold is accumulated after each training calculation until it reaches a predetermined value before switching). The reward accumulation threshold can leave sufficient buffer space to prevent obstacle avoidance from affecting other tasks to the greatest extent, and can react quickly when obstacles arrive quickly and switch to obstacle avoidance mode quickly.

[0124] In one embodiment, the switching threshold includes a first switching threshold and a second switching threshold; and according to the switching threshold, the first control mode reward, the second control mode reward, and the third control mode reward are trained as a whole to obtain the operation strategy of the robot arm, including:

[0125] Sampling training data from the first experience pool, the second experience pool, and the third experience pool according to a preset ratio;

[0126] Using the first control mode reward as a reward function, iterating the weights of the deep Q network model according to the training data;

[0127] Make real-time judgment on the current state of the robot arm;

[0128] When the current state of the robot arm satisfies the first switching threshold, taking the second control mode reward as the reward function, and iterating the weight of the deep Q network model according to the training data;

[0129] When the current state of the robot arm satisfies the second switching threshold, taking the third control mode reward as the reward function, and iterating the weight of the deep Q network model according to the training data;

[0130] When the number of iterations reaches the preset number or the loss function converges, the overall training is completed and the operation strategy of the robotic arm is obtained.

[0131] In the present application, the first switching threshold and the second switching threshold may be thresholds calculated by a specific calculation formula, or may be cumulative thresholds obtained by accumulating values ​​calculated by the calculation formula.

[0132] In the present application, the first control mode reward is used as the reward function, and the weights of the deep Q network model are iterated according to the training data, which is similar to the preliminary training of the first control mode reward based on the reinforcement learning method mentioned above, except that the number of iterations is limited by the switching threshold.

[0133] In the present application, the first switching threshold is used to determine when to switch from the first control mode to the second control mode.

[0134] In the present application, the second switching threshold is used to determine when to switch from the second control mode to the third control mode.

[0135] In this application, each control mode is first preliminarily trained separately and given a reward function suitable for independent training. Subsequently, each control mode is migrated from a single scenario to an overall scenario of multi-switching threshold control. In this way, multi-mode integrated control has a certain amount of experience in the early stage, which serves as a preprocessing to prepare for the final configuration.

[0136] In the present application, training data is sampled from the first, second and third experience pools according to a preset ratio. Such sampling can ensure that the model takes into account the experience of different control modes during training.

[0137] In this application, the first control mode reward is used as the reward function, and the weights of the deep Q network model are initially iterated according to the sampled training data. This step is to establish the model's learning of the initial control mode.

[0138] In the present application, in each iteration, the current state of the robot arm is monitored in real time so as to make a switching decision according to the current environment.

[0139] In the present application, when the current state of the robot arm meets the first switching threshold, it switches to the second control mode reward. The second control mode reward is used as a reward function, and the weights of the deep Q network model are iterated according to the training data.

[0140] In the present application, when the preset number of iterations is reached or the loss function converges, the overall training is completed, and finally the optimized operation strategy of the robotic arm is obtained.

[0141] In the present application, it should be noted that the loss function in the overall training process is obtained by adding the loss functions of the preliminary training of the first control mode, the second control mode, and the third control mode rewards based on the reinforcement learning method, thereby unifying the loss function of the entire overall training to realize the judgment of whether the loss converges.

[0142] An embodiment of the present application provides a multi-mode integrated control device for a robotic arm with embodied intelligence for a surgical robot, which is used to execute the multi-mode integrated control method for a robotic arm with embodied intelligence for a surgical robot described in the above content of the present application. The multi-mode integrated control device for a robotic arm with embodied intelligence for a surgical robot is described in detail below.

[0143] like Figure 2 As shown, the multi-mode integrated control device for a robotic arm with embodied intelligence for a surgical robot comprises:

[0144] An inverse reinforcement learning module 101, which is used to obtain a first reward function based on inverse reinforcement learning processing;

[0145] The obstacle acquisition module 102 is used to acquire the target position and obstacle information, wherein the obstacle information includes static information and dynamic information;

[0146] A reward construction module 103, which is used to construct a second reward function based on the target position and a third reward function based on obstacle information;

[0147] A reinforcement learning module 104, which is used to train the operation strategy of the robot arm through reinforcement learning according to the first reward function, the second reward function, and the third reward function;

[0148] The robot arm control module 105 is used to select the running trajectory with the highest cumulative reward after the training is completed, and control the robot arm to execute the running trajectory.

[0149] In one embodiment, the inverse reinforcement learning module 101 is further used to:

[0150] Obtain recorded data of any object operating a robotic arm, wherein the recorded data includes historical trajectory data; preprocess the historical trajectory data; infer a corresponding reward function from the preprocessed historical trajectory data based on an inverse reinforcement learning algorithm; verify the reward function based on a reinforcement learning algorithm, and determine a final reward function after the verification passes.

[0151] In one implementation, the obstacle acquisition module 102 is further configured to:

[0152] The surgical scene video stream is obtained and divided into continuously set depth image frames; the depth image frames are input into an obstacle recognition model to obtain obstacle recognition results; the obstacle recognition results include static obstacles, dynamic obstacles and target positions; the continuous depth image frames marked with obstacle recognition results are input into a spatiotemporal prediction model to obtain the spatial motion trajectory of the dynamic obstacle.

[0153] In one embodiment, the reinforcement learning module 104 is further configured to:

[0154] Generate a first control mode reward, a second control mode reward, and a third control mode reward according to the first reward function, the second reward function, and the third reward function;

[0155] Based on the reinforcement learning method, preliminary training is performed on the first control mode reward, the second control mode reward, and the third control mode reward;

[0156] Constructing switching thresholds between the first control mode reward, the second control mode reward, and the third control mode reward;

[0157] According to the switching threshold, the first control mode reward, the second control mode reward, and the third control mode reward are trained as a whole to obtain the operation strategy of the robot arm.

[0158] In one embodiment, the first control mode reward is determined by a second reward function and a third reward function, and the second control mode reward and the third control mode reward are both determined by the first reward function, the second reward function, and the third reward function as a whole.

[0159] In one embodiment, the reinforcement learning module 104 is further configured to:

[0160] A simulation space is constructed according to the target position and obstacle information; a state space and an action space are constructed according to the robot arm parameters; the action space includes multiple joint axes, each of which has a forward rotation action and a reverse rotation action; a deep Q network model and a first experience pool are constructed; the first control mode reward, the second control mode reward, and the third control mode reward have corresponding first experience pools, second experience pools, and third experience pools; taking the first control mode reward as the reward function, sampling from the experience replay pool, and calculating the Q value based on the deep Q network model; updating the weights of the deep Q network model by minimizing the loss function until a predetermined number of iterations is reached.

[0161] In one embodiment, the switching threshold includes a first switching threshold and a second switching threshold; the reinforcement learning module 104 is further used to:

[0162] The training data are sampled from the first experience pool, the second experience pool and the third experience pool according to a preset ratio; the first control mode reward is used as a reward function, and the weights of the deep Q network model are iterated according to the training data; the current state of the robot arm is determined in real time; when the current state of the robot arm meets the first switching threshold, the second control mode reward is used as a reward function, and the weights of the deep Q network model are iterated according to the training data; when the current state of the robot arm meets the second switching threshold, the third control mode reward is used as a reward function, and the weights of the deep Q network model are iterated according to the training data; when the number of iterations reaches a preset number, or the loss function converges, the overall training is completed to obtain the operation strategy of the robot arm.

[0163] The multi-mode integrated control device for robotic arms with embodied intelligence for surgical robots provided in the above-mentioned embodiments of the present application has a corresponding relationship with the multi-mode integrated control method for robotic arms with embodied intelligence for surgical robots provided in the embodiments of the present application. Therefore, the specific content in the device has a corresponding relationship with the multi-mode integrated control method for robotic arms with embodied intelligence for surgical robots. The specific content can be referred to the record in the multi-mode integrated control method for robotic arms with embodied intelligence for surgical robots, and will not be repeated in the present application.

[0164] The multi-mode integrated control device for robotic arms with embodied intelligence for surgical robots provided in the above-mentioned embodiments of the present application and the multi-mode integrated control method for robotic arms with embodied intelligence for surgical robots provided in the embodiments of the present application are based on the same inventive concept and have the same beneficial effects as the methods adopted, run or implemented by the application programs stored therein.

[0165] The above describes the internal functions and structure of the multi-mode integrated control device for the robotic arm of the surgical robot embodied intelligence. Figure 3 As shown, in practice, the multi-mode integrated control device for the robotic arm of the surgical robot with embodied intelligence can be realized as an electronic device, including: a memory 301 and a processor 303.

[0166] The memory 301 may be configured to store programs.

[0167] In addition, the memory 301 may also be configured to store various other data to support operations on the electronic device. Examples of such data include instructions for any application or method operating on the electronic device, contact data, phone book data, messages, pictures, videos, etc.

[0168] The memory 301 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.

[0169] The processor 303 is coupled to the memory 301 and is configured to execute a program in the memory 301 to:

[0170] Obtaining a first reward function based on inverse reinforcement learning processing;

[0171] Acquiring target position and obstacle information, wherein the obstacle information includes static information and dynamic information;

[0172] Constructing a second reward function based on the target position and a third reward function based on obstacle information;

[0173] According to the first reward function, the second reward function, and the third reward function, the operation strategy of the robot arm is trained through reinforcement learning;

[0174] After training is completed, the running trajectory with the highest cumulative reward is selected and the robot arm is controlled to execute the running trajectory.

[0175] In one implementation, the processor 303 is further configured to:

[0176] Obtain recorded data of any object operating a robotic arm, wherein the recorded data includes historical trajectory data; preprocess the historical trajectory data; infer a corresponding reward function from the preprocessed historical trajectory data based on an inverse reinforcement learning algorithm; verify the reward function based on a reinforcement learning algorithm, and determine a final reward function after the verification passes.

[0177] In one implementation, the processor 303 is further configured to:

[0178] The surgical scene video stream is obtained and divided into continuously set depth image frames; the depth image frames are input into an obstacle recognition model to obtain obstacle recognition results; the obstacle recognition results include static obstacles, dynamic obstacles and target positions; the continuous depth image frames marked with obstacle recognition results are input into a spatiotemporal prediction model to obtain the spatial motion trajectory of the dynamic obstacle.

[0179] In one implementation, the processor 303 is further configured to:

[0180] A first control mode reward, a second control mode reward and a third control mode reward are generated according to the first reward function, the second reward function and the third reward function; the first control mode reward, the second control mode reward and the third control mode reward are preliminarily trained based on the reinforcement learning method; a switching threshold between the first control mode reward, the second control mode reward and the third control mode reward is constructed; according to the switching threshold, the first control mode reward, the second control mode reward and the third control mode reward are trained as a whole to obtain the operation strategy of the robotic arm.

[0181] In one embodiment, the first control mode reward is determined by a second reward function and a third reward function, and the second control mode reward and the third control mode reward are both determined by the first reward function, the second reward function, and the third reward function as a whole.

[0182] In one implementation, the processor 303 is further configured to:

[0183] A simulation space is constructed according to the target position and obstacle information; a state space and an action space are constructed according to the robot arm parameters; the action space includes multiple joint axes, each of which has a forward rotation action and a reverse rotation action; a deep Q network model and a first experience pool are constructed; the first control mode reward, the second control mode reward, and the third control mode reward have corresponding first experience pools, second experience pools, and third experience pools; taking the first control mode reward as the reward function, sampling from the experience replay pool, and calculating the Q value based on the deep Q network model; updating the weights of the deep Q network model by minimizing the loss function until a predetermined number of iterations is reached.

[0184] In one implementation, the switching threshold includes a first switching threshold and a second switching threshold; the processor 303 is further configured to:

[0185] The training data are sampled from the first experience pool, the second experience pool and the third experience pool according to a preset ratio; the first control mode reward is used as a reward function, and the weights of the deep Q network model are iterated according to the training data; the current state of the robot arm is determined in real time; when the current state of the robot arm meets the first switching threshold, the second control mode reward is used as a reward function, and the weights of the deep Q network model are iterated according to the training data; when the current state of the robot arm meets the second switching threshold, the third control mode reward is used as a reward function, and the weights of the deep Q network model are iterated according to the training data; when the number of iterations reaches a preset number, or the loss function converges, the overall training is completed to obtain the operation strategy of the robot arm.

[0186] In the present application, the processor is also specifically used to execute all processes and steps of the above-mentioned multi-mode integrated control method of the robotic arm for embodied intelligence of the surgical robot. The specific content can be referred to the records in the multi-mode integrated control method of the robotic arm for embodied intelligence of the surgical robot, and will not be repeated in this application.

[0187] In this application, Figure 3 Only some components are shown schematically, which does not mean that the electronic device only includes Figure 3 Components shown.

[0188] The electronic device provided in this embodiment is based on the same inventive concept as the multi-mode integrated control method for a surgical robot's embodied intelligence provided in the embodiment of the present application, and has the same beneficial effects as the method adopted, run or implemented by the application program stored therein.

[0189] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-readable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.

[0190] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0191] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0192] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process in the computer or other programmable device. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0193] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0194] The memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0195] The present application also provides a computer-readable storage medium corresponding to the multi-mode integrated control method of a robotic arm for embodied intelligence of a surgical robot provided in the aforementioned embodiment, on which a computer program (i.e., a program product) is stored. When the computer program is run by a processor, it will execute the multi-mode integrated control method of a robotic arm for embodied intelligence of a surgical robot provided in any of the aforementioned embodiments.

[0196] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0197] The computer-readable storage medium provided in the above-mentioned embodiments of the present application and the multi-mode integrated control method of the robotic arm for embodied intelligence of a surgical robot provided in the embodiments of the present application are based on the same inventive concept and have the same beneficial effects as the methods adopted, run or implemented by the application programs stored therein.

[0198] It should be noted that in the description provided herein, a large number of specific details are described. However, it is understood that the embodiments of the present application can be practiced without these specific details. In some instances, well-known structures and technologies are not shown in detail so as not to obscure the understanding of this description.

[0199] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0200] The above is only an embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included in the scope of the claims of the present application.

Claims

1. A multi-mode integrated control method for a robotic arm of a surgical robot with embodied intelligence, characterized in that: include: Obtaining a first reward function based on inverse reinforcement learning processing; Obtaining target position and obstacle information, wherein the obstacle information includes static information and dynamic information; Constructing a second reward function based on the target position and a third reward function based on obstacle information; According to the first reward function, the second reward function, and the third reward function, the operation strategy of the robot arm is trained through reinforcement learning; After training is completed, the running trajectory with the highest cumulative reward is selected and the robot arm is controlled to execute the running trajectory.

2. The multi-mode integrated control method for a robotic arm of a surgical robot with embodied intelligence according to claim 1, characterized in that: The obtaining of a first reward function based on inverse reinforcement learning processing comprises: Acquire recorded data of any object operating a robotic arm, wherein the recorded data includes historical trajectory data; Preprocessing the historical trajectory data; Based on the inverse reinforcement learning algorithm, inferring the corresponding reward function from the preprocessed historical trajectory data; The reward function is verified based on a reinforcement learning algorithm, and a final reward function is determined after the verification passes.

3. The multi-mode integrated control method for a robotic arm of a surgical robot with embodied intelligence according to claim 1, characterized in that: The method of obtaining and parsing a real surgical scene video stream to obtain target location and obstacle information includes: Acquire a surgical scene video stream and segment it into continuously set depth image frames; Inputting the depth image frame into an obstacle recognition model to obtain an obstacle recognition result; the obstacle recognition result includes static obstacles, dynamic obstacles and target positions; The continuous depth image frames with labeled obstacle recognition results are input into the spatiotemporal prediction model to obtain the spatial motion trajectory of the dynamic obstacle.

4. The multi-mode integrated control method for a robotic arm of a surgical robot with embodied intelligence according to any one of claims 1 to 3, characterized in that: The step of training the operation strategy of the robot arm by reinforcement learning according to the first reward function, the second reward function, and the third reward function includes: Generate a first control mode reward, a second control mode reward, and a third control mode reward according to the first reward function, the second reward function, and the third reward function; Based on the reinforcement learning method, preliminary training is performed on the first control mode reward, the second control mode reward, and the third control mode reward; Constructing switching thresholds between the first control mode reward, the second control mode reward, and the third control mode reward; According to the switching threshold, the first control mode reward, the second control mode reward, and the third control mode reward are trained as a whole to obtain the operation strategy of the robotic arm.

5. The multi-mode integrated control method for a robotic arm of a surgical robot with embodied intelligence according to claim 4, characterized in that: The first control mode reward is determined by the second reward function and the third reward function, and the second control mode reward and the third control mode reward are both determined by the first reward function, the second reward function, and the third reward function as a whole.

6. The multi-mode integrated control method for a robotic arm of a surgical robot with embodied intelligence according to claim 4, characterized in that: The preliminary training of the first control mode reward based on the reinforcement learning method includes: Construct simulation space based on target location and obstacle information; According to the parameters of the robot arm, a state space and an action space are constructed; the action space includes a plurality of joint axes, each of which has a forward rotation action and a reverse rotation action; Constructing a deep Q network model and a first experience pool; the first control mode reward, the second control mode reward, and the third control mode reward have corresponding first experience pools, second experience pools, and third experience pools; Taking the first control mode reward as the reward function, sampling from the experience replay pool, and calculating the Q value based on the deep Q network model; By minimizing the loss function, the weights of the deep Q network model are updated until the predetermined number of iterations is reached.

7. The multi-mode integrated control method for a robotic arm of a surgical robot with embodied intelligence according to claim 4, characterized in that: The switching threshold includes a first switching threshold and a second switching threshold; and according to the switching threshold, the first control mode reward, the second control mode reward, and the third control mode reward are trained as a whole to obtain an operation strategy of the robot arm, including: Sampling training data from the first experience pool, the second experience pool, and the third experience pool according to a preset ratio; Using the first control mode reward as a reward function, iterating the weights of the deep Q network model according to the training data; Make real-time judgment on the current state of the robot arm; When the current state of the robot arm satisfies the first switching threshold, taking the second control mode reward as the reward function, and iterating the weight of the deep Q network model according to the training data; When the current state of the robot arm satisfies the second switching threshold, taking the third control mode reward as the reward function, and iterating the weight of the deep Q network model according to the training data; When the number of iterations reaches the preset number or the loss function converges, the overall training is completed and the operation strategy of the robotic arm is obtained.

8. A multi-mode integrated control device for a robotic arm of a surgical robot with embodied intelligence, characterized in that: include: An inverse reinforcement learning module, which is used to obtain a first reward function based on inverse reinforcement learning processing; An obstacle acquisition module, which is used to acquire target position and obstacle information, wherein the obstacle information includes static information and dynamic information; A reward building module, which is used to build a second reward function based on the target position and a third reward function based on obstacle information; A reinforcement learning module, which is used to train the operation strategy of the robot arm through reinforcement learning according to the first reward function, the second reward function, and the third reward function; The robotic arm control module is used to select the running trajectory with the highest cumulative reward after training is completed, and control the robotic arm to execute the running trajectory.

9. An electronic device, characterized in that: include: Memory and processor; The memory is used to store programs; The processor, coupled to the memory, is configured to execute the program to: Obtaining a first reward function based on inverse reinforcement learning processing; Obtaining target position and obstacle information, wherein the obstacle information includes static information and dynamic information; Constructing a second reward function based on the target position and a third reward function based on obstacle information; According to the first reward function, the second reward function, and the third reward function, the operation strategy of the robot arm is trained through reinforcement learning; After training is completed, the running trajectory with the highest cumulative reward is selected and the robot arm is controlled to execute the running trajectory.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: The program is executed by the processor to implement the multi-mode comprehensive control method of the robotic arm for embodied intelligence of a surgical robot as described in any one of claims 1-7.

Images