Surgical robot control method and device based on body intelligence

Abstract

The invention provides a surgical robot control method and device based on body intelligence. The method comprises the steps that a reward function based on inverse reinforcement learning processing is acquired; determining constraint conditions based on differences between past experience scenes and application scenes; based on multi-objective optimization, the operation strategy of the mechanical arm of the surgical robot is trained through reinforcement learning; after training is completed, selecting a moving track with the highest accumulated reward, and controlling a mechanical arm of the surgical robot to execute the moving track; the multi-objective optimization comprises a plurality of objective functions, and the objective functions at least comprise the reward functions and the constraint conditions. According to the method, the difference is converted into the constraint condition, strategy training is carried out in a multi-objective optimization mode, and therefore the problem that adaptation is difficult to carry out due to the difference between operation experience and data of a new surgical robot is solved.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and more specifically, to a surgical robot control method and device based on embodied intelligence. Background Art

[0002] Embodied intelligence refers to intelligent entities that have bodies and support interaction with the physical world, such as robots and driverless cars. By processing multiple sensor data inputs, a control center such as a large model generates motion commands to drive the intelligent entity, replacing the traditional motion-driving method based on rules or mathematical formulas, and achieving a deep integration of virtual and reality.

[0003] The surgical robot can be used as a perfect carrier of embodied intelligence to realize the functions of the surgical robot based on embodied intelligence. In the specific implementation process, the past experience of operating the surgical robot can be summarized first, and then a new control center can be generated based on the summarized experience.

[0004] However, there will be certain differences between the operating experience of past surgical robots and the data of new surgical robots. How to generate a new control center based on the summarized experience such as reward function in the current situation is a difficult problem to solve. Summary of the invention

[0005] The problem that this application solves is that it is currently difficult to generate new control centers through reward functions.

[0006] To solve the above problems, the first aspect of the present application provides a surgical robot control method based on embodied intelligence, comprising:

[0007] Obtain a reward function based on inverse reinforcement learning processing;

[0008] Determine constraints based on past experience scenarios and differences in application scenarios;

[0009] Based on multi-objective optimization, the operation strategy of the surgical robot arm is trained through reinforcement learning;

[0010] After the training is completed, the running trajectory with the highest cumulative reward is selected, and the surgical robot arm is controlled to execute the running trajectory; the multi-objective optimization includes multiple objective functions, and the objective function includes at least the reward function and the constraint condition.

[0011] The second aspect of the present application provides a surgical robot control device based on embodied intelligence, which includes:

[0012] A reward acquisition module, which is used to obtain a reward function based on inverse reinforcement learning processing;

[0013] A difference determination module, which is used to determine the constraint conditions based on the differences between past experience scenarios and application scenarios;

[0014] A strategy training module, which is used to train the operation strategy of the surgical robot arm through reinforcement learning based on multi-objective optimization;

[0015] A robotic arm control module is used to select the running trajectory with the highest cumulative reward after training is completed, and control the surgical robot robotic arm to execute the running trajectory; the multi-objective optimization includes multiple objective functions, and the objective function includes at least the reward function and the constraint condition.

[0016] A third aspect of the present application provides an electronic device, comprising: a memory and a processor;

[0017] The memory is used to store programs;

[0018] The processor, coupled to the memory, is configured to execute the program to:

[0019] Obtain a reward function based on inverse reinforcement learning processing;

[0020] Determine constraints based on past experience scenarios and differences in application scenarios;

[0021] Based on multi-objective optimization, the operation strategy of the surgical robot arm is trained through reinforcement learning;

[0022] After the training is completed, the running trajectory with the highest cumulative reward is selected, and the surgical robot arm is controlled to execute the running trajectory; the multi-objective optimization includes multiple objective functions, and the objective function includes at least the reward function and the constraint condition.

[0023] A fourth aspect of the present application provides a computer-readable storage medium having a computer program stored thereon, wherein the program is executed by a processor to implement the above-mentioned surgical robot control method based on embodied intelligence.

[0024] In the present application, the differences are converted into constraints and strategy training is performed through multi-objective optimization, thereby solving the problem of difficulty in adaptation due to differences in operating experience and data of the new surgical robot. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Figure 1 is a flow chart of a surgical robot control method based on embodied intelligence according to an embodiment of the present application;

[0026] Figure 2 A flowchart of a strategy training of a surgical robot control method based on embodied intelligence according to an embodiment of the present application;

[0027] Figure 3 is a structural block diagram of a surgical robot control device based on embodied intelligence according to an embodiment of the present application;

[0028] Figure 4 is a structural block diagram of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0029] In order to make the above-mentioned purposes, features and advantages of the present application more obvious and easy to understand, the specific embodiments of the present application are described in detail below in conjunction with the accompanying drawings. Although the exemplary embodiments of the present application are shown in the accompanying drawings, it should be understood that the present application can be implemented in various forms and should not be limited by the embodiments described herein. On the contrary, these embodiments are provided in order to enable a more thorough understanding of the present application and to fully convey the scope of the present application to those skilled in the art.

[0030] It should be noted that, unless otherwise specified, the technical terms or scientific terms used in this application should have the common meanings understood by technicians in the field to which this application belongs.

[0031] The surgical robot can be used as a perfect carrier of embodied intelligence to realize the functions of the surgical robot based on embodied intelligence. In the specific implementation process, the past experience of operating the surgical robot can be summarized first, and then a new control center can be generated based on the summarized experience.

[0032] However, for surgical robots, the past experience they learn is the experience of manual control of surgical robots, and the scenarios based on the application of experience are scenarios for automatic control of surgical robots. There are differences in the usage environment between their past experience and the application scenarios. How to reduce the impact of this difference in the process of generating a new control center is a problem that is currently difficult to solve.

[0033] In response to the above problems, the present application provides a new skeleton registration solution, which performs coarse registration through digital twins to solve the problem of low accuracy of current feature point registration.

[0034] The present application embodiment provides a surgical robot control method based on embodied intelligence, and the specific scheme of the method is as follows: Figure 1-Figure 2 As shown, the method can be performed by a surgical robot control device based on embodied intelligence, and the surgical robot control device based on embodied intelligence can be integrated into electronic devices such as computers, servers, computers, server clusters, and data centers. Figure 1 As shown, it is a flow chart of a surgical robot control method based on embodied intelligence according to an embodiment of the present application; wherein the surgical robot control method based on embodied intelligence includes:

[0035] S101, obtaining a reward function based on inverse reinforcement learning processing;

[0036] S102, determining constraint conditions based on differences between past experience scenarios and application scenarios;

[0037] For example, in past experience scenarios, the historical trajectory data is the data obtained by an expert doctor operating the smooth control of a robotic arm. In this operation scenario, the expert doctor holds the robotic arm and applies force to the robotic arm to control its operation. However, in the application scenario of active control of the robotic arm, obstacle avoidance control is required. In this case, the robotic arm needs to be controlled to stay away from the expert doctor to avoid interference, which is contrary to past experience scenarios.

[0038] Therefore, it is necessary to determine the corresponding constraints based on the differences in past experience scenarios and application scenarios, so as to avoid adverse consequences caused by the differences.

[0039] S103, training the operation strategy of the surgical robot arm through reinforcement learning based on multi-objective optimization;

[0040] S104, after the training is completed, select the running trajectory with the highest cumulative reward, and control the surgical robot arm to execute the running trajectory; the multi-objective optimization includes multiple objective functions, and the objective function at least includes the reward function and the constraint condition.

[0041] In the present application, the differences are converted into constraints and strategy training is performed through multi-objective optimization, thereby solving the problem of difficulty in adaptation due to differences in operating experience and data of the new surgical robot.

[0042] In one embodiment, the determining of the constraint conditions based on the difference between the past experience scenario and the application scenario includes:

[0043] Obtain past experience scenario information and application scenario information;

[0044] Determine the constraint direction of past scenes based on past experience scene information;

[0045] Determine the constraint direction of the application scenario based on the application scenario information;

[0046] Determine a difference direction according to a constraint direction of the past scene application scenario, wherein the difference direction belongs to the application scenario and does not belong to the past scene;

[0047] A constraint condition is determined according to the difference direction.

[0048] In the present application, the past experience scene information may be video information of past scenes and preoperative planning information.

[0049] In the present application, the application scenario information may be video information and preoperative planning information of the current application scenario.

[0050] In the present application, the past experience scene information determines the constraint direction of the past scene, which can be obtained by qualitative analysis of the past experience scene information, or can be summarized by an expert doctor.

[0051] In the present application, the constraint direction of the application scenario is determined based on the application scenario information, which can be obtained by qualitative analysis of the application scenario information or summarized by an expert doctor.

[0052] In this application, the difference direction is the direction that is not present in the previous scene application scenario but is present in the application scenario information. If it is present in the previous scene application scenario but not in the application scenario information, it is generally not considered as the difference direction unless the difference direction conflicts with the application scenario information or its constraints.

[0053] In one embodiment, the constraint conditions include obstacle constraints, and / or safety boundary constraints, and / or end tool orientation constraints, and / or mechanical constraints.

[0054] In the present application, when there are multiple constraints, the multiple constraints can be combined by adding weights, or the constraint is considered to be satisfied when all constraints are satisfied.

[0055] Among them, mechanical constraints can be: tool-bone contact force: during the cutting process, the robot must accurately control the force applied by the tool on the bone to prevent bone breakage or unnecessary damage; vibration control: during the cutting process, the vibration of the tool needs to be minimized to improve the accuracy of the surgery and the comfort of the patient.

[0056] The safety boundary constraint may be: preventing the tool from deviating from the path during actual execution.

[0057] The end tool orientation constraint may be to prevent the tool from being oriented toward the worker during actual execution.

[0058] In the present application, the specific settings of the safety boundary constraints, and / or the end tool orientation constraints, and / or the mechanical constraints can be determined based on actual conditions and are not limited or restricted in the present application.

[0059] In one embodiment, in combination Figure 2 As shown, the S103, based on multi-objective optimization, trains the operation strategy of the surgical robot arm through reinforcement learning, including:

[0060] S301, obtaining operating environment data and planning data, and constructing a simulation space;

[0061] S302, constructing a state space and an action space based on the inverse reinforcement learning process and the operating environment data and planning data; the action space includes a plurality of joint axes, each of which has a forward rotation action and a reverse rotation action;

[0062] S303, constructing a deep Q network model;

[0063] S304, performing a first training on the deep Q network model based on the deep Q network learning, using the obtained reward function;

[0064] S305, taking the difference constraint as the reward function, and performing a second training on the deep Q network model based on deep Q network learning;

[0065] S306: Alternately perform the first training and the second training until a preset condition is met.

[0066] In this way, by using the obtained reward function and the different constraints for alternating training, the reward function is optimized first, and then, under the premise of satisfying the constraints, the strategy is gradually adjusted to approach the optimized reward while satisfying the constraints as much as possible. This is done alternately until a solution that satisfies both the reward function and the constraints is found.

[0067] In one embodiment, S304, first training the deep Q network model based on deep Q network learning using the obtained reward function;

[0068] Use the obtained reward function as the reward function;

[0069] Constructing an experience replay pool, wherein the experience replay pool includes the action, state, reward, and next state of the agent;

[0070] Sampling from the experience replay pool and calculating the Q value based on the deep Q network model;

[0071] By minimizing the loss function, the weights of the deep Q network model are updated until the training is completed.

[0072] In one embodiment, the step S305, taking the difference constraint as the reward function and performing a second training on the deep Q network model based on deep Q network learning, includes:

[0073] Construct a reward function based on the constraints of diversity;

[0074] Constructing an experience replay pool, wherein the experience replay pool includes the action, state, reward, and next state of the agent;

[0075] Sampling from the experience replay pool and calculating the Q value based on the deep Q network model;

[0076] By minimizing the loss function, the weights of the deep Q network model are updated until the training is completed.

[0077] In this application, the specific process of deep Q reinforcement learning and the setting of the corresponding deep Q network model can be carried out with reference to the prior art and will not be repeated in this application.

[0078] In the present application, the specific processes of the first training and the second training are similar, and only the reward functions are different.

[0079] In one implementation, the obstacle constraint calculation process is:

[0080] Get the robot arm posture data in the current state;

[0081] Get the current state of the environment data;

[0082] Determine the minimum distance between the robot arm and environmental obstacles based on the robot arm posture data and environmental data;

[0083] When the minimum distance between the robot arm and the environmental obstacle is greater than a preset distance, it is determined that the obstacle constraint is satisfied.

[0084] The preset distance may be determined according to actual conditions.

[0085] It should be noted that the shapes of the robotic arm and environmental obstacles are irregular, making it difficult to make accurate calculations and estimates.

[0086] In one embodiment, determining the minimum distance between the robotic arm and the environmental obstacle according to the robotic arm posture data and the environmental data includes:

[0087] Performing equivalent processing on the robot arm posture data, the robot arm is equivalent to a plurality of line segments connected in sequence and a preset thickness;

[0088] Perform equivalent processing on the environmental data, and equate the obstacles in the environment to spheres and cylinders that wrap the obstacles;

[0089] Calculate the distance between each line segment of the robot arm and the center of the sphere and the axis segment of the cylinder;

[0090] The minimum distance is determined according to the calculated distance, the preset thickness, the radius of the sphere, and the radius of the cylinder.

[0091] In this application, the workload of minimum distance calculation is greatly reduced by equating the robot arm and obstacles to simple geometric bodies.

[0092] In this way, the robotic arm is equivalent to a line segment and the obstacle is equivalent to a point or a line segment. Then, it is only necessary to calculate the distance between the line segments and the distance between the line segment and the point to obtain the distance between the robotic arm itself and the obstacle.

[0093] Among them, the distance between the line segment and the point is to calculate the projection point from the point to the line segment. If the projection point is on the line segment, the distance from the point to the projection point is the required distance; if the projection point is not on the line segment, the smaller value of the distance between the point and the two end points of the line segment is the required distance.

[0094] Among them, the distance between line segments is determined by parameterizing the line segments and using the least squares method to solve the two parameters, determining the distance between the nearest points on the line segments, or checking the distance between the endpoints of the line segments.

[0095] Wherein, S101, obtaining a reward function based on inverse reinforcement learning processing, includes:

[0096] Acquire recorded data of any subject operating a surgical robot arm, wherein the recorded data includes historical trajectory data;

[0097] Preprocessing the historical trajectory data;

[0098] Based on the inverse reinforcement learning algorithm, inferring the corresponding reward function from the preprocessed historical trajectory data;

[0099] The reward function is verified based on a reinforcement learning algorithm, and a final reward function is determined after the verification passes.

[0100] In one implementation, the robotic arm is a six-axis robotic arm, and the historical trajectory data is angle change data of six motors corresponding to the six-axis robotic arm over time.

[0101] In one embodiment, the recorded data also includes operating environment data; and the historical trajectory data is the motion data of the robotic arm under compliant control.

[0102] In one embodiment, the inference of the corresponding reward function from the preprocessed historical trajectory data based on the inverse reinforcement learning algorithm includes:

[0103] Build simulation space based on operating environment data;

[0104] Construct state space based on operating environment data and historical trajectory data;

[0105] Setting an action space, wherein the action space includes a plurality of joint axes, each joint axis having a forward rotation action and a reverse rotation action;

[0106] Based on the state space and action space, a complete running trajectory is constructed;

[0107] Construct a reward function model;

[0108] Model the maximum entropy of trajectory probability to obtain the maximum entropy model;

[0109] By maximizing the log-likelihood of the expert trajectory, the parameters of the reward function model are iteratively optimized to obtain the optimal reward function parameters.

[0110] In one implementation, the reward function model is:

[0111] R(s t ,a t )=α·exp(-‖p t -g‖ 2 )-β·‖Δq t ‖ 2 -γ·exp(‖u t ‖ 2 )

[0112] Among them, R is the reward value, a t is the tth action, s t is the tth state, α, β, γ are weight coefficients, p t is the position of the end effector of the robot arm, g is the target position, Δq t is the tth comprehensive change value of the joint angle, u t is the joint angle vector.

[0113] In one embodiment, the maximum entropy model is:

[0114]

[0115] Among them, τ is the trajectory, is a possible trajectory, R(s t ,a t ) is to execute action a t Get status s t The reward value is the normalized function, To perform actions Get Status The reward value, are the possible states and possible actions corresponding to the tth state and action, and P(τ) is the probability of trajectory τ.

[0116] In one implementation, the preprocessing of the historical trajectory data includes:

[0117] Determine a unit time point according to a maximum value of the angle change data;

[0118] Divide the angle change data based on unit time points;

[0119] Based on the segmented angle change data, the angle value of the corresponding motor at each unit time point is determined.

[0120] Among them, the historical trajectory data includes the angle change data of the motor over time; based on the angle change data, first select the measurement unit, and count the angle change data of all joint motors over time, and count the shortest time required for the joint motor to change by one unit (the unit can be selected according to actual conditions, or it can be set by itself to facilitate accuracy constraints). The shortest time is used as the unit time point. In this way, the angle change of all motors within the unit time point does not exceed 1.

[0121] Among them, after determining the unit time point, the angle change data is divided with the unit time point as the duration; each unit time corresponds to an angle change data and a current angle value (determined by the angle value at the starting point or end point of the time point); the current angle value corresponding to each time point is processed by approximation or rounding down to obtain an approximate angle value, which is an integer, and the angle values ​​of adjacent time points differ by ±1 or 0.

[0122] In this way, after preprocessing, it is easy to obtain the corresponding action and state of each joint motor.

[0123] In the present application, by preprocessing the historical trajectory data, the continuous joint motor movements are converted into discrete movements that are easy to identify and use, thereby greatly reducing the number of movements.

[0124] It should be noted that in this application, if the motor action of each joint of the six-axis robot is set to three actions of ±1 or 0, then all the action combinations of the six motors are three to the sixth power. This will result in an overly large action space in the entire reinforcement learning and inverse reinforcement learning, and require longer training time.

[0125] In one implementation, the unit time point is divided twice, and each unit time point is divided into six virtual time points, which respectively execute the motor actions from the first joint to the end joint.

[0126] That is to say, at the first virtual time point, the motor action of the first joint is executed (the motor action is the action at the corresponding unit time point), but the motors of other joints remain unchanged; other virtual time points remain unchanged. In this way, each virtual time point corresponds to three actions, and six virtual time points correspond to 18 actions, which greatly reduces the combination of action space.

[0127] Preferably, after each unit time point is divided into six virtual time points, the action corresponding to each virtual time point is determined, and the state corresponding to each virtual time point is generated through the action and the state.

[0128] In the present application, the state can be split into the operating environment state and the robotic arm state. After each unit time point is divided into six virtual time points, the operating environment state remains unchanged (it is set to unchanged), and the robotic arm state changes accordingly with the motor action, thereby combining the changed robotic arm state and the unchanged operating environment state into a new state corresponding to the virtual time point.

[0129] Preferably, a reward function is determined based on the preprocessed inverse reinforcement learning method. After a new action sequence is obtained by training through a new reinforcement learning method based on the reward function, the actions corresponding to six consecutive virtual time points need to be synthesized into an action corresponding to a unit time point, so as to make the actions of the robotic arm coherent.

[0130] An embodiment of the present application provides a surgical robot control device based on embodied intelligence, which is used to execute the surgical robot control method based on embodied intelligence described in the above content of the present application. The surgical robot control device based on embodied intelligence is described in detail below.

[0131] like Figure 3 As shown, the surgical robot control device based on embodied intelligence includes:

[0132] A reward acquisition module 101, which is used to obtain a reward function based on inverse reinforcement learning processing;

[0133] A difference determination module 102, which is used to determine constraint conditions based on the differences between past experience scenarios and application scenarios;

[0134] A strategy training module 103, which is used to train the operation strategy of the surgical robot arm through reinforcement learning based on multi-objective optimization;

[0135] The robotic arm control module 104 is used to select the running trajectory with the highest cumulative reward after the training is completed, and control the surgical robot robotic arm to execute the running trajectory; the multi-objective optimization includes multiple objective functions, and the objective function includes at least the reward function and the constraint condition.

[0136] In one embodiment, the difference determination module 102 is further configured to:

[0137] Acquire past experience scenario information and application scenario information; determine the constraint direction of the past scenario based on the past experience scenario information; determine the constraint direction of the application scenario based on the application scenario information; determine the difference direction based on the constraint direction of the application scenario of the past scenario, the difference direction belongs to the application scenario and does not belong to the past scenario; determine the constraint condition based on the difference direction.

[0138] In one embodiment, the constraint conditions include obstacle constraints, and / or safety boundary constraints, and / or end tool orientation constraints, and / or mechanical constraints.

[0139] In one embodiment, the strategy training module 103 is further used to:

[0140] Acquire operating environment data and planning data to construct a simulation space; construct a state space and an action space based on an inverse reinforcement learning process and the operating environment data and planning data; the action space includes multiple joint axes, each of which has a forward rotation action and a reverse rotation action; construct a deep Q network model; perform a first training on the deep Q network model based on deep Q network learning using the acquired reward function; perform a second training on the deep Q network model based on deep Q network learning using differential constraints as the reward function; alternately perform the first training and the second training until a preset condition is met.

[0141] In one embodiment, the strategy training module 103 is further used to:

[0142] A reward function is constructed according to the differential constraints; an experience replay pool is constructed, which includes the action, state, reward, and next state of the agent; samples are taken from the experience replay pool, and the Q value is calculated based on the deep Q network model; and the weights of the deep Q network model are updated by minimizing the loss function until the training is completed.

[0143] In one embodiment, the strategy training module 103 is further used to:

[0144] Obtain the robot arm posture data in the current state; obtain the environmental data in the current state; determine the minimum distance between the robot arm and the environmental obstacle based on the robot arm posture data and the environmental data; when the minimum distance between the robot arm and the environmental obstacle is greater than the preset distance, determine that the obstacle constraint is satisfied.

[0145] In one embodiment, the strategy training module 103 is further used to:

[0146] The robot arm posture data is equivalently processed, and the robot arm is equivalent to a plurality of line segments connected in sequence and a preset thickness; the environmental data is equivalently processed, and the obstacles in the environment are equivalent to spheres and cylinders that wrap the obstacles; the distance between each line segment of the robot arm and the center of the sphere and the axis segment of the cylinder is calculated; the minimum distance is determined according to the calculated distance, the preset thickness, the radius of the sphere, and the radius of the cylinder.

[0147] The surgical robot control device based on embodied intelligence provided in the above-mentioned embodiments of the present application has a corresponding relationship with the surgical robot control method based on embodied intelligence provided in the embodiments of the present application. Therefore, the specific content in the device has a corresponding relationship with the surgical robot control method based on embodied intelligence. The specific content can be referred to the records in the surgical robot control method based on embodied intelligence, and will not be repeated in this application.

[0148] The surgical robot control device based on embodied intelligence provided by the above-mentioned embodiments of the present application and the surgical robot control method based on embodied intelligence provided by the embodiments of the present application are based on the same inventive concept and have the same beneficial effects as the methods adopted, run or implemented by the application programs stored therein.

[0149] The above describes the internal functions and structure of the surgical robot control device based on embodied intelligence, such as Figure 4 As shown, in practice, the surgical robot control device based on embodied intelligence can be implemented as an electronic device, including: a memory 301 and a processor 303.

[0150] The memory 301 may be configured to store programs.

[0151] In addition, the memory 301 may also be configured to store various other data to support operations on the electronic device. Examples of such data include instructions for any application or method operating on the electronic device, contact data, phone book data, messages, pictures, videos, etc.

[0152] The memory 301 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.

[0153] The processor 303 is coupled to the memory 301 and is configured to execute a program in the memory 301 to:

[0154] Obtain a reward function based on inverse reinforcement learning processing;

[0155] Determine constraints based on past experience scenarios and differences in application scenarios;

[0156] Based on multi-objective optimization, the operation strategy of the surgical robot arm is trained through reinforcement learning;

[0157] After the training is completed, the running trajectory with the highest cumulative reward is selected, and the surgical robot arm is controlled to execute the running trajectory; the multi-objective optimization includes multiple objective functions, and the objective function includes at least the reward function and the constraint condition.

[0158] In one implementation, the processor 303 is further configured to:

[0159] Acquire past experience scenario information and application scenario information; determine the constraint direction of the past scenario based on the past experience scenario information; determine the constraint direction of the application scenario based on the application scenario information; determine the difference direction based on the constraint direction of the application scenario of the past scenario, the difference direction belongs to the application scenario and does not belong to the past scenario; determine the constraint condition based on the difference direction.

[0160] In one embodiment, the constraint conditions include obstacle constraints, and / or safety boundary constraints, and / or end tool orientation constraints, and / or mechanical constraints.

[0161] In one implementation, the processor 303 is further configured to:

[0162] Acquire operating environment data and planning data to construct a simulation space; construct a state space and an action space based on an inverse reinforcement learning process and the operating environment data and planning data; the action space includes multiple joint axes, each of which has a forward rotation action and a reverse rotation action; construct a deep Q network model; perform a first training on the deep Q network model based on deep Q network learning using the acquired reward function; perform a second training on the deep Q network model based on deep Q network learning using differential constraints as the reward function; alternately perform the first training and the second training until a preset condition is met.

[0163] In one implementation, the processor 303 is further configured to:

[0164] A reward function is constructed according to the differential constraints; an experience replay pool is constructed, which includes the action, state, reward, and next state of the agent; samples are taken from the experience replay pool, and the Q value is calculated based on the deep Q network model; and the weights of the deep Q network model are updated by minimizing the loss function until the training is completed.

[0165] In one implementation, the processor 303 is further configured to:

[0166] Obtain the robot arm posture data in the current state; obtain the environmental data in the current state; determine the minimum distance between the robot arm and the environmental obstacle based on the robot arm posture data and the environmental data; when the minimum distance between the robot arm and the environmental obstacle is greater than the preset distance, determine that the obstacle constraint is satisfied.

[0167] In one implementation, the processor 303 is further configured to:

[0168] The robot arm posture data is equivalently processed, and the robot arm is equivalent to a plurality of line segments connected in sequence and a preset thickness; the environmental data is equivalently processed, and the obstacles in the environment are equivalent to spheres and cylinders that wrap the obstacles; the distance between each line segment of the robot arm and the center of the sphere and the axis segment of the cylinder is calculated; the minimum distance is determined according to the calculated distance, the preset thickness, the radius of the sphere, and the radius of the cylinder.

[0169] In the present application, the processor is also specifically used to execute all the processes and steps of the above-mentioned surgical robot control method based on embodied intelligence. The specific content can be referred to the records in the surgical robot control method based on embodied intelligence, and will not be repeated in this application.

[0170] In this application, Figure 4 Only some components are shown schematically, which does not mean that the electronic device only includes Figure 4 Components shown.

[0171] The electronic device provided in this embodiment is based on the same inventive concept as the surgical robot control method based on embodied intelligence provided in the embodiment of the present application, and has the same beneficial effects as the method adopted, run or implemented by the application program stored therein.

[0172] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-readable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.

[0173] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0174] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0175] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0176] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0177] The memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash memory (Flash RAM). The memory is an example of a computer-readable medium.

[0178] The present application also provides a computer-readable storage medium corresponding to the surgical robot control method based on embodied intelligence provided in the aforementioned embodiments, on which a computer program (i.e., a program product) is stored. When the computer program is run by a processor, it will execute the surgical robot control method based on embodied intelligence provided in any of the aforementioned embodiments.

[0179] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0180] The computer-readable storage medium provided in the above-mentioned embodiments of the present application and the surgical robot control method based on embodied intelligence provided in the embodiments of the present application are based on the same inventive concept and have the same beneficial effects as the methods adopted, run or implemented by the application programs stored therein.

[0181] It should be noted that in the description provided herein, a large number of specific details are described. However, it is understood that the embodiments of the present application can be practiced without these specific details. In some instances, well-known structures and technologies are not shown in detail so as not to obscure the understanding of this description.

[0182] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0183] The above is only an embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included in the scope of the claims of the present application.

Claims

1. A surgical robot control method based on embodied intelligence, characterized in that: include: Obtain a reward function based on inverse reinforcement learning processing; Determine constraints based on past experience scenarios and differences in application scenarios; Based on multi-objective optimization, the operation strategy of the surgical robot arm is trained through reinforcement learning; After the training is completed, the running trajectory with the highest cumulative reward is selected, and the surgical robot arm is controlled to execute the running trajectory; the multi-objective optimization includes multiple objective functions, and the objective function includes at least the reward function and the constraint condition.

2. The surgical robot control method based on embodied intelligence according to claim 1, characterized in that: The determination of constraints based on differences between past experience scenarios and application scenarios includes: Obtain past experience scenario information and application scenario information; Determine the constraint direction of past scenes based on past experience scene information; Determine the constraint direction of the application scenario based on the application scenario information; Determine a difference direction according to a constraint direction of the past scene application scenario, wherein the difference direction belongs to the application scenario and does not belong to the past scene; A constraint condition is determined according to the difference direction.

3. The surgical robot control method based on embodied intelligence according to claim 2, characterized in that: The constraint conditions include obstacle constraints, and / or safety boundary constraints, and / or end tool orientation constraints, and / or mechanical constraints.

4. The surgical robot control method based on embodied intelligence according to claim 3, characterized in that: The operation strategy of the surgical robot arm is trained by reinforcement learning based on multi-objective optimization, including: Obtain operating environment data and planning data to build a simulation space; Based on the inverse reinforcement learning process and the operating environment data and planning data, a state space and an action space are constructed; the action space includes a plurality of joint axes, each of which has a forward rotation action and a reverse rotation action; Build a deep Q network model; The obtained reward function is used to first train the deep Q network model based on deep Q network learning; Taking the difference constraint as the reward function, the deep Q network model is trained for the second time based on deep Q network learning; The first training and the second training are performed alternately until a preset condition is met.

5. The surgical robot control method based on embodied intelligence according to claim 4, characterized in that: The second training of the deep Q network model based on deep Q network learning using the difference constraint as the reward function includes: Construct a reward function based on the constraints of diversity; Constructing an experience replay pool, wherein the experience replay pool includes the action, state, reward, and next state of the agent; Sampling from the experience replay pool and calculating the Q value based on the deep Q network model; By minimizing the loss function, the weights of the deep Q network model are updated until the training is completed.

6. The surgical robot control method based on embodied intelligence according to claim 5, characterized in that: The calculation process of the obstacle constraint is: Get the robot arm posture data in the current state; Get the current state of the environment data; Determine the minimum distance between the robot arm and environmental obstacles based on the robot arm posture data and environmental data; When the minimum distance between the robot arm and the environmental obstacle is greater than a preset distance, it is determined that the obstacle constraint is satisfied.

7. The surgical robot control method based on embodied intelligence according to claim 6, characterized in that: Determining the minimum distance between the robot arm and the environmental obstacle according to the robot arm posture data and the environmental data includes: Performing equivalent processing on the robot arm posture data, the robot arm is equivalent to a plurality of line segments connected in sequence and a preset thickness; Perform equivalent processing on the environmental data, and equate the obstacles in the environment to spheres and cylinders that wrap the obstacles; Calculate the distance between each line segment of the robot arm and the center of the sphere and the axis segment of the cylinder; The minimum distance is determined according to the calculated distance, the preset thickness, the radius of the sphere, and the radius of the cylinder.

8. A surgical robot control device based on embodied intelligence, characterized in that: include: A reward acquisition module, which is used to obtain a reward function based on inverse reinforcement learning processing; A difference determination module is used to determine the constraint conditions based on the difference between past experience scenarios and application scenarios; A strategy training module, which is used to train the operation strategy of the surgical robot arm through reinforcement learning based on multi-objective optimization; A robotic arm control module is used to select the running trajectory with the highest cumulative reward after training is completed, and control the surgical robot robotic arm to execute the running trajectory; the multi-objective optimization includes multiple objective functions, and the objective function includes at least the reward function and the constraint condition.

9. An electronic device, characterized in that: include: Memory and processor; The memory is used to store programs; The processor, coupled to the memory, is configured to execute the program to: Obtain a reward function based on inverse reinforcement learning processing; Determine constraints based on past experience scenarios and differences in application scenarios; Based on multi-objective optimization, the operation strategy of the surgical robot arm is trained through reinforcement learning; After the training is completed, the running trajectory with the highest cumulative reward is selected, and the surgical robot arm is controlled to execute the running trajectory; the multi-objective optimization includes multiple objective functions, and the objective function includes at least the reward function and the constraint condition.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: The program is executed by the processor to implement the surgical robot control method based on embodied intelligence as described in any one of claims 1-7.

Images