Mechanical arm control information generation method and device and surgical robot

Abstract

The invention relates to a mechanical arm control information generation method and device and a surgical robot. The method comprises the following steps: determining a target operation position; through kinematics modeling, motion control information of the mechanical arm is determined based on the target operation position; inputting the motion control information into a pre-trained neural network model to obtain a predicted operation position corresponding to the motion control information; on the basis of the target operation position and the predicted operation position, under the condition that it is determined that the predicted operation position does not meet the requirement, the motion control information is adjusted, the adjusted motion control information is input into the neural network model, and the predicted operation position corresponding to the motion control information is obtained till the motion control information is obtained on the basis of the target operation position and the predicted operation position. And under the condition that it is determined that the predicted operation position meets the requirement, the final adjusted motion control information serves as control information of the mechanical arm. By adopting the method, the control precision of the mechanical arm can be improved.

Description

Technical Field

[0001] The present application relates to the field of medical device control technology, and in particular to a method and device for generating control information for a robotic arm and a surgical robot. Background Art

[0002] When a puncture or ablation device is operated by a robotic arm controlling a surgical instrument, it is often assumed that the tissue is not offset. Position-level inverse kinematics is used to directly calculate the motion matrix of each robotic arm, and the robotic arm is controlled based on the motion matrix so that the surgical instrument reaches the target position.

[0003] However, biological tissue itself is often non-rigid, and is subjected to force when the surgical instrument reaches the target position, causing the tissue to deviate to a certain extent. In addition, the design accuracy of the robotic arm itself has certain errors, which often leads to a certain deviation between the target position and the ideal position of the surgical instrument.

[0004] Therefore, there is an urgent need for a method that can accurately control the robotic arm so that the target position of the surgical instrument is consistent with the ideal position under the control of the robotic arm. Summary of the invention

[0005] Based on this, it is necessary to provide a method, device and surgical robot for generating robotic arm control information that can make the target position of the surgical instrument consistent with the ideal position in order to address the above technical problems.

[0006] In a first aspect, the present application provides a method for generating robot arm control information, the method comprising:

[0007] Determine the target operating location;

[0008] Determining motion control information of the robot arm based on the target operation position through kinematic modeling;

[0009] Inputting the motion control information into a pre-trained neural network model to obtain a predicted operation position corresponding to the motion control information;

[0010] When it is determined based on the target operation position and the predicted operation position that the predicted operation position does not meet the requirements, the motion control information is adjusted, and the adjusted motion control information is input into a pre-trained neural network model to obtain the predicted operation position corresponding to the motion control information, until it is determined based on the target operation position and the predicted operation position that the predicted operation position meets the requirements, and the final adjusted motion control information is used as the control information of the robotic arm.

[0011] In one embodiment, determining the target operation position includes:

[0012] Obtaining the image target location in medical images;

[0013] Acquire the conversion relationship between the image coordinate system of the medical image and the physical space coordinate system;

[0014] The image target position is converted into a physical space coordinate system based on the conversion relationship to obtain a target operation position.

[0015] In one embodiment, determining the motion control information of the robot arm based on the target operation position through kinematic modeling includes:

[0016] Performing kinematic modeling based on the physical parameters of the robotic arm to obtain a kinematic model;

[0017] Performing position-level inverse kinematics calculation based on the kinematic model and the target operation position to obtain a motion matrix of each joint of the robotic arm;

[0018] Performing position-level forward kinematics calculation based on the kinematic model and the motion matrix to obtain force information on each joint and connecting rod of the robotic arm;

[0019] The motion matrices of the joints and the force information on the joints and connecting rods of the robotic arm are used as the motion control information of the robotic arm.

[0020] In one embodiment, before inputting the motion control information into a pre-trained neural network model to obtain a predicted operation position corresponding to the motion control information, the method further includes:

[0021] Obtaining the operation position of each sample;

[0022] Determine each sample motion control information of the robot arm based on each sample operation position through kinematic modeling;

[0023] Controlling the robotic arm based on each of the sample motion control information to obtain each actual operation position of the robotic arm;

[0024] The sample motion control information of each of the robotic arms is taken as input, and the actual operation position of each of the robotic arms is taken as output, and a neural network model is trained to obtain a trained neural network model.

[0025] In one embodiment, after inputting the motion control information into a pre-trained neural network model to obtain a predicted operation position corresponding to the motion control information, the method further includes:

[0026] Based on the conversion relationship, the predicted operation position is converted into the image coordinate system of the medical image to obtain the image predicted operation position;

[0027] When a deviation between the image predicted operation position and the image target position is less than a preset deviation, determining that the predicted operation position meets the requirement;

[0028] When the image predicted operation position and the image target position are greater than or equal to the preset deviation, it is determined that the predicted operation position does not meet the requirement.

[0029] In one embodiment, after converting the predicted operation position into the image coordinate system of the medical image to obtain the image predicted operation position based on the conversion relationship, the method further includes:

[0030] The image-predicted operation position is displayed in the medical image.

[0031] In one embodiment, the adjusting the motion control information includes:

[0032] When the target operation position and the predicted operation position differ only in angle, adjusting the angle of the last joint of the robot arm;

[0033] When there is a deviation in distance between the target operation position and the predicted operation position, adjusting the angles of the multiple joints of the robotic arm;

[0034] When there is a deviation in both angle and distance between the target operation position and the predicted operation position, the angle of the last joint of the robotic arm is adjusted after adjusting the distance by adjusting multiple joints of the robotic arm.

[0035] In a second aspect, the present application further provides a robot arm control information generating device, the device comprising:

[0036] A target operation position determination module, used to determine the target operation position;

[0037] A motion control information determination module, configured to determine the motion control information of the robotic arm based on the target operation position through kinematic modeling;

[0038] A prediction module, used for inputting the motion control information into a pre-trained neural network model to obtain a predicted operation position corresponding to the motion control information;

[0039] A control module is used to adjust the motion control information when it is determined based on the target operation position and the predicted operation position that the predicted operation position does not meet the requirements, and input the adjusted motion control information into a pre-trained neural network model to obtain the predicted operation position corresponding to the motion control information, until it is determined based on the target operation position and the predicted operation position that the predicted operation position meets the requirements, and then use the final adjusted motion control information as the control information of the robotic arm.

[0040] In a third aspect, the present application further provides a surgical robot, comprising:

[0041] Robotic arms, and

[0042] A processor, wherein the processor is used to execute the method described in any one of the above embodiments to determine the control information of the robotic arm, and control the robotic arm based on the control information of the robotic arm.

[0043] In one embodiment, it also includes:

[0044] A medical image acquisition module, used for acquiring medical images;

[0045] The processor is also used to obtain the image target position in the medical image; obtain the conversion relationship between the image coordinate system and the physical space coordinate system of the medical image; and convert the image target position to the physical space coordinate system based on the conversion relationship to obtain the target operation position.

[0046] The above-mentioned method, device and surgical robot for generating control information of the robotic arm first determine the motion control information of the robotic arm based on the target operation position, and then input the motion control information into a pre-trained neural network model to obtain a predicted operation position corresponding to the motion control information. When the predicted operation position does not meet the requirements, the motion control information is adjusted. When it is determined based on the target operation position and the predicted operation position that the predicted operation position meets the requirements, the final adjusted motion control information is used as the control information of the robotic arm. In this way, the motion control information of the robotic arm is corrected through the predicted operation position of the neural network, so that the target position of the surgical instrument is finally made consistent with the ideal position, thereby ensuring the control accuracy of the robotic arm. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] In order to more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the drawings required for use in the embodiments of the present application or related technical descriptions will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0048] FIG. 1a is a schematic diagram of a surgical control system according to an embodiment;

[0049] Figure 1b is a schematic structural diagram of a patient trolley in one embodiment;

[0050] FIG2 is a schematic flow chart of a method for generating robot arm control information in one embodiment;

[0051] FIG3 is a schematic diagram of a situation where a target operation position and a predicted operation position are different in one embodiment;

[0052] FIG4 is a schematic diagram of a motion control information adjustment step in one embodiment;

[0053] FIG5 is a schematic diagram of conversion between an image coordinate system and a physical space coordinate system in one embodiment;

[0054] FIG6 is a kinematic model obtained by DH method modeling in one embodiment;

[0055] FIG7 is a flow chart of steps for generating motion control information in one embodiment;

[0056] FIG8 is a schematic diagram showing a deviation between an actual operating position and a target operating position obtained by controlling a robotic arm according to motion control information in one embodiment;

[0057] FIG9 is a schematic diagram of the structure of a neural network model in one embodiment;

[0058] FIG10 is a flow chart of the training steps of a neural network model in one embodiment;

[0059] FIG11 is a schematic diagram of training of an RBF model in one embodiment;

[0060] FIG12 is a schematic diagram of an output of an RBF model in one embodiment;

[0061] FIG13 is a schematic diagram of training of an LSTM model in one embodiment;

[0062] FIG14 is a schematic flow chart of a method for generating robot arm control information in another embodiment;

[0063] FIG15 is a block diagram of a device for generating control information for a robot arm according to an embodiment;

[0064] FIG. 16 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0065] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0066] The method for generating robotic arm control information provided in the embodiment of the present application can be applied to the surgical control system as shown in Figure 1a. The surgical robot control system includes a doctor's console, a patient trolley and an image trolley. The doctor's console includes a doctor's console, which is loaded with a main control arm, a controller, and a display. In a dual-doctor scenario, there will be two doctor's consoles, one for the surgeon and one for the assistant doctor; the patient trolley is loaded with a robotic arm, and the robotic arm is loaded with surgical instruments; the image trolley is loaded with a display, an image host, a cold light source, etc.

[0067] The robotic arm mounted on the patient cart can be manually controlled by the doctor's console, or controlled based on an automatic control algorithm to move the surgical instrument to the target operating position for subsequent operations. One point that needs to be explained is that the method for generating robotic arm control information involved in this application does not involve a specific diagnosis and treatment process, but is only for assistance, for example, before surgery, by controlling the robotic arm to move the surgical instrument to the target operating position.

[0068] Specifically, combined Figure 1b As shown, Figure 1b The schematic diagram of the structure of a patient trolley in an embodiment, in which the patient trolley includes a trolley motion module, a workbench, a medical image acquisition module and a mechanical arm. The workbench is arranged on the trolley motion module, and the medical image acquisition module and the mechanical arm assembly are installed on the workbench. The trolley motion module includes a sliding shaft at the bottom of the trolley, an upper and lower telescopic shaft, and a left and right rotating shaft to realize the movement of the trolley with six degrees of freedom, wherein the upper and lower telescopic shafts slide left and right at the bottom of the trolley, the medical image acquisition module can acquire medical images, and the medical image acquisition module can also be an ultrasonic probe for acquiring ultrasonic images. The mechanical arm includes a support rod, a mechanical arm positioning shaft, a mechanical arm needle insertion shaft and a surgical instrument, the mechanical arm positioning shaft is fixed on the support rod, the mechanical arm needle insertion shaft is connected to the mechanical arm positioning shaft, and the surgical instrument is installed on the mechanical arm needle insertion shaft. The processor corresponding to the patient trolley can execute the mechanical arm control information generation method of the present application to move the surgical instrument to the target operating position. In an exemplary embodiment, as shown in FIG2, a method for generating mechanical arm control information is provided, which is described by taking the method applied to the patient trolley in FIG1a as an example, and includes the following steps 202 to 208. Among them:

[0069] S202: Determine a target operation position.

[0070] The target operation position is the target position of the end of the robot arm, such as the target position of the surgical instrument on the end of the robot arm. Optionally, the target operation position is the target operation position in the physical space coordinate system. Optionally, the image target position can be determined based on the medical image first, and then the image target position can be converted to the physical space coordinate system to obtain the target operation position. The physical space is the real space.

[0071] S204: Determine motion control information of the robot arm based on the target operation position through kinematic modeling.

[0072] After the target operation position is determined, the motion control information of the robot arm is determined based on position-level inverse kinematics with the target operation position as input.

[0073] The motion control information includes the motion matrix of each joint, the joints of the robot arm, and the force information on the connecting rod.

[0074] Optionally, a corresponding kinematic model can be first established based on the structure of the robotic arm, and then the motion matrix of each joint can be determined based on the target operation position. Then, position-level positive kinematics calculations can be performed based on the motion matrix and kinematic model of each joint to obtain the force information on each joint and connecting rod of the robotic arm.

[0075] S206: Input the motion control information into a pre-trained neural network model to obtain a predicted operation position corresponding to the motion control information.

[0076] The pre-trained neural network model is trained based on samples, the input of which is the motion control information of the robot arm, and the output is the actual position reached by the robot arm after being controlled by the motion control information. In this way, the introduction of the neural network can correct the deviation caused by tissue force and the design accuracy of the robot arm itself.

[0077] Therefore, by inputting the motion control information into the neural network model, the predicted operation position corresponding to the motion control information can be obtained. The predicted operation position fully considers the deviation caused by tissue force and the design accuracy of the robot arm itself.

[0078] S208: When it is determined based on the target operation position and the predicted operation position that the predicted operation position does not meet the requirements, the motion control information is adjusted, and the adjusted motion control information is input into the pre-trained neural network model to obtain the predicted operation position corresponding to the motion control information, until it is determined based on the target operation position and the predicted operation position that the predicted operation position meets the requirements, and the final adjusted motion control information is used as the control information of the robotic arm.

[0079] Among them, the predicted operation position can be the predicted operation position in the physical space coordinate system. In this way, when the target operation position and the predicted operation position are not the same, the predicted operation position does not meet the requirements. Otherwise, when the target operation position and the predicted operation position are the same, the predicted operation position meets the requirements. That is to say, the adjusted motion control information can enable the end of the robotic arm to reach the target operation position.

[0080] Among them, in combination with Figure 3, Figure 3 is a schematic diagram of a situation in which the target operation position and the predicted operation position are different in an embodiment. When the target operation position and the predicted operation position are different, the motion control information can be adjusted. Specifically, in combination with Figure 4, Figure 4 is a schematic diagram of the motion control information adjustment step in an embodiment, so that the adjusted motion control information is input into the pre-trained neural network model to obtain the predicted operation position corresponding to the adjusted motion control information, and continue to judge whether the target operation position and the predicted operation position are the same. If they are not the same, continue to adjust the motion control information until the target operation position and the predicted operation position are the same, then use the final adjusted motion control information as the control information of the robotic arm, so that the end of the robotic arm can reach the target operation position.

[0081] In one of the optional embodiments, adjusting the motion control information includes: when there is a deviation in angle between the target operation position and the predicted operation position only, adjusting the angle of the last joint of the robotic arm; when there is a deviation in distance between the target operation position and the predicted operation position, adjusting the angles of multiple joints of the robotic arm; when there is a deviation in both angle and distance between the target operation position and the predicted operation position, adjusting the angle of the last joint of the robotic arm after adjusting the distance by adjusting multiple joints of the robotic arm.

[0082] Among them, in combination with FIG. 4, the adjustment of the motion control information can be performed based on the deviation between the target operation position and the predicted operation position. For example, if the target operation position and the predicted operation position in FIG. 4 only have a deviation in angle, the angle of the last joint of the robot arm can be adjusted so that the predicted operation position at the end of the robot arm reaches the target operation position. If there is a deviation in distance, the angles of multiple joints of the robot arm can be adjusted so that the predicted operation position at the end of the robot arm reaches the target operation position. If there is both a deviation in distance and a deviation in angle, the angles of multiple joints of the robot arm can be adjusted first, and then the angle of the last joint of the robot arm can be adjusted so that the predicted operation position at the end of the robot arm reaches the target operation position. The above adjustment of the motion control information is only an example. Those skilled in the art can adjust the motion control information based on the actual situation until the predicted operation position obtained by the neural network model based on the adjusted motion control information is consistent with the target operation position.

[0083] The above-mentioned method for generating robot arm control information first determines the motion control information of the robot arm based on the target operation position, and then inputs the motion control information into a pre-trained neural network model to obtain a predicted operation position corresponding to the motion control information. When the predicted operation position does not meet the requirements, the motion control information is adjusted. When it is determined based on the target operation position and the predicted operation position that the predicted operation position meets the requirements, the robot arm is controlled based on the final adjusted motion control information. In this way, the motion control information of the robot arm is corrected through the predicted operation position of the neural network, and finally the target position of the surgical instrument is consistent with the ideal position, thereby ensuring the control accuracy of the robot arm.

[0084] In one of the optional embodiments, determining the target operation position includes: obtaining an image target position in a medical image; obtaining a conversion relationship between an image coordinate system of the medical image and a physical space coordinate system; and converting the image target position to a physical space coordinate system based on the conversion relationship to obtain the target operation position.

[0085] Specifically, as shown in FIG5 , FIG5 is a schematic diagram of the conversion between the image coordinate system and the physical space coordinate system in an embodiment. In this embodiment, a medical image of the target area can be acquired by a medical image acquisition device, for example, a medical image of the lesion area of ​​the patient can be acquired. The medical image can be a nuclear magnetic resonance, CT or ultrasound image, and the image target position in the medical image can be determined by lesion recognition. In addition, the conversion relationship between the image coordinate system of the medical image and the physical space coordinate system can be obtained by feature point recognition or markers, and finally the image target position is converted to the physical space coordinate system based on the conversion relationship to obtain the target operation position. In conjunction with FIG5 , the target in the plane is the image target position to be reached, and the pivot is the initial position of the end of the robot arm. The spatial distance between them is known, and the image target position can be converted into the target operation position through the translation matrix and the scale matrix.

[0086] In one of the optional embodiments, the motion control information of the robotic arm is determined based on the target operating position through kinematic modeling, including: performing kinematic modeling based on the physical parameters of the robotic arm to obtain a kinematic model; performing position-level inverse kinematics calculations based on the kinematic model and the target operating position to obtain the motion matrix of each joint of the robotic arm; performing position-level forward kinematics calculations based on the kinematic model and the motion matrix to obtain the force information on each joint and connecting rod of the robotic arm; and using the motion matrix of each joint and the force information on each joint and connecting rod of the robotic arm as the motion control information of the robotic arm.

[0087] The physical parameters of the robot arm include the length of each connecting rod, the position of the joint, the center of mass of the connecting rod, etc. A kinematic model can be established based on these physical parameters, wherein the modeling methods include but are not limited to T matrix modeling, DH modeling, MDH modeling, MCPC modeling, etc.

[0088] In combination with Figure 6, Figure 6 is a kinematic model obtained by DH method modeling in one embodiment. The mechanical arm shown in Figure 6 is a mechanical arm with two connecting rods, the bottom triangle represents the base, and the circle between the base and the connecting rod represents the joint.

[0089] The DH method is used to establish the local coordinate system of the connecting rod as shown in Figure 6, with the z axis perpendicular to the plane and upward, where the reference coordinate system is x0y0z0, where δ i , α i ,d i 、a iRespectively represent the joint angle, connecting rod torsion angle, connecting rod distance, and connecting rod length data. Apply the DH method to establish the x1y1z1 coordinate system. The x1 axis is obtained by rotating the x0 axis around the z0 axis by θ1. Since there is no displacement along the z0 axis between the x1 and x0 axes, d1=0 is obtained. The displacement along the x1 axis is a1. With the x1 axis as the rotation axis, there is no rotation between the axes z1 and z0, so α1=0. Similarly, the four DH parameters between the end effector coordinate system x2y2z2 of the second connecting rod and the x1y1z1 coordinate system are obtained. i-1 Y i-1 Z i-1 Perform rotation and translation transformations in sequence to obtain the new coordinate system X i Y i Z i The purpose of this application is that the coordinates of the end effector of the connecting rod are the target operating position.

[0090] In this way, the various parameters of the robot arm can be obtained, as shown in Table 1 below:

[0091] Table 1:

[0092]

[0093] DH parameters of the robot arm: The DH parameters of the two links are shown in Figure 6. If point p2 is in the coordinate system x2y2z2, the transformation method to convert it into the coordinates in the x0y0z0 coordinate system is:

[0094]

[0095] Among them, S and C represent the sin function and cos function respectively, and Ciθ represents cosθ i , Siθ represents sinθ i , that is, when the coordinates of (Px2 Py2 Pz2) are known, the coordinates (Px0 Py0 Pz0) relative to the original base can be deduced. Conversely, when the coordinates represented by the original base coordinate system are known, the coordinates in the link 2 coordinate system can also be deduced. When the coordinates of the image operation position are known, they are converted into the target operation position in the real physical coordinates. The motion parameters of each link can be obtained by analytical method, geometric method or iterative method, that is, the motion matrix of each joint is obtained.

[0096] Specifically, in conjunction with FIG. 7, FIG. 7 is a flow chart of the steps of generating motion control information in an embodiment. In this embodiment, the image operation position in the image coordinate system is converted into the target operation position in the physical space coordinate system, and then the motion matrix of each joint is obtained based on the target operation position through position-level inverse kinematics. It is subsequently assumed that the motion matrix of each joint is input into the manipulator, and then it becomes a position-level positive kinematics problem, that is, the solution of the space velocity-level positive kinematics under the conditions of the positions and postures of each part of the space manipulator system. In this way, in the inertial system, the center of mass position vector of each manipulator link is known, and the center of mass velocity of the manipulator link can be known, so that the angular velocity and velocity are derived to obtain the angular acceleration and linear acceleration. Because the center of mass of each link is known, all inertial forces and moments of inertia are known, so that the external forces and moments of inertia on the link can be deduced.

[0097] In the above embodiment, after the target operation position is known, the motion control information of the robot arm can be obtained through kinematic modeling.

[0098] As shown in Figure 8, Figure 8 is a schematic diagram of the deviation between the actual operating position and the target operating position obtained by controlling the robotic arm according to motion control information in an embodiment. In this embodiment, since the robotic arm itself contains positioning errors (that is, each joint has friction, coupling, and motor drive has errors), and during operation, the end of the robotic arm is subjected to external force from the tissue. This external force is not a definite constant, so it will affect the kinematic speed level and various parameters of the dynamic force and torque of the robotic arm. At the same time, the tissue is also affected by the force of the robotic arm, resulting in an offset, causing a positioning error in the robotic arm. The actual operating position and the target operating position do not coincide, thereby reducing the accuracy. In order to avoid this situation, in this application, the end of the robotic arm is first predicted based on the motion control parameters to obtain a predicted operating position, and the motion control parameters are adjusted to make the predicted operating position consistent with the target operating position, and then the robotic arm is controlled by adjusting the motion control parameters to improve the control accuracy of the robotic arm.

[0099] In one of the optional embodiments, before the motion control information is input into a pre-trained neural network model to obtain the predicted operation position corresponding to the motion control information, it also includes: obtaining each sample operation position; determining each sample motion control information of the robotic arm based on each sample operation position through kinematic modeling; controlling the robotic arm based on each sample motion control information to obtain each actual operation position of the robotic arm; and training the neural network model using the sample motion control information of each robotic arm as input and the actual operation position of each robotic arm as output to obtain a trained neural network model.

[0100] In this embodiment, the training steps of the neural network model are given, wherein each training sample includes an input and an output, the input is the sample motion control information, and the output is the actual operating position of the robotic arm.

[0101] As shown in Figure 9, the sample motion control information, i.e., the kinematic parameters of each joint and the dynamic parameters of the robot arm, corresponds to an actual position output, i.e., the actual operation position of the robot arm. Then, the nonlinear mapping of a combination parameter corresponding to a position output can be fitted by a neural network. Optionally, an RBF neural network, an LSTM neural network, or other deep learning architectures can be used for fitting, such as transformerd. As shown in Figure 9, X i (n) is the input layer, C i (n) is the hidden layer, and Y(n) is the output layer. The input layer inputs the sample motion control information, that is, the various parameters of the robot's kinematics and dynamics; the hidden layer learning is usually unsupervised, and is used to determine the basis function parameters, that is, to determine the center and width, and adjust the weight value; the output layer is used to determine the real physical coordinates of the output relative to the initial position.

[0102] As shown in Figure 10, after the network is initialized, input excitation is added, that is, the sample motion control information mentioned above. The first stage calculates the output of the hidden layer neurons, that is, the K-means algorithm is used. Its task is to use the self-organizing clustering method to determine the appropriate data center for the radial basis function of the hidden layer nodes. The second stage is supervised learning, which calculates the output of the entire network and adjusts the weights until the global error of the network meets the accuracy requirements. Its task is to train the output layer weights with a supervised learning algorithm. The gradient method can be used. When the final error converges to a sufficiently small threshold or the number of iterations reaches a sufficiently large number, the convergence stops and the model training ends.

[0103] As shown in FIG. 11 , FIG. 11 is a schematic diagram of the training of the RBF model in an embodiment, and FIG. 12 is a schematic diagram of the output of the RBF model in an embodiment, wherein the input layer is the sample motion control information, i.e., the various parameters of the kinematics and dynamics of the robot arm. Assuming that the DH method is used for modeling, the actual needle insertion data each time includes the kinematics and dynamics parameters (δ i , α i ,d i 、a i 、V i ,ω i , i , C bi , C θi ...), where δ i , α i ,d i 、ai 、V i ,ω i , i , C bi , C θi Respectively represent the parameters such as the joint angle data, connecting rod torsion angle data, connecting rod distance data, connecting rod length data, connecting rod velocity data, connecting rod angular velocity data, connecting rod acceleration data, connecting rod nonlinear force, connecting rod nonlinear torque, etc., and use them as a set of inputs for training. There are many forms of radial basis functions, and the commonly used ones are: Gaussian function, multi-quadric function (multi_x0002_quadricfunction), inverse multi-quadric function, thin plate spline function, etc. The neural network can use Gaussian function, and the k-th node of RBF completes the linear combination of the output of the hidden layer nodes, which can be expressed as:

[0104]

[0105] Hidden layer learning is usually unsupervised and is used to determine the basis function parameters, that is, to determine the center and width, and adjust the weight value. There are generally two ways to select the data center: select the data center from the sample input and dynamically adjust the method. Generally, the input data is used to select the center vector and the standardization parameters. The output layer is supervised learning, which mainly determines the weight. The network center and width are known through hidden layer learning. At this time, the input layer introduces the offset term b i is the error between the actual coordinates and the true coordinates of each set of data, where (e x1 e x2 e x3 ...e xn ) is the horizontal coordinate X of each set of coordinates and the actual coordinates i The error is finally outputted and regressed to the horizontal coordinates of the actual coordinates of each set of data (x1 x2 x3 ....x n ), the same is true for the y-axis and z-axis, and finally LMS and RLS can be used to solve the weights. That is, after the entire model obtains the trained weights, the kinematic and dynamic data can be input to obtain the actual reality prediction coordinates, and the mapping of the actual coordinates in the image coordinates can be displayed by the software through conversion with the image coordinates.

[0106] As shown in FIG. 13 , FIG. 13 is a schematic diagram of the training of the LSTM model in one embodiment, the input layer processes the sample motion control information, that is, the original kinematics and dynamics parameter data of the robot arm, and constructs a feature sequence (δ i , α i ,d i 、a i 、V i ,ω i , i , C bi , Cθi ...) to meet the network input conditions. Then the input of the input layer enters the hidden layer. The main task of the hidden layer is network training. The processed input is input to the hidden layer. The hidden layer contains M isomorphic LSTM cell units connected in sequence. The output of X after the hidden layer is compared with the theoretical output, and the loss function is calculated. The minimization of the loss function is used as the optimization goal. The initial values ​​such as the learning rate and the number of training iterations are set, and the network weights are continuously updated to determine the final hidden layer network. Similarly, the final X, Y, and Z are the coordinate values ​​on each coordinate axis output by each feature sequence. The maximum number of iterations can be set or the iteration stops after the loss function converges to a certain value, and the model training is completed.

[0107] In one of the optional embodiments, after the motion control information is input into a pre-trained neural network model to obtain the predicted operation position corresponding to the motion control information, it also includes: based on the conversion relationship, converting the predicted operation position to the image coordinate system of the medical image to obtain the image predicted operation position; when the deviation between the image predicted operation position and the image target position is less than a preset deviation, determining that the predicted operation position meets the requirements; when the image predicted operation position and the image target position are greater than or equal to the preset deviation, determining that the predicted operation position does not meet the requirements.

[0108] The determination of whether the predicted operation position meets the requirements can be based on the image coordinate system or the physical space coordinate system, wherein the determination based on the physical space coordinate system can directly compare the target operation position with the predicted operation position. The determination based on the image coordinate system can be based on a conversion relationship, converting the predicted operation position into the image coordinate system of the medical image to obtain the image predicted operation position, and then comparing the image predicted operation position with the image target position.

[0109] By converting to the image coordinate system, the image predicted operation position and the image target position can be directly displayed, so that the operator can judge whether the predicted operation position meets the requirements, or the operator can automatically judge whether the predicted operation position meets the requirements based on the image predicted operation position and the image target position.

[0110] Optionally, based on the conversion relationship, after converting the predicted operation position into the image coordinate system of the medical image to obtain the image predicted operation position, the method further includes: displaying the image predicted operation position in the medical image.

[0111] For ease of understanding, FIG14 is combined with FIG14, which is a flow chart of a method for generating control information of a robotic arm in another embodiment. First, the image target position in the image coordinate system is converted into the target operation position in the physical space coordinate system, and then kinematic modeling is performed based on the target operation position in the physical space coordinate system to obtain motion control information of the robotic arm; the motion control information of the robotic arm is input into a pre-trained neural network model to obtain a predicted operation position, and the predicted operation position is converted into the image coordinate system to obtain an image predicted operation position, and the image predicted operation position is displayed, so that it can be determined whether the image predicted operation position in the image coordinate system coincides with the image target position. If so, the motion control information of the robotic arm is used as the final control information of the robotic arm. Otherwise, the motion control information can be adjusted based on the relationship between the image predicted operation position in the image coordinate system and the image target position, and then the adjusted motion control information is input into the pre-trained neural network model to obtain a new predicted operation position, and the predicted operation position is continued to be converted into the image coordinate system to obtain the image predicted operation position, until the image predicted operation position in the image coordinate system coincides with the image target position, and the motion control information of the robotic arm is used as the final control information of the robotic arm.

[0112] In the above embodiment, by collecting data on the characteristics of the robotic arm and various parameters and using a neural network to predict the actual output, it is more accurate than simply using position-level inverse dynamics to calculate the motion matrix of each robotic arm. Through the predictive training model and the conversion between images and real-world coordinates, the kinematic input of the robotic arm can be displayed in real time in the image coordinate system, which can achieve the generation of more accurate control information for the robotic arm, thereby improving the accuracy of subsequent puncture or ablation, reducing the pain of repeated punctures for patients, and ensuring that the operation is more in line with expectations.

[0113] It should be understood that, although the steps in the flowcharts involved in the above embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps is not strictly limited in order, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above embodiments may include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0114] Based on the same inventive concept, the embodiment of the present application also provides a robot arm control information generating device for implementing the robot arm control information generating method involved above. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above method, so the specific limitations in one or more robot arm control information generating device embodiments provided below can refer to the limitations of the robot arm control information generating method above, and will not be repeated here.

[0115] In an exemplary embodiment, as shown in FIG15 , a device for generating control information of a robot arm is provided, comprising: a target operation position determination module 1501, a motion control information determination module 1502, a prediction module 1503 and a control module 1504, wherein:

[0116] The target operation position determination module 1501 is used to determine the target operation position;

[0117] A motion control information determination module 1502 is used to determine the motion control information of the robot arm based on the target operation position through kinematic modeling;

[0118] Prediction module 1503, used to input motion control information into a pre-trained neural network model to obtain a predicted operation position corresponding to the motion control information;

[0119] The control module 1504 is used to adjust the motion control information when it is determined that the predicted operation position does not meet the requirements based on the target operation position and the predicted operation position, and input the adjusted motion control information into the pre-trained neural network model to obtain the predicted operation position corresponding to the motion control information, until it is determined that the predicted operation position meets the requirements based on the target operation position and the predicted operation position, and the final adjusted motion control information is used as the control information of the robotic arm.

[0120] In one of the optional embodiments, the above-mentioned target operation position determination module 1501 is specifically used to obtain the image target position in the medical image; obtain the conversion relationship between the image coordinate system and the physical space coordinate system of the medical image; and convert the image target position to the physical space coordinate system based on the conversion relationship to obtain the target operation position.

[0121] In one of the optional embodiments, the motion control information determination module 1502 is specifically used to perform kinematic modeling based on the physical parameters of the robotic arm to obtain a kinematic model; perform position-level inverse kinematics calculations based on the kinematic model and the target operation position to obtain the motion matrix of each joint of the robotic arm; perform position-level forward kinematics calculations based on the kinematic model and the motion matrix to obtain the force information on each joint and connecting rod of the robotic arm; and use the motion matrix of each joint and the force information on each joint and connecting rod of the robotic arm as the motion control information of the robotic arm.

[0122] In one of the optional embodiments, the above-mentioned device also includes: a model training module, used to obtain each sample operation position; through kinematic modeling, determine each sample motion control information of the robotic arm based on each sample operation position; control the robotic arm based on each sample motion control information to obtain each actual operation position of the robotic arm; use the sample motion control information of each robotic arm as input and the actual operation position of each robotic arm as output, perform neural network model training, and obtain a trained neural network model.

[0123] In one of the optional embodiments, the above-mentioned device also includes: a conversion module, which is used to convert the predicted operation position into the image coordinate system of the medical image based on a conversion relationship to obtain the image predicted operation position; when the deviation between the image predicted operation position and the image target position is less than a preset deviation, it is determined that the predicted operation position meets the requirements; when the image predicted operation position and the image target position are greater than or equal to the preset deviation, it is determined that the predicted operation position does not meet the requirements.

[0124] In one of the optional embodiments, the above-mentioned device also includes: a display module, used to display the image predicted operation position in the medical image.

[0125] In one of the optional embodiments, the motion control information determination module 1502 is specifically used to adjust the angle of the last joint of the robotic arm when there is a deviation in angle between the target operation position and the predicted operation position; to adjust the angles of multiple joints of the robotic arm when there is a deviation in distance between the target operation position and the predicted operation position; and to adjust the angle of the last joint of the robotic arm after adjusting the distance by adjusting multiple joints of the robotic arm when there is a deviation in angle and distance between the target operation position and the predicted operation position.

[0126] Each module in the above-mentioned robot arm control information generating device can be implemented in whole or in part by software, hardware and their combination. Each of the above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to each of the above modules.

[0127] In an exemplary embodiment, a computer device is provided, which may be a terminal, and its internal structure diagram may be shown in FIG16. The computer device includes a processor, a memory, an input / output interface, a communication interface, a display unit, and an input device. The processor, the memory, and the input / output interface are connected via a system bus, and the communication interface, the display unit, and the input device are connected to the system bus via the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner may be implemented through WIFI, a mobile cellular network, near field communication (NFC) or other technologies. When the computer program is executed by the processor, a method for generating control information of a robotic arm is implemented. The display unit of the computer device is used to form a visually visible picture, which may be a display screen, a projection device, or a virtual reality imaging device. The display screen can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device can be a touch layer covering the display screen, or a button, trackball or touchpad set on the computer device shell, or an external keyboard, touchpad or mouse.

[0128] Those skilled in the art will understand that the structure shown in FIG. 16 is merely a block diagram of a partial structure related to the scheme of the present application, and does not constitute a limitation on the computer device to which the scheme of the present application is applied. The specific computer device may include more or fewer components than shown in the figure, or combine certain components, or have a different arrangement of components.

[0129] In one embodiment, a computer device is further provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps in the above method embodiments when executing the computer program.

[0130] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments are implemented.

[0131] In one embodiment, a computer program product is provided, including a computer program, which implements the steps in the above method embodiments when executed by a processor.

[0132] A person of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment method can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, an artificial intelligence (AI) processor, etc., but are not limited to this.

[0133] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0134] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A method for generating control information of a robot arm, characterized in that: The method comprises: Determine the target operating location; Determining motion control information of the robot arm based on the target operation position through kinematic modeling; Inputting the motion control information into a pre-trained neural network model to obtain a predicted operation position corresponding to the motion control information; When it is determined based on the target operation position and the predicted operation position that the predicted operation position does not meet the requirements, the motion control information is adjusted, and the adjusted motion control information is input into a pre-trained neural network model to obtain the predicted operation position corresponding to the motion control information, until it is determined based on the target operation position and the predicted operation position that the predicted operation position meets the requirements, and the final adjusted motion control information is used as the control information of the robotic arm.

2. The method according to claim 1, characterized in that The determining of the target operation position comprises: Obtaining the image target location in medical images; Acquire the conversion relationship between the image coordinate system of the medical image and the physical space coordinate system; The image target position is converted into a physical space coordinate system based on the conversion relationship to obtain a target operation position.

3. The method according to claim 2, characterized in that The step of determining the motion control information of the robot arm based on the target operation position through kinematic modeling includes: Performing kinematic modeling based on the physical parameters of the robotic arm to obtain a kinematic model; Performing position-level inverse kinematics calculation based on the kinematic model and the target operation position to obtain a motion matrix of each joint of the robotic arm; Performing position-level forward kinematics calculation based on the kinematic model and the motion matrix to obtain force information on each joint and connecting rod of the robotic arm; The motion matrices of the joints and the force information on the joints and connecting rods of the robotic arm are used as the motion control information of the robotic arm.

4. The method according to claim 3, characterized in that Before inputting the motion control information into a pre-trained neural network model to obtain a predicted operation position corresponding to the motion control information, the method further includes: Obtaining the operation position of each sample; Determine each sample motion control information of the robot arm based on each sample operation position through kinematic modeling; Controlling the robotic arm based on each of the sample motion control information to obtain each actual operation position of the robotic arm; The sample motion control information of each of the robotic arms is taken as input, and the actual operation position of each of the robotic arms is taken as output, and a neural network model is trained to obtain a trained neural network model.

5. The method according to any one of claims 2 to 4, characterized in that: After inputting the motion control information into a pre-trained neural network model to obtain a predicted operation position corresponding to the motion control information, the method further includes: Based on the conversion relationship, the predicted operation position is converted into the image coordinate system of the medical image to obtain the image predicted operation position; When a deviation between the image predicted operation position and the image target position is less than a preset deviation, determining that the predicted operation position meets the requirement; When the image predicted operation position and the image target position are greater than or equal to the preset deviation, it is determined that the predicted operation position does not meet the requirement.

6. The method according to claim 5, characterized in that After converting the predicted operation position into the image coordinate system of the medical image based on the conversion relationship to obtain the image predicted operation position, the method further includes: The image-predicted operation position is displayed in the medical image.

7. The method according to claim 1, characterized in that The adjusting the motion control information includes: When the target operation position and the predicted operation position differ only in angle, adjusting the angle of the last joint of the robot arm; When there is a deviation in distance between the target operation position and the predicted operation position, adjusting the angles of the multiple joints of the robotic arm; When there is a deviation in both angle and distance between the target operation position and the predicted operation position, the angle of the last joint of the robotic arm is adjusted after adjusting the distance by adjusting multiple joints of the robotic arm.

8. A robot arm control information generating device, characterized in that: The device comprises: A target operation position determination module, used to determine the target operation position; A motion control information determination module, configured to determine the motion control information of the robotic arm based on the target operation position through kinematic modeling; A prediction module, used for inputting the motion control information into a pre-trained neural network model to obtain a predicted operation position corresponding to the motion control information; A control module is used to adjust the motion control information when it is determined based on the target operation position and the predicted operation position that the predicted operation position does not meet the requirements, and input the adjusted motion control information into a pre-trained neural network model to obtain the predicted operation position corresponding to the motion control information, until it is determined based on the target operation position and the predicted operation position that the predicted operation position meets the requirements, and then use the final adjusted motion control information as the control information of the robotic arm.

9. A surgical robot, characterized in that: include: Robotic arms, and A processor, wherein the processor is used to execute the method described in any one of claims 1 to 7 to determine the control information of the robotic arm, and control the robotic arm based on the control information of the robotic arm.

10. The surgical robot according to claim 9, characterized in that: Also includes: A medical image acquisition module, used for acquiring medical images; The processor is also used to obtain the image target position in the medical image; obtain the conversion relationship between the image coordinate system and the physical space coordinate system of the medical image; and convert the image target position to the physical space coordinate system based on the conversion relationship to obtain the target operation position.

Images