Methods, devices, and surgical robots for generating control information for robotic arms

Abstract

This application relates to a method, apparatus, and surgical robot for generating control information for a robotic arm. The method includes: determining a target operating position; determining motion control information for the robotic arm based on the target operating position through kinematic modeling; inputting the motion control information into a pre-trained neural network model to obtain a predicted operating position corresponding to the motion control information; if, based on the target operating position and the predicted operating position, the predicted operating position does not meet the requirements, adjusting the motion control information and inputting the adjusted motion control information into the neural network model to obtain a predicted operating position corresponding to the motion control information; until, based on the target operating position and the predicted operating position, the predicted operating position meets the requirements, the final adjusted motion control information is used as the control information for the robotic arm. This method can improve the control accuracy of the robotic arm.

Description

Technical Field

[0001] This application relates to the field of medical device control technology, and in particular to a method, apparatus and surgical robot for generating control information for a robotic arm. Background Technology

[0002] When puncture or ablation devices operate surgical instruments with robotic arms, it is often assumed that the tissue is not deflected. Therefore, position-level inverse kinematics is used to directly calculate the motion matrix of each robotic arm, and the robotic arms are controlled based on the motion matrix so that the surgical instruments reach the target position.

[0003] However, biological tissues are often non-rigid bodies and are subjected to forces when surgical instruments reach the target position, causing the tissue to deviate to some extent. In addition, the robotic arm itself has certain errors in design precision, which often leads to a certain deviation between the target position and the ideal position of the surgical instruments.

[0004] Therefore, there is an urgent need for a method that can accurately control the robotic arm so that the target position of the surgical instruments matches the ideal position under the control of the robotic arm. Summary of the Invention

[0005] Therefore, it is necessary to provide a method, device, and surgical robot for generating robotic arm control information that enables the target position of surgical instruments to match the ideal position, in order to address the aforementioned technical problems.

[0006] In a first aspect, this application provides a method for generating robotic arm control information, the method comprising:

[0007] Determine the target operation location;

[0008] The motion control information of the robotic arm is determined based on the target operating position through kinematic modeling.

[0009] The motion control information is input into a pre-trained neural network model to obtain the predicted operation position corresponding to the motion control information;

[0010] If, based on the target operating position and the predicted operating position, it is determined that the predicted operating position does not meet the requirements, the motion control information is adjusted, and the adjusted motion control information is input into a pre-trained neural network model to obtain the predicted operating position corresponding to the motion control information. This process continues until, based on the target operating position and the predicted operating position, it is determined that the predicted operating position meets the requirements, at which point the final adjusted motion control information is used as the control information for the robotic arm.

[0011] In one embodiment, determining the target operation location includes:

[0012] To obtain the location of a target image in a medical image;

[0013] Obtain the transformation relationship between the image coordinate system and the physical space coordinate system of the medical image;

[0014] Based on the transformation relationship, the image target position is transformed to the physical space coordinate system to obtain the target operation position.

[0015] In one embodiment, determining the motion control information of the robotic arm based on the target operating position through kinematic modeling includes:

[0016] A kinematic model is obtained by kinematic modeling based on the physical parameters of the robotic arm;

[0017] Based on the kinematic model and the target operating position, position-level inverse kinematics calculations are performed to obtain the motion matrix of each joint of the robotic arm;

[0018] Based on the kinematic model and the motion matrix, position-level positive kinematics calculations are performed to obtain the force information on each joint and link of the robotic arm;

[0019] The motion matrix of each joint, the force information of each joint of the robotic arm, and the connecting rods are used as the motion control information of the robotic arm.

[0020] In one embodiment, before inputting the motion control information into a pre-trained neural network model to obtain the predicted operation position corresponding to the motion control information, the method further includes:

[0021] Obtain the operation location for each sample;

[0022] By using kinematic modeling, the motion control information of each sample of the robotic arm is determined based on the operation position of each sample;

[0023] The robotic arm is controlled based on the motion control information of each sample to obtain the actual operating positions of the robotic arm.

[0024] The sample motion control information of each robotic arm is used as input, and the actual operating position of each robotic arm is used as output to train the neural network model, thereby obtaining the trained neural network model.

[0025] In one embodiment, after inputting the motion control information into a pre-trained neural network model to obtain the predicted operation position corresponding to the motion control information, the method further includes:

[0026] Based on the transformation relationship, the predicted operation position is transformed into the image coordinate system of the medical image to obtain the image predicted operation position;

[0027] If the deviation between the image prediction operation position and the image target position is less than a preset deviation, the prediction operation position is determined to meet the requirements.

[0028] If the image prediction operation position and the image target position are greater than or equal to the preset deviation, it is determined that the prediction operation position does not meet the requirements.

[0029] In one embodiment, after transforming the predicted operation position to the image coordinate system of the medical image based on the transformation relationship to obtain the image predicted operation position, the method further includes:

[0030] The location of the image prediction operation is displayed in the medical image.

[0031] In one embodiment, adjusting the motion control information includes:

[0032] If the target operating position and the predicted operating position deviate only in angle, adjust the angle of the last joint of the robotic arm;

[0033] If there is a distance deviation between the target operation position and the predicted operation position, adjust the angles of multiple joints of the robotic arm;

[0034] When there are deviations in both angle and distance between the target operation position and the predicted operation position, the angle of the last joint of the robotic arm is adjusted after adjusting the distance of multiple joints of the robotic arm.

[0035] Secondly, this application also provides a robotic arm control information generation device, the device comprising:

[0036] The target operation location determination module is used to determine the target operation location;

[0037] The motion control information determination module is used to determine the motion control information of the robotic arm based on the target operating position through kinematic modeling.

[0038] The prediction module is used to input the motion control information into a pre-trained neural network model to obtain the predicted operation position corresponding to the motion control information;

[0039] The control module is configured to adjust the motion control information when, based on the target operation position and the predicted operation position, the predicted operation position does not meet the requirements, and input the adjusted motion control information into a pre-trained neural network model to obtain the predicted operation position corresponding to the motion control information, until, based on the target operation position and the predicted operation position, the predicted operation position meets the requirements, and then use the final adjusted motion control information as the control information of the robotic arm.

[0040] Thirdly, this application also provides a surgical robot, comprising:

[0041] robotic arm, and

[0042] A processor is configured to execute the method described in any of the above embodiments to determine the control information of the robotic arm, and to control the robotic arm based on the control information of the robotic arm.

[0043] In one embodiment, it further includes:

[0044] Medical image acquisition module, used to acquire medical images;

[0045] The processor is also used to acquire the image target position in the medical image; acquire the transformation relationship between the image coordinate system and the physical space coordinate system of the medical image; and transform the image target position to the physical space coordinate system based on the transformation relationship to obtain the target operation position.

[0046] The aforementioned robotic arm control information generation method, device, and surgical robot first determine the motion control information of the robotic arm based on the target operation position. Then, the motion control information is input into a pre-trained neural network model to obtain a predicted operation position corresponding to the motion control information. If the predicted operation position does not meet the requirements, the motion control information is adjusted. Based on the target operation position and the predicted operation position, if the predicted operation position meets the requirements, the final adjusted motion control information is used as the control information of the robotic arm. In this way, the motion control information of the robotic arm is corrected by the predicted operation position of the neural network, so that the target position of the surgical instrument is consistent with the ideal position, thus ensuring the control accuracy of the robotic arm. Attached Figure Description

[0047] To more clearly illustrate the technical solutions in the embodiments of this application or related technologies, the drawings used in the description of the embodiments of this application or related technologies will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0048] Figure 1a This is a schematic diagram of the surgical control system in one embodiment;

[0049] Figure 1b This is a schematic diagram of the patient trolley in one embodiment;

[0050] Figure 2 This is a flowchart illustrating a method for generating robotic arm control information in one embodiment;

[0051] Figure 3 This is a schematic diagram illustrating a scenario where the target operation position and the predicted operation position are different in one embodiment;

[0052] Figure 4 This is a schematic diagram of the motion control information adjustment steps in one embodiment;

[0053] Figure 5 This is a schematic diagram illustrating the transformation between the image coordinate system and the physical space coordinate system in one embodiment;

[0054] Figure 6 A kinematic model obtained by modeling the DH method in one embodiment;

[0055] Figure 7 A flowchart of the motion control information generation steps in one embodiment;

[0056] Figure 8 This is a schematic diagram illustrating the deviation between the actual operating position and the target operating position obtained by controlling the robotic arm according to motion control information in one embodiment.

[0057] Figure 9 This is a schematic diagram of the structure of a neural network model in one embodiment;

[0058] Figure 10 This is a flowchart of the training steps of a neural network model in one embodiment;

[0059] Figure 11 This is a schematic diagram illustrating the training of an RBF model in one embodiment;

[0060] Figure 12 This is a schematic diagram of the output of the RBF model in one embodiment;

[0061] Figure 13 This is a schematic diagram illustrating the training of an LSTM model in one embodiment;

[0062] Figure 14 This is a flowchart illustrating the robotic arm control information generation method in another embodiment;

[0063] Figure 15This is a structural block diagram of a robotic arm control information generation device in one embodiment;

[0064] Figure 16 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0065] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0066] The robotic arm control information generation method provided in this application embodiment can be applied to, for example... Figure 1a The surgical control system shown includes a surgeon's console, a patient carriage, and an image carriage. The surgeon's console includes a main control arm, a controller, and a display. In a dual-surgeon scenario, there are two surgeon's consoles, one for the lead surgeon and one for the assistant surgeon. The patient carriage carries a robotic arm with surgical instruments. The image carriage carries a display, an image host, and a cold light source.

[0067] The robotic arm mounted on the patient trolley can be manually controlled by the doctor's console or controlled based on an automatic control algorithm to move surgical instruments to the target operating position for subsequent operations. It should be noted that the robotic arm control information generation method involved in this application does not involve the specific diagnosis and treatment process, but is only for assistance, such as controlling the robotic arm to move surgical instruments to the target operating position before surgery.

[0068] Specifically, in combination Figure 1b As shown, Figure 1b This is a schematic diagram of a patient cart in one embodiment. In this embodiment, the patient cart includes a cart motion module, a worktable, a medical image acquisition module, and a robotic arm. The worktable is disposed within the cart motion module, and the medical image acquisition module and robotic arm assembly are mounted on the worktable. The cart motion module includes a bottom sliding shaft, a vertical telescopic shaft, and a left-right rotation shaft to achieve six degrees of freedom of movement for the cart. The vertical telescopic shaft slides left and right at the bottom of the cart. The medical image acquisition module can acquire medical images, and can also be an ultrasound probe for acquiring ultrasound images. The robotic arm includes a support rod, a robotic arm positioning shaft, a robotic arm needle insertion shaft, and surgical instruments. The robotic arm positioning shaft is fixed to the support rod, the robotic arm needle insertion shaft is connected to the robotic arm positioning shaft, and the surgical instruments are mounted on the robotic arm needle insertion shaft. The processor corresponding to the patient cart can execute the robotic arm control information generation method of this application to move the surgical instruments to the target operating position. In an exemplary embodiment, such as... Figure 2 As shown, a method for generating control information for a robotic arm is provided, which can be applied to... Figure 1a Taking the patient trolley as an example, the explanation includes steps 202 to 208. Among them:

[0069] S202: Determine the target operation location.

[0070] The target operation position is the target position at the end effector of the robotic arm, such as the target position of a surgical instrument on the end effector. Optionally, this target operation position is the target operation position in a physical space coordinate system. Optionally, the target position can be determined first based on medical images, and then the image target position can be transformed to a physical space coordinate system to obtain the target operation position. This physical space is the real space.

[0071] S204: Determine the motion control information of the robotic arm based on the target operating position through kinematic modeling.

[0072] Once the target operating position is determined, the motion control information of the robotic arm is determined based on the position-level inverse kinematics, using the target operating position as input.

[0073] The motion control information includes the motion matrix of each joint, the force information of each joint of the robotic arm, and the links.

[0074] Optionally, a corresponding kinematic model can be established based on the structure of the robotic arm. Then, the motion matrix of each joint can be determined based on the target operation position. Finally, position-level positive kinematics calculations can be performed based on the motion matrix of each joint and the kinematic model to obtain the force information on each joint and link of the robotic arm.

[0075] S206: Input the motion control information into the pre-trained neural network model to obtain the predicted operation position corresponding to the motion control information.

[0076] The pre-trained neural network model is trained based on samples. The input of these samples is the motion control information of the robotic arm, and the output is the actual position reached by the robotic arm after being controlled by this motion control information. In this way, by introducing a neural network, deviations caused by tissue stress and other factors can be corrected, as well as the design accuracy of the robotic arm itself.

[0077] Therefore, by inputting motion control information into the neural network model, a predicted operation position corresponding to the motion control information can be obtained. This predicted operation position takes into full account the deviations caused by tissue stress and the design accuracy of the robotic arm itself.

[0078] S208: If, based on the target operation position and the predicted operation position, it is determined that the predicted operation position does not meet the requirements, the motion control information is adjusted, and the adjusted motion control information is input into the pre-trained neural network model to obtain the predicted operation position corresponding to the motion control information. This process continues until, based on the target operation position and the predicted operation position, it is determined that the predicted operation position meets the requirements, and then the final adjusted motion control information is used as the control information of the robotic arm.

[0079] The predicted operation position can be the predicted operation position in the physical space coordinate system. In this case, if the target operation position and the predicted operation position are different, the predicted operation position does not meet the requirements. Otherwise, if the target operation position and the predicted operation position are the same, the predicted operation position meets the requirements. That is to say, the adjusted motion control information can enable the end effector of the robotic arm to reach the target operation position.

[0080] Among them, combined Figure 3 As shown, Figure 3 This is a schematic diagram illustrating a scenario where the target operation position and the predicted operation position are different in one embodiment. When the target operation position and the predicted operation position are different, the motion control information can be adjusted. Specifically, in conjunction with... Figure 4 As shown, Figure 4 This is a schematic diagram of the motion control information adjustment step in one embodiment. The adjusted motion control information is input into a pre-trained neural network model to obtain the predicted operation position corresponding to the adjusted motion control information. The target operation position and the predicted operation position are then compared. If they are not the same, the motion control information is adjusted again until the target operation position and the predicted operation position are the same. Finally, the adjusted motion control information is used as the control information of the robotic arm so that the end effector of the robotic arm can reach the target operation position.

[0081] In one optional embodiment, adjusting the motion control information includes: adjusting the angle of the last joint of the robotic arm when the target operating position and the predicted operating position deviate only in angle; adjusting the angles of multiple joints of the robotic arm when the target operating position and the predicted operating position deviate in distance; and adjusting the angle of the last joint of the robotic arm after adjusting the distance by adjusting multiple joints of the robotic arm when the target operating position and the predicted operating position deviate in both angle and distance.

[0082] Among them, continue to combine Figure 4 As shown, adjustments to motion control information can be made based on the deviation between the target operating position and the predicted operating position, for example... Figure 4If the target operation position and the predicted operation position deviate only in angle, the angle of the last joint of the robotic arm can be adjusted to make the predicted operation position at the end of the robotic arm reach the target operation position. If there is a distance deviation, the angles of multiple joints of the robotic arm can be adjusted to make the predicted operation position at the end of the robotic arm reach the target operation position. If there are both distance and angle deviations, the angles of multiple joints of the robotic arm can be adjusted first, and then the angle of the last joint of the robotic arm can be adjusted to make the predicted operation position at the end of the robotic arm reach the target operation position. The above adjustments to motion control information are only illustrative examples. Those skilled in the art can adjust the motion control information based on actual conditions until the predicted operation position obtained by the neural network model based on the adjusted motion control information matches the target operation position.

[0083] The above-described method for generating robotic arm control information first determines the motion control information of the robotic arm based on the target operation position. Then, the motion control information is input into a pre-trained neural network model to obtain a predicted operation position corresponding to the motion control information. If the predicted operation position does not meet the requirements, the motion control information is adjusted. Based on the target operation position and the predicted operation position, if the predicted operation position meets the requirements, the robotic arm is controlled based on the final adjusted motion control information. In this way, the motion control information of the robotic arm is corrected by the predicted operation position of the neural network, so that the target position of the surgical instrument is consistent with the ideal position, thus ensuring the control accuracy of the robotic arm.

[0084] In one optional embodiment, determining the target operation position includes: acquiring the image target position in the medical image; acquiring the transformation relationship between the image coordinate system and the physical space coordinate system of the medical image; and transforming the image target position to the physical space coordinate system based on the transformation relationship to obtain the target operation position.

[0085] Specifically, in combination Figure 5 As shown, Figure 5 This is a schematic diagram illustrating the transformation between the image coordinate system and the physical space coordinate system in one embodiment. In this embodiment, medical images of the target area can be acquired using a medical imaging acquisition device, such as medical images of a patient's lesion area. These medical images can be MRI, CT, or ultrasound images, and the image target location in the medical image can be determined through lesion identification, etc. Furthermore, the transformation relationship between the image coordinate system and the physical space coordinate system of the medical image can be obtained through feature point recognition or markers. Finally, based on the transformation relationship, the image target location is transformed to the physical space coordinate system to obtain the target operation location. Combined with... Figure 5In the plane, target is the image target position to be reached, and pivot is the initial position of the robotic arm end effector. The spatial distance between them is known. The image target position can be converted into the target operation position through the translation matrix and the scale matrix.

[0086] In one optional embodiment, kinematic modeling is used to determine the motion control information of the robotic arm based on the target operating position. This includes: performing kinematic modeling based on the physical parameters of the robotic arm to obtain a kinematic model; performing position-level inverse kinematics calculation based on the kinematic model and the target operating position to obtain the motion matrix of each joint of the robotic arm; performing position-level forward kinematics calculation based on the kinematic model and the motion matrix to obtain the force information on each joint and link of the robotic arm; and using the motion matrix of each joint and the force information on each joint and link of the robotic arm as the motion control information of the robotic arm.

[0087] The physical parameters of the robotic arm include the length of each link, the position of the joints, and the center of mass of the links. Based on these physical parameters, a kinematic model can be established, and the modeling methods include, but are not limited to, T-matrix modeling, DH modeling, MDH modeling, MCPC modeling, etc.

[0088] Among them, combined Figure 6 As shown, Figure 6 The kinematic model is obtained by modeling the DH method in one embodiment. Figure 6 The robotic arm shown is a two-link robotic arm. The triangle at the bottom represents the base, and the circle between the base and the link represents the joint.

[0089] Using the DH method to establish such Figure 6 The connecting rod uses a local coordinate system with the z-axis perpendicular to the plane and pointing upwards. The reference coordinate system is x0y0z0, where δ... i α i d i a i These represent the joint rotation angle, link torsion angle, link distance, and link length data, respectively. Applying the DH method, a coordinate system x1y1z1 is established. The x1 axis is obtained by rotating x0 around the z0 axis by θ1. Since there is no displacement along the z0 axis between x1 and x0, d1=0. The displacement along the x1 axis is a1. Using x1 as the rotation axis, there is no rotation between axes z1 and z0, so α1=0. Similarly, the four DH parameters between the end effector coordinate system x2y2z2 and the x1y1z1 coordinate system of the second link are obtained. The previous coordinate system X... i-1 Y i-1 Z i-1 By performing rotation and translation transformations sequentially, the new coordinate system X can be obtained. i Y i Z iThe purpose of this application is to determine the target operating position by specifying the coordinates of the end effector of the linkage.

[0090] This allows us to obtain the various parameters of the robotic arm, as shown in Table 1 below:

[0091] Table 1:

[0092]

[0093] Robotic arm DH parameters: DH parameters for two links are as follows Figure 6 As shown. Let point p2 be in the coordinate system x2y2z2. The transformation to convert it to the coordinate system x0y0z0 is as follows:

[0094]

[0095] Where S and C represent the sin function and cos function respectively, and Ciθ represents cosθ. i Siθ represents sinθ i In other words, knowing the coordinates (Px2, Py2, Pz2) allows us to deduce the coordinates (Px0, Py0, Pz0) relative to the original base. Conversely, knowing the coordinates represented by the original base coordinate system allows us to deduce the coordinates in the link 2 coordinate system. Once the coordinates of the image operation position are known, they are converted into the target operation position in the real physical coordinate system. The motion parameters of each link can then be obtained using analytical, geometric, or iterative methods, thus yielding the motion matrix of each joint.

[0096] Specifically, in combination Figure 7 As shown, Figure 7 This is a flowchart of the motion control information generation steps in one embodiment. In this embodiment, the image operation position in the image coordinate system is converted into the target operation position in the physical space coordinate system, and then the motion matrix of each joint is obtained based on the target operation position through position-level inverse kinematics. Subsequently, it is assumed that the motion matrix of each joint is input into the robotic arm, which then becomes a position-level forward kinematics problem, i.e., solving for the spatial velocity-level forward kinematics under the pose conditions of each part of the spatial robotic arm system. Thus, in the inertial frame, the position vector of the center of mass of each robotic arm link is known, and the velocity of the center of mass of the robotic arm link can be known. Therefore, the angular velocity and linear acceleration can be obtained by differentiating the angular velocity and velocity. Furthermore, since the center of mass of each link is known, all inertial forces and inertial torques are known, thus allowing the recursive derivation of the external forces and torques acting on the links.

[0097] In the above embodiments, after the target operation position is known, the motion control information of the robotic arm can be obtained through kinematic modeling.

[0098] Combination Figure 8 As shown, Figure 8This is a schematic diagram illustrating the deviation between the actual operating position and the target operating position obtained by controlling the robotic arm according to motion control information in one embodiment. In this embodiment, the robotic arm itself contains positioning errors (i.e., each joint has friction, coupling, and motor drive errors), and during operation, the end of the robotic arm is subjected to external forces from the tissue. These external forces are not a fixed constant, which affects the kinematic velocity level, dynamic force, and torque parameters of the robotic arm. At the same time, the tissue is also subjected to the force of the robotic arm, resulting in displacement and causing positioning errors in the robotic arm. The actual operating position and the target operating position do not coincide, thereby reducing accuracy. To avoid this situation, this application first predicts the end of the robotic arm based on motion control parameters to obtain a predicted operating position. By adjusting the motion control parameters to make the predicted operating position consistent with the target operating position, the robotic arm is controlled by the adjusted motion control parameters to improve the control accuracy of the robotic arm.

[0099] In one optional embodiment, before inputting motion control information into a pre-trained neural network model to obtain the predicted operation position corresponding to the motion control information, the method further includes: acquiring the operation position of each sample; determining the motion control information of each sample of the robotic arm based on the operation position of each sample through kinematic modeling; controlling the robotic arm based on the motion control information of each sample to obtain the actual operation position of the robotic arm; and training the neural network model by taking the motion control information of each robotic arm as input and the actual operation position of each robotic arm as output to obtain the trained neural network model.

[0100] This embodiment provides the training steps for the neural network model, where each training sample includes input and output. The input is the sample motion control information, and the output is the actual operating position of the robotic arm.

[0101] Combination Figure 9 As shown, the sample motion control information, namely the kinematic parameters of each joint and the dynamic parameters of the robotic arm, corresponds exactly to the output of an actual position, i.e., the actual operating position of the robotic arm. This nonlinear mapping of a combination of parameters corresponding to a position output can be fitted using a neural network. Optionally, an RBF neural network, an LSTM neural network, or other deep learning architectures such as Transformer can be used for fitting. Figure 9 As shown, X i (n) represents the input layer, C iY(n) is the hidden layer, and Y(n) is the output layer. The input layer takes in the sample motion control information, that is, the various parameters of the robot arm's kinematics and dynamics; the hidden layer learning is usually unsupervised and is used to determine the basis function parameters, that is, to determine the center and width, and to adjust the weight values; the output layer is used to determine the real physical coordinates of the output relative to the initial position.

[0102] Combination Figure 10 As shown, after network initialization, input stimuli are added, namely the sample motion control information mentioned above. The first stage calculates the output of the hidden layer neurons, which is to use the K-means algorithm. Its task is to use the self-organizing clustering method to determine the appropriate data centers for the radial basis functions of the hidden layer nodes. The second stage is supervised learning, which calculates the output of the entire network and adjusts the weights until the global error of the network reaches the accuracy requirement. Its task is to train the output layer weights using a supervised learning algorithm, such as the gradient method. When the final error converges to a sufficiently small threshold or the number of iterations reaches a sufficiently large number, the convergence stops and the model training ends.

[0103] Combination Figure 11 As shown, Figure 11 This is a schematic diagram illustrating the training of an RBF model in one embodiment. Figure 12 This is a schematic diagram of the output of the RBF model in one embodiment. The input layer consists of sample motion control information, i.e., various parameters of the robotic arm's kinematics and dynamics. Assuming the DH method is used for modeling, each actual needle insertion data includes kinematic and dynamic parameters (δ). i α i d i a i V i ω i , ɑ i C bi C θi ...), where δ i α i d i a i V i ω i , ɑ i C bi C θiThese represent parameters such as joint rotation angle, link torsion angle, link distance, link length, link velocity, link angular velocity, link acceleration, link nonlinear force, and link nonlinear torque, which are used as a set of inputs during training. Radial basis functions (RBFs) can take various forms, commonly including Gaussian functions, multi-quadric functions, inverse multi-quadric functions, and thin-plate spline functions. This neural network can use a Gaussian function; the linear combination of the outputs of the hidden layer nodes by the k-th node of the RBF can be represented as:

[0104]

[0105] Hidden layer learning is typically unsupervised, used to determine the basis function parameters, i.e., the center and width, and to adjust the weight values. There are generally two methods for determining the center: selecting it from the sample input and dynamic adjustment. Typically, the center vector and standardized parameters are selected using the input data. The output layer is supervised learning, mainly determining the weights. Having learned the network center and width through hidden layer learning, the input layer introduces the offset term b. i The error between the actual coordinates and the true coordinates of each set of data is given by (e). x1 e x2 e x3 ...e xn (x) represents the x-coordinate of each set of coordinates and the actual coordinates. i The error is calculated, and the final output is the x-coordinate of the actual coordinates of each data set (x1 x2 x3 ....x). n Similarly, the y-axis and z-axis are calculated in the same way. Finally, LMS and RLS can be used to solve for the weights. That is, after the entire model has been trained with weights, it can be used with input kinematic and dynamic data to obtain the actual predicted coordinates. By converting them to image coordinates, the mapping of the actual coordinates to image coordinates can be displayed by the software.

[0106] Combination Figure 13 As shown, Figure 13 This is a schematic diagram of the training of an LSTM model in one embodiment. The input layer processes the sample motion control information, i.e., the original kinematics and dynamic parameters of the robotic arm, to construct a feature sequence (δ). i α i d i a i V i ω i , ɑ i C bi C θi...), ensuring it meets the network input conditions. Then, the input from the input layer is fed into the hidden layer. The main function of the hidden layer is network training. The processed input is compared with the theoretical output X by the hidden layer, and the loss function is calculated. Minimizing the loss function is the optimization objective. Initial values ​​such as the learning rate and the number of training iterations are set, and the network weights are continuously updated to determine the final hidden layer network. Similarly, the final X, Y, and Z are the coordinate values ​​of each feature sequence output on each coordinate axis. A maximum number of iterations can be set, or iteration can stop when the loss function converges to a certain value, completing the model training.

[0107] In one optional embodiment, after inputting motion control information into a pre-trained neural network model to obtain the predicted operation position corresponding to the motion control information, the method further includes: transforming the predicted operation position into the image coordinate system of the medical image based on the transformation relationship to obtain the image predicted operation position; determining that the predicted operation position meets the requirements if the deviation between the image predicted operation position and the image target position is less than a preset deviation; and determining that the predicted operation position does not meet the requirements if the deviation between the image predicted operation position and the image target position is greater than or equal to the preset deviation.

[0108] Determining whether the predicted operation location meets the requirements can be based on either the image coordinate system or the physical space coordinate system. Using the physical space coordinate system involves directly comparing the target operation location with the predicted operation location. Using the image coordinate system involves transforming the predicted operation location into the image coordinate system of the medical image to obtain the image predicted operation location, and then comparing the image predicted operation location with the image target location.

[0109] By converting to the image coordinate system, the image prediction operation position and the image target position can be displayed directly. This allows the operator to determine whether the prediction operation position meets the requirements, or it can be automatically determined based on the image prediction operation position and the image target position.

[0110] Optionally, after transforming the predicted operation position to the image coordinate system of the medical image based on the transformation relationship to obtain the image predicted operation position, the method further includes: displaying the image predicted operation position in the medical image.

[0111] For ease of understanding, combine Figure 14 As shown, Figure 14The flowchart of the robotic arm control information generation method in another embodiment is as follows: First, the image target position in the image coordinate system is converted into the target operation position in the physical space coordinate system. Then, kinematic modeling is performed based on the target operation position in the physical space coordinate system to obtain the motion control information of the robotic arm. The motion control information of the robotic arm is input into a pre-trained neural network model to obtain the predicted operation position. The predicted operation position is then converted into the image coordinate system to obtain the image predicted operation position, and the image predicted operation position is displayed. This determines whether the image predicted operation position in the image coordinate system coincides with the image target position. If so, the motion control information of the robotic arm is used as the final control information of the robotic arm. Otherwise, the motion control information can be adjusted based on the relationship between the image predicted operation position and the image target position in the image coordinate system. The adjusted motion control information is then input into the pre-trained neural network model to obtain a new predicted operation position. The predicted operation position is then converted into the image coordinate system to obtain the image predicted operation position again, until the image predicted operation position in the image coordinate system coincides with the image target position. Finally, the motion control information of the robotic arm is used as the final control information of the robotic arm.

[0112] In the above embodiments, by collecting data on the characteristics of the robotic arm and its various parameters, and using a neural network to predict the actual output, the method is more accurate than simply using position-level inverse dynamics to calculate the motion matrix of each robotic arm. By training the predictive model and converting it between images and real-world coordinates, the kinematic input of the robotic arm can be displayed in real time in the image coordinate system. This enables the generation of more precise control information for the robotic arm, thereby improving the accuracy of subsequent punctures or ablations, reducing the pain of repeated punctures for patients, and ensuring that the surgery is performed more in line with expectations.

[0113] It should be understood that although the steps in the flowcharts of the above embodiments are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the above embodiments may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0114] Based on the same inventive concept, this application also provides a robotic arm control information generation device for implementing the robotic arm control information generation method described above. The solution provided by this device is similar to the solution described in the above method; therefore, the specific limitations in one or more embodiments of the robotic arm control information generation device provided below can be found in the limitations of the robotic arm control information generation method described above, and will not be repeated here.

[0115] In one exemplary embodiment, such as Figure 15 As shown, a robotic arm control information generation device is provided, including: a target operation position determination module 1501, a motion control information determination module 1502, a prediction module 1503, and a control module 1504, wherein:

[0116] The target operation position determination module 1501 is used to determine the target operation position;

[0117] The motion control information determination module 1502 is used to determine the motion control information of the robotic arm based on the target operation position through kinematic modeling.

[0118] The prediction module 1503 is used to input motion control information into a pre-trained neural network model to obtain the predicted operation position corresponding to the motion control information;

[0119] The control module 1504 is used to adjust the motion control information when the predicted operation position does not meet the requirements based on the target operation position and the predicted operation position, and input the adjusted motion control information into a pre-trained neural network model to obtain the predicted operation position corresponding to the motion control information, until the predicted operation position meets the requirements based on the target operation position and the predicted operation position, and then use the final adjusted motion control information as the control information of the robotic arm.

[0120] In one optional embodiment, the target operation position determination module 1501 is specifically used to obtain the image target position in the medical image; obtain the transformation relationship between the image coordinate system and the physical space coordinate system of the medical image; and transform the image target position to the physical space coordinate system based on the transformation relationship to obtain the target operation position.

[0121] In one optional embodiment, the motion control information determination module 1502 is specifically used to perform kinematic modeling based on the physical parameters of the robotic arm to obtain a kinematic model; perform position-level inverse kinematics calculation based on the kinematic model and the target operation position to obtain the motion matrix of each joint of the robotic arm; perform position-level forward kinematics calculation based on the kinematic model and the motion matrix to obtain the force information on each joint and link of the robotic arm; and use the motion matrix of each joint and the force information on each joint and link of the robotic arm as the motion control information of the robotic arm.

[0122] In one optional embodiment, the above-mentioned device further includes: a model training module for acquiring the operation positions of each sample; determining the motion control information of each sample of the robotic arm based on the operation positions of each sample through kinematic modeling; controlling the robotic arm based on the motion control information of each sample to obtain the actual operation positions of the robotic arm; and training a neural network model by taking the motion control information of each robotic arm as input and the actual operation positions of each robotic arm as output to obtain a trained neural network model.

[0123] In one optional embodiment, the above-mentioned device further includes: a conversion module, configured to convert the predicted operation position to the image coordinate system of the medical image based on the conversion relationship to obtain the image predicted operation position; if the deviation between the image predicted operation position and the image target position is less than a preset deviation, determine that the predicted operation position meets the requirements; if the deviation between the image predicted operation position and the image target position is greater than or equal to the preset deviation, determine that the predicted operation position does not meet the requirements.

[0124] In one alternative embodiment, the apparatus further includes a display module for displaying the image prediction operation location in a medical image.

[0125] In one optional embodiment, the motion control information determination module 1502 is specifically used to adjust the angle of the last joint of the robotic arm when the target operation position and the predicted operation position deviate only in angle; to adjust the angles of multiple joints of the robotic arm when the target operation position and the predicted operation position deviate in distance; and to adjust the angle of the last joint of the robotic arm after adjusting the distance by adjusting multiple joints of the robotic arm when the target operation position and the predicted operation position deviate in both angle and distance.

[0126] Each module in the aforementioned robotic arm control information generation device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device as software, so that the processor can call and execute the operations corresponding to each module.

[0127] In one exemplary embodiment, a computer device is provided, which may be a terminal, and its internal structure diagram may be as follows: Figure 16 As shown, the computer device includes a processor, memory, input / output interface, communication interface, display unit, and input device. The processor, memory, and input / output interface are connected via a system bus, and the communication interface, display unit, and input device are also connected to the system bus via the input / output interface. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The input / output interface is used for exchanging information between the processor and external devices. The communication interface is used for wired or wireless communication with external terminals; wireless communication can be achieved through Wi-Fi, mobile cellular networks, Near Field Communication (NFC), or other technologies. When the computer program is executed by the processor, it implements a method for generating control information for a robotic arm. The display unit is used to form a visually visible image and can be a display screen, projection device, or virtual reality imaging device. The display screen can be an LCD screen or an e-ink screen. The input device of the computer device can be a touch layer covering the display screen, or buttons, trackballs, or touchpads set on the casing of the computer device, or external keyboards, touchpads, or mice, etc.

[0128] Those skilled in the art will understand that Figure 16 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0129] In one embodiment, a computer device is also provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps in the above method embodiments.

[0130] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon that, when executed by a processor, implements the steps in the above method embodiments.

[0131] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps in the above method embodiments.

[0132] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, artificial intelligence (AI) processors, etc., and are not limited to these.

[0133] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0134] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A robot arm control information generation method characterized by comprising: The method comprises: determining a target operation position, comprising: obtaining an image target position in a medical image; obtaining a conversion relationship between an image coordinate system of the medical image and a physical space coordinate system; converting the image target position to the physical space coordinate system based on the conversion relationship to obtain the target operation position; determining motion control information of the mechanical arm based on the target operation position through kinematic modeling; inputting the motion control information into a pre-trained neural network model to obtain a predicted operation position corresponding to the motion control information; in a case where it is determined based on the target operation position and the predicted operation position that the predicted operation position does not meet the requirements, adjusting the motion control information, inputting the adjusted motion control information into the pre-trained neural network model to obtain a predicted operation position corresponding to the motion control information, until it is determined based on the target operation position and the predicted operation position that the predicted operation position meets the requirements, and taking the final adjusted motion control information as control information of the mechanical arm; based on the conversion relationship, converting the predicted operation position into the image coordinate system of the medical image to obtain an image predicted operation position; in a case where a deviation between the image predicted operation position and the image target position is less than a preset deviation, determining that the predicted operation position meets the requirements; in a case where the image predicted operation position and the image target position are greater than or equal to the preset deviation, determining that the predicted operation position does not meet the requirements; displaying the image predicted operation position in the medical image.

2. The method of claim 1, wherein, The method comprises: based on the physical parameters of the mechanical arm, performing kinematic modeling to obtain a kinematic model; based on the kinematic model and the target operation position, performing position-level inverse kinematics calculation to obtain a motion matrix of each joint of the mechanical arm; based on the kinematic model and the motion matrix, performing position-level forward kinematics calculation to obtain force information on each joint and connecting rod of the mechanical arm; taking the motion matrix of each joint, the force information on each joint and connecting rod of the mechanical arm as the motion control information of the mechanical arm.

3. The method of claim 2, wherein, Before the motion control information is input into the pre-trained neural network model to obtain the predicted operation position corresponding to the motion control information, the method further comprises: obtaining each sample operation position; based on each sample operation position, determining each sample motion control information of the mechanical arm through kinematic modeling; controlling the mechanical arm based on each sample motion control information to obtain each actual operation position of the mechanical arm; respectively taking each sample motion control information of the mechanical arm as input and each actual operation position of the mechanical arm as output to perform neural network model training to obtain a trained neural network model.

4. The method of claim 1, wherein, The method comprises: in the case that only an angle deviation exists between the target operation position and the predicted operation position, adjusting an angle of a last joint of the mechanical arm; in the case that a distance deviation exists between the target operation position and the predicted operation position, adjusting angles of multiple joints of the mechanical arm; in the case that both an angle deviation and a distance deviation exist between the target operation position and the predicted operation position, adjusting an angle of a last joint of the mechanical arm after adjusting distances by adjusting multiple joints of the mechanical arm.

5. A robot arm control information generation device characterized by comprising: The device comprises: a target operation position determination module, configured to determine a target operation position, including: acquiring an image target position in a medical image; acquiring a conversion relationship between an image coordinate system of the medical image and a physical space coordinate system; converting the image target position to the physical space coordinate system based on the conversion relationship to obtain the target operation position; a motion control information determination module, configured to determine motion control information of the mechanical arm based on the target operation position through kinematic modeling; a prediction module, configured to input the motion control information into a neural network model trained in advance to obtain a predicted operation position corresponding to the motion control information; a control module, configured to, in the case that it is determined based on the target operation position and the predicted operation position that the predicted operation position does not meet a requirement, adjust the motion control information, and input the adjusted motion control information into the neural network model trained in advance to obtain a predicted operation position corresponding to the motion control information, until in the case that it is determined based on the target operation position and the predicted operation position that the predicted operation position meets the requirement, taking the finally adjusted motion control information as control information of the mechanical arm; a conversion module, configured to convert the predicted operation position to an image predicted operation position in the image coordinate system of the medical image based on the conversion relationship; in the case that a deviation between the image predicted operation position and the image target position is less than a preset deviation, determining that the predicted operation position meets the requirement; in the case that the deviation between the image predicted operation position and the image target position is greater than or equal to the preset deviation, determining that the predicted operation position does not meet the requirement; a display module, configured to display the image predicted operation position in the medical image.

6. The apparatus of claim 5, wherein, The motion control information determination module is specifically configured to: perform kinematic modeling based on physical parameters of the mechanical arm to obtain a kinematic model; perform position-level inverse kinematics calculation based on the kinematic model and the target operation position to obtain a motion matrix of each joint of the mechanical arm; perform position-level forward kinematics calculation based on the kinematic model and the motion matrix to obtain force information on each joint and connecting rod of the mechanical arm; take the motion matrix of each joint, the force information on each joint and connecting rod of the mechanical arm as the motion control information of the mechanical arm.

7. The apparatus of claim 6, wherein, The device further comprises a model training module configured to: acquire each sample operation position; determine, based on each sample operation position, sample motion control information of the robot arm through kinematic modeling; control the robot arm based on each sample motion control information to obtain an actual operation position of the robot arm; train a neural network model by taking, as input, each sample motion control information of the robot arm and, as output, each actual operation position of the robot arm, to obtain a trained neural network model.

8. The apparatus of claim 5, wherein, The motion control information determination module is specifically configured to: adjust an angle of a last joint of the robot arm in a case where the target operation position and the predicted operation position only have an angle deviation; adjust angles of multiple joints of the robot arm in a case where the target operation position and the predicted operation position have a distance deviation; adjust the angle of the last joint of the robot arm after adjusting distances of the multiple joints of the robot arm in a case where the target operation position and the predicted operation position have both an angle deviation and a distance deviation.

9. A surgical robot, characterized by comprise: a robot arm, and a processor configured to execute the method of any one of claims 1 to 4 to determine control information of the robot arm and control the robot arm based on the control information of the robot arm.

10. The surgical robot of claim 9, wherein, further comprise: a medical image acquisition module configured to acquire a medical image.

Images