Instrument off-screen display method, device and equipment of minimally invasive surgery robot and medium

Abstract

The embodiment of the invention provides an instrument off-screen display method and device for a minimally invasive surgery robot, equipment and a medium. The method comprises the steps that if it is determined that the instrument tail end of the minimally invasive surgery robot meets the off-screen display requirement, multiple pieces of first position information are determined according to first sensor data of the minimally invasive surgery robot; carrying out fusion processing on the plurality of pieces of first position information to obtain a target positioning result of the tail end of the instrument; and if it is determined that the instrument tail end is off-screen according to the target positioning result, performing off-screen display on the instrument tail end according to the target positioning result. The method is used for improving the accuracy of off-screen display of the minimally invasive surgery robot and further improving the off-screen display performance.

Description

Technical Field

[0001] This application relates to the fields of artificial intelligence and intelligent medical device technology, and in particular to a method, device, equipment and medium for off-screen display of instruments in a minimally invasive surgical robot. Background Technology

[0002] Minimally invasive surgical robots, as an important tool in modern surgery, can provide surgeons with higher operational precision, flexibility and stability through the video images provided by the endoscope and in a master-slave teleoperation mode.

[0003] However, due to the limited field of view provided by the endoscope, the end effector of the minimally invasive surgical robot may move to the edge of the endoscope's field of view or just leave the field of view. At this time, the off-screen display function may be activated, and the end effector of the minimally invasive surgical robot becomes particularly important.

[0004] In related technologies, the position of the instrument end effector is mainly estimated based on the kinematic model of the minimally invasive surgical robot. The estimation accuracy of the kinematic model depends on the absolute accuracy of the robot kinematic model and the hand-eye calibration matrix. However, in actual clinical applications, factors such as model error, calibration error and deformation error can affect the accuracy of instrument end effector positioning, and thus affect the accuracy of off-screen display. Summary of the Invention

[0005] This application provides a method, apparatus, device, and medium for off-screen display of instruments in a minimally invasive surgical robot, in order to improve the accuracy of off-screen display of the minimally invasive surgical robot and thus improve the performance of off-screen display.

[0006] In a first aspect, embodiments of this application provide a method for off-screen display of instruments in a minimally invasive surgical robot, comprising:

[0007] If it is determined that the instrument end of the minimally invasive surgical robot meets the off-screen display requirements, then multiple first position information are determined based on the first sensor data of the minimally invasive surgical robot.

[0008] The multiple first location information are fused to obtain the target positioning result of the instrument end effect;

[0009] If, based on the target positioning result, it is determined that the device end is off-screen, then, based on the target positioning result, the device end is displayed off-screen.

[0010] In one possible implementation, the first sensor data includes at least joint encoder data and endoscopic video stream data; the first position information includes at least kinematic position information and visual position information; based on the first sensor data of the minimally invasive surgical robot, multiple pieces of first position information are determined, including:

[0011] The kinematic position information is determined based on the joint encoder data and the preset kinematic model;

[0012] Based on the target detection model, the instrument end included in the endoscopic video stream data is identified and processed to obtain the visual position information.

[0013] In one possible implementation, the plurality of first location information are fused to obtain the target positioning result of the instrument end effector, including:

[0014] Obtain the joint error information of the minimally invasive surgical robot, and determine the first detection error based on the joint error information;

[0015] Determine the model error information of the target detection model, and determine the second detection error based on the model error information;

[0016] Based on the first detection error and the second detection error, determine the fusion weight information;

[0017] Based on the fusion weight information, the kinematic position information and the visual position information are fused to obtain the target localization result.

[0018] In one possible implementation, determining fusion weight information based on the first detection error and the second detection error includes:

[0019] Based on the joint motion information of the device end, the motion state of the device end is predicted to obtain the predicted position information and the prediction error information.

[0020] The fusion weight information is determined based on the predicted error information, the first detection error, and the second detection error.

[0021] The step of fusing the kinematic position information and the visual position information according to the fusion weight information to obtain the target localization result includes:

[0022] The predicted location information, the kinematic location information, and the visual location information are fused based on the fusion weight information to obtain the target localization result.

[0023] In one possible implementation, determining that the end effector of the minimally invasive surgical robot meets off-screen display requirements includes:

[0024] Based on the second sensor data of the minimally invasive surgical robot, at least one second location information is determined;

[0025] If, based on the second location information, it is determined that the instrument tip of the minimally invasive surgical robot is located within the edge area of ​​the robot's field of vision, then the instrument tip of the minimally invasive surgical robot meets the off-screen display requirement.

[0026] In one possible implementation, the method further includes:

[0027] Obtain the physical dimensions of the instrument's end effector;

[0028] The edge region range is determined based on the physical size information and / or the location type corresponding to the second location information;

[0029] The visual field edge region is determined based on the visual field boundary line of the minimally invasive surgical robot and the range of the edge region.

[0030] In one possible implementation, the method further includes:

[0031] If it is determined that the instrument end of the minimally invasive surgical robot does not meet the off-screen display requirements, then the matching position detection type is determined based on the second position information;

[0032] Based on the matched position detection type and the third sensor data of the minimally invasive surgical robot, the target positioning result of the instrument end is determined.

[0033] Secondly, embodiments of this application provide an off-screen display device for a minimally invasive surgical robot, comprising:

[0034] The determining unit is used to determine multiple first position information based on the first sensor data of the minimally invasive surgical robot if it is determined that the instrument end of the minimally invasive surgical robot meets the off-screen display requirements.

[0035] The fusion unit is used to fuse the multiple first position information to obtain the target positioning result of the instrument end effector;

[0036] An off-screen display unit is used to display the instrument end off-screen according to the target positioning result if it is determined that the instrument end is off-screen.

[0037] In one possible implementation, the first sensor data includes at least joint encoder data and endoscopic video stream data; the first position information includes at least kinematic position information and visual position information; in this case, the determining unit is configured to:

[0038] The kinematic position information is determined based on the joint encoder data and the preset kinematic model;

[0039] Based on the target detection model, the instrument end included in the endoscopic video stream data is identified and processed to obtain the visual position information.

[0040] In one possible implementation, the fusion unit is used for:

[0041] Obtain the joint error information of the minimally invasive surgical robot, and determine the first detection error based on the joint error information;

[0042] Determine the model error information of the target detection model, and determine the second detection error based on the model error information;

[0043] Based on the first detection error and the second detection error, determine the fusion weight information;

[0044] Based on the fusion weight information, the kinematic position information and the visual position information are fused to obtain the target localization result.

[0045] In one possible implementation, the fusion unit is used for:

[0046] Based on the joint motion information of the device end, the motion state of the device end is predicted to obtain the predicted position information and the prediction error information.

[0047] The fusion weight information is determined based on the predicted error information, the first detection error, and the second detection error.

[0048] The step of fusing the kinematic position information and the visual position information according to the fusion weight information to obtain the target localization result includes:

[0049] The predicted location information, the kinematic location information, and the visual location information are fused based on the fusion weight information to obtain the target localization result.

[0050] In one possible implementation, the determining unit is used for:

[0051] Based on the second sensor data of the minimally invasive surgical robot, at least one second location information is determined;

[0052] If, based on the second location information, it is determined that the instrument tip of the minimally invasive surgical robot is located within the edge area of ​​the robot's field of vision, then the instrument tip of the minimally invasive surgical robot meets the off-screen display requirement.

[0053] In one possible implementation, the device is also used for:

[0054] Obtain the physical dimensions of the instrument's end effector;

[0055] The edge region range is determined based on the physical size information and / or the location type corresponding to the second location information;

[0056] The visual field edge region is determined based on the visual field boundary line of the minimally invasive surgical robot and the range of the edge region.

[0057] In one possible implementation, the device is also used for:

[0058] If it is determined that the instrument end of the minimally invasive surgical robot does not meet the off-screen display requirements, then the matching position detection type is determined based on the second position information;

[0059] Based on the matched position detection type and the third sensor data of the minimally invasive surgical robot, the target positioning result of the instrument end is determined.

[0060] Thirdly, embodiments of this application provide a computer device, including: a memory and a processor;

[0061] The memory stores computer-executed instructions;

[0062] The processor executes computer execution instructions stored in the memory, causing the processor to perform the first aspect and / or various possible implementations of the first aspect as described above.

[0063] Fourthly, embodiments of this application provide a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, are used to implement the first aspect and / or various possible implementations of the first aspect.

[0064] Fifthly, embodiments of this application provide a computer program product, including a computer program that, when executed by a processor, implements the first aspect and / or various possible implementations of the first aspect.

[0065] The off-screen display method, apparatus, device, and medium for minimally invasive surgical robots provided in this application embodiment can, when it is determined that the end effector of the minimally invasive surgical robot meets the off-screen display requirements, determine multiple first position information based on the first sensor data of the minimally invasive surgical robot, and then fuse the multiple first position information to obtain the target positioning result of the end effector. At this time, by fusing multiple first position information, the defects of single position information can be compensated for, thereby improving the accuracy of the target positioning result of the end effector. Subsequently, if it is determined that the end effector is off-screen based on the target positioning result, the end effector is displayed off-screen according to the target positioning result. This implementation method can improve the accuracy of determining whether the end effector is off-screen based on accurate target positioning results, thereby improving the off-screen display performance of the minimally invasive surgical robot. Attached Figure Description

[0066] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0067] Figure 1 A flowchart illustrating a method for off-screen display of instruments in a minimally invasive surgical robot, provided as an embodiment of this application;

[0068] Figure 2 A flowchart illustrating another method for off-screen display of instruments in a minimally invasive surgical robot provided in this application embodiment;

[0069] Figure 3 This is a schematic diagram of the structure of an off-screen display device for a minimally invasive surgical robot provided in an embodiment of this application;

[0070] Figure 4 This is a schematic diagram of the structure of a computer device provided in an embodiment of this application.

[0071] The accompanying drawings have illustrated specific embodiments of this application, which will be described in more detail below. These drawings and descriptions are not intended to limit the scope of the concept in any way, but rather to illustrate the concept of this application to those skilled in the art through reference to specific embodiments. Detailed Implementation

[0072] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0073] In this document, the term "and / or" merely describes a relationship, indicating that three relationships can exist. For example, A and / or B can represent three cases: A alone, A and B simultaneously, and B alone. Furthermore, the term "at least one" in this document means any combination of at least two of any one or more elements. For example, including at least one of A, B, and C can mean including any one or more elements selected from the set consisting of A, B, and C.

[0074] Minimally invasive surgical robots, as an important tool in modern surgery, can provide surgeons with higher operational precision, flexibility and stability through the limited field of view video provided by the endoscope and in a master-slave teleoperation mode.

[0075] However, due to the load on the surgical area, the large range of instrument operation, and the inherent limitations of the endoscopic field of view, the tip of the surgical instrument may move out of the endoscopic field of view. This results in the surgeon losing direct visual feedback on the position and orientation of the instruments. In this situation, not only is the operation time prolonged, but there is also a risk of scratching or accidentally injuring non-target tissues, thus posing certain surgical risks.

[0076] To alleviate this problem, "off-screen display" was developed. This function can be activated when the instrument tip moves to the edge of the endoscope's field of view or just leaves the field of view. After the instrument tip leaves the endoscope's field of view, the doctor is prompted in real time at the edge of the main display screen in the form of a virtual arrow or indicator to indicate the direction of the instrument tip relative to the current field of view boundary, thereby helping the doctor to efficiently and safely reposition the instrument back to the center of the field of view.

[0077] In related technologies, when determining whether to activate the off-screen display function, the kinematic model of the minimally invasive surgical robot is mainly used to locate the end effector and then the off-screen display is performed based on the determined location of the end effector.

[0078] In practice, the angle data of each joint of the minimally invasive surgical robot can be collected in real time by an encoder. Based on a pre-calibrated robot kinematic model (e.g., established using the DH parameter method), the three-dimensional pose of the instrument's end effector in the robot base coordinate system is obtained through forward kinematics calculations. Then, through a fixed transformation relationship between the endoscope camera and the robot base or robotic arm, the three-dimensional pose of the instrument's end effector is transformed into the camera coordinate system. Finally, using the camera's intrinsic parameter model, it is determined whether the instrument's end effector is within the field of view. If its projection point is outside the field of view, an off-screen indicator is triggered at the corresponding edge of the screen based on its relative orientation to the image center, thus displaying the position of the instrument's end effector off-screen.

[0079] The estimation accuracy of the kinematic model depends on the absolute accuracy of the robot's kinematic model and the hand-eye calibration matrix. Furthermore, in actual clinical applications, the introduction of various error factors leads to a certain deviation between the theoretical calculation value of the kinematic model and the physical reality.

[0080] For example, introduced error factors may include, but are not limited to: model error, calibration error, and deformation error. Model error refers to the parameter error of the kinematic model. For instance, the DH parameter in the kinematic model is a theoretical design value; actual manufacturing tolerances of the links, assembly clearances, etc., can lead to mismatch between the kinematic model and the actual object. Calibration error is the residual error existing in the calibrated hand-eye conversion matrix. Deformation error refers to the slight deformation that occurs in the robotic arm of a minimally invasive surgical robot under gravity or external forces; this deformation, after being transmitted over long distances, will produce pose deviations at the end effector.

[0081] When the instrument tip is within the endoscope's field of view, doctors can eliminate the impact of these errors through visual positional comparison. However, when the instrument tip moves to the edge of the endoscope's field of view or has just left the field of view, this implementation method affects the accuracy of instrument tip positioning, which in turn affects the activation of the off-screen display function and the accuracy of the off-screen display of the instrument tip.

[0082] Therefore, there is an urgent need for a method that can compensate for the positioning error of the kinematic model, thereby providing high-precision and robust positioning of the instrument end in key areas such as the edge of the field of vision, thus improving the accuracy and reliability of off-screen display functions.

[0083] The instrument off-screen display method for minimally invasive surgical robots provided in this application, when determining that the instrument end-effector meets the off-screen display requirements, can compensate for the positioning error caused by a single position detection method by fusing multiple first position information detected by various position detection methods, thereby improving the accuracy of instrument end-effector positioning and thus enhancing the precision of instrument end-effector off-screen display. For example, the position information calculated by the kinematic model and the position information calculated by the visual algorithm can be fused, thereby providing real-time external feedback to the kinematic model based on visual information, thus compensating for the model error of the kinematic model; while the kinematic model can provide stable position prediction even in the event of a brief visual absence. This complementary approach can improve the accuracy and robustness of instrument end-effector positioning in key areas such as the edge of the field of vision, further enhancing the precision of instrument end-effector off-screen display.

[0084] The technical solution of this application and how the technical solution of this application solves the above-mentioned technical problems are described in detail below with specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments. The embodiments of this application will be described below with reference to the accompanying drawings.

[0085] Figure 1 This application provides a flowchart illustrating a method for off-screen display of instruments in a minimally invasive surgical robot, as shown in the embodiments below. Figure 1 As shown, the method includes:

[0086] S101. If it is determined that the instrument end of the minimally invasive surgical robot meets the off-screen display requirements, then multiple first position information are determined based on the first sensor data of the minimally invasive surgical robot.

[0087] In one example, if the end effector of the minimally invasive surgical robot meets the off-screen display requirement, it means that the end effector of the minimally invasive surgical robot is located at the edge of the endoscope's field of view or has just left the field of view.

[0088] At this point, the position of the instrument tip of the minimally invasive surgical robot can be determined based on at least one position detection method, and whether the instrument tip of the minimally invasive surgical robot meets the off-screen display requirements can be determined based on whether the position is located at the edge of the endoscope's field of view or within the field of view edge area corresponding to just leaving the field of view.

[0089] For example, if the location is within the edge of the field of vision, then the instrument end of the minimally invasive surgical robot is determined to meet the off-screen display requirements; otherwise, the instrument end of the minimally invasive surgical robot is determined not to meet the off-screen display requirements.

[0090] In one example, a minimally invasive surgical robot may include a variety of types of sensors, such as, but not limited to: position / attitude sensors (e.g., encoders, tiltmeters, and IMUs), force / tactile sensors (e.g., six-dimensional force sensors, piezoelectric / piezoresistive tactile sensors), vision sensors (e.g., binocular cameras, high-definition cameras, and fluorescence cameras), and navigation and positioning sensors (e.g., electromagnetic sensors, optical positioning sensors).

[0091] At this point, the first sensor data indicates sensor data that can be fused. The first sensor data may include at least two types of sensor data; for example, it may include sensor data acquired by a position / attitude sensor and sensor data acquired by a force / tactile sensor, or it may include sensor data acquired by a position / attitude sensor, sensor data acquired by a vision sensor, and sensor data acquired by a navigation and positioning sensor, etc.

[0092] At this point, multiple first position information can be determined based on the data from the first sensor.

[0093] S102. Multiple first position information are fused to obtain the target positioning result of the instrument end.

[0094] Optionally, the fusion of multiple first position information can be achieved by weighted summation; alternatively, the fusion of multiple first position information can be achieved by using a filtering algorithm to predict and correct multiple first position information; alternatively, the fusion of multiple first position information can be achieved by minimizing the residual function corresponding to multiple first position information; alternatively, the fusion of multiple first position information can be achieved based on a deep learning model. Here, the method of fusion of multiple first position information is not limited, and the method that can be implemented shall prevail.

[0095] S103. If the instrument end is determined to be off-screen based on the target positioning result, then the instrument end is displayed off-screen based on the target positioning result.

[0096] In one example, if the target localization result is outside the field of view boundary, it is determined that the instrument tip is off the screen. At this time, based on the target localization result, instrument indication information can be determined and displayed at the corresponding field of view boundary. The instrument indication information indicates the direction of the instrument tip.

[0097] In one example, the instrument indication information can be in the form of an arrow or a finger, etc. In addition, the instrument indication information can be red or green, etc. There is no limitation on the display color of the instrument indication information, as long as it can be accurately distinguished.

[0098] As described above, this embodiment of the application, when it is determined that the end effector of the minimally invasive surgical robot meets the off-screen display requirements, determines multiple first position information based on the first sensor data of the minimally invasive surgical robot. Then, it fuses these multiple first position information to obtain the target positioning result of the end effector. At this point, by fusing multiple first position information, the deficiencies of single position information can be compensated for, thereby improving the accuracy of the target positioning result of the end effector. Subsequently, if it is determined that the end effector is off-screen based on the target positioning result, the end effector is displayed off-screen based on the target positioning result. This implementation method can improve the accuracy of determining whether the end effector is off-screen based on precise target positioning results, thereby improving the off-screen display performance of the minimally invasive surgical robot.

[0099] Figure 2 This is a flowchart illustrating another method for off-screen display of instruments in a minimally invasive surgical robot provided in an embodiment of this application, as shown below. Figure 2As shown, in this embodiment... Figure 1 Based on the embodiments, the method for off-screen display of instruments in a minimally invasive surgical robot is described in detail. The method includes:

[0100] S201. Determine at least one second position information based on the second sensor data of the minimally invasive surgical robot.

[0101] In one example, the second sensor data can be used to initially determine the location of the instrument end effector of the minimally invasive surgical robot. Exemplarily, the second sensor data may include at least one of encoder data, camera image data, and positioning data.

[0102] At this point, at least one second position information can be determined based on the second sensor data, and based on this second position information, it can be determined whether the end effector of the minimally invasive surgical robot meets the off-screen display requirements, as described in the process below. The off-screen display requirements can be used to indicate the critical operating conditions required to activate the off-screen display function.

[0103] In one possible implementation, the second position information can be the position information determined by the sensor data used in the previous moment, or the second position information can be the position information determined by selecting the sensor data with a faster processing speed based on the position detection speed of the sensor data. Here, the method of determining the second position information is not limited, as long as it can be implemented.

[0104] S202. If, based on the second position information, it is determined that the end effector of the minimally invasive surgical robot is located within the edge of the field of vision of the minimally invasive surgical robot, then it is determined that the end effector of the minimally invasive surgical robot meets the off-screen display requirements.

[0105] In one example, if the instrument tip is located at the edge of the endoscopic field of view of the minimally invasive surgical robot, it indicates that the instrument tip of the minimally invasive surgical robot is at the edge of the endoscopic field of view or has just left the endoscopic field of view. At this time, the off-screen display function can be activated, and it can be determined that the off-screen display requirements are met.

[0106] In the above embodiments, it is possible to quickly determine whether the instrument end meets the off-screen display requirements based on any second position information, thereby improving the response speed of the instrument off-screen display of the minimally invasive surgical robot.

[0107] In one example, the second position information can indicate the wrist point position information of the instrument end of the minimally invasive surgical robot, or it can indicate the endpoint information of the instrument end of the minimally invasive surgical robot. Here, the position type of the instrument end indicated by the second position information is not limited, and the actual needs shall prevail.

[0108] At this point, the physical dimensions of the instrument's end effector can be acquired. Then, based on the position type corresponding to the physical dimensions and / or the second position information, the extent of the edge region is determined. Finally, the edge region of the field of view is determined based on the visual field boundary line and the extent of the edge region of the minimally invasive surgical robot.

[0109] In the above embodiments, the flexibility and diversity of determining the edge region of the field of view can be improved based on the physical size information of the instrument end and / or the position type corresponding to the second position information, thereby improving the accuracy of whether the instrument end meets the off-screen display requirements.

[0110] S203. If it is determined that the instrument end of the minimally invasive surgical robot meets the off-screen display requirements, then multiple first position information are determined based on the first sensor data of the minimally invasive surgical robot.

[0111] In one example, this step can be referred to the content described in S101 above, and will not be repeated in detail here.

[0112] In one possible implementation, the first sensor data includes at least joint encoder data and endoscopic video stream data, and the first position information includes at least kinematic position information and visual position information.

[0113] Based on this, kinematic position information can be determined according to joint encoder data and preset kinematic model; and visual position information can be obtained by identifying the instrument end in the endoscopic video stream data according to the target detection model.

[0114] In practice, the controller of the minimally invasive surgical robot can read the joint encoder data of each joint encoder in real time. Then, based on the pre-calibrated robot kinematic model and hand-eye conversion matrix, the kinematic position information can be determined through a preset kinematic model (e.g., a forward kinematic model).

[0115] In one example, the controller of the minimally invasive surgical robot can receive endoscopic video stream data and identify and locate the instrument region in the image in real time based on computer vision algorithms (e.g., algorithms based on color segmentation, contour extraction, or deep learning object detection networks), and determine the visual position information corresponding to the instrument end based on the instrument region.

[0116] In one possible implementation, the embodiments of this application can unify kinematic position information and visual position information into the camera coordinate system and convert them into three-dimensional spatial coordinates of the instrument end effector, thereby enabling accurate fusion processing of kinematic position information and visual position information in subsequent steps.

[0117] In one possible implementation, after obtaining the aforementioned multiple first position information, the multiple first position information can be fused to obtain the target positioning result of the instrument end effector, as described in the process below.

[0118] S204. Obtain the joint error information of the minimally invasive surgical robot, and determine the first detection error based on the joint error information.

[0119] In one example, the joint error information of a minimally invasive surgical robot can be determined based on the spatial position information actually measured by the joint encoder and the labeled spatial position information.

[0120] S205. Determine the model error information of the target detection model, and determine the second detection error based on the model error information.

[0121] In one example, the model error information of the object detection model can be determined based on the deviation between the detection results of the object detection model and the true annotation results.

[0122] S206. Determine the fusion weight information based on the first detection error and the second detection error.

[0123] Optionally, the fusion weight information can be determined based on the error proportions corresponding to the first detection error and the second detection error, respectively.

[0124] Optionally, the fusion weight information can be determined based on a filtering algorithm. Specifically, the minimized error can be obtained through a two-step prediction-update process, and the fusion weight information can be determined based on the minimized error.

[0125] The method for determining the fusion weight information is not limited here; the method that can be implemented shall prevail.

[0126] S207. Based on the fusion weight information, the kinematic position information and visual position information are fused to obtain the target localization result.

[0127] In one example, different methods of determining the fusion weights can correspond to different fusion processing procedures. For instance, if the fusion weights are determined based on the aforementioned error proportions, the kinematic position information and visual position information can be weighted and summed to obtain the target localization result. If the fusion weights are determined based on the aforementioned method of minimizing error, the kinematic position information, visual position information, and predicted position information can be fused using the fusion weights to obtain the target localization result.

[0128] In the above embodiments, the fusion weight information can be determined based on the real-time detection error under different location detection methods, thereby dynamically determining the fusion weight information, improving the accuracy of the fusion weight information, and further improving the accuracy of the target positioning result determined based on the fusion weight information.

[0129] In one possible implementation, the filtering algorithm for determining the fusion weight information described above can be the Kalman filter algorithm.

[0130] At this point, when determining the fusion weight information based on the first detection error and the second detection error, the motion state of the device's end effector can be predicted first based on the joint motion information of the device's end effector, obtaining the predicted position information and the predicted error information. Then, the fusion weight information is determined based on the predicted error information, the first detection error, and the second detection error.

[0131] In one example, the pose (e.g., position and orientation) of the end effector in the camera coordinate system can be used as the system state variables to establish a state-space model of the system. Then, based on the robot's current motion state (which can be determined by joint velocities), a state transition equation is established to predict the pose state at the next moment, obtaining the predicted position information and the corresponding prediction error information (i.e., the predicted covariance matrix).

[0132] Then, the Kalman gain can be determined based on the predicted covariance matrix, the first detection error, and the second detection error, that is, the fusion weight information.

[0133] In practice, the first detection error and the second detection error can be summed to obtain the total detection error. Then, the fusion weight information, i.e., the Kalman gain, can be determined according to the following formula (1).

[0134] (1)

[0135] in, This represents the Kalman gain at time k. This represents the covariance matrix of the prediction at the previous time k-1. This represents the total detection error.

[0136] At this point, when fusing kinematic position information and visual position information according to fusion weight information to obtain target localization results, the predicted position information, kinematic position information and visual position information can be fused according to fusion weight information to obtain target localization results.

[0137] In the above embodiments, by fusing multiple first position information according to the Kalman filtering method, the real-time performance of the fusion of multiple first position information can be improved, and the dynamic fusion of multiple first position information can be realized, thereby improving the real-time performance and accuracy of the fusion of multiple first position information, and thus improving the accuracy of the target positioning result.

[0138] S208. If the instrument end is determined to be off-screen based on the target positioning result, the instrument end is displayed off-screen based on the target positioning result.

[0139] In one example, this step can be referred to the content described in S103 above, and will not be repeated in detail here.

[0140] In one possible implementation, if it is determined that the end effector of the minimally invasive surgical robot does not meet the off-screen display requirements, then a matching position detection type is determined based on the second position information; then, based on the matching position detection type and the third sensor data of the minimally invasive surgical robot, the target positioning result of the end effector is determined.

[0141] In one example, the matched position detection type indicates either visual position detection or kinematic position detection. Under visual position detection, the third sensor data is endoscopic video stream data, which can be used to determine the target localization of the instrument's end effector. Under kinematic position detection, the third sensor data is joint encoder data, which can be used to determine the target localization of the instrument's end effector.

[0142] Optionally, in practical application scenarios, the third sensor data may include endoscopic video stream data and joint encoder data. The endoscopic video stream data is used to determine visual position information, and the joint encoder data is used to determine kinematic position information. Then, based on the matched position detection type, the position weights corresponding to the visual position information and kinematic position information can be determined respectively. Based on these position weights, the visual position information and kinematic position information are weighted and fused to obtain the target localization result.

[0143] For example, if the matched position detection type indicates a visual position detection type, then the position weight corresponding to the kinematic position information can be determined to be a minimum value; if the matched position detection type indicates a kinematic position detection type, then the position weight corresponding to the visual position information can be determined to be a minimum value.

[0144] This implementation method ensures the accuracy and stability of the calculation when using one or more location information to determine the target positioning result in different scenarios, thereby improving the quality of the target positioning result.

[0145] Figure 3This is a schematic diagram of the structure of an off-screen display device for a minimally invasive surgical robot provided in an embodiment of this application, as shown below. Figure 3 As shown, the instrument off-screen display device 30 for the minimally invasive surgical robot provided in this embodiment includes:

[0146] The determining unit 301 is used to determine multiple first position information based on the first sensor data of the minimally invasive surgical robot if it is determined that the instrument end of the minimally invasive surgical robot meets the off-screen display requirements.

[0147] The fusion unit 302 is used to fuse multiple first position information to obtain the target positioning result of the instrument end.

[0148] The off-screen display unit 303 is used to display the instrument end off-screen according to the target positioning result if it is determined that the instrument end is off-screen.

[0149] In one possible implementation, the first sensor data includes at least joint encoder data and endoscopic video stream data; the first position information includes at least kinematic position information and visual position information; in this case, the determining unit 301 is used to:

[0150] Based on the joint encoder data and the preset kinematic model, determine the kinematic position information;

[0151] Based on the target detection model, the instrument ends included in the endoscopic video stream data are identified and processed to obtain visual position information.

[0152] In one possible implementation, the fusion unit 302 is used for:

[0153] Obtain joint error information of the minimally invasive surgical robot, and determine the first detection error based on the joint error information;

[0154] Determine the model error information of the target detection model, and determine the second detection error based on the model error information;

[0155] The fusion weight information is determined based on the first detection error and the second detection error;

[0156] Based on the fusion weight information, the kinematic position information and visual position information are fused to obtain the target localization result.

[0157] In one possible implementation, the fusion unit 302 is used for:

[0158] Based on the joint motion information at the end of the device, the motion state of the end of the device is predicted, and the predicted position information and prediction error information are obtained.

[0159] Based on the predicted error information, the first detection error, and the second detection error, the fusion weight information is determined.

[0160] Based on the fusion weight information, the kinematic position information and visual position information are fused to obtain the target localization result, including:

[0161] The predicted location information, kinematic location information, and visual location information are fused based on the fusion weight information to obtain the target localization result.

[0162] In one possible implementation, the determining unit 301 is configured to:

[0163] Based on the data from the second sensor of the minimally invasive surgical robot, at least one second location information is determined;

[0164] If, based on the second position information, it is determined that the end effector of the minimally invasive surgical robot is located within the edge of the robot's field of vision, then the end effector of the minimally invasive surgical robot meets the off-screen display requirement.

[0165] In one possible implementation, the device is also used for:

[0166] Obtain the physical dimensions of the instrument's end effector;

[0167] The edge region range is determined based on the location type corresponding to the physical size information and / or the second location information;

[0168] The edge region of the field of view is determined based on the boundary line and edge region range of the minimally invasive surgical robot.

[0169] In one possible implementation, the device is also used for:

[0170] If it is determined that the end effector of the minimally invasive surgical robot does not meet the off-screen display requirements, then the matching position detection type is determined based on the second position information.

[0171] Based on the matched position detection type, and using the third sensor data of the minimally invasive surgical robot, the target positioning result of the instrument end is determined.

[0172] The instrument off-screen display device for the minimally invasive surgical robot provided in this embodiment can execute the method provided in the above method embodiment. Its implementation principle and technical effect are similar, and will not be described in detail here.

[0173] Figure 4 This is a schematic diagram of the structure of a computer device provided in an embodiment of this application. Figure 4As shown, the computer device 40 provided in this embodiment includes at least one processor 401 and a memory 402. Optionally, the computer device 40 further includes a communication component 403. The processor 401, memory 402, and communication component 403 are connected via a bus 404.

[0174] In a specific implementation, at least one processor 401 executes computer execution instructions stored in memory 402, causing at least one processor 401 to perform the above-described method.

[0175] The specific implementation process of processor 401 can be found in the above method embodiments, and its implementation principle and technical effect are similar. It will not be repeated here.

[0176] In the above embodiments, it should be understood that the processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), etc. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in this invention can be directly implemented by a hardware processor, or implemented by a combination of hardware and software modules within the processor.

[0177] The memory may include random access memory (RAM) and may also include non-volatile memory (NVM), such as at least one disk storage device.

[0178] The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of illustration, the buses shown in the accompanying drawings are not limited to a single bus or a single type of bus.

[0179] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the above-described method.

[0180] This application also provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, implement the above-described method.

[0181] The aforementioned readable storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk. The readable storage medium can be any available medium accessible to a general-purpose or special-purpose computer.

[0182] An exemplary readable storage medium is coupled to a processor, enabling the processor to read information from and write information to the readable storage medium. Of course, the readable storage medium can also be a component of the processor. The processor and the readable storage medium can reside in an Application Specific Integrated Circuit (ASIC). Alternatively, the processor and the readable storage medium can exist as discrete components in the device.

[0183] The division of units is merely a logical functional division; in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be indirect coupling or communication connection through some interfaces, devices, or units, and may be electrical, mechanical, or other forms.

[0184] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0185] In addition, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0186] If a function is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0187] Those skilled in the art will understand that all or part of the steps of the above-described method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When executed, the program performs the steps of the above-described method embodiments; and the aforementioned storage medium includes various media capable of storing program code, such as ROM, RAM, magnetic disks, or optical disks.

[0188] Finally, it should be noted that other embodiments of the invention will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This invention is intended to cover any variations, uses, or adaptations of the invention that follow the general principles of the invention and include common knowledge or customary techniques in the art not disclosed herein, and is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of the invention is limited only by the appended claims.

Claims

1. A method for off-screen display of instruments in a minimally invasive surgical robot, characterized in that, The method includes: If it is determined that the instrument end of the minimally invasive surgical robot meets the off-screen display requirements, then multiple first position information are determined based on the first sensor data of the minimally invasive surgical robot. The multiple first location information are fused to obtain the target positioning result of the instrument end effect; If, based on the target positioning result, it is determined that the device end is off-screen, then, based on the target positioning result, the device end is displayed off-screen.

2. The method according to claim 1, characterized in that, The first sensor data includes at least joint encoder data and endoscopic video stream data; the first position information includes at least kinematic position information and visual position information; Based on the first sensor data of the minimally invasive surgical robot, multiple first location information are determined, including: The kinematic position information is determined based on the joint encoder data and the preset kinematic model; Based on the target detection model, the instrument end included in the endoscopic video stream data is identified and processed to obtain the visual position information.

3. The method according to claim 2, characterized in that, The multiple first location information are fused to obtain the target positioning result of the instrument end effector, including: Obtain the joint error information of the minimally invasive surgical robot, and determine the first detection error based on the joint error information; Determine the model error information of the target detection model, and determine the second detection error based on the model error information; Based on the first detection error and the second detection error, determine the fusion weight information; Based on the fusion weight information, the kinematic position information and the visual position information are fused to obtain the target localization result.

4. The method according to claim 3, characterized in that, Based on the first detection error and the second detection error, the fusion weight information is determined, including: Based on the joint motion information of the device end, the motion state of the device end is predicted to obtain the predicted position information and the prediction error information. The fusion weight information is determined based on the predicted error information, the first detection error, and the second detection error. The step of fusing the kinematic position information and the visual position information according to the fusion weight information to obtain the target localization result includes: The predicted location information, the kinematic location information, and the visual location information are fused based on the fusion weight information to obtain the target localization result.

5. The method according to any one of claims 1-4, characterized in that, The instrument end effector of the minimally invasive surgical robot was determined to meet off-screen display requirements, including: Based on the second sensor data of the minimally invasive surgical robot, at least one second location information is determined; If, based on the second location information, it is determined that the instrument tip of the minimally invasive surgical robot is located within the edge area of ​​the robot's field of vision, then the instrument tip of the minimally invasive surgical robot meets the off-screen display requirement.

6. The method according to claim 5, characterized in that, The method further includes: Obtain the physical dimensions of the instrument's end effector; The edge region range is determined based on the physical size information and / or the location type corresponding to the second location information; The visual field edge region is determined based on the visual field boundary line of the minimally invasive surgical robot and the range of the edge region.

7. The method according to claim 5, characterized in that, The method further includes: If it is determined that the instrument end of the minimally invasive surgical robot does not meet the off-screen display requirements, then the matching position detection type is determined based on the second position information; Based on the matched position detection type and the third sensor data of the minimally invasive surgical robot, the target positioning result of the instrument end is determined.

8. An off-screen display device for instruments in a minimally invasive surgical robot, characterized in that, include: The determining unit is used to determine multiple first position information based on the first sensor data of the minimally invasive surgical robot if it is determined that the instrument end of the minimally invasive surgical robot meets the off-screen display requirements. The fusion unit is used to fuse the multiple first position information to obtain the target positioning result of the instrument end effector; An off-screen display unit is used to display the instrument end off-screen according to the target positioning result if it is determined that the instrument end is off-screen.

9. A computer device, characterized in that, include: Memory, processor; The memory stores computer-executed instructions; The processor executes computer execution instructions stored in the memory, causing the processor to perform the method as described in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, are used to implement the method as described in any one of claims 1-7.

Images