Projection method, device, equipment and medium for robotic laparoscopic surgery

Abstract

Embodiments of the present application provide a projection method, device, equipment and medium for robotic endoscopic surgery. The method comprises: judging whether visual impairment exists in robotic endoscopic surgery, if so, obtaining a target transformation matrix, calculating a target position of the instrument end of the instrument holding arm according to the target transformation matrix and a theoretical spatial position of the instrument end of the instrument holding arm in an endoscope coordinate system at the current moment, and projecting the instrument end of the instrument holding arm based on the target position. The target transformation matrix is a transformation matrix corresponding to the last unimpaired moment before the occurrence of visual impairment of the instrument end of the instrument holding arm, and is used to represent the conversion relationship between the actual spatial position and the theoretical spatial position of the instrument end of the instrument holding arm in the endoscope coordinate system. The method can improve the display accuracy of the instrument end and improve the safety and continuity of robotic endoscopic surgery.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and in particular to a projection method, apparatus, device, and medium for robotic laparoscopic surgery. Background Technology

[0002] With the widespread adoption of minimally invasive and precision medical concepts in modern medicine, clinicians have placed higher demands on trauma control, operational precision, and postoperative recovery levels in surgical procedures. The deep integration of cutting-edge technologies such as precision mechanics, intelligent control, and high-definition 3D imaging with clinical medicine has provided strong support for innovation in surgical procedures. Robotic laparoscopic surgery aligns with the high-quality development trend of minimally invasive surgery, possessing clinical advantages such as minimal trauma, high precision, and rapid recovery. It effectively meets the needs of refined diagnosis and treatment of surgical diseases and is now widely used in urology, gynecology, thoracic surgery, and other fields, representing an important direction for promoting the transformation and upgrading of minimally invasive surgery and standardized clinical treatment. Because surgical procedures heavily rely on real-time high-definition visual feedback provided by the endoscope, the stability and clarity of the intraoperative visual field are crucial. However, during robotic laparoscopic surgery, conditions such as smoke obstruction, lens contamination, or abnormal lighting can easily lead to a decline in intraoperative visual quality, making accurate display of the instrument tip impossible and affecting the continuity and safety of the surgery.

[0003] Existing technologies mainly employ two independent methods to achieve intraoperative instrument end-effector display: one is the robotic arm kinematics calibration method, which relies on static calibration of DH parameters, solves the spatial pose of the instrument end-effector through the forward kinematics of the robotic arm, and completes visual projection reconstruction to achieve instrument end-effector display; the other is the endoscopic visual recognition method, which completes instrument end-effector display by performing feature reasoning and target segmentation recognition on images acquired by the endoscope.

[0004] However, both of the above methods have limitations. If a sudden situation occurs during the operation that leads to a decline in visual quality, the former will lose real-time correction of visual feedback, and the latter will fail directly or produce misidentification due to the loss of target features. Neither can maintain the continuity and accuracy of the instrument end display under visual impairment, thus affecting the safety of the operation. Summary of the Invention

[0005] This invention provides a projection method, apparatus, device, and medium for robotic laparoscopic surgery to improve the display accuracy of the instrument tip and ensure the safety and continuity of robotic laparoscopic surgery.

[0006] In a first aspect, embodiments of the present invention provide a projection method for robotic laparoscopic surgery, comprising:

[0007] Determine if visual impairment occurs during robotic laparoscopic surgery;

[0008] If it exists, the target transformation matrix is ​​obtained; the target transformation matrix is ​​the transformation matrix corresponding to the last undamaged moment of the end of the robotic arm before visual impairment occurs, and is used to represent the transformation relationship between the actual spatial position and the theoretical spatial position of the end of the robotic arm in the endoscope coordinate system.

[0009] Based on the target transformation matrix and the theoretical spatial position of the end of the holding arm in the endoscope coordinate system at the current moment, calculate the target position of the end of the holding arm.

[0010] Based on the target position, the end of the holding arm is projected.

[0011] In one possible implementation, determining whether visual impairment occurs during robotic laparoscopic surgery includes:

[0012] Acquire the field of view images of the robotic laparoscopic surgery;

[0013] The visual field image is input into a segmentation model, a lighting condition assessment model, and a visual noise model, respectively, and a score is obtained from each model. The score output by the segmentation model is positively correlated with the confidence level of the pixel position of the segmented end of the robotic arm. The score output by the lighting condition assessment model is positively correlated with the illumination quality of the visual field image. The score output by the visual noise model is positively correlated with the visual purity of the visual field image.

[0014] Based on the scores output by the segmentation model, the light state assessment model, and the visual noise model, it is determined whether visual impairment exists during the robotic laparoscopic surgery.

[0015] In one possible implementation, determining whether visual impairment exists during the robotic laparoscopic surgery based on the scores output by the segmentation model, the light state assessment model, and the visual noise model includes:

[0016] The scores output by the segmentation model, the light state evaluation model, and the visual noise model are weighted and summed to obtain the visual score of the robotic laparoscopic surgery.

[0017] If the visual score is greater than the preset score, it is determined that there is no visual impairment during the robotic laparoscopic surgery;

[0018] If the visual score is less than or equal to the preset score, then visual impairment is determined to exist during the robotic laparoscopic surgery.

[0019] In one possible implementation, the method further includes:

[0020] If there is no visual impairment during the robotic laparoscopic surgery at the current moment, then calculate the spatial deviation between the actual spatial position and the theoretical spatial position of the end effector arm in the endoscope coordinate system;

[0021] The transformation matrix is ​​determined based on the spatial deviation.

[0022] In one possible implementation, the method further includes:

[0023] If the robotic laparoscopic surgery presents with visual impairment at the current moment, but not at the previous moment, then the transformation matrix determined at the previous moment is locked as the target transformation matrix.

[0024] In one possible implementation, before calculating the spatial deviation between the actual and theoretical spatial positions of the end effector arm in the endoscopic coordinate system if there is no visual impairment during the robotic laparoscopic surgery at the current moment, the method further includes:

[0025] The visual field image of the robotic laparoscopic surgery is processed by a segmentation model to obtain the pixel position of the end effector of the robotic arm; wherein, the visual field image is a binocular image;

[0026] Based on the binocular endoscope parameters, the pixel positions are reconstructed to obtain the actual spatial position of the end of the holding arm in the endoscope coordinate system.

[0027] The theoretical spatial position of the end effector of the holding arm in the endoscope coordinate system is calculated using forward kinematics.

[0028] In one possible implementation, projecting the end effector of the robotic arm based on the target location includes:

[0029] Based on the target position, the end effector of the robotic arm is projected in OBB mode.

[0030] In a second aspect, embodiments of the present invention provide a projection device for robotic laparoscopic surgery, comprising:

[0031] The judgment module is used to determine whether there is visual impairment during robotic laparoscopic surgery;

[0032] The acquisition module is used to acquire a target transformation matrix when visual impairment is determined to occur during robotic laparoscopic surgery. The target transformation matrix is ​​the transformation matrix corresponding to the last unimpaired moment before the occurrence of visual impairment of the end of the robotic arm instrument, and is used to represent the transformation relationship between the actual spatial position and the theoretical spatial position of the end of the robotic arm instrument in the endoscopic coordinate system.

[0033] The calculation module is used to calculate the target position of the end of the robotic arm based on the target transformation matrix and the theoretical spatial position of the end of the robotic arm in the endoscope coordinate system at the current moment.

[0034] The projection module is used to project the end effector of the robotic arm based on the target position.

[0035] In one possible implementation, the determining module is specifically used for:

[0036] Acquire the field of view images of the robotic laparoscopic surgery;

[0037] The visual field image is input into a segmentation model, a lighting condition assessment model, and a visual noise model, respectively, and a score is obtained from each model. The score output by the segmentation model is positively correlated with the confidence level of the pixel position of the segmented end of the robotic arm. The score output by the lighting condition assessment model is positively correlated with the illumination quality of the visual field image. The score output by the visual noise model is positively correlated with the visual purity of the visual field image.

[0038] Based on the scores output by the segmentation model, the light state assessment model, and the visual noise model, it is determined whether visual impairment exists during the robotic laparoscopic surgery.

[0039] In one possible implementation, the determining module is further specifically used for:

[0040] The scores output by the segmentation model, the light state evaluation model, and the visual noise model are weighted and summed to obtain the visual score of the robotic laparoscopic surgery.

[0041] If the visual score is greater than the preset score, it is determined that there is no visual impairment during the robotic laparoscopic surgery;

[0042] If the visual score is less than or equal to the preset score, then visual impairment is determined to exist during the robotic laparoscopic surgery.

[0043] In one possible implementation, the acquisition module is specifically used for:

[0044] If the robotic laparoscopic surgery presents with visual impairment at the current moment, but not at the previous moment, then the transformation matrix determined at the previous moment is locked as the target transformation matrix.

[0045] In one possible implementation, the projection module is specifically used for:

[0046] Based on the target position, the end effector of the robotic arm is projected in OBB mode.

[0047] In one possible implementation, the projection device for robotic laparoscopic surgery further includes:

[0048] The determination module is used to calculate the spatial deviation between the actual and theoretical spatial positions of the end effector of the robotic laparoscopic surgery in the endoscope coordinate system if there is no visual impairment at the current moment; and to determine the transformation matrix based on the spatial deviation.

[0049] The determining module is further configured to process the visual field image of the robotic laparoscopic surgery using a segmentation model to obtain the pixel position of the end effector of the robotic arm; wherein the visual field image is a binocular image; depth reconstruction is performed on the pixel position according to the binocular endoscope parameters to obtain the actual spatial position of the end effector of the robotic arm in the endoscope coordinate system; and the theoretical spatial position of the end effector of the robotic arm in the endoscope coordinate system is calculated using forward kinematics.

[0050] Thirdly, embodiments of the present invention provide a projection device for robotic laparoscopic surgery, comprising:

[0051] Memory and processor;

[0052] The memory stores computer-executed instructions;

[0053] The processor executes computer execution instructions stored in the memory, causing the processor to perform the first aspect and / or various possible implementations of the first aspect as described above.

[0054] Fourthly, embodiments of the present invention provide a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, are used to implement the first aspect and / or various possible implementations of the first aspect.

[0055] The projection method, apparatus, device, and medium for robotic laparoscopic surgery provided in this invention, when visual impairment is detected during robotic laparoscopic surgery, obtains a target transformation matrix calculated at the last unimpaired moment before visual impairment occurs. This matrix represents the transformation relationship between the actual and theoretical spatial positions of the end-effector of the robotic arm in the endoscopic coordinate system. Based on this target transformation matrix and the theoretical spatial position of the end-effector of the robotic arm in the endoscopic coordinate system at the current moment, a target position is calculated. Then, the end-effector of the robotic arm is projected based on the target position. This allows for a relatively accurate estimation and projection display of the end-effector position even when vision is impaired, using the target transformation matrix from when vision is normal. This ensures the safety and continuity of robotic laparoscopic surgery. Attached Figure Description

[0056] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention.

[0057] Figure 1 A schematic diagram illustrating an application scenario of the projection method for robotic laparoscopic surgery provided by this invention;

[0058] Figure 2 A flowchart of a first embodiment of the projection method for robotic laparoscopic surgery provided by the present invention;

[0059] Figure 3 A flowchart of Embodiment 2 of the projection method for robotic laparoscopic surgery provided by the present invention;

[0060] Figure 4 A flowchart of Embodiment 3 of the projection method for robotic laparoscopic surgery provided by the present invention;

[0061] Figure 5 A flowchart of Embodiment 4 of the projection method for robotic laparoscopic surgery provided by the present invention;

[0062] Figure 6 A schematic diagram of the projection method for robotic laparoscopic surgery provided by the present invention;

[0063] Figure 7 This is a schematic diagram of the display state under normal visual conditions provided by the present invention;

[0064] Figure 8 This is a schematic diagram comparing display states under different visual conditions provided by the present invention.

[0065] Figure 9 A schematic diagram of the projection device for robotic laparoscopic surgery provided by the present invention;

[0066] Figure 10 This is a schematic diagram of the projection device for robotic laparoscopic surgery provided by the present invention.

[0067] The accompanying drawings have illustrated specific embodiments of the invention, which will be described in more detail below. These drawings and descriptions are not intended to limit the scope of the invention in any way, but rather to illustrate the concept of the invention to those skilled in the art through reference to particular embodiments. Detailed Implementation

[0068] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numerals in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present invention. Rather, they are merely examples of apparatuses and methods consistent with some aspects of the invention as detailed in the appended claims.

[0069] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this invention are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, use and processing of related data must comply with relevant laws, regulations and standards, and corresponding operation entry points are provided for users to choose to authorize or refuse.

[0070] Robotic laparoscopic surgery is a high-precision, minimally invasive surgical procedure that uses a robotic system (such as the da Vinci Surgical System) to insert an endoscope and surgical instruments into the body through a tiny incision, and is then performed by a surgeon controlling a robotic arm from a control console.

[0071] For example, Figure 1 This is a schematic diagram illustrating an application scenario of the projection method for robotic laparoscopic surgery provided by the present invention, such as... Figure 1 As shown, the robot system comprises several robotic arms, a robot control host, and a display terminal. The display terminal can be integrated into the robot control host; several robotic arms are integrated onto the same base, and a spatial Cartesian coordinate system fixed to this base can be defined as the robot's base coordinate system, serving as a unified reference for describing the spatial poses of all robotic arms and instrument ends; different surgical instruments (such as…) are mounted on the robot. Figure 1 The robotic arms of surgical instruments 1 and 2 are called the robotic arms, and the robotic arm carrying the endoscope is called the endoscope arm. The specific workflow is as follows: the robot control host coordinates the operation of the whole machine and issues motion control commands. The robotic arms carrying surgical instruments perform surgical operations according to the motion control commands. The endoscope arm carrying the endoscope collects images of the surgical area. The images of the surgical area are transmitted to the image processing unit integrated inside the robot control host for processing. The processed clear images of the surgical area are pushed to the display terminal for real-time display. The surgeon uses the images of the surgical area displayed on the display terminal to control each robotic arm to complete precise minimally invasive operations.

[0072] Current methods for displaying end-effectors in robotic laparoscopic surgery typically employ two approaches: robotic arm kinematic calibration and endoscopic visual recognition. The robotic arm kinematic calibration method uses statically calibrated Denavit-Hartenberg (DH) parameters and current joint variables. Based on these parameters and variables, it calculates the spatial pose of the end-effector in the robot's base coordinate system using a forward kinematics model. This pose is then transformed to the endoscopic coordinate system, projected onto the surgical area image, and displayed on the terminal. The endoscopic visual recognition method relies on real-time images acquired by the endoscope. It utilizes feature extraction, target detection, and semantic segmentation to identify and locate the end-effector, obtaining its pixel coordinates within the image plane. These coordinates are then superimposed and annotated onto the display terminal for final display.

[0073] The kinematic calibration method for robotic arms is based on the assumption of an ideal rigid structure and does not consider the mechanical deformation characteristics under actual surgical conditions. During surgery, the robotic arm is affected by multiple factors such as its own weight, the flexible deformation of the links, and the disturbance of the surgical load, which will produce real-time dynamic non-rigid deviations. The static calibration DH parameters are fixed offline values ​​before surgery and cannot be dynamically corrected and compensated for as the surgical conditions change. This results in inherent deviations in the spatial pose solution of the instrument, which in turn affects the accuracy of the instrument end-effector display. On the other hand, the endoscopic visual recognition method is highly dependent on high-quality, interference-free intraoperative imaging conditions. It relies on clear and complete image features for learning and reasoning. When visual impairment occurs during surgery, such as smoke obstruction, lens contamination, or abnormal lighting, the effective image features are greatly attenuated or even completely lost, making it impossible to complete target segmentation and feature matching. Ultimately, this leads to the failure of the instrument end-effector display. Users can only continue the subsequent surgical process after cleaning the dirt that causes visual impairment. This is difficult to meet the safety and continuity requirements of complex surgical scenarios.

[0074] In summary, both of the above methods suffer from low end-effector display accuracy, which affects the safety and continuity of robotic laparoscopic surgery.

[0075] To address the aforementioned problems, this invention provides a projection method for robotic laparoscopic surgery. The technical concept is as follows: When visual impairment occurs during surgery, the end-effector display fails, affecting the safety and continuity of the procedure. However, the theoretical spatial position of the end-effector in the endoscopic coordinate system, calculated using the robotic arm kinematic calibration method, is unaffected by visual impairment. This method can be used to calculate the theoretical spatial position of the end-effector. However, actual robotic arms are not ideal rigid structures, and the theoretical spatial position calculated solely through forward kinematics has deviations. Therefore, when vision is intact, a transformation matrix needs to be calculated in real-time. This transformation matrix represents the transformation relationship between the actual and theoretical spatial positions of the end-effector in the endoscopic coordinate system. When vision is impaired, the deviation of the theoretical spatial position at the current moment is corrected using the target transformation matrix calculated at the last unimpaired moment before visual impairment, yielding the target position of the end-effector. Finally, the end-effector is projected based on this target position. In this way, even when visual impairment occurs during surgery due to smoke obstruction, lens contamination, or abnormal lighting, the system can still accurately display the image based on the theoretical spatial position of the end effector of the robotic arm, thus improving the safety and continuity of robotic laparoscopic surgery.

[0076] The technical solution of the present invention and how the technical solution of the present invention solves the above-mentioned technical problems are described in detail below with specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments. The embodiments of the present invention will now be described with reference to the accompanying drawings.

[0077] Figure 2 The flowchart of the projection method for robotic laparoscopic surgery provided by the present invention is as follows: Figure 2 As shown, the method includes:

[0078] S201: Determine if visual impairment exists during robotic laparoscopic surgery.

[0079] Reference Figure 1 Robotic laparoscopic surgery is a minimally invasive surgical procedure performed collaboratively by a robot control unit, an instrument arm (carrying surgical instruments), an endoscope arm (carrying an endoscope), and a display terminal.

[0080] Visual impairment refers to the inability of the endoscope to accurately identify the instrument tip in the current endoscopic field of view. It can manifest as smoke obscuring the image, blood stains, water mist diffusion, strong reflection, local darkness, and blurring during rapid movement. It can also manifest as incomplete instrument outlines, decreased edge contrast, and key areas being obscured by tissue.

[0081] In practice, the image processing unit integrated inside the robot control host continuously receives the field-of-view images output by the endoscope and evaluates and scores the quality of the current field-of-view image at frame intervals or fixed time windows. When the score is lower than a preset threshold, it is determined that there is visual impairment at the current moment.

[0082] It should be understood that the specific implementation process and principles of this step will be explained later. Figure 3 The embodiments shown are described in detail here, and will not be repeated here.

[0083] S202: If it exists, obtain the target transformation matrix.

[0084] Among them, the target transformation matrix is ​​the transformation matrix corresponding to the last undamaged moment of the end of the orthogonal arm before visual impairment occurs. It is used to represent the transformation relationship between the actual spatial position and the theoretical spatial position of the end of the orthogonal arm in the endoscopic coordinate system.

[0085] In one possible implementation, if there is visual impairment during the robotic laparoscopic surgery at the current moment, but no visual impairment at the previous moment, then the transformation matrix determined at the previous moment is locked as the target transformation matrix.

[0086] The target transformation matrix includes translational and rotational components. The translational component is used to compensate for positional deviations caused by joint clearance, self-weight sagging, and force deformation; the rotational component is used to compensate for the effects of instrument orientation and installation posture errors on end-effector spatial positioning.

[0087] The end effector of a robotic arm refers to the working area of ​​the surgical instrument end that is installed on the robotic arm and can perform operations such as clamping, cutting, electrocoagulation, separation, or suturing.

[0088] The endoscope coordinate system is a three-dimensional spatial coordinate system constructed based on the endoscope imaging system. The transformation relationship from the robot base coordinate system to the endoscope coordinate system can be determined through hand-eye calibration, and the coordinate unification between the robot base coordinate system and the endoscope coordinate system can be completed by combining the intrinsic and extrinsic parameters of the binocular endoscope.

[0089] In practice, under normal visual conditions, the transformation matrix is ​​continuously calculated and updated. The calculation process includes obtaining the theoretical spatial position of the instrument's end effector in the endoscopic coordinate system and its actual spatial position at the same moment. The transformation matrix is ​​then obtained based on these two spatial positions. When S201 determines that there is visual impairment at the current moment, the actual spatial position is no longer calculated based on the current visual field image. Instead, the saved target transformation matrix is ​​directly read and used in subsequent S203 to calculate the target spatial position of the instrument's end effector.

[0090] Optionally, when vision is normal, the transformation matrix is ​​updated in real time at a fixed frequency, and the result of each update is written to the cache or register. After visual impairment is detected for the first time, the transformation matrix of the previous moment is locked, and the transformation matrix calculation is stopped using the current visual field image, thereby preventing erroneous transformation matrices from being written to the cache or register.

[0091] It should be understood that the specific implementation process and principles of this step will be explained later. Figure 4 The embodiments shown are described in detail here, and will not be repeated here.

[0092] S203: Calculate the target position of the end of the arm based on the target transformation matrix and the theoretical spatial position of the end of the arm in the endoscope coordinate system at the current moment.

[0093] The target position is the direct input for subsequent projection, reflecting the predicted spatial position of the instrument's end effector in the endoscopic coordinate system under visual impairment scenarios. Compared to the theoretical spatial position calculated solely based on forward kinematics, the target transformation matrix used in the target position calculation considers the spatial deviation between the theoretical and actual spatial positions, and can correct the current theoretical spatial position. Therefore, the calculated target position is closer to the real spatial state.

[0094] In practice, the theoretical spatial position of the end effector of the arm at the current moment in the endoscope coordinate system is first obtained, and then multiplied by the target transformation matrix to obtain the target position of the end effector of the arm.

[0095] S204: Project the end effector of the robotic arm based on the target position.

[0096] Projection refers to mapping the calculated three-dimensional target position of the instrument tip in the endoscopic coordinate system onto a two-dimensional image plane and displaying it as a graphic overlay on the current endoscopic field of view. This projection result can be represented as dot markers, crosshairs, outlines, rotational bounding boxes, or directional cues aligned with the instrument's axis, allowing the surgeon to still perceive the approximate position of the instrument tip and its relative relationship to the target tissue even with visual impairment.

[0097] In one possible implementation, the end effector of the robotic arm is projected under an Oriented Bounding Box (OBB) based on the target position.

[0098] The OBB mode is used to overlay the approximate area and orientation of the instrument tip onto the endoscopic field-of-view image in the form of a tilted rectangular box, based on the target location of the instrument tip. This maintains high visibility even when the field of view is affected by smoke, water mist, or blood. The bounding box in this OBB mode can determine its center point, long side direction, and bounding size according to the spatial orientation corresponding to the target location, thus ensuring that the displayed result is consistent with the actual orientation of the instrument tip.

[0099] In practice, after obtaining the target position, it is first mapped onto the field of view image. Then, the rotation angle and bounding box boundary parameters are calculated based on the posture information of the instrument's end effector, and a rotated bounding box is generated accordingly. The generated rotated bounding box is displayed in the field of view image in a graphic overlay manner, which can provide continuous prompts for the end effector of the instrument arm without obstructing too much of the effective field of view. This rotated bounding box can be drawn in real time by the image processing unit, and the frame color and line width can be adapted according to the display terminal resolution to ensure good recognition under different image sharpness.

[0100] This invention provides a projection method for robotic laparoscopic surgery, comprising: first, determining whether visual impairment exists during the robotic laparoscopic surgery; if so, obtaining a target transformation matrix; then, calculating the target position of the endoscope's distal end based on the target transformation matrix and the theoretical spatial position of the endoscope's distal end in the endoscopic coordinate system at the current moment; and finally, projecting the endoscope's distal end based on the target position. The target transformation matrix is ​​the transformation matrix corresponding to the last unimpaired moment before visual impairment occurs, used to represent the transformation relationship between the actual and theoretical spatial positions of the endoscope's distal end in the endoscopic coordinate system. In this embodiment, by continuously calculating and updating the transformation relationship between the actual and theoretical spatial positions during the unimpaired phase, and locking and calling the target transformation matrix calculated at the last unimpaired moment before visual impairment occurs to correct the current theoretical spatial position, and then projecting the corrected result onto the endoscopic image in real time, the continuity of the instrument's distal end projection display can be maintained in visually impaired scenarios such as smoke, pollution, blurriness, and occlusion. This method avoids the failure of endoscopic visual recognition when vision is impaired, and also reduces the positioning error of robotic arm kinematic calibration caused by load or link deformation. As a result, the projection results can still reflect the actual spatial position of the end of the robotic arm in the endoscopic coordinate system even when vision is impaired, providing continuous and stable operation reference for the surgeon.

[0101] Figure 3 The flowchart of Embodiment 2 of the projection method for robotic laparoscopic surgery provided by the present invention is as follows: Figure 3 As shown, S201 can be achieved through the following steps:

[0102] S301: Acquire visual images of the robotic laparoscopic surgery.

[0103] Among them, the visual field image refers to the real-time image captured by the endoscope during the operation. This visual field image can be preprocessed by the image processing unit, such as filtering, cropping, data enhancement, etc., and then displayed through the display terminal.

[0104] In practical applications, the field of view image includes the end effector of the robotic arm, the surface of human tissue, and the internal space environment.

[0105] S302: Input the field-of-view image into the segmentation model, the lighting condition assessment model, and the visual noise model respectively, and obtain the score output by each model.

[0106] Among them, the score output by the segmentation model is positively correlated with the confidence level of the pixel position of the segmented end of the robotic arm; the score output by the lighting condition assessment model is positively correlated with the lighting quality of the visual field image; and the score output by the visual noise model is positively correlated with the visual purity of the visual field image.

[0107] Specifically, the segmentation model is used to perform pixel-level identification of the instrument tip region in the visual field image and outputs a corresponding confidence score. A higher score indicates a clearer instrument tip location. The lighting condition assessment model is used to analyze the brightness, uniformity, and local overexposure of the visual field image and outputs a score reflecting the lighting quality. A higher score indicates better lighting conditions. The visual noise model is used to detect smoke, blood, water mist, jitter artifacts, and other interfering factors and outputs a score reflecting visual purity. A higher score indicates higher visual purity.

[0108] Specifically, during the model training phase, training sets for the three models are constructed based on multiple sample field-of-view images and the annotation information for each sample field-of-view image. Specifically, for the segmentation model, the annotation information describes the end effector of the holding arm in the sample field-of-view image; for the lighting condition assessment model, the annotation information describes the lighting score in the sample field-of-view image; and for the visual noise model, the annotation information describes the visual purity score in the sample field-of-view image.

[0109] It should be understood that the above-mentioned annotation information can be obtained through doctor calibration or other technical means. This application embodiment does not specifically limit the method of obtaining the annotation information.

[0110] Furthermore, for each model, it is trained using the corresponding training set until the training cutoff condition is met.

[0111] In practice, the visual field images can be acquired in real time by the endoscope and sent to the image processing unit. The image processing unit can call pre-trained segmentation models, lighting condition evaluation models, and visual noise models to score the visual field images acquired at the same time.

[0112] S303: Based on the scores output by the segmentation model, the lighting condition assessment model, and the visual noise model, determine whether there is visual impairment during robotic laparoscopic surgery.

[0113] In one possible implementation, the scores output by the segmentation model, the lighting condition assessment model, and the visual noise model are weighted and summed to obtain the visual score of the robotic laparoscopic surgery. If the visual score is greater than the preset score, it is determined that there is no visual impairment in the robotic laparoscopic surgery; if the visual score is less than or equal to the preset score, it is determined that there is visual impairment in the robotic laparoscopic surgery.

[0114] Among them, the visual score is used to characterize the comprehensive quantitative result of the current field image quality in robotic laparoscopic surgery. The higher the score, the more the segmentation confidence of the end effector of the robotic arm, the illumination quality of the surgical field, and the visual purity meet the criteria for normal vision.

[0115] The preset score is used as a threshold benchmark for judging visual impairment. It can be preset according to the minimum requirements of robotic laparoscopic surgery for image continuity, instrument end recognition stability and intraoperative guidance reliability, and can be adjusted in combination with different surgical methods, different lens magnifications or different imaging devices.

[0116] During weighted summation, the weights of each model can be determined based on their impact on the usability of the surgical field. For example, in scenarios where instrument edges are easily obscured, the weight of the segmentation model score can be increased; in scenarios with strong reflections or local underexposure, the weight of the lighting condition assessment model score can be increased; and in scenarios with heavy electrocoagulation smoke, the weight of the visual noise model score can be increased. After comparing the weighted visual score with the preset score, the result determining whether visual impairment exists during robotic laparoscopic surgery can be output.

[0117] In practice, the visual score of robotic laparoscopic surgery can be calculated in real time using the following formula:

[0118]

[0119] in, for Time-segmentation model scoring for The scoring of the light condition assessment model at any time. for Time-of-flight visual noise model scoring The weights for the segmentation model scores, The weights for the scoring of the light condition assessment model. Weights for scoring the visual noise model. , and The specific value can be set according to the actual scenario, and the sum of the three is 1.

[0120] For example, the scoring can be preset according to actual needs. and ,when At that time, it was determined that there was no visual impairment during robotic laparoscopic surgery; when At that time, it was determined that visual impairment occurred during robotic laparoscopic surgery.

[0121] This invention provides a multi-dimensional quality assessment of images within the same field of view, simultaneously incorporating instrument end-effector separability, lighting suitability, and image purity into the judgment criteria. This eliminates reliance on a single feature for visual impairment identification. When smoke obscures the surgical field, lens contamination occurs, or illumination is insufficient, a visual impairment determination can be output promptly, providing triggering conditions for subsequent invocation of the target transformation matrix and instrument end-effector projection.

[0122] Next, through Figure 4 The process of determining the transformation matrix is ​​explained in detail.

[0123] Figure 4 The flowchart of Embodiment 3 of the projection method for robotic laparoscopic surgery provided by the present invention is as follows: Figure 4 As shown, the method includes:

[0124] S401: If there is no visual impairment during the robotic laparoscopic surgery at the current moment, calculate the spatial deviation between the actual and theoretical spatial positions of the end effector arm in the endoscopic coordinate system.

[0125] Among them, spatial deviation is used to characterize the offset between the actual spatial position and the theoretical spatial position of the end of the holding arm in the endoscope coordinate system. It can reflect both the difference in three-dimensional position coordinates and the direction and magnitude relationship of the position offset.

[0126] The transformation matrix is ​​used to describe the spatial transformation relationship between the actual spatial location and the theoretical spatial location, and can be used as a basis for correcting the theoretical spatial location when vision is impaired.

[0127] In practice, the actual spatial position is compared with the theoretical spatial position to obtain the spatial deviation. The spatial deviation can be represented as a coordinate difference vector, or as a combination of rotation and translation deviations, and then used to fit the mapping relationship between the actual and theoretical spatial positions.

[0128] In one possible implementation, the difference between the actual spatial position and the theoretical spatial position at the current moment can be defined as the spatial deviation.

[0129] In another possible implementation, the difference between the actual spatial position and the theoretical spatial position at the current moment can be determined as the initial spatial deviation. The initial spatial deviation at the current moment is then weighted and summed with the spatial deviation at the previous moment to obtain the spatial deviation at the current moment.

[0130] In the implementation, the spatial deviation at the current moment can be expressed by the following formula:

[0131]

[0132] in, For theoretical spatial location, For actual spatial location, The spatial deviation at time t-1 The spatial deviation at time t, The function representing the spatial dynamic deviation at time t is used to calculate the spatial deviation at time t based on the actual spatial position at time t, the theoretical spatial position at time t, and the spatial deviation at time t-1.

[0133] S402: Determine the transformation matrix based on the spatial deviation.

[0134] In practical implementation, a coordinate transformation model can be constructed based on spatial deviations, and a transformation matrix can be obtained through least-squares fitting, matrix inversion, or pose registration. This transformation matrix can reflect the true spatial relationships under the combined effects of the robotic arm, instrument load, and intraoperative environment. In practical applications, this transformation matrix can also be updated smoothly by incorporating multi-frame deviations to reduce the impact of instantaneous noise on the results.

[0135] Optionally, the relationship between the transformation matrix, the actual spatial location, and the theoretical spatial location can be expressed by the following formula:

[0136]

[0137] in, This is the transformation matrix.

[0138] This invention calculates the spatial deviation between the actual and theoretical spatial positions when visual impairment is absent, and determines a transformation matrix accordingly. This allows for continuous correction of the mapping relationship between the actual and theoretical spatial positions of the instrument tip when visual conditions are good, resulting in higher accuracy and timeliness of the target transformation matrix when visual impairment occurs. Consequently, it reduces projection drift caused by mechanical errors, tissue contact disturbances, or changes in instrument load under visual impairment conditions, improving the continuity of instrument tip positioning and guidance stability in endoscopic images, and ensuring a high degree of consistency between the projection results and the actual instrument position during robotic laparoscopic surgery.

[0139] Figure 5 The flowchart of Embodiment 4 of the projection method for robotic laparoscopic surgery provided by the present invention is as follows: Figure 5 As shown, before S401 calculates the spatial deviation, the following steps are also included:

[0140] S501: The field of view image of robotic laparoscopic surgery is processed by a segmentation model to obtain the pixel position of the end effector of the robotic arm.

[0141] Among them, the visual field image is a binocular image, specifically the left and right eye images acquired simultaneously by a binocular endoscope, which includes the texture, edge and spatial parallax information of the instrument tip and surrounding tissue.

[0142] The segmentation model is used to extract the instrument end region from the field of view image and output the corresponding pixel position. The segmentation model can adopt convolutional neural network, encoder-decoder network, semantic segmentation (Seg) model or other model structure suitable for medical image segmentation to improve the stability of end-point localization.

[0143] For example, when the segmentation model is the Seg model, firstly, the left and right visual images are input into the pre-trained Seg model. The Seg model classifies each pixel in the left and right visual images, accurately identifying and segmenting all pixel regions belonging to the end effector of the surgical arm (such as an electrocautery hook, needle forceps, or bipolar electrocautery forceps), and outputs a binary mask of the same size as the input image. Then, morphological post-processing is performed on the binary mask, using closing operations to fill small holes and remove isolated noise regions to obtain a complete and coherent connected region of the end effector. Finally, the coordinate set of all pixels within this connected region is extracted and output as the pixel position of the end effector of the surgical arm, thus obtaining the pixel position of the end effector of the surgical arm in the left visual image and the pixel position of the end effector of the surgical arm in the right visual image.

[0144] S502: Based on the binocular endoscope parameters, the pixel position is reconstructed to obtain the actual spatial position of the end of the holding arm in the endoscope coordinate system.

[0145] The binocular endoscope parameters include the intrinsic and distortion parameters of the left and right cameras, as well as the extrinsic parameters between the left and right cameras. These parameters can be used to convert pixel positions into depth information and recover three-dimensional coordinates.

[0146] Before robotic laparoscopic surgery, the binocular endoscope needs to be calibrated to obtain its parameters and ensure the accuracy of image projection and stereo matching. During binocular endoscope calibration, a checkerboard or planar target is typically used to acquire multiple image pairs in different poses. Subsequently, the intrinsic parameters (such as focal length and optical center) and distortion parameters (including radial and tangential distortion) of a single camera can be solved using Zhang Zhengyou's calibration method. Then, the extrinsic parameters describing the relative rotation and translation relationship between the two cameras (i.e., the rotation matrix and translation vector of the right eye relative to the left eye) are solved using a binocular calibration algorithm. After obtaining these parameters, epipolar correction is performed on the images to ensure that the matching points of the left and right views are on the same horizontal line. By completing the entire parameter calibration process, an accurate binocular imaging model can be constructed, effectively ensuring subsequent intraoperative image projection transformation and 3D reconstruction.

[0147] In practice, firstly, using the calibrated intrinsic parameters and distortion parameters of the left and right cameras, distortion correction is performed on the pixel positions of the segmented instrument tip in the right eye image and the left eye image, respectively, to obtain distortion-free normalized coordinates. Then, based on the extrinsic parameters between the left and right cameras, and using the left eye image as the guiding view, triangulation calculations are performed on the matching normalized coordinates in the left and right views to solve for the spatial intersection of the projection rays originating from the optical centers of the left and right cameras. This allows the actual spatial position of the instrument tip pixel in the endoscope coordinate system to be reconstructed, thus achieving depth reconstruction from two-dimensional pixels to three-dimensional space.

[0148] S503: Calculate the theoretical spatial position of the end effector of the orthokeratology arm in the endoscope coordinate system using forward kinematics.

[0149] In practice, firstly, by using the real-time angles of each joint of the robotic arm, forward kinematics is solved based on the DH coordinate system and DH parameters to obtain the real-time pose of the instrument end effector in the robot base coordinate system. Then, forward kinematics is solved on the end effector arm to obtain the real-time pose of the end effector flange. This pose is then multiplied by the pre-calibrated hand-eye calibration matrix (i.e., the rigid body transformation matrix from the end effector flange to the endoscope optical center) to obtain the real-time pose of the endoscope optical center in the robot base coordinate system. Finally, the theoretical spatial position of the instrument end effector in the endoscope coordinate system is obtained using the real-time poses of the instrument end effector and the endoscope optical center in the robot base coordinate system.

[0150] This invention, by simultaneously obtaining the actual and theoretical spatial positions of the instrument's end effector when vision is normal, facilitates subsequent calculation of spatial deviations and updating of the transformation matrix, thereby improving the continuity and accuracy of instrument projection before and after visual impairment. Since the actual spatial position is derived from binocular reconstruction and the theoretical spatial position from forward kinematics calculations, the two complement each other, reducing the accumulation of errors caused by single visual recognition or single kinematics calculations, making the instrument's end effector positioning in robotic laparoscopic surgery more stable and reliable.

[0151] Optionally, in some embodiments, if there is no visual impairment during the robotic laparoscopic surgery at the current moment, the Seg mode is used to display the pixel position of the end-effector in the endoscopic field of view based on the actual spatial state.

[0152] To more clearly explain the projection method of robotic laparoscopic surgery in the above embodiments, a surgical robot system equipped with a binocular endoscope is taken as an example. This surgical robot system includes multiple robotic arms (such as a scope-holding robotic arm and a surgical instrument-holding robotic arm) mounted on the same base, a robot control host (including an image processing unit), and a display terminal. The scope-holding arm has a binocular endoscope fixed at its end, and the surgical instrument-holding arm has surgical instruments such as electrocoagulation hooks installed at its end. The left and right eye images are transmitted to the image processing unit for processing in real time and displayed in real time through the display terminal.

[0153] Based on this, Figure 6 This is a schematic diagram of the projection method for robotic laparoscopic surgery provided by the present invention, as shown below. Figure 6 As shown, it includes the following steps:

[0154] S601: Before surgery, the system is first subjected to dual-target calibration, hand-eye calibration, and multi-arm calibration.

[0155] Specifically, the binocular endoscope is first calibrated to obtain the intrinsic parameters, distortion parameters, and extrinsic parameters of the left and right cameras, which are used for subsequent image projection and stereo matching. Secondly, it is confirmed that the robot control host can accurately output the real-time pose of the end effector arm relative to the robot's base coordinate system. Thirdly, through hand-eye calibration, the endoscope mounting pose given by the robot control host is converted into the real-time pose of the endoscope's optical center relative to the robot's base coordinate system. This includes a fixed transformation from the end effector flange to the endoscope's optical center. Finally, using the fixed relationship between the end-arm and the robotic arm provided by the robot control host, the end effector pose is uniformly transformed to the endoscope coordinate system, completing the multi-arm calibration. This yields the theoretical spatial position of the end effector in the endoscope coordinate system, serving as the basis for subsequent projection and display.

[0156] The theoretical spatial location can be expressed by the following formula:

[0157]

[0158] in, This refers to the transformation matrix of the end effector arm in the endoscopic coordinate system at time t. It refers to the set of coordinates of the point cloud on the end face of the robotic arm in its own coordinate system.

[0159] S602: During the operation, the actual spatial position and the theoretical spatial position are calculated in real time.

[0160] Specifically, the endoscopic field-of-view images are processed using a segmentation model to obtain the pixel positions of the end effector of the robotic arm. Depth reconstruction is then performed on these pixel positions based on binocular endoscope parameters to obtain the actual spatial position of the end effector in the endoscopic coordinate system. Finally, the theoretical spatial position of the end effector in the endoscopic coordinate system is calculated using forward kinematics.

[0161] S603: Calculate the spatial deviation based on the actual spatial location and the theoretical spatial location.

[0162] Specifically, a transformation matrix can be determined based on the spatial deviation, which represents the transformation relationship between the actual and theoretical spatial positions of the end effector arm's distal end in the endoscopic coordinate system. For a detailed implementation process, please refer to [reference needed]. Figure 4 The embodiments shown are not described in detail here.

[0163] S604: During the surgery, the visual field images are scored in real time to determine whether there is visual impairment.

[0164] For details, please refer to the scoring process. Figure 3 The embodiments shown are not described in detail here.

[0165] If yes, enter damaged mode and execute S605. If not, enter normal mode and execute S606.

[0166] S605: Fixed spatial deviation, contour prediction.

[0167] Specifically, if a latched target transformation matrix does not currently exist, the transformation matrix from the previous time step is latched and used as the target transformation matrix; if a latched target transformation matrix exists, it is directly acquired. Then, based on the target transformation matrix and the theoretical spatial position of the instrument tip in the endoscopic coordinate system at the current time step, the target position of the instrument tip is calculated. The instrument tip is then projected based on the target position, using 0BB mode to generate a rotated bounding box, which is superimposed on the image as a semi-transparent dashed box to indicate the surgeon's approximate position and orientation of the instrument tip.

[0168] S606: Spatial deviation is continuously updated, and the outline is visualized.

[0169] Specifically, when vision is normal, the actual spatial position and theoretical spatial position of the end of the robotic arm in the endoscopic coordinate system are continuously updated and the spatial deviation between the two is calculated. The real-time segmented contour is displayed on the screen using Seg mode.

[0170] Furthermore, Figure 7 This is a schematic diagram of the display state under normal visual conditions provided by the present invention, such as... Figure 7 As shown, in the endoscopic coordinate system, when the surgical arm is holding the surgical instrument and moving it, and the visual state is normal, the display terminal can clearly present the position of the instrument tip at different times and the fine outline of the instrument tip in real time.

[0171] Figure 8 This is a schematic diagram comparing display states under different visual conditions provided by the present invention, such as... Figure 8 As shown, in the endoscopic coordinate system, the surgical arm holds surgical instruments to perform various surgical operations, and the binocular endoscope mounted on the surgical arm collects visual field images in real time and displays them through the display terminal.

[0172] Among them, when the visual state is good, the visual field image can be referenced. Figure 8 The upper right portion. Simultaneously, the Seg mode is used to project the end effector of the robotic arm, clearly displaying its fine contours. See the attached image for details. Figure 8 Middle right side.

[0173] Optionally, if the user is currently in a visually impaired state, the end effector of the robotic arm is projected using OBB mode, with a directed bounding box indicating the position of the end effector. For details, please refer to [reference needed]. Figure 8 Lower right side.

[0174] As can be seen from the above, the projection method for robotic laparoscopic surgery proposed in this invention can project and display the end effector of the robotic arm when vision is impaired, which helps to maintain the safety and continuity of robotic laparoscopic surgery.

[0175] Figure 9 This is a schematic diagram of the projection device for robotic laparoscopic surgery provided by the present invention, as shown below. Figure 9 As shown, the projection device 90 for robotic laparoscopic surgery provided in this embodiment includes:

[0176] The judgment module 901 is used to determine whether visual impairment occurs during robotic laparoscopic surgery;

[0177] The acquisition module 902 is used to acquire the target transformation matrix when it is determined that there is visual impairment in the robotic laparoscopic surgery. The target transformation matrix is ​​the transformation matrix corresponding to the last unimpaired moment before the occurrence of visual impairment of the end of the robotic arm instrument. It is used to represent the transformation relationship between the actual spatial position and the theoretical spatial position of the end of the robotic arm instrument in the endoscope coordinate system.

[0178] The calculation module 903 is used to calculate the target position of the end of the arm based on the target transformation matrix and the theoretical spatial position of the end of the arm in the endoscope coordinate system at the current moment.

[0179] Projection module 904 is used to project the end effector of the robotic arm based on the target position.

[0180] In one possible implementation, the judgment module 901 is used to acquire the visual field image of the robotic laparoscopic surgery; input the visual field image into the segmentation model, the lighting condition assessment model, and the visual noise model respectively, and obtain the score output by each model; wherein, the score output by the segmentation model is positively correlated with the confidence of the pixel position of the segmented end of the robotic arm instrument; the score output by the lighting condition assessment model is positively correlated with the illumination quality of the visual field image; the score output by the visual noise model is positively correlated with the visual purity of the visual field image; and based on the scores output by the segmentation model, the lighting condition assessment model, and the visual noise model respectively, it is determined whether there is visual impairment during the robotic laparoscopic surgery.

[0181] In one possible implementation, the judgment module 901 is further used to perform a weighted summation of the scores output by the segmentation model, the light state assessment model, and the visual noise model to obtain a visual score for the robotic laparoscopic surgery; if the visual score is greater than a preset score, it is determined that there is no visual impairment in the robotic laparoscopic surgery; if the visual score is less than or equal to the preset score, it is determined that there is visual impairment in the robotic laparoscopic surgery.

[0182] In one possible implementation, the acquisition module 902 is used to lock the transformation matrix determined in the previous moment as the target transformation matrix if there is visual impairment during the robotic laparoscopic surgery at the current moment and there was no visual impairment at the previous moment.

[0183] In one possible implementation, the projection module 904 is used to project the end effector of the robotic arm in OBB mode based on the target position.

[0184] In one possible implementation, the projection device 90 for robotic laparoscopic surgery further includes:

[0185] The determination module 905 is used to calculate the spatial deviation between the actual and theoretical spatial positions of the end effector of the robotic laparoscopic surgery arm in the endoscopic coordinate system if there is no visual impairment at the current moment; and to determine the transformation matrix based on the spatial deviation.

[0186] In one possible implementation, the determining module 905 is further configured to process the visual field image of the robotic laparoscopic surgery using a segmentation model to obtain the pixel position of the end effector of the robotic arm; wherein the visual field image is a binocular image; depth reconstruction is performed on the pixel position according to the binocular endoscope parameters to obtain the actual spatial position of the end effector of the robotic arm in the endoscope coordinate system; and the theoretical spatial position of the end effector of the robotic arm in the endoscope coordinate system is calculated using forward kinematics.

[0187] The projection device for robotic laparoscopic surgery provided in this embodiment can execute the method provided in the above-described method embodiment. Its implementation principle and technical effect are similar, and will not be described in detail here.

[0188] Figure 10 This is a schematic diagram of the projection device for robotic laparoscopic surgery provided by the present invention. Figure 10 As shown, the robotic laparoscopic surgery projection device 100 provided in this embodiment includes at least one processor 1001 and a memory 1002. Optionally, the robotic laparoscopic surgery projection device 100 further includes a communication component 1003. The processor 1001, memory 1002, and communication component 1003 are connected via a bus 1004.

[0189] In a specific implementation, at least one processor 1001 executes computer execution instructions stored in memory 1002, causing at least one processor 1001 to perform the above-described method.

[0190] The specific implementation process of processor 1001 can be found in the above method embodiments, and its implementation principle and technical effect are similar. It will not be repeated here.

[0191] In the above embodiments, it should be understood that the processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), etc. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in this invention can be directly implemented by a hardware processor, or implemented by a combination of hardware and software modules within the processor.

[0192] The memory may include random access memory (RAM) and may also include non-volatile memory (NVM), such as at least one disk storage device.

[0193] The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of illustration, the buses shown in the accompanying drawings are not limited to a single bus or a single type of bus.

[0194] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the above-described method.

[0195] The present invention also provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, implement the above-described method.

[0196] The aforementioned readable storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as Static Random-Access Memory (SRAM), Electrically Erasable Programmable Read-Only Memory (EEPROM), Erasable Programmable Read-Only Memory (EPROM), Programmable Read-Only Memory (PROM), Read-Only Memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk. The readable storage medium can be any available medium accessible to a general-purpose or special-purpose computer.

[0197] An exemplary readable storage medium is coupled to a processor, enabling the processor to read information from and write information to the readable storage medium. Of course, the readable storage medium can also be a component of the processor. The processor and the readable storage medium can reside in an application-specific integrated circuit (ASIC). Alternatively, the processor and the readable storage medium can exist as discrete components in the device.

[0198] The division of units is merely a logical functional division; in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be indirect coupling or communication connection through some interfaces, devices, or units, and may be electrical, mechanical, or other forms.

[0199] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0200] In addition, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0201] If a function is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, ROM, RAM, magnetic disks, or optical disks.

[0202] Those skilled in the art will understand that all or part of the steps of the above-described method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When executed, the program performs the steps of the above-described method embodiments; and the aforementioned storage medium includes various media capable of storing program code, such as ROM, RAM, magnetic disks, or optical disks.

[0203] Finally, it should be noted that other embodiments of the invention will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This invention is intended to cover any variations, uses, or adaptations of the invention that follow the general principles of the invention and include common knowledge or customary techniques in the art not disclosed herein, and is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of the invention is limited only by the appended claims.

Claims

1. A projection method for robotic laparoscopic surgery, characterized in that, include: Determine if visual impairment occurs during robotic laparoscopic surgery; If it exists, obtain the target transformation matrix; The target transformation matrix is ​​the transformation matrix corresponding to the last undamaged moment of the end of the robotic arm before visual impairment occurs. It is used to represent the transformation relationship between the actual spatial position and the theoretical spatial position of the end of the robotic arm in the endoscopic coordinate system. Based on the target transformation matrix and the theoretical spatial position of the end of the holding arm in the endoscope coordinate system at the current moment, calculate the target position of the end of the holding arm. Based on the target position, the end of the holding arm is projected.

2. The method according to claim 1, characterized in that, The determination of whether visual impairment occurs during robotic laparoscopic surgery includes: Acquire the field of view images of the robotic laparoscopic surgery; The visual field image is input into a segmentation model, a lighting condition assessment model, and a visual noise model, respectively, and a score is obtained from each model. The score output by the segmentation model is positively correlated with the confidence level of the pixel position at the end of the segmented robotic arm. The score output by the lighting condition assessment model is positively correlated with the illumination quality of the visual field image. The score output by the visual noise model is positively correlated with the visual purity of the visual field image. Based on the scores output by the segmentation model, the light state assessment model, and the visual noise model, it is determined whether visual impairment exists during the robotic laparoscopic surgery.

3. The method according to claim 2, characterized in that, The step of determining whether visual impairment exists during robotic laparoscopic surgery based on the scores output by the segmentation model, the light state assessment model, and the visual noise model includes: The scores output by the segmentation model, the light state evaluation model, and the visual noise model are weighted and summed to obtain the visual score of the robotic laparoscopic surgery. If the visual score is greater than the preset score, it is determined that there is no visual impairment during the robotic laparoscopic surgery; If the visual score is less than or equal to the preset score, then visual impairment is determined to exist during the robotic laparoscopic surgery.

4. The method according to any one of claims 1-3, characterized in that, The method further includes: If there is no visual impairment during the robotic laparoscopic surgery at the current moment, then calculate the spatial deviation between the actual spatial position and the theoretical spatial position of the end effector arm in the endoscope coordinate system; The transformation matrix is ​​determined based on the spatial deviation.

5. The method according to any one of claims 1-3, characterized in that, The method further includes: If the robotic laparoscopic surgery presents with visual impairment at the current moment, but not at the previous moment, then the transformation matrix determined at the previous moment is locked as the target transformation matrix.

6. The method according to claim 4, characterized in that, Before calculating the spatial deviation between the actual and theoretical spatial positions of the end effector arm in the endoscopic coordinate system if there is no visual impairment during the robotic laparoscopic surgery at the current moment, the method further includes: The visual field image of the robotic laparoscopic surgery is processed by a segmentation model to obtain the pixel position of the end effector of the robotic arm; wherein, the visual field image is a binocular image; Based on the binocular endoscope parameters, the pixel positions are reconstructed to obtain the actual spatial position of the end of the holding arm in the endoscope coordinate system. The theoretical spatial position of the end effector of the holding arm in the endoscope coordinate system is calculated using forward kinematics.

7. The method according to any one of claims 1-3 and 6, characterized in that, The projection of the distal end of the robotic arm based on the target position includes: Based on the target position, the end effector of the robotic arm is projected in the Rotating Bounding Box (OBB) mode.

8. A projection device for robotic laparoscopic surgery, characterized in that, include: The judgment module is used to determine whether there is visual impairment during robotic laparoscopic surgery; The acquisition module is used to acquire the target transformation matrix when visual impairment is determined to occur during robotic laparoscopic surgery; The target transformation matrix is ​​the transformation matrix corresponding to the last undamaged moment of the end of the robotic arm before visual impairment occurs. It is used to represent the transformation relationship between the actual spatial position and the theoretical spatial position of the end of the robotic arm in the endoscopic coordinate system. The calculation module is used to calculate the target position of the end of the robotic arm based on the target transformation matrix and the theoretical spatial position of the end of the robotic arm in the endoscope coordinate system at the current moment. The projection module is used to project the end effector of the robotic arm based on the target position.

9. A projection device for robotic laparoscopic surgery, characterized in that, include: Memory and processor; The memory stores computer-executed instructions; The processor executes computer execution instructions stored in the memory, causing the processor to perform the method as described in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, are used to implement the method as described in any one of claims 1-7.

Images