Image display method of surgical robot system, surgical robot system and related device
By generating and overlaying virtual images of surgical instruments in a surgical robot system, the problem of difficult observation of the position and orientation of surgical instruments is solved, thereby improving the accuracy and safety of surgery.
Patent Information
- Application Number
- CN202411347333.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-25
- Publication Date
- 2026-03-27
AI Technical Summary
In surgical robot systems, the changes in the position of surgical instruments are difficult to observe directly through the surgical screen, especially when the instruments are outside the field of view of the image acquisition device or are obstructed, which leads to high operational difficulty and low safety.
The system acquires the positional information of surgical instruments and generates virtual images that are overlaid on the surgical screen. The processor generates virtual images of surgical instruments to indicate their positional information.
It improves the accuracy and safety of surgical procedures, enabling doctors to simultaneously observe changes in posture while operating surgical instruments.
Smart Images
Figure CN121730987A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of image display, and more particularly to an image display method for a surgical robot system, a surgical robot system, an electronic device, and a computer-readable storage medium. Background Technology
[0002] In surgical scenarios utilizing robotic surgical systems, the surgical field of view is typically captured by an image acquisition device inside the human body and presented to the surgeon. The surgeon can then control the surgical instruments inserted into the body based on this image. However, when robotically driven surgical instruments are inside the body, it is often difficult for the surgeon to visually observe changes in their position and orientation, especially when the instruments or part of them are outside the image acquisition device's field of view or obstructed by other objects within that field of view. This makes observing changes in instrument position and orientation even more challenging and reduces the safety of surgical procedures. Summary of the Invention
[0003] To address the aforementioned technical problems, this application provides an image display method for a surgical robot system, a surgical robot system, an electronic device, and a computer-readable storage medium. The technical solution is as follows:
[0004] According to a first aspect of this application, an image display method for a surgical robot system is provided, the method comprising:
[0005] The surgical images captured by the image acquisition device are acquired and displayed on the display device.
[0006] Obtain the position and orientation information of the surgical instruments;
[0007] A virtual image of the surgical instrument is generated based on the positional information of the surgical instrument, and the virtual image of the surgical instrument indicates the positional information of the surgical instrument.
[0008] The virtual image of the surgical instrument is overlaid on the surgical screen.
[0009] According to a second aspect of this application, a surgical robot system is provided, the system comprising:
[0010] Image acquisition equipment to capture surgical images;
[0011] The display device displays the surgical scene;
[0012] Surgical instruments;
[0013] The processor is configured to acquire the pose information of the surgical instrument, generate a virtual image of the surgical instrument based on the pose information, and overlay the virtual image of the surgical instrument onto the surgical screen, wherein the virtual image of the surgical instrument indicates the pose information of the surgical instrument.
[0014] According to a third aspect of this application, an electronic device is provided, the electronic device comprising:
[0015] processor;
[0016] Memory used to store processor-executable instructions;
[0017] The processor is configured to implement the method as described in the first aspect.
[0018] According to a fourth aspect of this application, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the steps of the method as described in the first aspect.
[0019] The technical solution provided in this application acquires the positional information of surgical instruments in a surgical robot system, and generates and overlays virtual images that indicate the positional information of surgical instruments on the surgical screen based on the positional information. This allows doctors to observe the positional changes of surgical instruments simultaneously on the surgical screen while operating them, thereby performing surgery more efficiently and improving the accuracy and safety of surgery using surgical instruments.
[0020] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and do not limit this application. Attached Figure Description
[0021] To more clearly illustrate the technical solutions in the embodiments or related technologies of this application, the drawings used in the description of the embodiments or related technologies will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this application. For those skilled in the art, other drawings can be obtained based on these drawings.
[0022] Figure 1A This is a schematic diagram of the structure of a surgical robot system according to an embodiment of this application;
[0023] Figure 1B This is a schematic diagram of the structure of a surgical robot system according to an embodiment of this application;
[0024] Figure 1C This is a schematic diagram of a surgical scene according to an embodiment of this application;
[0025] Figure 2This is a schematic diagram of the structure of a surgical robot system according to an embodiment of this application;
[0026] Figure 3 This is a schematic diagram illustrating the overlay of virtual images onto a surgical screen according to an embodiment of this application;
[0027] Figure 4 This is a schematic diagram illustrating the overlay of virtual images onto a surgical screen according to another embodiment of this application;
[0028] Figure 5 This is a schematic diagram of the rim structure of a cup with an LED halo in related technologies;
[0029] Figure 6 This is a schematic diagram illustrating the overlay of virtual images onto a surgical screen according to another embodiment of this application;
[0030] Figure 7 This is a flowchart illustrating an image display method for a surgical robot system according to this application;
[0031] Figure 8 This is a schematic diagram of the structure of an electronic device according to an embodiment of this application. Detailed Implementation
[0032] To enable those skilled in the art to better understand the technical solutions in this application, the technical solutions in the embodiments of this application will be described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art should fall within the scope of protection of this application.
[0033] Figures 1A-1B A schematic diagram of a surgical robot system A is shown. See also Figures 1A-1B The surgical robot system A may include at least a console 1 and a robotic arm system 2. The console 1 may include a display device for showing the surgical instrument environment (such as the surgical screen) and a processor. The display device has an observation window (also called a stereoscopic display) for the surgeon to observe, and the processor is used for information processing and program execution. Furthermore, the console 1 may also include some or all of the following: an operation control mechanism, armrests, and control switches (not shown in the figure). The actions of the operation control mechanism correspond to the actions of the surgical instruments; by controlling the operation control mechanism, the position of the surgical instruments can be adjusted. The armrests are used to support the surgeon's arms. The control switches are conveniently accessible by hand or foot for touch or pressing, used for various functional operations and human-machine interaction.
[0034] The robotic arm system 2 includes several robotic arms 21, each comprising several connecting arms. Adjacent connecting arms move relative to each other with specific degrees of freedom, allowing the end effector of the robotic arm 21 to achieve multiple degrees of freedom (e.g., 7 degrees of freedom, depending on the instrument). An instrument actuator, surgical instrument, or image acquisition device is mounted on the end effector of the robotic arm 21. The image acquisition device can be an endoscope (e.g., a 3D endoscope) and is detachably mounted on the instrument actuator. The images acquired by the image acquisition device (e.g., surgical footage) can be displayed on the display device of the control console 1. It is understood that the image acquisition device disclosed herein can also be of other types, such as a laparoscope.
[0035] Several robotic arms 21 can be mounted on one or more bases, and at least one robotic arm can be mounted on each base. For ease of description, each robotic arm on each base can be defined as a robotic arm subsystem. For example, in Figure 1B In the example shown, the robotic arm system 2 has two bases located next to the patient C on bed B. The two bases respectively house the robotic arm subsystems 22 and 23. The robotic arm subsystem 22 can hold an image acquisition device, while the robotic arm subsystem 23 can hold surgical instruments, such as a uterine manipulator. Of course, the implementation of this application is not limited to this.
[0036] Optionally, the surgical robot system A described above may further include an image processing system 3. The image processing system 3 includes a display device (e.g., a screen) that can display images acquired by the image acquisition device. The image acquisition device can acquire images (such as surgical scenes) and send them to the image processing system 3, so that the display device included in the image processing system 3 can display the acquired images. The image acquisition device can send the acquired images directly or via the console 1 to the image processing system 3. The image acquisition device can perform image processing (e.g., decoding) on the acquired images using its own processor or via the processor of the image processing system 3. The processed images can also be sent to other image processing devices for further image processing (e.g., noise reduction, contrast enhancement, sharpness improvement, etc.).
[0037] In related technologies, when using surgical robot systems for surgery, the surgical scene is typically captured by an image acquisition device inside the human body, displaying the surgical images within its field of view to the surgeon. The surgeon can then control the surgical instruments inside the body based on this image. However, when surgical instruments driven by the surgical robot enter the body, the surgeon often finds it difficult to visually observe changes in their position and orientation, especially when the instruments or part of them are outside the field of view of the image acquisition device, or obstructed by other objects within the device's field of view. This makes it even more difficult to observe changes in the instruments' position and orientation, resulting in greater difficulty and lower safety when performing surgery using surgical instruments. A specific scenario will illustrate this below. Please see [link to documentation]. Figure 1B In this specific scenario, the surgical instrument can be a uterine manipulator 232, and the image acquisition device 222 can be an endoscope or laparoscope, such as a three-dimensional endoscope. The robotic arm 21 can include a robotic arm subsystem 22 and a robotic arm subsystem 23, wherein the robotic arm subsystem 22 is used to hold the endoscope, and the robotic arm subsystem 23 is used to hold the uterine manipulator. Figure 1B As shown, the uterine manipulator 232 can enter the human body through natural passages (such as the vagina) to precisely manipulate the uterus, such as adjusting its tension and position, so that the surgeon can smoothly remove the uterus using cutting instruments based on the surgical images captured by laparoscopic or other imaging equipment. This layout allows the surgeon to directly control the uterine manipulator 232, thereby reducing the need for bedside assistants and improving surgical efficiency and accuracy.
[0038] like Figure 1C As shown, as an example, the uterine manipulator 232, which enters the human body through a natural passage (such as the vagina 5), may have a tilting joint 2321 at its front end (the end near the uterus 4). This tilting joint 2321 can tilt within a certain angle range (e.g., it can move within a range of -50 degrees to +90 degrees). The tilting joint 2321 may include a uterine cup 23211 (used to expose the vaginal fornix boundary line 6 in the surgical image captured by the image acquisition device 222, i.e., as a baseline for removing the uterus 4) and a uterine manipulator head 23212 (before the uterine cup 23211). The rear end 2322 (backend, BKD) of the uterine manipulator 232, i.e., the part of the uterine manipulator 232 that can be connected to the manipulator assembly of the robotic arm subsystem 23, can interface with the instrument drive module (Motor Pack, MPK) of the manipulator assembly, and the uterine manipulator 232 is driven by the motor of the MPK.
[0039] It is worth noting that the above description of a scenario for performing surgery using a surgical robot system is merely an illustrative example. In practical applications, there may be other types of surgeries performed using surgical robot systems, and no specific limitations are made for these scenarios.
[0040] In the aforementioned specific scenarios, surgeons often find it difficult to visually observe changes in the position and orientation of surgical instruments through the surgical screen, resulting in greater difficulty and lower safety when performing surgery using surgical instruments.
[0041] Based on this, see Figure 2 As shown, this application displays the surgical scene captured by the image acquisition device 25 through the display device 12, that is, displays the surgical scene on the display device 12, obtains the pose information of the surgical instrument 24 through the processor 11, generates a virtual image of the surgical instrument 24 based on the pose information of the surgical instrument 24, and superimposes the virtual image of the surgical instrument 24 onto the surgical scene, wherein the virtual image of the surgical instrument 24 indicates the pose information of the surgical instrument 24.
[0042] This application acquires the positional information of surgical instruments in a surgical robot system, and generates and overlays virtual images that indicate the positional information of the surgical instruments on the surgical screen based on the positional information. This allows doctors to observe the positional changes of the surgical instruments simultaneously on the surgical screen while operating the instruments, thereby performing surgery more efficiently and improving the accuracy and safety of surgery using surgical instruments.
[0043] As an example, both the processor 11 and the display device 12 can be configured as follows: Figure 1A The image acquisition device 25 on the console 1 of the surgical robot system A shown can be configured as follows: Figure 1A The surgical instrument 24 may be configured on one robotic arm subsystem of the robotic arm system 2. Of course, the implementation of this application is not limited to this.
[0044] The image acquisition device 25 can acquire surgical images in various ways. For example, in a hysterectomy, the image acquisition device 25 can be a laparoscope. The laparoscope can enter the body through an opening in the abdomen to acquire images of the surgical site (such as the uterine lesion area). These images can then be displayed on the display device 12. As another example, in other surgical scenarios, the image acquisition device can be an endoscope. The endoscope can enter the body through an opening or through a natural passage (such as the esophagus through the mouth) to acquire images of the surgical site. Therefore, the specific method by which the image acquisition device acquires surgical images is not limited.
[0045] The pose information of the surgical instrument 24 can be implemented in various ways. As an example, the pose information of the surgical instrument 24 may include the pose information of the surgical instrument 24 itself; as another example, the pose information of the surgical instrument 24 may include the relative pose information between the surgical instrument 24 and objects outside the surgical screen acquired by the image acquisition device 25; as another example, the pose information of the surgical instrument may include the relative pose information between the surgical instrument 24 and objects within the surgical screen acquired by the image acquisition device 25; as yet another example, the pose information of the surgical instrument 24 may include at least one of the following: the pose information of the surgical instrument 24 itself, the relative pose information between the surgical instrument 24 and objects outside the surgical screen, or the relative pose information between the surgical instrument 24 and objects within the surgical screen, such as including one, two, or all of them.
[0046] As an example, when the image acquisition device 25 acquires a surgical image, at least a portion of the surgical instrument 24 may be invisible in the surgical image. "Invisible" can include various situations. For example, at least a portion of the surgical instrument 24 being invisible in the surgical image could mean that although at least a portion of the surgical instrument 24 is within the field of view of the image acquisition device 25, it is obscured by other objects within that field of view and therefore invisible in the surgical image. As another example, at least a portion of the surgical instrument 24 being invisible in the surgical image could also mean that at least a portion of the surgical instrument 24 is not within the field of view of the image acquisition device 25 and is therefore invisible in the surgical image. For instance, taking the surgical instrument 24 as a uterine manipulator, at least a portion of the surgical instrument 24 being invisible in the surgical image could mean that the entire uterine manipulator is invisible in the surgical image, or it could mean that a part of the uterine manipulator is invisible in the surgical image, such as the uterine cup portion of the uterine manipulator being invisible in the surgical image.
[0047] It is worth noting that the above description of the situation where at least a part of the surgical view of the surgical instrument 24 is not visible is only an exemplary demonstration. In actual application, other situations may exist, and no specific limitation is made here.
[0048] As another example, when the image acquisition device 25 acquires the surgical image, the surgical instrument 24 may also be visible in the surgical image. Therefore, there is no specific limitation on whether the surgical instrument 24 is visible in the surgical image.
[0049] As an example, if the surgical instrument 24 is a uterine manipulator, the surgical robot system may further include a first manipulator robot, which may include a first manipulator assembly, allowing the uterine manipulator to move under the actuation of the first manipulator assembly. It is understood that the first manipulator robot may correspond to... Figure 1BIn the robotic arm subsystem 23, the first manipulator component may correspond to the robotic arm 231 of the robotic arm subsystem 23.
[0050] As an example, the surgical robot system may also include a second manipulator robot, which may include at least one second manipulator assembly that drives the movement of a surgical tool, which includes at least one of an image acquisition device 25 or a surgical instrument, other than surgical instrument 24. It is understood that the second manipulator robot may correspond to... Figure 1B In the robotic arm subsystem 22, the second manipulator assembly may correspond to the robotic arm 221 of the robotic arm subsystem 22.
[0051] As an example, the first manipulator robot can be separately configured with the second manipulator robot. It is understood that the first manipulator robot can be separately configured with the second manipulator robot, and can correspond to... Figure 1B The robotic arm system 2 shown has two bases distributed next to the patient, and the two bases respectively have robotic arm subsystems 22 and 23 included in the robotic arm system 2.
[0052] There are multiple ways for the processor 11 to acquire the posture information of the surgical instrument 24. As an example, if the surgical robot system includes the aforementioned first manipulator robot and the first manipulator robot includes the aforementioned first manipulator assembly, then the first manipulator assembly may include several connecting arms, wherein adjacent connecting arms are connected by joints, and the joints can drive the connecting arms to move under the drive of joint motors. The processor 11 can acquire the operating parameters of the joint motors and determine the posture information of the surgical instrument 24 based on the operating parameters. It is worth noting that the above description of the methods for acquiring the posture information of the surgical instrument 24 is only an exemplary demonstration. In practical applications, other acquisition methods may exist, and no specific limitation is made here.
[0053] It is understandable that different specific implementations of the pose information of the surgical instrument 24 obtained by the processor 11 can affect the specific implementation of the virtual image of the surgical instrument 24 generated by the processor 11. The following is an exemplary description of one way in which the processor 11 generates a virtual image that can indicate the pose information of the surgical instrument 24:
[0054] As an example, when the pose information of the surgical instrument 24 includes the pose information of the surgical instrument 24 itself, the pose information of the surgical instrument 24 itself may include the overall tilt angle and the local tilt angle of the surgical instrument 24. The local tilt angle may be a movable part of the surgical instrument 24. Specifically, the processor 11 may be used to generate a virtual image of the surgical instrument 24 that changes with the change of the overall tilt angle and / or the local tilt angle based on the overall tilt angle and the local tilt angle of the surgical instrument 24. For example, if one of the overall tilt angle and the local tilt angle changes, a virtual image that changes accordingly is generated; if the overall tilt angle and the local tilt angle change simultaneously, a virtual image that changes with both is generated.
[0055] The aforementioned overall and local tilt angles can be implemented in various ways. As an example, surgical instrument 24 is used. Figure 1C Taking the uterine lifter 232 as an example, the overall tilt angle can describe the overall tilt state of the uterine lifter 232, while the local part of the uterine lifter 232 can refer to the tilting joint 2321 at the front end of the uterine lifter (e.g., Figure 1C As shown, the lifting joint 2321 is located at the end near the uterus 4 (including the uterine lifting cup 23211 and the uterine manipulator head 23212). The local lifting angle can describe the lifting state of the lifting joint 2321 at the front end of the uterine manipulator. It is worth noting that the above description of the specific implementation of the overall lifting angle and the local lifting angle is only an exemplary demonstration. In practical applications, other specific implementations may exist, which are not limited here.
[0056] Please see Figure 3 The following is an example of an overlay method for virtual images that change with the position and pose information of the surgical instrument 24 on the surgical screen:
[0057] like Figure 3 As shown, taking surgical instrument 24 as an example of a uterine lifter, the generated virtual image that changes with the overall and / or local tilting angle of the surgical instrument can be superimposed on the surgical screen captured by image acquisition device 25 (i.e., as shown). Figure 3The endoscopic field of view shown is 7). As an example, the virtual image can be superimposed on the surgical screen in the form of a semi-transparent 3D image 71. The semi-transparency can minimize the impact on the doctor's observation of the original surgical screen. As another example, the virtual image can be a side view of the uterine manipulator. From the side view, the overall tilt angle and / or local tilt angle changes of the uterine manipulator can be better observed. By observing the 3D image of the uterine manipulator from the side view, it is possible to intuitively observe whether the uterine manipulator has reached the limit of the joint tilt angle, thereby further improving the safety and accuracy of the operation. Furthermore, when the uterine manipulator cannot be driven before reaching the limit, the doctor can also judge the operation of the uterine manipulator through the 3D image of the uterine manipulator, thereby promptly identifying problems and taking corresponding measures.
[0058] The following is an example of another method for generating virtual images that can indicate the pose information of surgical instruments:
[0059] As an example, when the pose information of the surgical instrument 24 includes the relative pose information between the surgical instrument 24 and an object outside the surgical screen, the relative pose information between the surgical instrument 24 and the object outside the surgical screen may include the overall yaw angle between the surgical instrument 24 and the object outside the surgical screen. Specifically, the processor 11 may be used to generate a virtual image of the surgical instrument 24 that changes with the overall yaw angle based on the overall yaw angle between the surgical instrument 24 and the object outside the surgical screen.
[0060] To visually demonstrate the relative pose between the surgical instrument 24 and an object outside the surgical screen, a virtual image of the object outside the surgical screen can also be displayed on the surgical screen. As an example, the processor 11 can also first generate a virtual image of the object outside the surgical screen, and then, based on the overall yaw angle between the surgical instrument 24 and the object outside the surgical screen, combine and superimpose the virtual image of the surgical instrument 24, which changes with the overall yaw angle, and the virtual image of the object outside the surgical screen on the surgical screen to indicate the yaw information between the surgical instrument 24 and the object outside the surgical screen.
[0061] Considering that objects outside the surgical field may undergo pose changes, and these changes will affect their relative pose relationship with the surgical instruments 24, as an example, the processor 11 can first acquire the pose information of the objects outside the surgical field, and then generate a virtual image of the objects based on this information. The virtual image of the objects outside the surgical field can also indicate their pose information. Therefore, when an object outside the surgical field undergoes a pose change, this change can also be displayed in its virtual image.
[0062] The object outside the surgical view can be implemented in various ways. Taking the surgical instrument 24 as a uterine manipulator 232 as an example, as an example, the object outside the surgical view can be the patient's lower limb. The processor 11 can be used to generate a virtual image of the patient's lower limb based on the midline of the lower limb, where the midline of the lower limb is parallel to the long side of the operating table. For example, such as... Figure 1B As shown, the midline of the patient's lower limb is parallel to the long side of the operating table (bed). In other surgical scenarios, as another example, the object outside the surgical view can also be other concrete implementations, such as the patient's upper limb or other patient parts. By displaying the relative positional relationship between the surgical instrument 24 and the corresponding part of the patient on the surgical screen, the doctor can intuitively observe whether the surgical instrument is about to approach the patient's body part, such as observing whether the uterine manipulator is about to contact the patient's leg, and thus make corresponding positional adjustments, further improving the safety and accuracy of the surgery.
[0063] Please see Figure 4 The following is an exemplary description of a method for overlaying virtual images onto the surgical screen, where the overall yaw angle between the surgical instrument 24 and objects outside the surgical screen changes:
[0064] like Figure 4 As shown, surgical instrument 24 is used as uterine manipulator 232, and the object outside the surgical view is the patient's lower limb (e.g., Figure 4 Taking the human leg shown as an example, the virtual image of the uterine lifter 232, which changes with the overall yaw angle between the uterine lifter 232 and the patient's lower limbs, and the virtual image of the patient's lower limbs, can be combined and superimposed on the surgical screen captured by the image acquisition device 25 (i.e., Figure 4 The endoscopic field of view shown is 7). As an example, the virtual image can be superimposed on the surgical screen in the form of a semi-transparent 3D image. The semi-transparency can minimize the impact on the doctor's observation of the original surgical screen. As another example, the virtual image can be a top view 72 of the uterine manipulator and a top view 73 of the patient's lower limbs. The top view allows for better observation of the change in the overall yaw angle between the uterine manipulator and the patient's lower limbs. By observing the 3D image of the uterine manipulator from the top view, it is possible to intuitively observe whether the uterine manipulator has reached the yaw angle that will contact the patient's legs. For example, if the angle between the patient's legs is 90 degrees, the overall yaw angle of the uterine manipulator needs to be less than 45 degrees. When the overall yaw angle of the uterine manipulator needs to be greater than or equal to 45 degrees, the position of the uterine manipulator needs to be adjusted in a timely manner to prevent the uterine manipulator from deviating beyond the safe position.
[0065] The following is an example of another method for generating virtual images that can indicate the pose information of surgical instruments 24:
[0066] As an example, when the pose information of the surgical instrument 24 includes the relative pose information between the surgical instrument 24 and the object in the surgical screen, the relative pose information between the surgical instrument 24 and the object in the surgical screen may include the position and / or angle between a specified part of the surgical instrument 24 and the object in the surgical screen. Specifically, the processor 11 may first generate a virtual image of the specified part of the surgical instrument 24 that changes with the position and / or angle between the specified part of the surgical instrument 24 and the object in the surgical screen, based on the position and / or angle between the specified part of the surgical instrument 24 and the object in the surgical screen, and then overlay the virtual image of the specified part of the surgical instrument 24 onto the surgical screen based on the position and / or angle between the specified part of the surgical instrument 24 and the object in the surgical screen.
[0067] The designated component of the surgical instrument 24 can be implemented in various ways. As an example, the surgical instrument can be a uterine manipulator; as another example, the designated component of the surgical instrument can be the rim of the uterine manipulator cup.
[0068] The objects in the surgical view can also have multiple specific implementations. As an example, the object in the surgical view can be the uterus.
[0069] Considering that in a hysterectomy scenario, the surgical images captured by the image acquisition device 25 require the rim of the uterine cup to be exposed, such as... Figure 1C The vaginal fornix boundary line 6 is shown to allow the surgeon to perform precise resection of the uterus 4 based on the vaginal fornix boundary line 6 in the surgical view. Please refer to [link to surgical procedure]. Figure 5 In related technologies, a partial LED halo 81 is set on the rim 8 of the uterine cup to create an LED halo in the vaginal fornix boundary line in the surgical view, which guides the doctor to cut the uterus along the halo. However, under the complex intra-abdominal light source, the uterine cup with LED halo has poor exposure effect on the vaginal fornix boundary line under the influence of other light sources, and the boundary of the LED halo is blurred, which is not conducive to the doctor's precise cutting. In addition, the uterine cup with LED halo requires additional light source equipment, which increases the cost and surgical setup time.
[0070] To address this issue, as an example, when the surgical instrument 24 is a uterine manipulator, and a designated component of the surgical instrument 24 is the rim of the uterine manipulator's cup, the positional information of the surgical instrument 24 can include the position and / or angle of the rim of the uterine manipulator's cup relative to the uterus in the surgical view. Specifically, the processor 11 can first generate a virtual image of the rim of the uterine manipulator's cup that changes with the position and / or angle of the rim relative to the uterus in the surgical view, based on the position and / or angle of the rim of the uterine manipulator's cup relative to the uterus in the surgical view, and then overlay the virtual image of the rim onto the surgical view based on the position and / or angle of the rim relative to the uterus in the surgical view. By overlaying the virtual image of the rim of the uterine manipulator onto the surgical view, a direct and accurate anatomical reference can be provided to the doctor, allowing the doctor to intuitively observe the correspondence between the rim of the uterine manipulator's cup and actual anatomical structures (such as the boundary between the uterus and the vaginal fornix), further improving the accuracy of the surgery.
[0071] like Figure 6 As shown, taking surgical instrument 24 as a uterine lifter, and the designated part of surgical instrument 24 as the rim of the uterine lifter cup as an example, the virtual image of the rim of the cup, generated by processor 11, which changes with the position and / or angle of the rim relative to the uterus in the surgical image, can be combined and superimposed on the surgical image captured by image acquisition device 25 (i.e., Figure 6 The endoscopic view shown is 7). As an example, the virtual image can be overlaid on the surgical screen as a semi-transparent 3D model. The semi-transparency minimizes the impact on the doctor's observation of the original surgical screen. By overlaying the virtual image of the rim of the uterine cup onto the surgical screen, the doctor can clearly identify the boundary line of the vaginal fornix using the virtual image of the cup rim, allowing for precise cutting operations based on the boundary line of the vaginal fornix.
[0072] As an example, of the three virtual images described above (a virtual image of the surgical instrument 24 that changes with the overall tilt angle and / or local tilt angle of the surgical instrument 24; a virtual image of the combination of the surgical instrument 24 and an object outside the surgical screen that changes with the overall yaw angle between the surgical instrument 24 and an object outside the surgical screen; and a virtual image of a designated part of the surgical instrument 24 that changes with the position and / or angle between a designated part of the surgical instrument 24 and an object within the surgical screen), at least one of the three virtual images can be displayed on the surgical screen, for example, one, two, or all of them can be displayed simultaneously. The specific display method of the three virtual images is not limited.
[0073] Considering that doctors need to promptly adjust any abnormal positions of surgical instruments to avoid affecting the surgery, it is necessary to consider which of the three virtual images to display on the surgical screen. The following provides examples of different display methods:
[0074] Regarding the virtual image of the surgical instrument 24 that changes with the overall tilt angle and / or local tilt angle, as an example, if the overall tilt angle of the surgical instrument 24 is greater than or equal to a first preset angle, and / or, the local tilt angle of the surgical instrument 24 is greater than or equal to a second preset angle, then the virtual image of the surgical instrument 24 that changes with the overall tilt angle and / or the local tilt angle will be superimposed on the surgical screen. As another example, when the overall tilt angle is less than the first preset angle and the local tilt angle is less than the second preset angle, the virtual image of the surgical instrument 24 that changes with the overall tilt angle and / or the local tilt angle may not be displayed on the surgical screen. That is, when both the overall and partial tilting angles of the surgical instrument 24 are normal (e.g., not reaching the limit joint angle), the corresponding virtual image may not be displayed on the surgical screen. Only when the overall or partial tilting angle of the surgical instrument 24 becomes abnormal (e.g., reaching the limit joint angle) will the corresponding virtual image be displayed on the surgical screen, allowing the surgeon to promptly notice any abnormal posture of the surgical instrument 24. As another example, the virtual image displayed on the surgical screen can flash at a preset frequency. Alternatively, the entire virtual image of the surgical instrument 24 may flash at a preset frequency, or only a portion of the virtual image of the surgical instrument 24 may flash at a preset frequency.
[0075] As another example, a virtual image of the surgical instrument 24, which changes with the overall tilt angle and / or the local tilt angle, can be continuously superimposed on the surgical screen. When the overall tilt angle of the surgical instrument is greater than or equal to a first preset angle, and / or the local tilt angle of the surgical instrument 24 is greater than or equal to a second preset angle, the virtual image displayed on the surgical screen can flash at a preset frequency. As another example, the virtual image of the surgical instrument 24 can be displayed flashing entirely at a preset frequency, or only a portion of the virtual image of the surgical instrument 24 can be displayed flashing at a preset frequency.
[0076] Regarding the virtual image combining the surgical instrument 24 and the object outside the surgical screen, which changes with the overall yaw angle relative to the object outside the surgical screen, as an example, if the overall yaw angle is greater than or equal to a third preset angle, the virtual image of the surgical instrument 24 changing with the overall yaw angle and the virtual image of the object outside the surgical screen will be combined and superimposed on the surgical screen. That is, when the overall yaw angle is normal (e.g., the surgical instrument 24 is far from the patient's lower limb), the corresponding virtual image may not be displayed on the surgical screen. Only when the overall yaw angle is abnormal (e.g., the surgical instrument 24 is about to contact the patient's lower limb) will the corresponding virtual image be displayed on the surgical screen, so that the doctor can notice the abnormal posture of the surgical instrument 24 in time. As another example, the virtual image displayed on the surgical screen can flash at a preset frequency.
[0077] As another example, the virtual image of the surgical instrument 24 and the object outside the surgical screen, which changes with the overall yaw angle between the surgical instrument 24 and the object outside the surgical screen, can also be continuously superimposed on the surgical screen. When the overall yaw angle is greater than or equal to a third preset angle, the virtual image displayed on the surgical screen can flash at a preset frequency.
[0078] Regarding the virtual image of a designated component of surgical instrument 24 that changes with the position and / or angle between the designated component and the object within the surgical screen, as an example, if the position of the designated component of surgical instrument 24 relative to the object within the surgical screen is at a preset abnormal position, and / or the angle between the designated component of surgical instrument 24 and the object within the surgical screen is greater than or equal to a fourth preset angle, then the virtual image of the designated component of surgical instrument 24 is superimposed on the surgical screen. That is, when the position and angle of the designated component of surgical instrument 24 relative to the object within the surgical screen are normal, the corresponding virtual image may not be displayed on the surgical screen; only when the position or angle is abnormal will the corresponding virtual image be displayed on the surgical screen, so that the doctor can promptly notice the abnormal position of surgical instrument 24. As another example, the virtual image displayed on the surgical screen can flash at a preset frequency.
[0079] As another example, a virtual image of a designated component of the surgical instrument 24, which changes with the position and / or angle between the designated component of the surgical instrument 24 and the object in the surgical screen, can also be continuously superimposed on the surgical screen. When the position is in a preset abnormal position and / or the angle is greater than or equal to a fourth preset angle, the virtual image displayed on the surgical screen can flash at a preset frequency.
[0080] Corresponding to the above embodiments of the surgical robot system, this application also provides an image display method for a surgical robot system, see [link to relevant documentation]. Figure 7 As shown, the method includes the following steps:
[0081] S701. Acquire the surgical image captured by the image acquisition device and display the surgical image on the display device.
[0082] S702. Obtain the position information of the surgical instrument.
[0083] S703. Generate a virtual image of the surgical instrument based on the position information of the surgical instrument.
[0084] The virtual image of the surgical instruments indicates the positional information of the surgical instruments.
[0085] S704. The virtual image of the surgical instrument is superimposed on the surgical screen.
[0086] As an example, at least a portion of the surgical instruments is not visible in the surgical view.
[0087] As an example, the pose information of the surgical instrument includes at least one of the following: the pose information of the surgical instrument itself, the relative pose information between the surgical instrument and an object outside the surgical screen, or the relative pose information between the surgical instrument and an object within the surgical screen.
[0088] As an example, the pose information of the surgical instrument itself includes the overall tilt angle and local tilt angle of the surgical instrument, wherein the local tilt angle refers to a movable part of the surgical instrument. Generating a virtual image of the surgical instrument based on its pose information includes:
[0089] Based on the overall tilt angle and local tilt angle of the surgical instrument, a virtual image of the surgical instrument is generated that varies with the changes in the overall tilt angle and / or the local tilt angle.
[0090] As an example, the step of overlaying the virtual image of the surgical instruments onto the surgical screen includes:
[0091] If the overall tilt angle is greater than or equal to a first preset angle, and / or the local tilt angle is greater than or equal to a second preset angle, then the virtual image of the surgical instrument, which changes with the change of the overall tilt angle and / or the change of the local tilt angle, will be superimposed on the surgical screen.
[0092] As an example, the relative pose information between the surgical instrument and an object outside the surgical view includes the overall yaw angle of the surgical instrument relative to the object outside the surgical view. Generating a virtual image of the surgical instrument based on its pose information includes:
[0093] Based on the overall yaw angle between the surgical instrument and an object outside the surgical field, a virtual image of the surgical instrument is generated that varies with the overall yaw angle.
[0094] As an example, the method also includes:
[0095] Generate virtual images of objects outside the surgical field;
[0096] The step of overlaying the virtual image onto the surgical screen includes:
[0097] Based on the overall yaw angle between the surgical instrument and an object outside the surgical screen, the virtual image of the surgical instrument, which changes with the overall yaw angle, and the virtual image of the object outside the surgical screen are combined and superimposed on the surgical screen to indicate the yaw information between the surgical instrument and the object outside the surgical screen.
[0098] As an example, overlaying virtual images of the surgical instruments onto the surgical screen includes:
[0099] If the overall yaw angle is greater than or equal to the third preset angle, then the virtual images of the surgical instruments that change with the overall yaw angle and the virtual images of objects outside the surgical screen will be combined and superimposed on the surgical screen.
[0100] As an example, the object outside the surgical view is the patient's lower limb, and generating a virtual image of the object outside the surgical view includes:
[0101] A virtual image of the patient's lower limbs is generated based on the midline of the patient's lower limbs, wherein the midline of the patient's lower limbs is parallel to the long side of the operating table.
[0102] As an example, the relative pose information between the surgical instrument and the object within the surgical view includes the position and / or angle of a designated component of the surgical instrument relative to the object within the surgical view. Generating a virtual image of the surgical instrument based on the pose information of the surgical instrument includes:
[0103] Based on the position and / or angle between a designated component of the surgical instrument and an object within the surgical screen, a virtual image of the designated component of the surgical instrument is generated, which changes as the position and / or angle between the designated component of the surgical instrument and the object within the surgical screen changes.
[0104] The step of overlaying the virtual image onto the surgical screen includes:
[0105] Based on the position and / or angle between a designated component of the surgical instrument and an object within the surgical screen, a virtual image of the designated component of the surgical instrument is overlaid on the surgical screen.
[0106] As an example, the surgical instrument is a uterine manipulator.
[0107] As an example, the positional information of the surgical instruments includes the position and / or angle of the rim of the uterine lifter cup relative to the uterus within the surgical view. Generating a virtual image of the surgical instruments based on the positional information includes:
[0108] Based on the position and / or angle between the cup rim and the uterus within the surgical view, a virtual image of the cup rim is generated that changes with the position and / or angle between the cup rim and the uterus within the surgical view.
[0109] The step of overlaying the virtual image onto the surgical screen includes:
[0110] Based on the position and / or angle between the cup rim and the uterus within the surgical screen, a virtual image of the cup rim is overlaid on the surgical screen.
[0111] As an example, the surgical robot system includes a manipulator assembly, under the drive of the surgical instruments.
[0112] As an example, the manipulator assembly includes several connecting arms, wherein adjacent connecting arms are connected by joints, and the joints drive the connecting arms to move under the drive of joint motors; acquiring the posture information of the surgical instrument includes:
[0113] The operating parameters of the joint motor are obtained, and the posture information of the surgical instrument is determined based on the operating parameters.
[0114] This application also provides an electronic device, such as Figure 8 As shown, the electronic device includes:
[0115] Processor 801;
[0116] Memory 802 is used to store processor-executable instructions;
[0117] The processor 801 is configured to implement the image display method of the surgical robot system described in any of the embodiments above.
[0118] This application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the image display method of the surgical robot system described in any of the embodiments above.
[0119] The above description is only a specific embodiment of this application. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of this application, and these improvements and modifications should also be considered within the scope of protection of this application.
Claims
1. An image display method for a surgical robot system, characterized in that, The surgical robot system includes an image acquisition device, a display device, and surgical instruments; the method includes: The surgical images captured by the image acquisition device are acquired and displayed on the display device. Obtain the position and orientation information of the surgical instruments; A virtual image of the surgical instrument is generated based on the positional information of the surgical instrument, and the virtual image of the surgical instrument indicates the positional information of the surgical instrument. The virtual image of the surgical instrument is overlaid on the surgical screen.
2. The method according to claim 1, characterized in that, The pose information of the surgical instrument includes at least one of the following: the pose information of the surgical instrument itself, the relative pose information between the surgical instrument and an object outside the surgical screen, or the relative pose information between the surgical instrument and an object within the surgical screen.
3. The method according to claim 2, characterized in that, The positional information of the surgical instrument itself includes the overall tilt angle and local tilt angle of the surgical instrument, wherein the local tilt angle refers to a movable part of the surgical instrument. Generating a virtual image of the surgical instrument based on its positional information includes: Based on the overall tilt angle and local tilt angle of the surgical instrument, a virtual image of the surgical instrument is generated that varies with the changes in the overall tilt angle and / or the local tilt angle.
4. The method according to claim 3, characterized in that, The step of overlaying the virtual image of the surgical instrument onto the surgical screen includes: If the overall tilt angle is greater than or equal to a first preset angle, and / or the local tilt angle is greater than or equal to a second preset angle, then the virtual image of the surgical instrument, which changes with the change of the overall tilt angle and / or the change of the local tilt angle, will be superimposed on the surgical screen.
5. The method according to claim 2, characterized in that, The relative pose information between the surgical instrument and an object outside the surgical view includes the overall yaw angle of the surgical instrument relative to the object outside the surgical view. Generating a virtual image of the surgical instrument based on the pose information includes: Based on the overall yaw angle between the surgical instrument and an object outside the surgical field, a virtual image of the surgical instrument is generated that varies with the overall yaw angle.
6. The method according to claim 5, characterized in that, The method further includes: Generate virtual images of objects outside the surgical field; The step of overlaying the virtual image onto the surgical screen includes: Based on the overall yaw angle between the surgical instrument and an object outside the surgical screen, the virtual image of the surgical instrument, which changes with the overall yaw angle, and the virtual image of the object outside the surgical screen are combined and superimposed on the surgical screen to indicate the yaw information between the surgical instrument and the object outside the surgical screen.
7. The method according to claim 6, characterized in that, The step of overlaying the virtual image of the surgical instrument onto the surgical screen includes: If the overall yaw angle is greater than or equal to the third preset angle, then the virtual images of the surgical instruments that change with the overall yaw angle and the virtual images of objects outside the surgical screen will be combined and superimposed on the surgical screen.
8. The method according to claim 6, characterized in that, The object outside the surgical view is the patient's lower limb, and generating a virtual image of the object outside the surgical view includes: A virtual image of the patient's lower limbs is generated based on the midline of the patient's lower limbs, wherein the midline of the patient's lower limbs is parallel to the long side of the operating table.
9. The method according to claim 2, characterized in that, The relative pose information between the surgical instrument and the object within the surgical view includes the position and / or angle of a designated component of the surgical instrument relative to the object within the surgical view. Generating a virtual image of the surgical instrument based on the pose information includes: Based on the position and / or angle between a designated component of the surgical instrument and an object within the surgical screen, a virtual image of the designated component of the surgical instrument is generated, which changes as the position and / or angle between the designated component of the surgical instrument and the object within the surgical screen changes. The step of overlaying the virtual image onto the surgical screen includes: Based on the position and / or angle between a designated component of the surgical instrument and an object within the surgical screen, a virtual image of the designated component of the surgical instrument is overlaid on the surgical screen.
10. The method according to any one of claims 1-9, characterized in that, The surgical instrument is a uterine manipulator.
11. The method according to claim 9, characterized in that, The surgical instrument is a uterine manipulator. The positional information of the surgical instrument includes the position and / or angle of the uterine manipulator's cup relative to the uterus within the surgical view. Generating a virtual image of the surgical instrument based on the positional information includes: Based on the position and / or angle between the cup rim and the uterus within the surgical view, a virtual image of the cup rim is generated that changes with the position and / or angle between the cup rim and the uterus within the surgical view. The step of overlaying the virtual image onto the surgical screen includes: Based on the position and / or angle between the cup rim and the uterus within the surgical screen, a virtual image of the cup rim is overlaid on the surgical screen.
12. The method according to any one of claims 1-9, characterized in that, The surgical robot system includes a manipulator assembly, under the drive of the surgical instruments.
13. The method according to claim 12, characterized in that, The manipulator assembly includes several connecting arms, wherein adjacent connecting arms are connected by joints, and the joints drive the connecting arms to move under the drive of joint motors; acquiring the posture information of the surgical instrument includes: The operating parameters of the joint motor are obtained, and the posture information of the surgical instrument is determined based on the operating parameters.
14. A surgical robot system, characterized in that, include: Image acquisition equipment to capture surgical images; The display device displays the surgical scene; Surgical instruments; The processor is configured to acquire the pose information of the surgical instrument, generate a virtual image of the surgical instrument based on the pose information, and overlay the virtual image of the surgical instrument onto the surgical screen, wherein the virtual image of the surgical instrument indicates the pose information of the surgical instrument.
15. The surgical robot system according to claim 14, characterized in that, The pose information of the surgical instrument includes at least one of the following: the pose information of the surgical instrument itself, the relative pose information between the surgical instrument and an object outside the surgical screen, or the relative pose information between the surgical instrument and an object within the surgical screen.
16. The surgical robot system according to claim 15, characterized in that, The positional information of the surgical instrument itself includes the overall tilt angle and the local tilt angle of the surgical instrument, wherein the local part is a movable part of the surgical instrument. The processor is specifically used to generate a virtual image of the surgical instrument that changes with the changes in the overall tilt angle and / or the local tilt angle based on the overall tilt angle and the local tilt angle of the surgical instrument.
17. The surgical robot system according to claim 15, characterized in that, The relative pose information between the surgical instrument and an object outside the surgical screen includes the overall yaw angle between the surgical instrument and the object outside the surgical screen. The processor is specifically used to generate a virtual image of the surgical instrument that changes with the overall yaw angle based on the overall yaw angle between the surgical instrument and the object outside the surgical screen.
18. The surgical robot system according to claim 17, characterized in that, The processor is also configured to generate a virtual image of an object outside the surgical screen, and based on the overall yaw angle between the surgical instrument and the object outside the surgical screen, combine and overlay the virtual image of the surgical instrument and the virtual image of the object outside the surgical screen on the surgical screen to indicate the yaw information between the surgical instrument and the object outside the surgical screen.
19. The surgical robot system according to claim 15, characterized in that, The relative pose information between the surgical instrument and the object in the surgical screen includes the position and / or angle between a designated component of the surgical instrument and the object in the surgical screen. Specifically, the processor is used to generate a virtual image of the designated component of the surgical instrument that changes with the position and / or angle between the designated component of the surgical instrument and the object in the surgical screen; and to overlay the virtual image of the designated component of the surgical instrument onto the surgical screen based on the position and / or angle between the designated component of the surgical instrument and the object in the surgical screen.
20. The surgical robot system according to any one of claims 15-19, characterized in that, The surgical instrument is a uterine manipulator.
21. The surgical robot system according to claim 20, characterized in that, The surgical robot system includes a first manipulator robot, which includes a first manipulator assembly, and the uterine manipulator moves under the drive of the first manipulator assembly.
22. The surgical robot system according to claim 21, characterized in that, The surgical robot system includes a second manipulator robot, which includes at least one second manipulator component that drives the movement of surgical tools, which include at least one of the image acquisition device or surgical instruments.
23. The surgical robot system according to claim 22, characterized in that, The first and second control robots are set up separately.
24. An electronic device, characterized in that, include: processor; Memory used to store processor-executable instructions; The processor is configured to implement the method of any one of claims 1 to 13.
25. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 13.