Method for displaying virtual surgical instruments at a surgeon console and surgeon console

Abstract

The present specification relates to the technical field of endoscopic surgery, and specifically discloses a method for displaying a virtual surgical instrument on a surgeon console and the surgeon console, wherein the method comprises: acquiring an image of a main control arm of the surgeon console; determining pose data of the main control arm according to the image of the main control arm; generating virtual pose data of a surgical instrument based on the pose data of the main control arm and a three-dimensional model of the surgical instrument, so as to display a virtual instruction instrument image in a stereoscopic monitor of the surgeon console. The above scheme can facilitate the surgeon to timely discover misoperation, timely discover instrument failure, assist the surgeon in operation, improve surgical efficiency, improve surgical quality and success rate, and improve the surgical experience of the surgeon and the patient by displaying the virtual instruction instrument image in the stereoscopic monitor.

Description

Technical Field

[0001] This manual relates to the field of endoscopic surgery technology, and in particular to a method for displaying virtual surgical instruments on a physician's console and a physician's console. Background Technology

[0002] In existing endoscopic robotic surgery, surgeons control surgical instruments within the endoscopic surgical environment via a main control arm and foot pedals on a control console. This requires coordinated hand and foot movements. The 3D monitor on the control console displays images of the surgical instruments within the endoscopic environment. However, surgeons rely solely on training or muscle memory from past experience for their foot and hand movements, making them prone to errors that can negatively impact surgical quality and success rates.

[0003] There is currently no effective solution to the above problems. Summary of the Invention

[0004] This specification provides a method for displaying virtual surgical instruments on a doctor's console and a doctor's console, in order to solve the problem of easy misoperation when performing surgery based solely on endoscopic images from a stereoscopic monitor in the prior art.

[0005] This specification provides a method for displaying virtual surgical instruments on a doctor's console, comprising: acquiring an image of the main control arm of the doctor's console; determining the posture data of the main control arm based on the image of the main control arm; and generating virtual posture data of the surgical instruments based on the posture data of the main control arm and a three-dimensional model of the surgical instruments, so as to display the image of the virtual control instrument on a stereoscopic monitor of the doctor's console.

[0006] In one embodiment, the method further includes: acquiring a foot pedal image of a doctor's console and / or an image of the operator's feet; generating foot pedal position data based on the foot pedal image, and / or generating foot position data based on the foot image, to display the foot pedal area and / or foot area in the stereoscopic monitor.

[0007] In one embodiment, the method further includes: highlighting one or more foot pedals that are effectively stepped on in the foot pedal area on the stereoscopic monitor when one or more foot pedals of the doctor's console are effectively stepped on.

[0008] In one embodiment, the images of the main control arm include the left and right eye images of the main control arm captured by a binocular camera.

[0009] In one embodiment, the image acquisition device includes a binocular camera and an infrared rangefinder.

[0010] In one embodiment, the image acquisition device includes an RGB-D (RGB-Depth) camera.

[0011] In one embodiment, the image acquisition device includes two vertically positioned monocular cameras.

[0012] In one embodiment, the image acquisition device includes two vertically positioned monocular cameras and an infrared rangefinder.

[0013] In one embodiment, determining the attitude data of the main control arm based on the image of the main control arm includes: acquiring the spatial position data of the left and right cameras in the binocular cameras; determining the spatial position data corresponding to each feature point based on the spatial position data and the position parameters of each feature point in the left and right images; and determining the attitude data of the main control arm based on the spatial position data corresponding to each feature point at multiple time points.

[0014] In one embodiment, generating virtual posture data of the surgical instrument based on the posture data of the main control arm and the three-dimensional model of the surgical instrument includes: generating control command information of the main control arm based on the posture data of the main control arm; and applying the control command information of the main control arm to the three-dimensional model of the surgical instrument to generate virtual posture data of the surgical instrument.

[0015] In one embodiment, generating virtual posture data of the surgical instrument based on the posture data of the main control arm and the three-dimensional model of the surgical instrument includes: generating virtual posture data of the surgical instrument using a preset control algorithm based on the posture data of the main control arm and the three-dimensional model of the surgical instrument.

[0016] In one embodiment, the stereo monitor also displays actual instrument images of surgical instruments acquired by the endoscope; after displaying virtual instruction instrument images on the stereo monitor of the doctor's console, the method further includes: comparing the actual instrument images with the virtual instruction instrument images; determining, based on the comparison results, target portions in the virtual instruction instrument images where the difference between the virtual instruction instrument images and the actual instrument images exceeds a preset range; the stereo monitor is also used to annotate and display the target portions.

[0017] This specification also provides a doctor's console, including: an image acquisition device for acquiring images of the main control arm of the doctor's console; an image processing device for determining the posture data of the main control arm based on the image of the main control arm; and for generating virtual posture data of the surgical instruments based on the posture data of the main control arm and a three-dimensional model of the surgical instruments; and a stereo monitor for displaying images of virtual command instruments based on the virtual posture data.

[0018] In one embodiment, the image acquisition device is further configured to acquire foot pedal images of the doctor's console and images of the operator's feet; the image processing device is further configured to generate foot pedal position data based on the foot pedal images, and / or generate foot position data based on the foot images; the stereo monitor is further configured to display a foot pedal area based on the foot pedal position data, and / or display a foot area based on the foot position data.

[0019] In one embodiment, the stereo monitor is also used to highlight one or more pedals that are effectively stepped on in the pedal area.

[0020] In one embodiment, the image acquisition device includes a binocular camera, which is used to acquire left and right eye images of the main control arm.

[0021] In one embodiment, the image processing device is specifically used to: acquire spatial position data of the left and right cameras in the binocular camera; determine the spatial position data corresponding to each feature point based on the spatial position data and the position parameters of each feature point in the left and right images of the main control arm; and determine the attitude data of the main control arm based on the spatial position data corresponding to each feature point at multiple times.

[0022] In one embodiment, the image processing device is specifically used to: generate control command information for the main control arm based on the posture data of the main control arm; and apply the control command information of the main control arm to a three-dimensional model of the surgical instrument to generate virtual posture data of the surgical instrument.

[0023] In one embodiment, the image processing device is specifically used to: generate virtual posture data of the surgical instrument using a preset control algorithm based on the posture data of the main control arm and the three-dimensional model of the surgical instrument.

[0024] In one embodiment, the stereo monitor also displays actual instrument images of surgical instruments acquired by the endoscope; the image processing device is further configured to: compare the actual instrument image with the virtual instruction instrument image; determine, based on the comparison result, a target portion in the virtual instruction instrument image whose difference from the actual instrument image exceeds a preset range; the stereo monitor is further configured to annotate and display the target portion.

[0025] This specification also provides a medical device, including a processor and a memory for storing processor-executable instructions, wherein the processor executes the instructions to implement the steps of the method for displaying virtual surgical instruments on a doctor's console as described in any of the above embodiments.

[0026] This specification also provides a computer-readable storage medium storing computer instructions that, when executed, implement the steps of the method for displaying virtual surgical instruments on a doctor's console as described in any of the above embodiments.

[0027] This specification provides a method for displaying virtual surgical instruments on a doctor's console. The method involves acquiring an image of the main control arm of the doctor's console, determining the posture data of the main control arm based on the image, and generating virtual posture data of the surgical instrument based on the posture data of the main control arm and a 3D model of the surgical instrument. This virtual control instrument image is then displayed on a stereoscopic monitor of the doctor's console. In this approach, the posture data of the main control arm is first determined based on its image. Since a master-slave relationship exists between the main control arm and the surgical instrument, virtual posture data of the surgical instrument can be generated based on the posture data of the main control arm and the 3D model of the surgical instrument. Finally, the virtual control instrument image is displayed on the stereoscopic monitor based on this virtual posture data. Displaying the virtual control instrument image on the stereoscopic monitor facilitates timely detection of operational errors and instrument malfunctions by the doctor, assists in doctor operations, improves surgical efficiency, enhances surgical quality and success rate, and improves the surgical experience for both doctors and patients. Attached Figure Description

[0028] The accompanying drawings, which are included to provide a further understanding of this specification and form part of it, do not constitute a limitation thereof. In the drawings:

[0029] Figure 1 A schematic diagram of the endoscopic surgical system according to an embodiment of this specification is shown;

[0030] Figure 2 A schematic diagram of the doctor's console in an embodiment of this specification is shown;

[0031] Figure 3A schematic diagram of the image acquisition device in an embodiment of this specification is shown;

[0032] Figure 4 A schematic diagram of the patient surgical platform in an embodiment of this specification is shown;

[0033] Figure 5 A schematic diagram of the image platform in an embodiment of this specification is shown;

[0034] Figure 6 A flowchart illustrating a method for displaying virtual surgical instruments on a doctor's console, as shown in an embodiment of this specification, is presented.

[0035] Figure 7 A structural block diagram of a doctor's console in an embodiment of this specification is shown;

[0036] Figure 8 A block diagram of the enhanced display control logic in an embodiment of this specification is shown;

[0037] Figure 9 A flowchart illustrating the display of virtual command device images in embodiments of this specification is shown;

[0038] Figure 10 A flowchart illustrating the association between the master hand pose and the virtual instrument pose in embodiments of this specification is shown;

[0039] Figure 11 A schematic diagram of the camera coordinate system and world coordinate system in the embodiments of this specification is shown;

[0040] Figure 12 A schematic diagram illustrating the transformation between the world coordinate system and the camera coordinate system in an embodiment of this specification is shown;

[0041] Figure 13 This diagram illustrates the virtual foot position and foot movement image display in the embodiments of this specification;

[0042] Figure 14 A schematic diagram illustrating the warning of excessive deviation in the position tracking of surgical instruments in the embodiments of this specification is shown;

[0043] Figure 15 A schematic diagram showing the position of the foot pedal panel in an embodiment of this specification is shown;

[0044] Figure 16 A schematic diagram showing the foot movement display and effective pedaling display in the embodiments of this specification is shown;

[0045] Figure 17 A schematic diagram of the operation process of the endoscopic surgical robot in the embodiments of this specification is shown;

[0046] Figure 18A schematic diagram of the virtual command device acquisition and display process in the embodiments of this specification is shown;

[0047] Figure 19 A flowchart illustrating the display of virtual foot pedal position and foot movements in an embodiment of this specification is shown.

[0048] Figure 20 A schematic diagram of the medical device composition structure in the embodiments of this specification is shown. Detailed Implementation

[0049] The principles and spirit of this specification will now be described with reference to several exemplary embodiments. It should be understood that these embodiments are given merely to enable those skilled in the art to better understand and implement this specification, and are not intended to limit the scope of this specification in any way. Rather, these embodiments are provided to make this disclosure more thorough and complete, and to fully convey the scope of this disclosure to those skilled in the art.

[0050] Those skilled in the art will recognize that the embodiments described in this specification can be implemented as a system, apparatus, method, or computer program product. Therefore, the disclosure of this specification can be specifically implemented in the following forms: entirely hardware, entirely software (including firmware, resident software, microcode, etc.), or a combination of hardware and software.

[0051] This specification provides a method for displaying virtual surgical instruments on a doctor's console and a doctor's console, which can be applied to endoscopic surgery systems. Figure 1 A schematic diagram of an endoscopic surgical system is shown. Figure 1 As shown, an endoscopic surgery system can consist of a physician's console, a patient surgical platform (including instruments and endoscopic instruments), and an imaging platform. The physician can operate from the physician's console, providing motion control commands to the instruments on the patient surgical platform. The endoscopic images are processed by the imaging platform and displayed on both the imaging platform and a stereoscopic monitor on the physician's console.

[0052] Please refer to Figure 2 This diagram illustrates a doctor's console as described in an embodiment of this specification. The doctor's console is the control center of the endoscopic surgery system, providing the doctor with clear images and the necessary control signal inputs for surgical procedures. The main components of the doctor's console may include: a stereoscopic monitor 201, a main operating hand (or main control arm) 202, a footpad 203, and an image acquisition device (…). Figure 2 (Not shown in the image).

[0053] A stereoscopic monitor can display the same image or video information on two separate screens, with the left eye viewing the left screen and the right eye viewing the right screen. The brain automatically synthesizes the images into a stereoscopic visual effect. The main functions of a stereoscopic monitor can include: displaying endoscopic visual images, displaying images of virtual control instruments, and displaying the position and movement of virtual foot pedals.

[0054] The foot pedal panel serves as an auxiliary control function, working in conjunction with the main hand / control arm to provide control signals to surgical instruments. The main functional components of the foot pedal panel can include: operation of surgical instruments, operation of the endoscope, and other auxiliary functions.

[0055] The doctor's console can be equipped with two master operating arms / master control arms, which receive operation signals from the operator's left and right hands respectively. The master control arm monitors the operator's hand movements and is the primary motion control input for the entire system. The operator can control the movement of the tool arm by operating the control handle at the end of the master control arm, thereby controlling the operation of the endoscope and surgical instruments.

[0056] Figure 3 A schematic diagram of an image acquisition device according to an embodiment of this specification is shown. The image acquisition device may include one or more binocular cameras. Binocular cameras can obtain object depth information by utilizing the difference in the position of feature point pixels in images captured by the two lenses. Using VR 3D technology, the position and posture of the object can be obtained. The image acquisition device primarily monitors the position of the operating hand and foot pedal panel, as well as foot movements.

[0057] In some embodiments of this specification, the image acquisition device may include a binocular camera and an infrared rangefinder. By additionally using an infrared rangefinder, it is easier to acquire depth information of the object, thereby obtaining the attitude data of the main control arm.

[0058] In some embodiments of this specification, the image acquisition device may include an RGB-D (RGB-Depth) camera. The depth map acquired by the RGB-D camera contains image channels that contain information related to the distance to the surface of objects in the viewpoint scene. The channels themselves are similar to grayscale images, and each pixel value is the actual distance from the object measured by the sensor. This allows the acquisition of object depth and position information, thereby obtaining the attitude data of the main control arm.

[0059] In some embodiments of this specification, the image acquisition device may include multiple monocular cameras. Multiple monocular cameras can be configured, such as two vertically aligned monocular cameras. The acquired images can be processed to obtain object depth information, and subsequently, the attitude data of the main control arm.

[0060] Please refer to Figure 4The diagram shows a schematic of the patient operating platform. The patient operating platform is the operating platform of the endoscopic surgery system located next to the patient's operating table. It consists of three main components: the operating trolley, the adjusting arm, and the tool arm (including the imaging arm). The instruments and endoscopes mounted on the tool arm act as driven mechanisms, receiving motion control commands from the physician's control console.

[0061] Please refer to Figure 5 The diagram illustrates the structure of the imaging platform. The imaging platform is the visual feedback subsystem of the endoscopic surgery system, and can include three main components: a 3D electronic endoscope, an endoscopic image processing host, and an imaging cart. It provides energy to the instruments, performs endoscopic visual processing, and displays images. The images synchronized with the doctor's stereoscopic monitor are primarily for viewing by personnel other than the operating surgeon.

[0062] Figure 6 A flowchart illustrating a method for displaying virtual surgical instruments on a physician's console according to one embodiment of this specification is shown. While this specification provides method steps or apparatus structures as shown in the embodiments or figures below, more or fewer steps or module units may be included in the method or apparatus based on conventional or non-inventive effort. The execution order of steps or the module structure of the apparatus in steps or structures where there is no logically necessary causal relationship is not limited to the execution order or module structure described in the embodiments and figures of this specification. When the method or module structure is applied in a practical device or end product, it can be executed sequentially or in parallel (e.g., in a parallel processor or multi-threaded processing environment, or even a distributed processing environment) according to the method or module structure shown in the embodiments or figures.

[0063] Specifically, such as Figure 6 As shown, a method for displaying virtual surgical instruments on a doctor's console according to one embodiment of this specification may include the following steps:

[0064] Step S601: Obtain an image of the main control arm of the doctor's console.

[0065] The methods described in the embodiments of this specification can be applied to an image processing device. The image processing device can acquire images of the main control arm of a doctor's console. The main control arm may include a left main control arm and a right main control arm.

[0066] In one embodiment, images of the main control arm during endoscopic surgery can be acquired using an image acquisition device. An image processing device can then acquire images of the main control arm from the image acquisition device. These images may include images of the left and right main control arms at at least one moment within the endoscopic surgical environment.

[0067] In one embodiment, the image acquisition device can acquire images of the main control arm during endoscopic surgery in real time. The image processing device can acquire images of the main control arm from the image acquisition device at preset time intervals.

[0068] Step S602: Determine the attitude data of the main control arm based on the image of the main control arm.

[0069] After acquiring an image of the main control arm, the image processing device can determine the main control arm's attitude data based on the image. The attitude data of the main control arm can include the attitude data of multiple feature points on the main control arm. The attitude of the main control arm can be characterized by the attitude data of multiple feature points. The attitude data of each feature point can include at least one of the following: position coordinates, displacement direction, displacement amount, rotation angle, rotation direction, etc. The image processing device can determine the attitude data of the main control arm by analyzing the images of the main control arm at two adjacent time points, thereby determining the main control arm's attitude data.

[0070] Step S603: Based on the posture data of the main control arm and the three-dimensional model of the surgical instrument, generate virtual posture data of the surgical instrument to display the virtual command instrument image on the stereoscopic monitor of the doctor's console.

[0071] Because of the master-slave relationship between the master control arm and the surgical instruments, the posture of the surgical instruments can be adjusted through the operation of the master control arm. Therefore, virtual posture data of the surgical instruments can be generated based on the posture data of the master control arm and the 3D model of the surgical instruments. The virtual posture data of the surgical instruments can include the posture data of the surgical instruments corresponding to the posture data of the master control arm.

[0072] In one embodiment, the image processing device may store a correspondence between the posture data of the main control arm and the posture data of the surgical instruments. The posture data of the surgical instruments may also include the posture data of multiple instrument feature points. The posture data of each instrument feature point may include at least one of the following: position coordinates, displacement direction, displacement amount, rotation angle, rotation direction, and other data.

[0073] After determining the posture data of the main control arm, the image processing device can generate virtual posture data of the surgical instruments based on the posture data of the main control arm and the 3D model of the surgical instruments. The 3D model of the surgical instruments can be pre-established model data of the surgical instruments in a 3D coordinate system. The virtual posture data of the surgical instruments refers to the posture data of the surgical instruments under the operation of the main control arm.

[0074] After generating virtual posture data for the surgical instruments, the virtual command instrument image can be displayed on a stereoscopic monitor in the doctor's console based on this data. In one embodiment, the stereoscopic monitor can simultaneously display both the virtual command instrument image and the actual instrument image captured by the endoscope.

[0075] In the above embodiments, the posture data of the main control arm can be determined first based on the image of the main control arm. Since there is a master-slave relationship between the main control arm and the surgical instruments, virtual posture data of the surgical instruments can be generated based on the posture data of the main control arm and the 3D model of the surgical instruments. Then, a virtual command instrument image is displayed on a stereoscopic monitor based on the virtual posture data of the surgical instruments. By displaying the virtual command instrument image on a stereoscopic monitor, doctors can more easily detect misoperations and instrument malfunctions in a timely manner, thereby reducing misoperations, improving surgical quality and success rate, and enhancing the surgical experience for both doctors and patients.

[0076] In some embodiments of this specification, the method may further include: acquiring a foot pedal image of a doctor's console and / or an image of the operator's feet; generating foot pedal position data based on the foot pedal image, and / or generating foot position data based on the foot image, to display the foot pedal area and / or foot area in the stereoscopic monitor.

[0077] Considering that the surgeon cannot see the foot pedals or foot movements during the operation and can only rely on experience to operate the foot pedals, the image acquisition device can capture images of the foot pedals on the doctor's console and images of the surgeon's feet within the foot pedal area. The image processing device can acquire the foot pedal and foot images captured by the image acquisition device.

[0078] Subsequently, the image processing device can generate foot pedal position data based on the foot pedal image. The foot pedal position data can include the position and size data of each foot pedal among multiple foot pedals in the doctor's console. Based on the foot pedal position data, the image processing device can display the foot pedal area on a stereoscopic monitor. The foot pedal area can include multiple foot pedal sub-areas, each corresponding to a single foot pedal.

[0079] The image processing device can also generate foot position data based on the foot image. The foot position data may include the position and size data of the operator's left and / or right feet. The image processing device can then display the foot area on a stereoscopic monitor based on the foot position data. The foot area may include a left foot area corresponding to the left foot and a right foot area corresponding to the right foot.

[0080] In some embodiments of this specification, the method may further include: highlighting one or more foot pedals that have been effectively pressed in the foot pedal area on the stereoscopic monitor when one or more foot pedals of the doctor's console are effectively pressed.

[0081] To allow operators to promptly know whether they have stepped on a particular foot pedal, when one or more foot pedals on the doctor's console are effectively stepped on, the one or more foot pedals that have been effectively stepped on can be highlighted in the stereo monitor.

[0082] In one embodiment, each of the multiple footstools on the doctor's console is equipped with a pressure sensor underneath. When the pressure detected by the pressure sensor exceeds a preset value, the corresponding footstool is considered to have been effectively stepped on. Each pressure sensor can be connected to an image processing device, which can determine whether the pressure value detected by each sensor exceeds a preset pressure threshold, and classify the footstool corresponding to the pressure sensor with the pressure value exceeding the preset pressure threshold as effectively stepped on. Based on the determination result, the image processing device highlights one or more effectively stepped footstools in the footstool area on a stereoscopic monitor. This method allows the operator to promptly know whether a footstool has been effectively stepped on or mistakenly stepped on, improving surgical quality and success rate, and enhancing the operator's experience.

[0083] In some embodiments of this specification, the images of the main control arm may include the left and right eye images of the main control arm captured by a binocular camera.

[0084] In this embodiment, the image acquisition device may include a binocular camera. Accordingly, the acquired images of the main control arm may include the left and right eye images of the main control arm acquired by the binocular camera. In this embodiment, by setting the image acquisition device as a binocular camera, the image processing device can obtain depth information based on the left and right eye images, thereby enabling stereoscopic display of the instrument.

[0085] In some embodiments of this specification, determining the attitude data of the main control arm based on the image of the main control arm may include: acquiring the spatial position data of the left and right cameras in the binocular cameras; determining the spatial position data corresponding to each feature point based on the spatial position data and the position parameters of each feature point in the left and right images of the main control arm; and determining the attitude data of the main control arm based on the spatial position data corresponding to each feature point at multiple times.

[0086] In this embodiment, the main control arm can be considered a rigid body. The position or orientation of three points not on a straight line can determine the position or orientation of the rigid body. Therefore, the orientation of the main control arm can be determined by the orientation of at least three feature points on the main control arm. Spatial position data of the left and right cameras in the binocular camera system can be acquired. Then, based on the spatial position data of the left and right cameras and the position parameters of at least three feature points on the main control arm, the spatial position data corresponding to each feature point, i.e., the spatial position data of each feature point in the geographic coordinate system, can be determined. After obtaining the spatial position data corresponding to each feature point, the orientation data of the main control arm can be determined based on the spatial position data corresponding to each feature point. Through the above method, the orientation data of the main control arm can be determined based on the obtained image of the main control arm.

[0087] In some embodiments of this specification, generating virtual posture data of the surgical instrument based on the posture data of the main control arm and the three-dimensional model of the surgical instrument may include: generating control command information of the main control arm based on the posture data of the main control arm; and applying the control command information of the main control arm to the three-dimensional model of the surgical instrument to generate virtual posture data of the surgical instrument.

[0088] Specifically, the image processing device can generate control command information for the main control arm based on its posture data. This control command information can include operational instructions for the robotic arm, which may include, but are not limited to, at least one of the following: translation by a specified distance, rotation by a specified angle, rotation direction, rotation axis, rotation location, opening by a specified angle, etc. In one embodiment, the image processing device may store a correspondence between the control arm's posture data and the control commands, or a conversion formula between the two. Based on the stored correspondence or conversion formula, the device can generate the main control arm's control command information according to the main control arm's posture data. After obtaining the main control arm's control command information, the corresponding control commands are applied to the 3D model of the surgical instrument, thus obtaining the virtual posture data of the surgical instrument. For example, the 3D model of the surgical instrument can be manipulated according to the operational instructions in the control commands to obtain the virtual posture data of the surgical instrument. In this way, virtual posture data of the surgical instrument can be generated based on the control arm's posture data and the 3D model of the surgical instrument.

[0089] In some embodiments of this specification, generating virtual posture data of the surgical instrument based on the posture data of the main control arm and the three-dimensional model of the surgical instrument may include: acquiring a target machine learning model; the target machine learning model being pre-established based on the three-dimensional model of the surgical instrument; and inputting the posture data of the main control arm into the target machine learning model to obtain the virtual posture data of the surgical instrument.

[0090] Specifically, after obtaining the posture data of the control arm and the 3D model of the surgical instrument, virtual posture data of the surgical instrument can be generated using a preset control algorithm. In one embodiment, the preset control algorithm may include multiple operators in a machine learning model. After inputting the posture data of the main control arm into the machine learning model, the virtual posture data of the surgical instrument can be output. The machine learning model can be pre-built based on the 3D model of the surgical instrument.

[0091] In some embodiments of this specification, before obtaining the target machine learning model, the process may further include: constructing a machine learning model based on a three-dimensional model of the surgical instrument; acquiring the posture data of the control arm and the corresponding posture data of the surgical instrument during the endoscopic surgery, and constructing a training sample set based on the posture data of the control arm and the corresponding posture data of the surgical instrument during the endoscopic surgery; and training the machine learning model using the constructed training sample set to obtain the target machine learning model.

[0092] A machine learning model can be constructed based on the 3D model of the surgical instruments. During endoscopic surgery, the posture data of the control arm and the corresponding surgical instruments can be acquired. A training sample set is constructed based on the acquired posture data of the control arm and the corresponding surgical instruments during the endoscopic surgery. The posture data of the control arm serves as the input data, and the posture data of the surgical instruments serves as the labels. The pre-established machine learning model is trained using the constructed training sample set to obtain a target machine learning model. This target machine learning model can determine the posture data of a specific surgical instrument based on the posture data of the control arm. In this way, virtual posture data of the surgical instruments can be directly generated based on the posture data of the control arm and the 3D model of the surgical instruments.

[0093] In some embodiments of this specification, the stereo monitor also displays actual instrument images of surgical instruments acquired by the endoscope; after displaying virtual instruction instrument images on the stereo monitor of the doctor's console, the method may further include: comparing the actual instrument images with the virtual instruction instrument images; determining, based on the comparison results, a target portion in the virtual instruction instrument image whose difference from the actual instrument image exceeds a preset range; the stereo monitor is also used to annotate and display the target portion.

[0094] In this embodiment, both the actual instrument image acquired by the endoscope and the virtual command instrument image determined based on the posture data of the main control arm can be simultaneously displayed on a stereoscopic monitor. The image processing device can compare the actual instrument image with the virtual command instrument image. In one embodiment, the image processing device can divide the surgical instrument into multiple parts and compare each part in the virtual command instrument image with its corresponding part in the actual instrument image. Based on the comparison results, target parts in the virtual command instrument image whose differences from the actual instrument image exceed a preset range can be identified. These target parts can be marked and displayed on the stereoscopic monitor. For example, if a part in the virtual command instrument image translates a distance greater than a preset distance relative to its corresponding part in the actual instrument image, that part is identified as a target part. Another example is if a part in the virtual command instrument image rotates an angle greater than a preset angle around a fixed axis relative to its corresponding part in the actual instrument image, then that part is identified as a target part. Yet another example is if a part in the virtual command instrument image rotates an angle greater than a preset angle around a fixed point relative to its corresponding part in the actual instrument image, then that part is identified as a target part.

[0095] After identifying the target area, the image processing device can control the display of the target area on a stereoscopic monitor. This display can involve highlighting the target area in different colors, such as red or yellow. Alternatively, the display can be blinked. In one embodiment, both the actual instrument image and the virtual control instrument image can be displayed in the same coordinate system and at the same scale, allowing the operator to observe the target area promptly, reducing errors or timely detection of instrument malfunctions, and improving surgical quality and success rate.

[0096] In some embodiments of this specification, after determining a target portion in the virtual instruction instrument image whose difference from the actual instrument image exceeds a preset range, the method may further include: determining the position data of the target portion in the surgical instrument; generating warning information based on the position data; and issuing a warning to the operator in a preset manner according to the warning information.

[0097] Specifically, the image processing device divides the surgical instruments into multiple parts and then identifies the target part. The position of the target part within the surgical instruments can be obtained according to the division rules. The position data may include the location of the target part within the surgical instruments and its specific component name, among other data. After determining the position data of the surgical instruments, warning information can be generated based on the position data. Warnings can be issued to the operator in a preset manner based on the warning information. For example, it can be displayed on a stereo monitor in voice broadcast or text form.

[0098] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to interchangeably. Each embodiment focuses on describing the differences from other embodiments. For details, please refer to the foregoing descriptions of the relevant processing embodiments; they will not be repeated here.

[0099] The foregoing has described specific embodiments of this specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in a different order than that shown in the embodiments and may still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require the specific or sequential order shown to achieve the desired result. In some embodiments, multitasking and parallel processing are possible or may be advantageous.

[0100] Based on the same inventive concept, this specification also provides a doctor's console, as described in the following embodiments. Since the principle by which the doctor's console solves the problem is similar to that of the method for displaying virtual surgical instruments on a doctor's console, the implementation of the doctor's console can refer to the implementation of the method for displaying virtual surgical instruments on a doctor's console, and repeated details will not be elaborated further. As used below, the terms "unit" or "module" can refer to a combination of software and / or hardware that performs a predetermined function. Although the apparatus described in the following embodiments is preferably implemented in software, hardware implementation, or a combination of software and hardware, is also possible and contemplated. Figure 7 This is a structural block diagram of a doctor's console in an embodiment of this specification, such as... Figure 7 As shown, it includes: an image acquisition device 701, an image processing device 702, and a stereo monitor 703. The structure is described below.

[0101] The image acquisition device 701 is used to acquire images of the main control arm of the doctor's console.

[0102] The image processing device 702 is used to determine the posture data of the main control arm based on the image of the main control arm; and is also used to generate virtual posture data of the surgical instrument based on the posture data of the main control arm and the three-dimensional model of the surgical instrument.

[0103] The stereo monitor 703 is used to display virtual command device images based on the virtual posture data.

[0104] In some embodiments of this specification, the image acquisition device is further configured to acquire foot pedal images of the doctor's console and images of the operator's feet; the image processing device is further configured to generate foot pedal position data based on the foot pedal images, and / or generate foot position data based on the foot images; the stereo monitor is further configured to display a foot pedal area based on the foot pedal position data, and / or display a foot area based on the foot position data.

[0105] In some embodiments of this specification, the stereo monitor is also used to highlight one or more pedals that are effectively stepped on in the pedal area.

[0106] In some embodiments of this specification, the image acquisition device includes a binocular camera used to acquire left and right eye images of the main control arm.

[0107] In some embodiments of this specification, the image processing device is specifically used for: acquiring spatial position data of the left and right cameras in the binocular camera; determining the spatial position data corresponding to each feature point based on the spatial position data and the position parameters of each feature point in the left and right images of the main control arm; and determining the attitude data of the main control arm based on the spatial position data corresponding to each feature point at multiple times.

[0108] In some embodiments of this specification, the image processing device is specifically used to: generate control command information for the main control arm based on the posture data of the main control arm; and apply the control command information of the main control arm to a three-dimensional model of the surgical instrument to generate virtual posture data of the surgical instrument.

[0109] In some embodiments of this specification, the image processing device is specifically used to: generate virtual posture data of the surgical instrument using a preset control algorithm based on the posture data of the main control arm and the three-dimensional model of the surgical instrument.

[0110] In some embodiments of this specification, the stereo monitor may also display actual instrument images of surgical instruments acquired by an endoscope; the image processing device may also be used to: compare the actual instrument image with the virtual instruction instrument image; determine, based on the comparison result, a target portion in the virtual instruction instrument image whose difference from the actual instrument image exceeds a preset range; the stereo monitor may also be used to annotate and display the target portion.

[0111] The above method and doctor console are described below with reference to a specific embodiment. However, it is worth noting that this specific embodiment is only for better illustration of this specification and does not constitute an undue limitation of this specification.

[0112] In this specific embodiment, an augmented reality doctor's console is provided. For example... Figure 1 As shown, the doctor's console may include a stereo monitor, a main control arm and foot pedal panel, and an image acquisition device.

[0113] Enhanced display primarily achieves the simultaneous display of endoscopic visual images, virtual control instrument images, foot pedal position images, and foot movement images on a stereo monitor. This real-time fusion display can provide operators with prompts and reduce operational errors. Please refer to [reference needed]. Figure 8 The diagram shows the block diagram of the enhanced display control logic. For example... Figure 8 As shown, augmented reality can include: displaying images of virtual command devices, and displaying images of virtual foot pedal positions and foot movements.

[0114] Please refer to Figure 9 The flowchart illustrating the display of virtual command device images is shown. For example... Figure 9 As shown, the image acquisition device, preferably binocular vision, uses depth information to obtain the master hand's operating posture, which is then fused with the master hand's control commands to obtain the virtual command device's pose. The virtual command device image is displayed on the operating platform via a switch that can control its on / off state. When a malfunction occurs with excessive deviation, a red warning will be displayed at the location of the excessive deviation.

[0115] Please refer to Figure 10 This illustrates a flowchart showing the association between the primary hand pose and the virtual instrument pose. (Example:) Figure 10 As shown, after startup, binocular vision is activated. The positions of multiple target feature points on the main control arm within the field of view are acquired through the left eye camera (camera 1). The positions of multiple target feature points on the main control arm within the field of view are also acquired through the right eye camera (camera 2). Based on the spatial positions of the two cameras and the positions of each feature point in the two fields of view, the spatial position of each target feature point is calculated. Through continuous image processing and position analysis, the continuous motion trajectory of each target feature point is obtained. The continuous motion trajectory of each feature point is correlated with the pose of the virtual instrument's 3D model to obtain the virtual instrument's pose data.

[0116] Please refer to Figure 11 The diagram above illustrates the camera coordinate system and the world coordinate system. As shown in the diagram: XcYcZc represents the camera coordinate system; XwYwZw represents the world coordinate system; and XY represents the imaging plane coordinate system. Based on the coordinate information of the target feature points on the main control arm as imaged in two directions, the position of the feature points in the three-dimensional coordinate system is calculated. Continuous image processing and position calculations yield the motion displacement / trajectory of the feature points. This is then correlated with the three-dimensional model of the virtual machine to obtain an animation effect where the virtual machine model follows the motion.

[0117] Please refer to Figure 12 This diagram illustrates the transformation between the world coordinate system and the camera coordinate system. The transformation from the world coordinate system to the camera coordinate system is a rigid body transformation, meaning the object does not deform; only rotation and translation are required. R represents the rotation matrix, and S represents the translation vector. The transformation formula is as follows:

[0118]

[0119] Where (x,y,z) are the coordinates in the camera coordinate system, (x... w ,y w ,z w (r) represents the coordinates in the world coordinate system. ij (i,j=1,2,3) are elements in the rotation matrix R, s k (k = x, y, z) are the elements in the translation vector S, and T is the matrix transpose.

[0120] Please refer to Figure 13 This diagram illustrates the virtual foot position and the display of foot movement images. (Example) Figure 13 As shown, using an image acquisition device, preferably binocular vision, depth information is utilized to acquire foot pedal position and foot movement information. The virtual foot pedal position is displayed in the stereo monitor to guide the doctor when switching pedals. The virtual foot movement is displayed in the stereo monitor, providing real-time visual feedback when the doctor moves their foot to switch pedals, preventing accidental pedaling. The calculation method for acquisition and display is as follows: using the center of the lower left foot pedal as the origin of the X, Y, Z coordinate system, the real-time displacement of the foot in the XYZ directions is acquired using a depth camera. This displacement is then displayed proportionally within a designated area on the stereo monitor.

[0121] Please refer to Figure 14 This diagram illustrates a warning for excessive deviation in the position tracking of surgical instruments. During operation, if the position of the instrument given the command deviates significantly from the position of the instrument being executed, a fault will be reported, and the corresponding fault location and text prompt will appear on the stereo monitor. Figure 14 As shown, areas with excessive deviation can be highlighted in red (marked in dark gray in the image). This enhanced display can intuitively and effectively pinpoint the fault location and provide solutions.

[0122] Please refer to Figure 15 This shows a schematic diagram indicating the location of the foot pedal panel. (For example...) Figure 15As shown, during operation, the foot pedal position and foot movement are displayed in real time. This aims to provide visual feedback to the operator during pedal switching, reducing the probability of errors. It provides both a clear indication of the pedal's effective depress position and a real-time display of the current foot's position, allowing for mutual correction and error prevention. Real-time display of foot movements during operation allows the doctor to confirm the intended pedal movement direction when the operator wants to switch pedals. When the pedal is pressed, color highlighting indicates whether the pedal function has been effectively activated. Both of these features assist doctors in ensuring error-free pedal switching. Please refer to [reference needed]. Figure 16 This diagram illustrates the display of foot movement and effective pedaling. Figure 16 As shown, the foot pedal is being effectively stepped on.

[0123] Please refer to Figure 17 The diagram illustrates the operational flow of an endoscopic surgical robot. Figure 17 As shown, the main process includes: Operation: The doctor uses the main hand and foot pedals on the platform to control the instruments on the surgical platform; Monitoring: The doctor uses an image acquisition device on the platform to acquire information on the main hand and foot pedal movements, virtually recreating the position of the controlled instruments. This position is then cross-corrected with the actual control commands. An alarm is triggered when the deviation is too large; Display: The endoscopic image is displayed on a stereo monitor, and the virtual controlled instruments, foot pedal positions, and foot movements are also displayed in real time on the monitor as real-time feedback information for the doctor's operation.

[0124] Please refer to Figure 18 This illustrates a flowchart of the virtual command device acquisition and display process. For example... Figure 18 As shown, the main process includes: displaying the virtual command instrument based on image acquisition in a stereo monitor; when a fault with excessive deviation occurs, displaying the faulty part with color highlighting and flashing prompts.

[0125] Please refer to Figure 19 This illustrates a flowchart of the virtual foot pedal position and foot movement display. For example... Figure 19 As shown, the main process includes: acquiring an image of the foot position based on the image acquisition device and displaying it in a stereo monitor, confirming that the foot is effectively highlighted; acquiring a real-time image of the foot's movement based on the image acquisition device and displaying it in a stereo monitor, providing visual feedback on the target's orientation and movement.

[0126] In the above scheme, an image acquisition module is installed on the doctor's control console to monitor the main hand and foot pedals. Through the acquired image information, the movement postures of the main hand and feet are identified. Then, based on the control algorithm's control commands for the instruments, VR display technology is used to virtually visualize and display the instrument's posture on the doctor's stereoscopic monitor. Based on the image acquisition device, VR technology is used to virtually display the foot pedal position and foot movements on the stereoscopic monitor to assist the doctor's operation. By adding image acquisition device monitoring the operator's movement posture and foot pedal position, and using VR technology to display and provide operational prompts in real time on the doctor's stereoscopic monitor, the doctor's operation is facilitated. For example, the foot pedal position and foot movements are displayed in real time on the doctor's stereoscopic monitor, allowing the doctor to clearly see the foot movement process and which foot needs to move to which foot pedal next. This can avoid misoperation, improve surgical efficiency, and the prompts during the operation can assist the doctor's operation.

[0127] As can be seen from the above description, the embodiments of this specification achieve the following technical effects: First, the posture data of the main control arm is determined based on the image of the main control arm. Since there is a master-slave relationship between the main control arm and the surgical instruments, virtual posture data of the surgical instruments can be generated based on the posture data of the main control arm and the three-dimensional model of the surgical instruments. Then, a virtual command instrument image is displayed on a stereoscopic monitor based on the virtual posture data of the surgical instruments. By displaying the virtual command instrument image on a stereoscopic monitor, doctors can more easily detect misoperations and instrument malfunctions in a timely manner, assisting doctors in their operations, improving surgical efficiency, surgical quality and success rate, and enhancing the surgical experience for both doctors and patients.

[0128] This specification also provides a medical device, which can be found in the following description. Figure 20 The diagram shown illustrates the structural composition of a medical device for displaying virtual surgical instruments on a doctor's console, based on an embodiment of this specification. Specifically, the medical device may include an input device 21, a processor 22, and a memory 23. The memory 23 stores processor-executable instructions. When the processor 22 executes these instructions, it implements the steps of the method for displaying virtual surgical instruments on a doctor's console as described in any of the above embodiments.

[0129] This specification also provides an endoscopic medical robot, including the doctor's console described in any of the above embodiments.

[0130] In this embodiment, the input device can specifically be one of the main devices for information exchange between the user and the computer system. The input device may include a keyboard, mouse, camera, scanner, light pen, handwriting input tablet, voice input device, etc.; the input device is used to input raw data and programs for processing these data into the computer. The input device can also receive data transmitted from other modules, units, and devices. The processor can be implemented in any suitable manner. For example, the processor can take the form of a microprocessor or processor and a computer-readable medium storing computer-readable program code (e.g., software or firmware) executable by the (micro)processor, logic gates, switches, application-specific integrated circuits (ASICs), programmable logic controllers, and embedded microcontrollers, etc. The memory can specifically be a memory device used to store information in modern information technology. The memory can include multiple layers; in digital systems, anything that can store binary data can be considered memory; in integrated circuits, a circuit without physical form but with storage function is also called memory, such as RAM, FIFO, etc.; in a system, a storage device with physical form is also called memory, such as a memory stick, TF card, etc.

[0131] In this embodiment, the specific functions and effects of the medical device can be explained by comparison with other embodiments, and will not be repeated here.

[0132] This specification also provides a computer storage medium based on a method for displaying virtual surgical instruments on a doctor's console, the computer storage medium storing computer program instructions that, when executed, implement the steps of the method for displaying virtual surgical instruments on a doctor's console described in any of the above embodiments.

[0133] In this embodiment, the storage medium includes, but is not limited to, Random Access Memory (RAM), Read-Only Memory (ROM), cache, hard disk drive (HDD), or memory card. The memory can be used to store computer program instructions. The network communication unit can be an interface configured according to standards specified in the communication protocol for network connection communication.

[0134] In this embodiment, the specific functions and effects implemented by the program instructions stored in the computer storage medium can be explained by comparison with other embodiments, and will not be repeated here.

[0135] Obviously, those skilled in the art will understand that the modules or steps of the embodiments described above can be implemented using general-purpose computing devices. They can be centralized on a single computing device or distributed across a network of multiple computing devices. Optionally, they can be implemented using computer-executable program code, thereby storing them in a storage device for execution by a computing device. In some cases, the steps shown or described can be performed in a different order than those presented herein, or they can be fabricated as separate integrated circuit modules, or multiple modules or steps can be fabricated as a single integrated circuit module. Thus, the embodiments of this specification are not limited to any particular combination of hardware and software.

[0136] It should be understood that the above description is for illustrative purposes and not for limitation. Many embodiments and applications beyond the provided examples will be apparent to those skilled in the art upon reading the above description. Therefore, the scope of this specification should not be determined by reference to the above description, but rather by reference to the foregoing claims and the full scope of their equivalents.

[0137] The above description is merely a preferred embodiment of this specification and is not intended to limit this specification. Various modifications and variations can be made to the embodiments described herein by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this specification should be included within the scope of protection of this specification.

Claims

1. A computer-readable storage medium storing computer instructions thereon, characterized in that, When the instructions are executed by the processor, they implement the steps of a method for displaying virtual surgical instruments on a physician's console, the method comprising: Acquire an image of the main control arm of the doctor's console; Based on the image of the main control arm, determine the attitude data of the main control arm; Based on the posture data of the main control arm and the three-dimensional model of the surgical instrument, virtual posture data of the surgical instrument is generated to display virtual command instrument images on the stereoscopic monitor of the doctor's console; the three-dimensional model of the surgical instrument is a pre-established model data of the surgical instrument in a three-dimensional coordinate system; the virtual posture data of the surgical instrument refers to the posture data of the surgical instrument under the operation of the main control arm. The process of generating virtual posture data of the surgical instrument based on the posture data of the main control arm and the three-dimensional model of the surgical instrument includes: acquiring a target machine learning model; the target machine learning model being pre-established based on the three-dimensional model of the surgical instrument; and inputting the posture data of the main control arm into the target machine learning model to obtain the virtual posture data of the surgical instrument.

2. The computer-readable storage medium according to claim 1, characterized in that, The method further includes: Acquire images of the foot pedals on the doctor's console and / or images of the operator's feet; Generate foot position data based on the foot image, and / or generate foot position data based on the foot image, to display the foot area and / or foot area in the stereoscopic monitor.

3. The computer-readable storage medium according to claim 2, characterized in that, The method further includes: When one or more foot pedals on the doctor's console are effectively pressed, the one or more foot pedals that are effectively pressed are highlighted in the stereoscopic monitor.

4. The computer-readable storage medium according to claim 1, characterized in that, The images of the main control arm include the left and right eye images of the main control arm captured by the binocular cameras.

5. The computer-readable storage medium according to claim 4, characterized in that, Based on the image of the main control arm, determine the attitude data of the main control arm, including: Obtain the spatial position data of the left and right cameras in the binocular camera system; Based on the spatial location data and the position parameters of each feature point in the left and right eye images of the main control arm, the spatial location data corresponding to each feature point is determined. The attitude data of the main control arm is determined based on the spatial position data of each feature point at multiple time points.

6. The computer-readable storage medium according to claim 1, characterized in that, Based on the posture data of the main control arm and the three-dimensional model of the surgical instrument, virtual posture data of the surgical instrument is generated, including: Based on the attitude data of the main control arm, control command information for the main control arm is generated; The control command information of the main control arm is applied to the three-dimensional model of the surgical instrument to generate virtual posture data of the surgical instrument.

7. The computer-readable storage medium according to claim 1, characterized in that, Before obtaining the target machine learning model, the method further includes: A machine learning model is constructed based on the three-dimensional model of the surgical instrument; Acquire the posture data of the control arm and the corresponding posture data of the surgical instruments during the endoscopic surgery, and construct a training sample set based on the posture data of the control arm and the corresponding posture data of the surgical instruments during the endoscopic surgery. The machine learning model is trained using the constructed training sample set to obtain the target machine learning model.

8. The computer-readable storage medium according to claim 1, characterized in that, The stereoscopic monitor also displays actual instrument images of surgical instruments captured by the endoscope; After displaying the virtual command instrument image on the stereoscopic monitor of the doctor's console, the method further includes: Compare the actual instrument image with the virtual command instrument image; Based on the comparison results, the target portion in the virtual command instrument image that differs from the actual instrument image by more than a preset range is identified, and the target portion is marked and displayed on the stereoscopic monitor.

9. The computer-readable storage medium according to claim 8, characterized in that, After identifying a target portion in the virtual command instrument image where the difference between the virtual command instrument image and the actual instrument image exceeds a preset range, the method further includes: Determine the positional data of the target portion within the surgical instrument; Early warning information is generated based on the location data, and the operator is alerted according to the early warning information in a preset manner.

10. A doctor's control console, characterized in that, include: Image acquisition device, used to acquire images of the main control arm of the doctor's console; An image processing device is used to determine the posture data of the main control arm based on an image of the main control arm; and to generate virtual posture data of the surgical instrument based on the posture data of the main control arm and a three-dimensional model of the surgical instrument; the three-dimensional model of the surgical instrument is a pre-established model data of the surgical instrument in a three-dimensional coordinate system; the virtual posture data of the surgical instrument refers to the posture data of the surgical instrument under the operation of the main control arm; wherein, the image processing device is specifically used to: acquire a target machine learning model; the target machine learning model is pre-established based on the three-dimensional model of the surgical instrument; input the posture data of the main control arm into the target machine learning model to obtain the virtual posture data of the surgical instrument; A stereo monitor is used to display images of virtual command instruments based on the virtual posture data.

11. The doctor's console according to claim 10, characterized in that, The image acquisition device is also used to acquire foot pedal images of the doctor's console and images of the operator's feet; the image processing device is also used to generate foot pedal position data based on the foot pedal images, and / or generate foot position data based on the foot images; the stereo monitor is also used to display the foot pedal area based on the foot pedal position data, and / or display the foot area based on the foot position data.

12. A medical device, characterized in that, It includes a processor and a memory for storing processor-executable instructions, wherein the processor, when executing the instructions, implements the steps of the method according to any one of claims 1 to 9.

13. An endoscopic medical robot, characterized in that, Includes the doctor console as described in claim 10 or 11.

Images