Surgical instrument manipulation system and control method for a surgical instrument manipulation system

Abstract

The application relates to a surgical instrument control system and a control method thereof. The surgical instrument control system comprises a plurality of operation consoles, each of which comprises a display module and a control module, and each of the control modules is connected with a to-be-operated instrument; the control module of the master operation console is used for controlling the to-be-operated instrument to move according to a received first driving control signal, fusing a scope image and a preset virtual image to obtain a target image, and transmitting the target image to the display module of the master operation console for display. The control module of the slave operation console is used for controlling a virtual instrument to move according to a received second driving control signal, generating the virtual instrument in the target image to obtain a virtual target image, and transmitting the target image or the virtual target image to the display module of the slave operation console for display. Therefore, an operator of the slave operation console can intuitively observe the operation process of an operator of the master operation console, and can carry out simulation practice, thereby improving the learning efficiency.

Description

Technical Field

[0001] This application relates to the field of medical device technology, and in particular to a surgical instrument control system and a control method for the surgical instrument control system. Background Technology

[0002] With the development of medical technology, minimally invasive surgery is becoming increasingly common. Compared with traditional surgery, minimally invasive surgery has advantages such as smaller incisions, fewer postoperative complications, and faster recovery, making it highly favored by patients. However, due to the small incisions and inconvenient operation, performing minimally invasive surgeries involving numerous internal sutures using traditional laparoscopic techniques is quite difficult, requiring surgeons to have extensive experience in open surgery and proficient in traditional laparoscopic techniques. Therefore, minimally invasive robotic surgical systems have been introduced. Surgeons can precisely control robotic arms to perform minimally invasive surgery with higher accuracy and better surgical outcomes, meeting the needs for minimally invasive and refined surgical procedures. However, minimally invasive robotic surgical systems are currently high-end medical devices, and doctors in many underdeveloped areas lack access to them, resulting in a severe shortage of surgeons skilled in operating such systems.

[0003] In traditional techniques, training for robotic laparoscopic surgery involves instructors explaining images or videos to train trainees.

[0004] However, training using traditional techniques is neither intuitive nor efficient. Summary of the Invention

[0005] Therefore, it is necessary to provide a surgical instrument control system and a control method for the surgical instrument control system that can facilitate the teaching of robotic laparoscopic surgery to trainees, in order to address the above-mentioned technical problems.

[0006] A surgical instrument control system includes: multiple operation consoles, each operation console including a display module and a control module, each control module being connected to an instrument to be operated; the control module of the main operation console is used to control the movement of the instrument to be operated according to a received first drive control signal, and to obtain a target image by fusing an endoscope image and a preset virtual image, and to transmit the target image to the display module of the main operation console for display; wherein, the endoscope image is a real-time image of the instrument to be operated and a preset range around the instrument, and the preset virtual image includes all feature elements in the endoscope image and virtual simulation images corresponding to the elements around the feature elements; the control module of the secondary operation console is used to control the movement of a virtual instrument according to a received second drive control signal, and to obtain the target image by fusing the endoscope image and the preset virtual image, and to generate a virtual instrument in the target image to obtain a virtual target image, and to transmit the target image or the virtual target image to the display module of the secondary operation console for display; wherein, the main operation console is one of the multiple operation consoles, and the operation consoles other than the main operation console are the secondary operation consoles.

[0007] In one embodiment, the operation consoles are connected via a bus; the master operation console is further configured to receive a switching instruction, switch to a new slave operation console according to the switching instruction, and send the switching instruction to a target slave operation console via the bus to instruct the target slave operation console to switch to a new master operation console; or, the target slave operation console is further configured to receive a switching instruction, switch to a new master operation console according to the switching instruction, and send the switching instruction to the master operation console via the bus to instruct the master operation console to switch to a new slave operation console; wherein, the target slave operation console is one of at least one slave operation console.

[0008] In one embodiment, each of the operation consoles further includes an input module, the input module including an operation terminal, the input module being connected to the control module, and the input module being used to send the drive control signal to the control module according to the posture of the operation terminal.

[0009] In one embodiment, the control module of the current master control console transmits the attitude of the operating terminal to the control module of the new master control console via the bus, so as to adjust the attitude of the operating terminal of the new master control console to a target attitude, wherein the target attitude is the same as the attitude of the operating terminal of the current master control console.

[0010] In one embodiment, the control module of the main operating console is used to determine the end position of the instrument to be operated based on the endoscope image, and to establish a mapping relationship between the first driving control signal, the end position of the instrument to be operated, and the position of the instrument to be operated in the target image based on the first driving control signal, the end position of the instrument to be operated, and the position of the instrument to be operated in the target image; each control module of the slave operating console is used to import the mapping relationship, and to control the virtual instrument and generate the virtual instrument in the target image based on the mapping relationship and the second driving control signal.

[0011] In one embodiment, the control module is used to acquire the endoscope image and medical image, obtain the preset virtual image based on the medical image, the preset virtual image including a three-dimensional model constructed from all feature elements in the endoscope image and elements surrounding the feature elements, and is also used to determine the instrument to be operated and three-dimensional point cloud data within a preset range around the instrument to be operated based on the endoscope image, and fuse the three-dimensional model and the three-dimensional point cloud data to obtain the target image.

[0012] In one embodiment, the control module is further configured to mark multiple feature locations in the three-dimensional model, determine feature point cloud data corresponding to at least one of the multiple feature locations in the three-dimensional point cloud data, and register and fuse the three-dimensional point cloud data with the three-dimensional model based on the feature point cloud data to obtain the target image.

[0013] In one embodiment, each of the operation consoles includes an alarm module connected to the control module, used to issue a prompt signal when the instrument to be operated or the virtual instrument moves to a sensitive location, wherein the sensitive location is the location of the sensitive tissue.

[0014] In one embodiment, each of the operation consoles includes: a recording module connected to the control module, used to record operation videos of the instrument to be operated and / or the virtual instrument; the display module is also used to play the operation videos according to playback instructions.

[0015] In one embodiment, each of the operation consoles includes: a scoring module connected to the control module, used to acquire a first operation video recording of operating the device to be operated and a second operation video recording of operating the virtual device, and compare the first operation video recording with the second operation video recording to determine the evaluation result of operating the virtual device.

[0016] A control method for a surgical instrument control system, the surgical instrument control system including multiple operation consoles, each operation console including a display module and a control module, each control module being connected to an instrument to be operated, the method comprising:

[0017] The movement of the instrument to be operated is controlled by the control module of the main operation console according to the received first drive control signal;

[0018] The target image is obtained by fusing the endoscope image and the preset virtual image, and the target image is transmitted to the display module of the main operation console for display; wherein, the endoscope image is a real-time image of the instrument to be operated and the preset range around the instrument, and the preset virtual image includes all feature elements in the endoscope image and the virtual simulation images corresponding to the elements around the feature elements;

[0019] The virtual machine is controlled to move via the control module from the operation console based on the received second drive control signal.

[0020] The target image is obtained by fusing the endoscope image and the preset virtual image, and a virtual instrument is generated in the target image to obtain a virtual target image. The target image or the virtual target image is then transmitted to the display module of the operation console for display.

[0021] The main operation console is one of the plurality of operation consoles, and the operation consoles other than the main operation console are the slave operation consoles.

[0022] The aforementioned surgical instrument control system and its control method utilize multiple operating consoles. Each console includes a display module and a control module, with each control module connected to the instrument to be operated on. Thus, each operating console can control the instrument. The control module of the main operating console controls the movement of the instrument based on a received first drive control signal, making the main operating console the control unit for the instrument. Furthermore, the control module of the main operating console can fuse the endoscopic image and a preset virtual image to obtain a target image, which is then transmitted to the display module of the main operating console for display. This allows the operator to view the surgical procedure on the main operating console's display module, facilitating the operation and control of the instruments. Furthermore, the endoscopic image is a real-time image of the instrument to be operated on and its surrounding preset range. The preset virtual image includes all feature elements in the endoscopic image and the corresponding virtual simulation images of the elements around those feature elements. Therefore, the image presented in the display module not only includes the real image visible through the endoscopic lens but also the virtual simulation image near the endoscopic image. This provides the operator with a wider field of view, allowing them to see the entire surgical area while operating the instrument, avoiding accidental contact with sensitive tissues outside the endoscopic image's field of view, thus improving surgical safety. The control module from the operating console controls the movement of the virtual instrument based on the received second drive control signal, and fuses the endoscopic image and preset virtual image to obtain a target image. Then, a virtual instrument is generated within the target image to obtain a virtual target image, which is then transmitted to the display module from the operating console for display. Both the main control console and the secondary control console display modules can present the same target image. The difference lies in that the secondary control console display module can also display a virtual target image that includes virtual instruments within the target image. Furthermore, the secondary control console can control the virtual instruments to perform actions, allowing the operator on the secondary control console to intuitively observe the operation process of the operator on the main control console. The operator can also practice using the virtual instruments, enabling intuitive and efficient learning of the main control console operator's techniques, thus improving learning efficiency. In summary, the device of this application, on the one hand, allows the operator on the secondary control console to intuitively observe the operation process of the operator on the main control console and to practice using simulations, improving learning efficiency; on the other hand, it can display the endoscope image and a virtual simulation image near the endoscope image on the display module, providing a wider surgical field of view and thus improving surgical safety. Attached Figure Description

[0023] To more clearly illustrate the technical solutions in the embodiments of this application or the conventional technology, the drawings used in the description of the embodiments or the conventional technology will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0024] Figure 1 This is a schematic diagram of the surgical instrument control system in one embodiment;

[0025] Figure 2 This is a schematic diagram of a surgical instrument control system in one embodiment;

[0026] Figure 3 This is a schematic diagram of an endoscopic imaging system in one embodiment;

[0027] Figure 4 This is a schematic diagram of a scanning imaging system in one embodiment;

[0028] Figure 5 This is a schematic diagram of a device entering the body in one embodiment;

[0029] Figure 6 This is a schematic diagram of the surgical instrument control system in another embodiment;

[0030] Figure 7 This is a schematic diagram of a surgical instrument control system in another embodiment;

[0031] Figure 8 This is a schematic diagram of the surgical instrument control system in yet another embodiment;

[0032] Figure 9 This is a schematic diagram of the operation console in one embodiment;

[0033] Figure 10 This is a schematic diagram of the surgical instrument control system in yet another embodiment;

[0034] Figure 11 This is a flowchart of the main operation console in recording mode in one embodiment;

[0035] Figure 12 This is a flowchart of an embodiment of the operation console in learning mode;

[0036] Figure 13 This is a flowchart of a control method for a surgical instrument control system in one embodiment. Detailed Implementation

[0037] To facilitate understanding of this application, a more complete description will be provided below with reference to the accompanying drawings, which illustrate embodiments of the present application. However, the present application can be implemented in many different forms and is not limited to the embodiments described herein. Rather, these embodiments are provided so that the disclosure of this application will be thorough and complete.

[0038] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the application.

[0039] It is understood that the terms “first,” “second,” etc., used in this application may be used herein to describe various elements, but these elements are not limited by these terms. These terms are only used to distinguish one element from another.

[0040] It should be noted that when one element is considered to be "connected" to another element, it can be directly connected to the other element or connected to the other element through an intermediary element. Furthermore, in the following embodiments, "connection" should be understood as "electrical connection," "communication connection," etc., if there is transmission of electrical signals or data between the connected objects.

[0041] When used herein, the singular forms of “a,” “an,” and “the” may also include the plural forms unless the context clearly indicates otherwise. It should also be understood that the terms “comprising,” “including,” or “having,” etc., specify the presence of the stated feature, whole, step, operation, component, part, or combination thereof, but do not preclude the possibility of the presence or addition of one or more other features, wholes, steps, operations, components, parts, or combinations thereof.

[0042] In one embodiment, such as Figure 1 As shown, a surgical instrument control system is provided, including: multiple operation consoles 100, each operation console 100 including a display module 11 and a control module 10, and each control module 10 is connected to the instrument 200 to be operated.

[0043] The control module 10 of the main control console 100 is used to control the movement of the instrument to be operated 200 according to the received first drive control signal, and to obtain a target image by fusing the endoscope image and a preset virtual image, and then transmit the target image to the display module 11 of the main control console 100 for display. The endoscope image is a real-time image of the instrument to be operated 200 and a preset range around the instrument to be operated 200, and the preset virtual image includes all feature elements in the endoscope image and the virtual simulation images corresponding to the elements surrounding the feature elements.

[0044] For example, such as Figure 2 As shown, the surgical instrument control system includes an operating console 100, a robotic arm 20, an image carriage 21, and a function carriage 22. The surgeon controls the robotic arm 20 and the image carriage 21 and function carriage 22 via the operating console 100. The surgical instruments 200 and an endoscope can be mounted on the robotic arm 20, allowing the surgeon to control the robotic arm 20's movements and thus move the surgical instruments 200. The endoscope can directly acquire endoscopic images. The surgeon can control the robotic arm 20's movements by moving the operating arm on the operating console 100. The operating arm also receives force information from human tissues and organs on the instruments and feeds it back to the surgeon's hand, allowing for a more intuitive understanding of the surgical procedure. The display module 11 on the operating console 100 is communicatively connected to the endoscope, enabling direct acquisition of endoscopic images.

[0045] Specifically, the endoscopic image is a real-time image of the instrument to be operated on 200 and a preset range around the instrument to be operated on 200, acquired directly by the endoscope, thus presenting a real-time surgical scene to facilitate the doctor's operation. The preset virtual image is a three-dimensional model created based on image data of the patient's target area obtained through computed tomography (CT) or magnetic resonance imaging (MRI).

[0046] For example, such as Figure 3 As shown, the end of the robotic arm 20 includes multiple manipulators, one of which is equipped with an endoscope 26 and another with an instrument to be operated on. Both the instrument to be operated on and the endoscope 26 can be inserted into the patient's body, and the endoscopic images captured by the endoscope can be directly displayed through the image trolley 21.

[0047] For example, such as Figure 4 As shown, the scanning imaging device 23 is used to scan the patient 25, thereby obtaining image information of the target part of the patient 25. The image information is displayed on the display screen of the imaging carriage 24. Then, based on the image information, three-dimensional modeling can be performed to obtain a three-dimensional model of the target part of the patient 25, that is, a preset virtual image is obtained.

[0048] The control module 10 of the operation console 100 is used to control the movement of the virtual instrument according to the received second drive control signal, and to obtain a target image by fusing the endoscope image and the preset virtual image, and to generate a virtual instrument in the target image to obtain a virtual target image, and to transmit the target image or virtual target image to the display module 11 of the operation console 100 for display.

[0049] Specifically, the control module 10 of the operation console 100 can generate a virtual device in the target image using augmented reality (AR) to obtain a virtual target image. The display module 11 of the operation console 100 can choose to display the target image or the virtual target image. When displaying the target image, the operator of the operation console 100 can only watch the operator of the main operation console 100 operate the device 200. When displaying the virtual target image, the operator of the operation console 100 can not only watch the operator of the main operation console 100 operate the device 200, but also control the virtual device to perform simulated practice.

[0050] For example, such as Figure 5 As shown, after the endoscope 26 enters the human body, the field of view (endoscopic image) is the endoscope field of view 300 in the figure, while the field of view of the target image is the target image field of view 400 in the figure. Thus, the doctor can directly observe the real picture within the endoscope field of view 300 through the endoscope, and at the same time, can see the simulated picture within the target image field of view 400 outside the endoscope field of view 300 through the target image. The instrument to be operated 200 is a real instrument that enters the human body, while the virtual instrument 27 is an instrument generated in the image through AR. The actions that the virtual instrument 27 can perform are completely consistent with those of the instrument to be operated 200, such as instrument translation, instrument rotation, instrument pitch, instrument yaw, instrument opening and closing, etc.

[0051] The master operation console 100 is one of multiple operation consoles 100, and the other operation consoles 100 are slave operation consoles 100. Only one master operation console 100 can be used at any given time, thus avoiding confusion in the control of the instrument 200 that would result from multiple operation consoles 100 controlling it simultaneously.

[0052] In this embodiment, multiple operation consoles 100 are provided. Each operation console 100 includes a display module 11 and a control module 10. Each control module 10 is connected to the instrument to be operated on 200, so that each operation console 100 can control the instrument to be operated on 200. The control module 10 of the main operation console 100 can control the movement of the instrument to be operated on 200 according to the received first drive control signal. Thus, the main operation console 100 is the console for controlling the instrument to be operated on 200. The control module 10 of the main operation console 100 can also obtain a target image by fusing the endoscope image and a preset virtual image, and transmit the target image to the display module 11 of the main operation console 100 for display. Thus, the surgical scene can be displayed on the display module 11 of the main operation console 100, which is convenient for the operator to control the instrument to be operated on 200 to perform surgery. Furthermore, the endoscope image is a real-time image of the instrument to be operated on 200 and the preset range surrounding the instrument 200. The preset virtual image includes all feature elements in the endoscope image and the virtual simulation images corresponding to the elements around the feature elements. Therefore, the image presented in the display module 11 not only includes the real image that can be seen by the endoscope, but also includes the virtual simulation image near the endoscope image. This allows the operator to have a wider field of view, enabling the operator to see the image within the surgical area completely when operating the instrument 200. This avoids accidental contact with sensitive tissues outside the field of view of the endoscope image, thereby improving the safety of the surgery. The control module 10 from the operation console 100 can control the movement of the virtual instrument according to the received second drive control signal, and obtain a target image by fusing the endoscope image and the preset virtual image. Then, a virtual instrument is generated in the target image to obtain a virtual target image, and the target image or virtual target image is transmitted to the display module 11 from the operation console 100 for display. Both the main control console 100 and the display module 11 of the secondary control console 100 can display the same target image. The difference is that the display module 11 of the secondary control console 100 can also display a virtual target image that includes a virtual instrument within the target image. Furthermore, the secondary control console 100 can control the virtual instrument to perform actions. This allows the operator of the secondary control console 100 to intuitively observe the operation process of the operator of the main control console 100 with the instrument 200, and to practice through simulation using the virtual instrument. This allows for intuitive and efficient learning of the operation methods used by the operator of the main control console 100, improving learning efficiency. In summary, the device of this application, on the one hand, allows the operator of the secondary control console 100 to intuitively observe the operation process of the operator of the main control console 100 with the instrument 200 and to practice through simulation, improving learning efficiency. On the other hand, it can display the endoscope image and the virtual simulation image near the endoscope image on the display module 11, providing a wider surgical field of view and thus improving surgical safety.

[0053] In one embodiment, such as Figure 6 As shown, each operation console 100 is connected via a bus. The master operation console 100 is also used to receive a switching instruction, switch to a new slave operation console 100 according to the switching instruction, and send the switching instruction to the target slave operation console 100 via the bus to instruct the target slave operation console 100 to switch to the new master operation console 100. Alternatively, the target slave operation console 100 is also used to receive a switching instruction, switch to a new master operation console 100 according to the switching instruction, and send the switching instruction to the master operation console 100 via the bus to instruct the master operation console 100 to switch to the new slave operation console 100. The target slave operation console 100 is one of at least one slave operation console 100.

[0054] Specifically, the various operation consoles 100 can be connected via network communication cables, fiber optic cables, or wireless signals. Each operation console 100 can act as a master operation console 100, and switching between the current master operation console 100 and any slave operation console 100 is possible. Alternatively, the master operation console 100 may receive a switching command, then switch to become the new slave operation console 100 according to the command, and send the switching command to the target slave operation console 100 via the bus. The target slave operation console 100 then switches to become the new master operation console 100 according to the command. Or, the target slave operation console 100 may receive a switching command, switch to become the new master operation console 100 according to the command, and send the switching command to the master operation console 100 via the bus. The master operation console 100 then switches to become the new slave operation console 100 according to the command.

[0055] For example, such as Figure 7 As shown, each operation console 100 is connected via a communication module 28, which controls the robotic arm 20, thereby driving the operation of the instrument 200 mounted on the robotic arm 20.

[0056] In this embodiment, the main operation console 100 and any slave operation console 100 can be switched, meaning that the control of the instrument to be operated 200 can be changed at any time. This allows operators of each operation console 100 to try to control the instrument to be operated 200 to perform surgery. In case of an accident, the operation can be switched to a senior instructor in a timely manner, which facilitates practical practice and ensures safety.

[0057] In one embodiment, such as Figure 8As shown, each operation console 100 also includes an input module 12, which includes an operation terminal 13. The input module 12 is connected to the control module 10 and is used to send drive control signals to the control module 10 according to the attitude of the operation terminal 13.

[0058] For example, such as Figure 9 As shown, the doctor can observe the images during surgery through the display module 11 on the control console 100, which facilitates operation. The doctor can send drive control signals by changing the posture of the operating end 13, such as adjusting the pitch angle of the operating end 13. Simultaneously, a feedback module is also provided within the control console 100, which can collect information on the force exerted by human tissues and organs on the instruments and feed it back to the operating end 13, transmitting it to the doctor's hands, allowing the doctor to more intuitively experience the surgical operation. The control console 100 also includes a pedal 14, the specific control functions of which can be determined according to actual needs.

[0059] For example, the operating end 13 includes redundant sensors. The drive control signal generated by the attitude change of the operating end 13 is processed by a forward kinematics algorithm to calculate the Cartesian pose contained in the drive control signal. Then, through a mapping relationship, the Cartesian pose of the device is determined. Then, the movement and rotation of the robotic arm are controlled according to the determined pose to adjust the pose of the device.

[0060] In this embodiment, by setting up an input module 12, the operator can send drive control signals to the control module 10 through the operation terminal 13 of the input module 12, thereby controlling the instrument.

[0061] In one embodiment, please see [link to embodiment]. Figure 7 and Figure 9 The control module 10 of the current master control console 100 transmits the attitude of the operation terminal 13 to the control module 10 of the new master control console 100 via the bus, so as to adjust the attitude of the operation terminal 13 of the new master control console 100 to the target attitude, wherein the target attitude is the same as the attitude of the operation terminal 13 of the current master control console 100.

[0062] Specifically, when switching between the current master control console 100 and the new master control console 100, in addition to transferring control of the instrument to be operated 200 to the new master control console 100, the posture of the operating terminal 13 of the new master control console 100 will also be adjusted to be the same as that of the current master control console 100.

[0063] For example, such as Figure 9As shown, the switching command between the current main control console 100 and the new main control console 100 can be issued through the pedal 14 on the current control console 100, through the operating terminal 13, or through an external interactive interface. During the switching process, the posture of the operating terminal 13 of the new main control console 100 will automatically match and follow the posture of the operating terminal 13 of the current main control console 100.

[0064] In this embodiment, when switching the main operation console 100, the posture of the operation terminal 13 is also matched and synchronized, so as to facilitate the operation of the new main operation console 100.

[0065] In one embodiment, please see [link to embodiment]. Figure 1 The control module 10 of the main operation console 100 is used to determine the end position of the instrument to be operated 200 according to the endoscope image, and to establish a mapping relationship between the first drive control signal, the end position of the instrument to be operated 200, and the position of the instrument to be operated 200 in the target image.

[0066] Specifically, after establishing the mapping relationship between the first drive control signal, the end position of the instrument to be operated 200, and the position of the instrument to be operated 200 in the target image, the end position of the instrument to be operated 200 and the position of the instrument to be operated 200 in the target image can be directly inferred based on the first drive control signal and the mapping relationship.

[0067] Each control module 10 of the operation console 100 is used to import mapping relationships, control the virtual device according to the mapping relationships and the second drive control signal, and generate the virtual device in the target image.

[0068] Specifically, by using the mapping relationship and the second drive control signal, the end position of the virtual device corresponding to the second drive control signal and the position of the virtual device in the target image can be obtained directly.

[0069] In this embodiment, a mapping relationship is first established between the first drive control signal, the end position of the instrument to be operated, and the position of the instrument 200 in the target image. This mapping relationship is then imported into the slave operation console 100, allowing the slave console 100 to control the virtual instrument in the same way, ensuring that the slave console 100's control of the virtual instrument is completely consistent with the master operation console 100's control of the instrument 200. Furthermore, the end position of the corresponding virtual instrument and its position in the target image can be inferred from the mapping relationship and the second drive control signal, facilitating the generation of a virtual instrument in the target image. The end position and position of this virtual instrument in the target image are consistent with the instrument 200 receiving the same drive control signal. This results in a more realistic virtual instrument, and the operator's manipulation of the virtual instrument from the slave operation console 100 is closer to that of a real instrument.

[0070] In one embodiment, the control module is used to acquire endoscope images and medical images, obtain a preset virtual image based on the medical images, the preset virtual image includes a three-dimensional model constructed from all feature elements in the endoscope image and elements surrounding the feature elements, and is also used to determine the instrument to be operated and three-dimensional point cloud data within a preset range around the instrument based on the endoscope image, and fuse the three-dimensional model and the three-dimensional point cloud data to obtain a target image.

[0071] Specifically, medical imaging refers to image data of the patient's target area obtained through computed tomography (CT) or magnetic resonance imaging (MRI). Then, three-dimensional finite element modeling is performed on the medical images to obtain a three-dimensional model of the patient's target area, i.e., a pre-defined virtual image. Based on the endoscopic images, a binocular vision-based three-dimensional reconstruction method can be used to determine the three-dimensional point cloud data of the instrument to be operated on and within a pre-defined range around the instrument.

[0072] In this embodiment, the three-dimensional model constructed from all feature elements in the endoscope image and the elements surrounding the feature elements is fused with the three-dimensional point cloud data of the instrument to be operated and the three-dimensional point cloud data of the instrument to be operated within a preset range determined from the endoscope image. That is, the feature elements in the three-dimensional point cloud data are matched with the same feature elements in the three-dimensional model, so as to achieve the fusion of the three-dimensional model and the three-dimensional point cloud data and obtain the target image.

[0073] In one embodiment, the control module is further configured to mark multiple feature locations in the 3D model, determine feature point cloud data corresponding to at least one of the multiple feature locations in the 3D point cloud data, and register and fuse the 3D point cloud data with the 3D model based on the feature point cloud data to obtain the target image.

[0074] In this embodiment, after the three-dimensional model is constructed, multiple feature positions are marked in the three-dimensional model. Then, feature point cloud data corresponding to at least one of the multiple feature positions is found in the three-dimensional point cloud data. The feature point cloud data is then registered with the corresponding feature position in the three-dimensional model, thereby integrating the three-dimensional point cloud data into the three-dimensional model as a whole. This achieves the registration and fusion of three-dimensional point cloud data and three-dimensional model to obtain the target image.

[0075] In one embodiment, such as Figure 10 As shown, each operation console 100 includes an alarm module 30. The alarm module 30 is connected to the control module 10 and is used to issue a prompt signal when the instrument to be operated 200 or the virtual instrument moves to a sensitive position, wherein the sensitive position is the location of the sensitive tissue.

[0076] Specifically, since the control module obtains the target image, which includes internal human tissue visible within the endoscopic field of view, as well as peripheral human tissue not visible within the endoscopic field of view, the control module can determine the location of all sensitive tissues within the target image field of view and mark these locations as sensitive locations. Sensitive tissues can include blood vessels, nerves, organs, etc. When the instrument to be operated 200 or the virtual instrument moves to a sensitive location, it may cause damage to the sensitive tissue. At this time, the alarm module 30 will issue a warning signal to alert the operator.

[0077] For example, the alarm module 30 can be an audible and visual alarm, which can alert the operator through sound or indicator lights, or it can provide enhanced prompts on the image displayed by the display module 11, such as marking sensitive locations with darker colors.

[0078] In this embodiment, by setting an alarm module, a prompt signal can be automatically issued to alert the operator when the instrument to be operated 200 or the virtual instrument moves to a sensitive position, so that the operator can be aware of the danger in time and avoid damage to sensitive tissues.

[0079] In one embodiment, please see [link to embodiment]. Figure 10 Each operation console 100 includes a recording module 32. The recording module 32 is connected to the control module 10 and is used to record the operation of the instrument 200 to be operated and / or the virtual instrument. The display module 11 is also used to play the operation recording according to the playback command.

[0080] For example, senior instructors and trainees use different operating consoles 100 to perform surgical operations. Senior instructors use the main operating console 100 to perform surgical operations and record the surgical operations through the video recording module 32 to obtain an operation video.

[0081] Trainees can view the operation recordings at any time. They can choose to follow the operation recordings in real time to control the virtual instruments from the control console to perform simulated surgical operations, or they can choose to watch the operation recordings first and then control the virtual instruments from the control console to perform simulated surgical operations. They can also choose to replay specific operation scenarios in the operation recordings.

[0082] For example, such as Figure 11 As shown, the main control console workflow in recording mode is as follows:

[0083] Step S1100: Perform the surgical procedure.

[0084] Step S1110: Record the surgical procedure to obtain a video recording.

[0085] like Figure 12 As shown, the process from the operation console is as follows:

[0086] Step S1200: Select the learning mode. If the selected learning mode is asynchronous, proceed to step S1210. If the selected learning mode is synchronous, proceed to step S1230.

[0087] Step S1210: Select the scene to be played in the operation recording.

[0088] Step S1220: Simulate the operation of virtual equipment based on the operation recording of the selected scenario.

[0089] Step S1230: Simulate the operation of the virtual device based on the operation video.

[0090] In this embodiment, by setting up a video recording module, the surgical procedure can be recorded for easy learning.

[0091] In one embodiment, please see [link to embodiment]. Figure 10 Each operation console includes a scoring module 31. The scoring module 31 is connected to the control module 10 and is used to acquire a first operation video recording of operating the instrument to be operated and a second operation video recording of operating the virtual instrument, and compare the first operation video recording with the second operation video recording to determine the evaluation result of operating the virtual instrument.

[0092] For example, experienced instructors and trainees simultaneously perform surgical procedures using different control consoles. The experienced instructor uses the main control console to perform the procedure and guides the trainee through the breakdown of surgical actions. The trainee observes the experienced instructor's real-time actions and performs simulated surgical procedures by controlling virtual instruments from the control console. The experienced instructor can explain the surgical requirements to the trainee in real time and guide the trainee through the simulation. The scoring module evaluates and scores the trainee's simulated surgical procedures using the virtual instruments, based on the experienced instructor's actions.

[0093] In this embodiment, by setting up a scoring module, scores can be given based on the comparison between the student's operation video of the virtual device and the instructor's operation video of the actual device, so that the student can intuitively understand their current level of proficiency.

[0094] In one embodiment, such as Figure 13 As shown, a control method for a surgical instrument control system is provided. The surgical instrument control system includes multiple operation consoles, each operation console including a display module and a control module. Each control module is connected to the instrument to be operated. The method includes:

[0095] Step S1300: Based on the received first drive control signal, control the movement of the instrument to be operated through the control module of the main operation console.

[0096] Step S1310: A target image is obtained by fusing the endoscope image and a preset virtual image, and the target image is transmitted to the display module of the main operation console for display. The endoscope image is a real-time image of the instrument to be operated and its surrounding preset range. The preset virtual image includes all feature elements in the endoscope image and the corresponding virtual simulation images of the elements surrounding those feature elements.

[0097] Step S1320: Control the movement of the virtual device through the control module from the operation console according to the received second drive control signal.

[0098] Step S1330: A target image is obtained by fusing the endoscope image and a preset virtual image, and a virtual instrument is generated in the target image to obtain a virtual target image. The target image or virtual target image is then transmitted to the display module of the operation console for display.

[0099] The main operation console is one of multiple operation consoles, and the operation consoles other than the main operation console are slave operation consoles.

[0100] In this embodiment, multiple operation consoles are set up, each including a display module and a control module. Each control module is connected to the instrument to be operated on, so each operation console can control the instrument. Based on the received first drive control signal, the control module of the main operation console can control the movement of the instrument. Thus, the main operation console is the control console for controlling the instrument. The control module of the main operation console can also fuse the endoscope image and a preset virtual image to obtain a target image, and transmit the target image to the display module of the main operation console for display. This allows the surgical scene to be displayed on the main operation console's display module, facilitating the operator's control of the instrument during surgery. Furthermore, the endoscopic image is a real-time image of the instrument to be operated on and its surrounding preset range. The preset virtual image includes all feature elements in the endoscopic image and the corresponding virtual simulation images of the elements around those feature elements. Therefore, the image presented in the display module not only includes the real image visible through the endoscopic lens but also the virtual simulation image near the endoscopic image. This provides the operator with a wider field of view, allowing them to see the entire surgical area while operating the instrument, avoiding accidental contact with sensitive tissues outside the endoscopic image's field of view, thus improving surgical safety. Based on the received second drive control signal, the virtual instrument's movement can be controlled via the control module on the operating console. A target image is obtained by fusing the endoscopic image and the preset virtual image, and then a virtual instrument is generated within the target image to obtain a virtual target image. The target image or virtual target image is then transmitted to the display module on the operating console for display. Both the main control console and the secondary control console display modules can present the same target image. The difference lies in that the secondary control console display module can also display a virtual target image that includes virtual instruments within the target image. Furthermore, the secondary control console can control the virtual instruments to perform actions, allowing the operator on the secondary control console to intuitively observe the operation process of the operator on the main control console. The operator can also practice using the virtual instruments, enabling intuitive and efficient learning of the main control console operator's techniques, thus improving learning efficiency. In summary, the device of this application, on the one hand, allows the operator on the secondary control console to intuitively observe the operation process of the operator on the main control console and to practice using simulations, improving learning efficiency; on the other hand, it can display the endoscope image and a virtual simulation image near the endoscope image on the display module, providing a wider surgical field of view and thus improving surgical safety.

[0101] It should be understood that, although Figure 11 , 12The steps in flowchart 13 are shown sequentially as indicated by the arrows; however, these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order in which these steps are performed, and they can be executed in other orders. Figure 11 , 12 At least some of the steps in 13 may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but may be executed at different times. The execution order of these steps or stages is not necessarily sequential, but may be executed in turn or alternately with other steps or at least some of the steps or stages in other steps.

[0102] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the methods described above. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, or optical storage, etc. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM), etc.

[0103] In the description of this specification, references to terms such as "some embodiments," "other embodiments," and "ideal embodiments" indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the invention. In this specification, the illustrative descriptions of the above terms do not necessarily refer to the same embodiments or examples.

[0104] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0105] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the invention patent. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this patent application should be determined by the appended claims.

Claims

1. A surgical instrument control system, characterized in that, include: Multiple operation consoles, each operation console including a display module and a control module, each control module being connected to the instrument to be operated; The control module of the main operating console is used to control the movement of the instrument to be operated according to the received first drive control signal, and to obtain a target image by fusing the endoscope image and a preset virtual image, and to transmit the target image to the display module of the main operating console for display; wherein, the endoscope image is a real-time image of the instrument to be operated and the preset range around the instrument to be operated, and the preset virtual image includes all feature elements in the endoscope image and the virtual simulation images corresponding to the elements around the feature elements; The control module of the secondary operation console controls the movement of the virtual instrument according to the received second drive control signal, and obtains the target image by fusing the endoscope image and the preset virtual image, and generates a virtual instrument in the target image to obtain a virtual target image, and transmits the target image or the virtual target image to the display module of the secondary operation console for display; wherein, when the display module of the secondary operation console displays the target image, the secondary operation console can only view the operation of the instrument to be operated by the main operation console; when the display module of the secondary operation console displays the virtual target image, the secondary operation console controls the virtual instrument to perform simulated operation according to the second drive control signal; The main operation console is one of the plurality of operation consoles, and the operation consoles other than the main operation console are the slave operation consoles.

2. The surgical instrument control system according to claim 1, characterized in that, The aforementioned operation consoles are connected via a bus; The master operation console is also used to receive a switching instruction, switch to a new slave operation console according to the switching instruction, and send the switching instruction to the target slave operation console via the bus to instruct the target slave operation console to switch to the new master operation console; Alternatively, the target slave operation console is also used to receive a switching instruction, switch to a new master operation console according to the switching instruction, and send the switching instruction to the master operation console via the bus to instruct the master operation console to switch to a new slave operation console; wherein, the target slave operation console is one of at least one slave operation console.

3. The surgical instrument control system according to claim 2, characterized in that, Each of the aforementioned operation consoles further includes an input module, the input module including an operation terminal, the input module being connected to the control module, and the input module being used to send the drive control signal to the control module according to the posture of the operation terminal.

4. The surgical instrument control system according to claim 3, characterized in that, The control module of the current main control console transmits the attitude of the operating terminal to the control module of the new main control console via the bus, so as to adjust the attitude of the operating terminal of the new main control console to the target attitude, wherein the target attitude is the same as the attitude of the operating terminal of the current main control console.

5. The surgical instrument control system according to claim 3, characterized in that, The control module of the main operation console is used to determine the end position of the instrument to be operated based on the endoscope image, and to establish a mapping relationship between the first drive control signal, the end position of the instrument to be operated, and the position of the instrument to be operated in the target image based on the first drive control signal, the end position of the instrument to be operated, and the position of the instrument to be operated in the target image. Each control module from the operation console is used to import the mapping relationship, control the virtual device according to the mapping relationship and the second drive control signal, and generate the virtual device in the target image.

6. The surgical instrument control system according to claim 1, characterized in that, The control module is used to acquire the endoscope image and medical image, obtain the preset virtual image based on the medical image, the preset virtual image includes a three-dimensional model constructed from all feature elements in the endoscope image and elements surrounding the feature elements, and is also used to determine the three-dimensional point cloud data of the instrument to be operated and the three-dimensional point cloud data within a preset range around the instrument based on the endoscope image, and fuse the three-dimensional model and the three-dimensional point cloud data to obtain the target image.

7. The surgical instrument control system according to claim 6, characterized in that, The control module is also used to mark multiple feature locations in the three-dimensional model, determine feature point cloud data corresponding to at least one of the multiple feature locations in the three-dimensional point cloud data, and register and fuse the three-dimensional point cloud data with the three-dimensional model based on the feature point cloud data to obtain the target image.

8. The surgical instrument control system according to any one of claims 1-7, characterized in that, Each of the aforementioned operation consoles includes: An alarm module, connected to the control module, is used to issue a prompt signal when the instrument to be operated or the virtual instrument moves to a sensitive position, wherein the sensitive position is the location of the sensitive tissue.

9. The surgical instrument control system according to any one of claims 1-7, characterized in that, Each of the aforementioned operation consoles includes: A video recording module, connected to the control module, is used to record the operation of the instrument to be operated and / or the virtual instrument. The display module is also used to play the operation video according to the playback command.

10. The surgical instrument control system according to claim 9, characterized in that, Each of the aforementioned operation consoles includes: The scoring module, connected to the control module, is used to acquire a first operation video recording of operating the instrument to be operated and a second operation video recording of operating the virtual instrument, and compare the first operation video recording with the second operation video recording to determine the evaluation result of operating the virtual instrument.

11. A control method for a surgical instrument control system, characterized in that, The surgical instrument control system includes multiple operation consoles, each operation console including a display module and a control module, each control module being connected to the instrument to be operated, and the method includes: The movement of the instrument to be operated is controlled by the control module of the main operation console according to the received first drive control signal; The target image is obtained by fusing the endoscope image and the preset virtual image, and the target image is transmitted to the display module of the main operation console for display; wherein, the endoscope image is a real-time image of the instrument to be operated and the preset range around the instrument, and the preset virtual image includes all feature elements in the endoscope image and the virtual simulation images corresponding to the elements around the feature elements; The virtual machine is controlled to move via the control module from the operation console based on the received second drive control signal. The target image is obtained by fusing the endoscope image and the preset virtual image, and a virtual instrument is generated in the target image to obtain a virtual target image. The target image or the virtual target image is then transmitted to the display module of the slave operation console for display. When displaying the target image, the display module of the slave operation console can only view the operation of the instrument by the master operation console through the slave operation console. When displaying the virtual target image, the display module of the slave operation console controls the virtual instrument to perform simulated operation according to the second drive control signal through the slave operation console. The main operation console is one of the plurality of operation consoles, and the operation consoles other than the main operation console are the slave operation consoles.

Images