Doctor main console and surgical robot system

Abstract

The invention provides a doctor main console and a surgical robot system. The surgical robot system comprises a master end and a slave end, a doctor master console of the master end integrates a first pedal, a second pedal, a master operation unit and a master end mark sharing device, and the master operation unit comprises a first operation arm and a second operation arm; the first pedal outputs an enabling mark instruction and enables the master end and / or the slave end to enter an operation locking state; the second pedal outputs an exit mark instruction and enables the master end and / or the slave end to enter an operation unlocking state; the first operation arm selects a main end mark according to an operation instruction input by a user; a second operation arm creates a main end mark on the initial medical image according to an operation instruction input by a user; the master end mark sharing device obtains an initial medical image, sends master end mark information to the slave end after obtaining a master end mark instruction, and fuses the master end mark information and / or the slave end mark information in the initial medical image; therefore, the problem of unsmooth information exchange between the master end and the slave end in the remote operation is solved.

Description

[0001] This application is a divisional application of application number 202111306714.1. Technical Field

[0002] This invention relates to the field of medical technology, specifically to a doctor's main control console and surgical robot system. Background Technology

[0003] With the advancement of technology, the development of 5G, and the rise of AI, telemedicine has found a rapid opportunity for development. Telemedicine refers to using computer technology, remote sensing, telemetry, and remote control technologies to fully leverage the medical technology and equipment advantages of large hospitals or specialized medical centers to provide remote diagnosis, treatment, and consultation for the wounded and sick in remote areas, islands, or on ships with poor medical conditions. The aim is to improve diagnostic and medical standards and reduce medical expenses.

[0004] Currently, telemedicine technology has evolved from initial television monitoring and remote telephone diagnosis to the comprehensive transmission of digital data, images, and voice using high-speed networks, and has achieved real-time voice and high-definition image communication, providing a broader development space for the application of modern medicine. The technologies currently used in telemedicine include: (1) whiteboard marking technology, which provides a remote real-time triage system, including image browsing, image marking, text input, and real-time audio and video synchronization; (2) medical image stereoscopic vision display, in which the computer acquires medical images through input devices, processes the images through software systems, and finally displays and performs visual linkage through stereoscopic vision modules, 3D modules, or 2D modules; (3) medical image remote real-time consultation, which provides a case real-time triage system, including a consultation creation and management module, an electronic whiteboard, an audio and video acquisition and playback module, a network data transmission module, and a central processing module.

[0005] The technologies mentioned above are mostly independent systems, primarily designed for remote consultation scenarios and whiteboard sharing systems for remote surgery. They focus on teaching and sharing, but lack the capability to integrate and synchronize data from multiple devices in real-time remote medical surgery, and thus fail to guide physicians in their surgeries. Summary of the Invention

[0006] To address the technical problems in the prior art, the present invention aims to provide a doctor's main control console and surgical robot system that enables real-time sharing of data tags between the master and slave ends during remote surgery. This effectively solves the problem of poor communication between the master doctor and the slave assistant doctor during remote robotic surgery, thereby significantly improving the surgical accuracy in this application scenario and ensuring the safe and smooth conduct of the surgery.

[0007] To achieve the above objectives, according to a first aspect of the present invention, a doctor's main control console is provided for use at the main end of a surgical robot system. The doctor's main control console integrates a first foot pedal, a second foot pedal, a main operating unit, and a main end marking and sharing device. The main operating unit includes a first operating arm and a second operating arm.

[0008] The first foot pedal is used to output an enable marking command, so that the master terminal starts the marking mode according to the enable marking command, so as to allow marking on the initial medical image and put the master terminal and / or the slave terminal of the surgical robot system into an operation lock state;

[0009] The second foot pedal is used to output an exit marking command, so that the master terminal exits the marking mode according to the exit marking command, prohibits marking the initial medical image, and puts the master terminal and / or the slave terminal into the operation unlock state;

[0010] The first operating arm is used to select the master end marker according to the operating instructions input by the user;

[0011] The second manipulator is used to create the master marker on the initial medical image according to the operation command input by the user, using the selected master marker;

[0012] The master-end marker sharing device is configured to: acquire the currently generated initial medical image; upon acquiring the master-end marker instruction input by the user, send the master-end marker information contained in the master-end marker instruction to the slave end in real time, and fuse the master-end marker information in the initial medical image; upon receiving the slave-end marker information sent in real time by the slave end, fuse the slave-end marker information in the initial medical image.

[0013] In some embodiments, after receiving the slave tagging information sent in real time by the slave end, the master tagging sharing device fuses the slave tagging information into the initial medical image that fuses the master tagging information; and / or, sends the initial medical image that fuses the master tagging information and / or the slave tagging information to the slave end in real time.

[0014] In some embodiments, the master-end marker sharing device fuses the master-end marker information in the initial medical image, including: reconstructing a master-end marker corresponding to the master-end marker information on the initial medical image based on the location information in the master-end marker information; and fusing the slave-end marker information in the initial medical image, including: reconstructing a slave-end marker corresponding to the slave-end marker information on the initial medical image based on the location information in the slave-end marker information.

[0015] In some embodiments, after the master-end marker sharing device fuses the first marker information and / or fuses the second marker information in the initial medical image, it further includes:

[0016] Obtain image change information of the currently generated initial medical image, and update the first marker information and / or the second marker information according to the image change information of the initial medical image, so that the first marker information and / or the second marker information follow the changes of the initial medical image in real time.

[0017] In some embodiments, the master-end tag sharing device is further configured to: acquire the tag status of the master end and / or the slave end, and output a master-slave control command or an enable tag command according to the tag status of the master end and / or the slave end; wherein the tagging functions of the master end and the slave end are mutually exclusive, and the master end and the slave end execute the tagging operation sequentially according to the priority level;

[0018] The master-end tag sharing device is further configured to: acquire the tag status of the slave end, and when it is determined that the slave end is currently in a non-tag state, activate the tag mode and send the local current tag status to the slave end in real time.

[0019] In some embodiments, the master-end tag sharing device includes:

[0020] The master-end image data acquisition module is used to acquire the currently generated initial medical images;

[0021] The master-end tag instruction acquisition module is used to acquire the master-end tag instruction input by the user, and also to acquire the slave-end tag information sent by the slave end in real time;

[0022] A master-end data fusion module is used to fuse the master-end labeling information and / or the slave-end labeling information in the initial medical image;

[0023] The master-end information sharing module is used to send the initial medical image, which integrates the master-end marking information and / or the slave-end marking information, to the slave end in real time, and can also receive data sent by the slave end;

[0024] The master display module is used to display the initial medical image that integrates the master marking information and / or the slave marking information.

[0025] In some embodiments, the main image data module is configured to process the real-time acquired image data using 3D modeling and artificial intelligence algorithms to generate the initial 3D medical image; the main display module is a 3D display screen to display the 3D medical image in stereoscopic form.

[0026] In some embodiments, the main display module includes a main interactive interface that displays the initial medical image; the main interactive interface provides drawing tools, a pen, and a main undo button; the first operating arm is used to select a graphic and mark a position on the main interactive interface; the second operating arm is used to perform a drawing operation based on the graphic selected by the first operating arm; the main undo button is used to provide an operation command to undo the currently drawn graphic corresponding to the main mark.

[0027] In some embodiments, the main terminal interaction interface is configured such that: when the main terminal enters the marking mode, the main terminal interaction interface displays a preset main terminal marking graphic and a main terminal undo button on the screen; when the main terminal exits the marking mode, the main terminal interaction interface hides the preset main terminal marking graphic and the main terminal undo button on the screen.

[0028] To achieve the above objectives, according to a second aspect of the present invention, a surgical robot system is provided, comprising a master end and a slave end connected in communication, the slave end comprising a surgical robot, and the master end comprising a physician master console as described in any one of the present invention;

[0029] The slave end is configured to: upon receiving master end marker information sent in real time by the master end, fuse the master end marker information into the initial medical image of the slave end, and then the slave end sends the initial medical image fused with the master end marker information and / or the slave end marker information to the master end in real time.

[0030] Compared with the prior art, the doctor's main control console and surgical robot system of the present invention have the following advantages:

[0031] This invention utilizes data fusion and tag-sharing technologies to achieve real-time sharing of data tags between the master and slave ends during surgery. This effectively solves the problem of unclear communication between the master surgeon and the slave assistant surgeon regarding surgical information, such as lesion location and surgical plan, during remote surgical robot procedures. This significantly improves surgical accuracy in telemedicine and ensures the safe and successful execution of surgeries. Thus, by sharing data tags in real time, this invention enables effective communication and exchange between the master surgeon and the slave assistant surgeon during surgery, allowing the master surgeon to better guide the slave assistant surgeon in performing surgical operations, thereby improving surgical efficiency and success rates.

[0032] This invention utilizes existing equipment in the surgical robot system, such as foot pedals, operating arms, keyboards, and mice, for marking operations, which facilitates the doctor's operation and effectively reduces the interference of marking operations on the surgery.

[0033] This invention utilizes 3D modeling and artificial intelligence algorithms to process real-time acquired image data to generate initial 3D medical images. Marking and communication between the master and slave ends are performed on the initial 3D medical images to improve the accuracy of reference value and significantly enhance the accuracy of remote surgery applications. Attached Figure Description

[0034] The features, properties, and advantages of the implementation methods and related embodiments of the present invention will be described in conjunction with the following drawings, wherein:

[0035] Figure 1 This is a schematic diagram of a surgical scenario in which the master and slave ends of a surgical robot system according to a preferred embodiment of the present invention are located in different places.

[0036] Figure 2 This is a structural block diagram of a surgical robot system according to a preferred embodiment of the present invention;

[0037] Figure 3 This is a schematic diagram of the structure of the doctor's main control console according to a preferred embodiment of the present invention;

[0038] Figure 4 This is a schematic diagram of the structure of a slave control device according to a preferred embodiment of the present invention;

[0039] Figure 5 This is a flowchart illustrating the process of creating a master-end marker according to a preferred embodiment of the present invention;

[0040] Figure 6 This is a schematic diagram illustrating the operational principle of creating a master-end marker according to a preferred embodiment of the present invention;

[0041] Figure 7 This is a schematic diagram of the main terminal display module displaying the main terminal marker, the main terminal cancel button, and the graphic selection window according to a preferred embodiment of the present invention.

[0042] Figure 8 This is a flowchart illustrating the sharing of a master terminal marker from a master terminal to a slave terminal according to a preferred embodiment of the present invention;

[0043] Figure 9 This is a flowchart illustrating the process of creating a slave marker according to a preferred embodiment of the present invention;

[0044] Figure 10 This is an operational principle diagram of creating a slave marker according to a preferred embodiment of the present invention;

[0045] Figure 11 This is a schematic diagram of a slave-end interactive interface displaying a slave-end marker, a marker button, a marker undo button, and a graphical selection window in a slave-end display module according to a preferred embodiment of the present invention.

[0046] Figure 12 This is a flowchart illustrating the sharing of a slave tag from the slave end to the master end according to a preferred embodiment of the present invention;

[0047] Figure 13 This is a schematic diagram illustrating the principle of data fusion in a surgical robot system according to a preferred embodiment of the present invention;

[0048] Figure 14 This is a schematic diagram illustrating the state of a surgical robot system according to a preferred embodiment of the present invention, where data is fused and displayed.

[0049] In the diagram: 100 - Main terminal; 101 - Doctor's main control console; 1011 - Main terminal image data acquisition module; 1012 - Main terminal marker instruction acquisition module; 1013 - Main terminal data fusion module; 1014 - Main terminal information sharing module; 1015 - Main terminal display module; 1016 - Main terminal audio and / or video acquisition module; 1017 - Main terminal data transmission module; 1041 - Left foot pedal; 1042 - Right foot pedal; 1043 - Left operating arm; 1044 - Right operating arm; 1045 - Left control handle; 1046 - Right control handle; G1 - Main terminal interactive interface; G2 - Main terminal drawing tools; G3 - Rectangular preset main terminal markers; G4 - Circular preset main terminal markers; G5 - Circular main terminal markers; G6 - Custom main terminal markers; G7 - Rectangular main terminal markers; G8 - Master Cancel Button; 200 - Slave End; 201 - Surgical Robot; 202 - Slave Control Device; 2021 - Slave Image Data Acquisition Module; 2022 - Slave Marking Instruction Acquisition Module; 2023 - Slave Data Fusion Module; 2024 - Slave Information Sharing Module; 2025 - Slave Display Module; 2026 - Slave Audio and / or Video Acquisition Module; 2027 - Slave Data Transmission Module; 2071 - Keyboard; 2072 - Keyboard; H1 - Slave Interactive Interface; H2 - Slave Drawing Tool; H3 - Preset Rectangular Slave Marks; H4 - Preset Circular Slave Marks; H5 - Circular Slave Marks; H6 - Custom Slave Marks; H7 - Rectangular Slave Marks; H8 - Slave Cancel Button; H9 - Marking Button; 203 - Image Cart; 204 - Sterile Table; 211 - Partition. Detailed Implementation

[0050] The following specific examples illustrate the implementation of the present invention. Those skilled in the art can easily understand other advantages and effects of the present invention from the content disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and various details in this specification can be modified or changed based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that the illustrations provided in this embodiment are only schematic representations of the basic concept of the present invention. Therefore, the drawings only show components related to the present invention and are not drawn according to the actual number, shape, and size of components in implementation. In actual implementation, the type, quantity, and proportion of each component can be arbitrarily changed, and the component layout may also be more complex. Furthermore, each embodiment described below has one or more technical features; however, this does not mean that the user of the present invention must simultaneously implement all technical features in any embodiment, or can only separately implement some or all technical features in different embodiments. In other words, provided implementation is possible, those skilled in the art can selectively implement some or all technical features in any embodiment, or selectively implement a combination of some or all technical features in multiple embodiments, based on the disclosure of the present invention and depending on design specifications or implementation needs, thereby increasing the flexibility of the implementation of the present invention. As used herein, the singular forms “a,” “an,” and “the” include plural objects, and the plural form “a plurality” includes two or more objects, unless otherwise expressly indicated. As used herein, the term “or” is generally used to include the meaning of “and / or,” unless otherwise expressly indicated, and the terms “install,” “connect,” and “link” should be interpreted broadly, for example, as a fixed connection, a detachable connection, or an integral connection. Connections can be mechanical or electrical. Connections can be direct or indirect through an intermediate medium, and can represent internal communication between two elements or an interaction between two elements. Those skilled in the art will understand the specific meaning of the above terms in this invention according to the specific circumstances.

[0051] To make the objectives, advantages, and features of the present invention clearer, the present invention will be further described in detail below with reference to the accompanying drawings. It should be noted that the drawings are all in a very simplified form and use non-precise proportions, and are only used to facilitate and clearly illustrate the objectives of the embodiments of the present invention. Unless otherwise specified, the following embodiments and features can complement or combine with each other.

[0052] Figure 1 An application scenario of a surgical robot system according to a preferred embodiment of the present invention is illustrated. For example... Figure 1As shown, a preferred embodiment of the present invention provides a surgical robot system, including a first surgical robot end and a second surgical robot end connected in communication. One of the first and second surgical robot ends is a master end 100, and the other is a slave end 200. For simplicity, the following description assumes that the first surgical robot end is the master end 100 and the second surgical robot end is the slave end 200. Those skilled in the art should be able to modify the following description, making appropriate modifications to the details, to apply the description to the case where the first surgical robot end is the slave end 200 and the second surgical robot end is the master end 100.

[0053] The master terminal 100 and slave terminal 200 can be located in different rooms, different hospitals, or different cities to achieve telemedicine. The master terminal 100 includes a doctor's master console 101, allowing the doctor to use their hands and feet to perform telemedicine. The slave terminal 200 includes a surgical robot 201, which includes a robotic arm. The end of the robotic arm is used to detachably connect to surgical instruments or image acquisition devices, including but not limited to endoscopes, to control the operation of the surgical instruments or image acquisition devices. During the operation, utilizing the master-slave control relationship formed by the doctor's master console 101, the robotic arm, and the surgical instruments, the doctor remotely operates the surgical robot 201 through the master operation unit of the doctor's master console 101, causing the robotic arm and surgical instruments to move according to the movement of the doctor's master console 101, such as according to the operation of the doctor's hands, thereby performing minimally invasive surgical treatment on the patient in the bed.

[0054] like Figure 1 and Figure 2 As shown, the slave end 200 also includes a slave control device 202, which is typically integrated with the image carriage 203. The image carriage 203 can also house endoscopes and other related equipment (such as a slave display module and some cables). The endoscope is used to acquire images of the surgical environment, including human tissues and organs, surgical instruments, blood vessels, and body fluids, and the endoscope and surgical instruments are inserted into the patient's position through incisions in the patient's body. The image carriage 203 is positioned next to the bed and is separate from the surgical robot 201.

[0055] The slave control device 202 communicates with the surgical robot 201 via wired or wireless communication. The doctor's main control console 101 generates and outputs master-slave control commands based on external instructions and sends them to the slave control device 202. The slave control device 202 controls the movement of the surgical robot 201 according to the received master-slave control commands. More specifically, the slave control device 202 outputs master-slave control commands based on the movement information sent by the doctor's main control console 101 and a preset master-slave mapping relationship, thereby controlling the surgical robot 201 to execute the master-slave control commands to drive the movement of surgical instruments or endoscopes. For example, the slave control device 202 controls the surgical robot 201 to drive the movement of surgical instruments or endoscopes based on the acquired movement speed of the master operating unit, and controls the surgical robot 201 to drive the rotation of surgical instruments or endoscopes based on the acquired rotation angle or rotation speed of the master operating unit. It can also control the surgical robot 201 to drive the bending of surgical instruments or endoscopes based on the acquired bending angle or bending direction of the master operating unit.

[0056] Furthermore, the slave end 200 may also include surgical auxiliary equipment, such as a sterile table, a ventilator, and a detection device. In this embodiment, the slave end 200 also includes a sterile table 204, which is set next to the bed for placing surgical instruments to establish a sterile barrier, prevent recontamination of sterile surgical instruments and dressings, and also prevent the leakage or loss of surgical instruments and dressings.

[0057] Furthermore, the surgical robot system in this embodiment aims to realize telemedicine and enable information exchange between the master and slave ends during telemedicine surgery. The exchanged information may include the location of tissues, blood vessels, and lesion boundaries between the master physician and the slave assistant physician, as well as the surgical path, surgical plan, and communication regarding unclear locations shown in medical images. This effectively solves the problem of poor information exchange between the master physician and the slave assistant physician during remote surgery, thereby effectively improving the surgical accuracy in this telemedicine scenario, ensuring the safe and smooth conduct of the surgery, and improving surgical efficiency.

[0058] More specifically, this embodiment provides a tag-sharing method for surgical robots, applicable to either the master end 100 or the slave end 200. Taking the master end 100 as an example, it includes the following steps:

[0059] Step 1: Acquire the currently generated initial medical image; the initial medical image can be a three-dimensional medical image, or in other cases, a two-dimensional medical image; in addition, the initial medical image data includes, but is not limited to, images acquired by an endoscope, and can also be image data acquired by other image acquisition devices, such as CT or MRI data acquired by imaging equipment; furthermore, the doctor's main console 101 can create initial medical images of predetermined objects in the body, such as target tissues, organs, or blood vessels, based on the initial medical image data, and share the created initial medical images with the slave device 200;

[0060] Step 2: Upon receiving the first marking instruction input by the user, which includes first marking information, the first marking information is sent to the slave device 200 in real time and fused into the initial medical image; the first marking information mainly includes coordinate information corresponding to the first marking.

[0061] Step 3: After receiving the second marker information sent in real time from the slave device 200, the second marker information is fused into the initial medical image; the second marker information mainly includes coordinate information corresponding to the second marker.

[0062] Preferably, the above-mentioned tag sharing method further includes: after receiving the second tag information sent in real time by the slave end 200, fusing the second tag information into the initial medical image that fuses the first tag information; thereby obtaining the initial medical image that fuses the first tag information and the second tag information.

[0063] Preferably, the tag sharing method further includes: sending the initial medical image, which fuses the first tag information and / or the second tag information, to the slave end 200 in real time. In this case, the slave end 200 can skip the data fusion step and directly obtain the initial medical image fused with the first tag and / or the second tag from the master end 100. The first tag corresponds to the first tag information, which may be graphic and / or text information; the second tag corresponds to the second tag information, which may also be graphic and / or text information.

[0064] Of course, the above tag sharing method can also be applied to the slave end 200. In this case, most of the above steps are the same. The difference is that in step two, the slave end 200 sends the first tag information to the master end 100 in real time, and in step three, after the slave end 200 receives the second tag information sent by the master end 100 in real time, it fuses the second tag information into the initial medical image. Another difference is that in the preferred step, the slave end 200 sends the initial medical image fused with the first tag information and / or the second tag information to the master end 100 in real time. In a further preferred step, the slave end 200 sends the initial medical image fused with the first tag information and / or the second tag information to the master end 100 in real time. Then, the master end 100 can also skip the data fusion step and directly obtain the initial medical image containing the first tag and / or the second tag from the slave end 200.

[0065] In this embodiment of the application, the above-mentioned tag sharing method is applicable to both the master end 100 and the slave end 200, so that the master end 100 and the slave end 200 can each perform data fusion to obtain an initial medical image containing tags (defined as a tagged medical image), and finally display the initial medical image fused with the first tag and / or the second tag on their respective display modules, so that the tags made by the master end doctor and the slave end assistant doctor on the initial medical image can be viewed.

[0066] Correspondingly, one embodiment of this application also provides a tag sharing device, which can be applied to the master end 100 or the slave end 200, and is used to execute the above tag sharing method.

[0067] Taking the master terminal 100 as an example, the label sharing device includes an image data acquisition module, a labeling instruction acquisition module, and a data fusion module. The image data acquisition module is used to acquire the currently generated initial medical image. The labeling instruction acquisition module is used to acquire a first labeling instruction input by the user and also to acquire second labeling information sent in real time by the slave terminal 200. The data fusion module is used to fuse the first labeling information and / or the second labeling information in the initial medical image. Preferably, the label sharing device further includes an information sharing module, used to send the initial medical image fused with the first labeling information and / or the second labeling information to the slave terminal 200 in real time. In this case, it should be understood that the first labeling instruction is the labeling instruction input by the master terminal doctor at the master terminal 100, and this first labeling instruction is defined as the master terminal labeling instruction, while the labeling instruction input by the slave terminal assistant doctor at the slave terminal 200 is defined as the second labeling instruction, the second labeling instruction including second labeling information, and this second labeling instruction is defined as the slave terminal labeling instruction.

[0068] When applied to the slave end 200, unlike the tag sharing device applied to the master end, the tag instruction acquisition module is used to acquire the second tag instruction sent in real time by the master end 100. In a preferred embodiment, the information sharing module is used to send the initial medical image, which integrates the first tag information and / or the second tag information, to the master end 100 in real time. In this case, the first tag instruction is the tag instruction input by the slave-end assistant physician at the slave end 200, and this first tag instruction is defined as the slave-end tag instruction. The second tag instruction is the tag instruction input by the master-end physician at the master end 100, and this second tag instruction is defined as the master-end tag instruction.

[0069] In this embodiment, both the master terminal 100 and the slave terminal 200 include a marker sharing device. For clarity, the marker sharing device of the master terminal 100 is defined as a master terminal marker sharing device, and the marker sharing device of the slave terminal 200 is defined as a slave terminal marker sharing device. Therefore, both the master terminal 100 and the slave terminal 200 can utilize data fusion technology to acquire initial medical images containing a first marker and / or a second marker, thereby facilitating the display of the initial medical images containing the first marker and / or the second marker on both the master terminal 100 and the slave terminal 200, and enabling timely and effective communication of information between the master terminal physician and the slave terminal assistant physician.

[0070] like Figure 2 As shown, the master-end marker sharing device includes a master-end image data acquisition module 1011, a master-end marker instruction acquisition module 1012, a master-end data fusion module 1013, and a master-end information sharing module 1014. The master-end image data acquisition module 1011 acquires the currently generated initial medical image; the master-end marker instruction acquisition module 1012 acquires the master-end marker instruction input by the user, the master-end marker instruction including master-end marker information, and is also used to acquire the slave-end marker information sent in real time by the slave end 200; the master-end data fusion module 1013 fuses the master-end marker information and / or the slave-end marker information in the initial medical image to obtain an initial medical image fused with the master-end marker information and / or the slave-end marker information. Further, the master-end information sharing module 1014 sends the initial medical image fused with the master-end marker information and / or the slave-end marker information to the slave end 200 in real time. Furthermore, the master-end marker sharing device also includes a master-end display module 1015 for displaying the initial medical image that integrates the master-end marker information and / or the slave-end marker information.

[0071] In this embodiment, the doctor's main console 101 integrates the main-end tagging and sharing device. The doctor can tag the initial medical images on the main console 101, and display the untagged initial medical images and the tagged initial medical images on the monitor of the main console 101. The untagged initial medical images and the tagged initial medical images can be displayed in different windows, or when the initial medical images are tagged, only the tagged initial medical images are displayed.

[0072] Continue reading Figure 2 The slave-end labeling and sharing device includes a slave-end image data acquisition module 2021, a slave-end labeling instruction acquisition module 2022, a slave-end data fusion module 2023, and a slave-end information sharing module 2024. The slave-end image data acquisition module 2021 acquires the currently generated initial medical image; the slave-end labeling instruction acquisition module 2022 acquires a slave-end labeling instruction input by the user, the slave-end labeling instruction including slave-end labeling information, and is also used to acquire master-end labeling information sent by master-end 100; the slave-end data fusion module 2023 fuses the master-end labeling information and / or the slave-end labeling information into the initial medical image to obtain an initial medical image fused with the master-end labeling information and / or the slave-end labeling information. Further, the slave-end information sharing module 2024 sends the initial medical image fused with the master-end labeling information and / or the slave-end labeling information to master-end 100 in real time. Furthermore, the slave-end tag sharing device also includes a slave-end display module 2025 for displaying the initial medical image that integrates the master-end tag information and / or the slave-end tag information.

[0073] In this embodiment, the slave control device 200, such as the image cart 203, integrates the slave labeling and sharing device. The slave-assisted physician can label the initial medical images on the slave control device 200 and display both unlabeled and labeled initial medical images on the display of the slave control device 200. In the slave labeling and sharing device, the unlabeled and labeled initial medical images can be displayed in different windows, or only the labeled initial medical image can be displayed after the initial medical image has been labeled.

[0074] It should be understood that the master-end data fusion module 1013 and the slave-end data fusion module 2023 respectively employ data fusion algorithms to achieve data fusion. The data fusion algorithm is mainly implemented using marked coordinates, and how to achieve data fusion is a well-known technology in the field; therefore, this application does not describe it in detail.

[0075] In one example, the tag sharing device reconstructs a master tag corresponding to the master tag information on the initial medical image based on the position information in the master tag information, and reconstructs a slave tag corresponding to the slave tag information on the initial medical image based on the position information in the slave tag information.

[0076] More specifically, the marker sharing device obtains an image in the initial medical image corresponding to the location of the master marker information based on the coordinate information in the master marker information, and generates a master marker on the image at the location of the master marker information, thereby fusing the master marker with the initial medical image. The marker sharing device also obtains an image in the initial medical image corresponding to the location of the slave marker information based on the coordinate information in the slave marker information, and generates a slave marker on the image at the location of the slave marker information, thereby fusing the slave marker with the initial medical image as well; finally, the master data fusion module 1013 obtains the initial medical image fused with the master marker and the slave marker.

[0077] In one specific embodiment, the tag sharing device can also acquire an enable or disable tagging instruction, and output a master-slave control instruction according to the disable tagging instruction. The master-slave control instruction is used to control the movement of the master end 100 or the slave end 200, and to start a tagging mode according to the enable tagging instruction, so as to allow the user to tag the initial medical image.

[0078] For the master device 100, the master-end tag sharing device, based on the acquired enable or disable tagging instruction, allows or disables the creation of master-end tags on the doctor's main console 101. When master-end tagging is enabled, the user can tag the initial medical image through the doctor's main console 101 and save the location information (including coordinates) of the master-end tag. Similarly, for the slave device 200, the slave-end tag sharing device, based on the acquired enable or disable tagging instruction, allows or disables the creation of slave-end tags on the slave-end control device 200. When slave-end tagging is enabled, the user can tag the initial medical image through the slave-end control device 200 and save the location information (including coordinates) of the slave-end tag.

[0079] Furthermore, in addition to sending tagging information and initial medical images containing tagging information to the slave end 200, the master-end information sharing module 1014 can also receive data sent from the slave end 200, such as slave tagging information, and transmit the received slave tagging information to the master-end tagging instruction acquisition module 1012. Furthermore, the master-end tagging instruction acquisition module 1012 can parse the slave tagging information to obtain the location information within it. Furthermore, the master-end tagging instruction acquisition module 1012 can package the master-end tagging information, and the packaged master-end tagging information is sent to the slave end 200 through the master-end information sharing module 1014.

[0080] Furthermore, in addition to sending tagging information and initial medical images containing tagging information to the master terminal 100, the slave information sharing module 2024 can also receive data sent by the master terminal 100, such as master tagging information, and transmit the received master tagging information to the slave tagging instruction acquisition module 2022. Furthermore, the slave tagging instruction acquisition module 2022 can parse the master tagging information to obtain the position information within it. Furthermore, the slave tagging instruction acquisition module 2022 can package the slave tagging information, and the packaged slave tagging information is sent to the master terminal 100 through the slave information sharing module 2024.

[0081] Non-limitingly, the master-end image data acquisition module 1011 can also establish initial medical images of predetermined objects such as tissues, organs, or blood vessels based on initial medical image data. Preferably, the master-end image data acquisition module 1011 provides 3D modeling functionality to acquire 3D medical images. Furthermore, the master-end display module 1015 is preferably a 3D display screen to display the 3D medical images in stereoscopic form, thereby improving the accuracy of reference and significantly increasing the precision of remote surgical applications. To improve surgical accuracy, the master-end image data acquisition module 1011 primarily utilizes 3D modeling and artificial intelligence algorithms to process the real-time acquired image data to generate the 3D initial medical images. Artificial intelligence algorithms enable fully automated modeling without manual repair, featuring high fidelity, no distortion, small model size, and convenient display, effectively reducing 3D modeling time and improving the accuracy of 3D modeling. Further, the slave-end display module 2025 can be a 2D display screen or a 3D display screen.

[0082] To further improve the smoothness of information exchange between the master terminal 100 and the slave terminal 200, in one embodiment, the tag sharing method further includes: the slave terminal 200 wirelessly transmitting audio data and / or video data to the master terminal 100 in real time. In one embodiment, the tag sharing method further includes: the master terminal 100 wirelessly transmitting audio data and / or video data to the slave terminal 200 in real time. Furthermore, the surgical robot system also includes an audio and / or video acquisition module for acquiring audio and / or video data.

[0083] like Figure 2 As shown, the doctor's main control console 101 is also equipped with a master audio and / or video acquisition module 1016, used to acquire audio and / or video data from the master terminal 100 and transmit the master terminal's audio and / or video data to the slave terminal 200 in real time. This facilitates online communication between the master terminal doctor and the slave terminal assistant doctor via voice and / or video, further improving communication efficiency. The master terminal 100 can transmit audio and video data between the master terminal 100 and the slave terminal 200 through the master terminal data transmission module 1017, and provides a real-time low-latency audio and video transmission scheme to ensure low latency and high quality during data transmission.

[0084] Similarly, the slave control device 202 is equipped with a slave audio and / or video acquisition module 2026, used to acquire audio and / or video data from the slave device 200 and transmit the slave device's audio and / or video data to the master device 100 in real time. This facilitates online communication between the master device doctor and the slave device assistant doctor via voice and / or video, further improving communication efficiency. The slave device 200 further transmits audio and video data between the master device 100 and the slave device 200 through a slave data transmission module 2027, and provides a real-time low-latency audio and video transmission scheme to ensure low latency and high quality during data transmission.

[0085] The master audio and / or video acquisition module 1016 and the slave audio and / or video acquisition module 2026 include, but are not limited to, audio and video devices such as cameras and microphones. Both the master audio and / or video acquisition module 1016 and the slave audio and / or video acquisition module 2026 can be connected to wired or wireless networks, preferably 5G Wi-Fi, for faster transmission speeds and better transmission quality. The slave audio and / or video acquisition module 2026 communicates with the surgical robot 201 to receive video data transmitted from the image acquisition device.

[0086] The surgical robot system also includes an image acquisition device, whereby the initial medical image data can be determined by acquiring images within the body using an image acquisition device (such as an endoscope). Furthermore, considering that the images acquired by the image acquisition device change in real time, to improve labeling efficiency, after fusing the master-end labeling information and / or the slave-end labeling information into the initial medical image, the system further includes: acquiring image change information of the currently generated initial medical image, and updating the master-end labeling information and / or the slave-end labeling information based on the image change information of the initial medical image, so that the master-end labeling information and / or the slave-end labeling information follow the changes in the initial medical image acquired by the image acquisition device in real time; thereby enabling the labels to dynamically follow the changes in the image, resulting in higher labeling efficiency and more convenient operation.

[0087] Furthermore, the tag sharing method further includes: activating a tag mode before receiving a first tag instruction input by the user, wherein the tag mode allows the user to input the first tag instruction. It is understood that when the first tag instruction is a master tag instruction, the master terminal 100 also activates the tag mode before receiving the master tag instruction input by the user, and only after activating the tag mode is the user allowed to input the master tag instruction on the master terminal 100. Conversely, when the first tag instruction is a slave tag instruction, the slave terminal 200 also activates the tag mode before receiving the slave tag instruction input by the user, and only after activating the tag mode is the user allowed to input the slave tag instruction on the slave terminal 200.

[0088] In one specific embodiment, the tag sharing method further includes: acquiring the tag status of the master terminal 100 and / or the slave terminal 200, and outputting a master-slave control command or a tag enable command based on the tag status of the master terminal 100 and / or the slave terminal 200. The master-slave control command is used to control the motion state (including operation state) of the master terminal 100 or the slave terminal 200, and the tag enable command is used to activate the tag mode.

[0089] Furthermore, the tag sharing method also includes: obtaining a user-inputted exit tag command, and exiting the tag mode according to the exit tag command.

[0090] Furthermore, the tag-sharing method further includes: according to the tag-enabling instruction, causing the master terminal 100 and / or the slave terminal 200 to enter an operation-locked state to prohibit the movement of the master terminal 100 and / or the slave terminal 200, so that the master terminal 100 and / or the slave terminal 200 are prohibited from controlling the movement of the surgical robot 201. For example, when the surgical robot system enters the tagging mode, the master operating unit of the doctor's main control console 101 is in an operation-locked state, and the master operating unit cannot remotely operate the surgical robot 201. For example, the slave terminal 200 receives the movement status of the master operating unit, but does not control the movement of the surgical robot accordingly. That is, the operating handle of the master operating unit cannot remotely control the movement of the surgical robot and the surgical operation, and / or, when the surgical robot system enters the tagging mode, the surgical robot 201 is automatically locked and cannot move.

[0091] Furthermore, the tag-sharing method further includes: according to the tag exit command, causing the master terminal 100 and / or the slave terminal 200 to enter an operation unlock state, thereby allowing the master terminal 100 and / or the slave terminal 200 to move, so that the master terminal 100 and / or the slave terminal 200 can control the movement of the surgical robot. For example, when the surgical robot system exits the tag mode, the master operation unit of the doctor's main control console 101 is in the operation unlock state, and the master operation unit can remotely operate the surgical robot 201. For example, the slave terminal 200 receives the movement status of the master operation unit and controls the movement of the surgical robot accordingly. That is, the operating handle of the master operation unit can realize remote operation control of the movement of the surgical robot and surgical operations, and / or, when the surgical robot system exits the tag mode, the surgical robot 201 automatically unlocks and can move.

[0092] It should be understood that when the surgical robot system enters the operation locked state, the surgical robot system locks the master-slave mapping relationship to facilitate the marking operation of the slave end 200 or the master end 100; while when the surgical robot system is in the operation unlocked state, the surgical robot system can rebuild the master-slave mapping relationship so that the master end 100 can remotely operate the slave end 200 according to the determined master-slave control instructions, so that the surgical robot 201 can control the movement of surgical instruments or endoscopes according to the master-slave control instructions.

[0093] See Figure 3In an exemplary embodiment, the main terminal 100 includes a left foot pedal 1041 and a right foot pedal 1042, which are integrated into the doctor's main control console 101. One of the left foot pedal 1041 and the right foot pedal 1042 is configured as a first foot pedal, and the other is configured as a second foot pedal. The first foot pedal is used to output an enable marking command, and the main terminal 100 initiates marking mode based on the enable marking command of the first foot pedal. The second foot pedal is used to output an exit marking command, and the main terminal 100 exits marking mode based on the exit marking command of the second foot pedal. In use, the doctor on the main terminal can enter or exit marking mode by controlling the corresponding foot pedals with both feet. For example, if the left foot pedal 1041 is currently in the operating state of being pressed and outputs an enable marking command, the main terminal 100 determines to enter marking mode; conversely, if the right foot pedal 1042 is currently in the operating state of being pressed and outputs an exit marking command, the main terminal 100 determines to exit marking mode.

[0094] Continue reading Figure 3 In one specific embodiment, the master end marker instruction acquisition module 1012 may include a left operating arm 1043 and a right operating arm 1044. These operating arms are also integrated into the doctor's main control console 101. The left operating arm 1043 and the right operating arm 1044 constitute a main operating unit, which can accept external instructions to output motion information, and one of the left operating arm 1043 and the right operating arm 1044 is configured as a first operating arm, and the other is configured as a second operating arm. The first operating arm is used to select a master end marker corresponding to the master end marker information according to the operation instructions input by the user. The second operating arm is used to create the master end marker on the initial medical image according to the operation instructions input by the user, using the selected master end marker.

[0095] like Figure 3 and Figure 7 As shown, the main display module 1015 includes a main display interface G1. By manipulating the first operating arm, the user can select drawing tools on the main interface. These drawing tools provide various drawing commands corresponding to preset graphics, allowing the user to select graphics from the drawing tools to create main markers corresponding to main marker instructions. Furthermore, the user can also manipulate the second operating arm to draw main markers corresponding to main marker instructions on the initial medical image based on preset graphics determined by the first operating arm. For example, the left operating arm 1043 is used to select drawing tools, and the right operating arm 1044 is used to draw graphics using the drawing tools.

[0096] In one specific embodiment, the doctor at the main end moves the cursor position by controlling the movement of the left operating arm 1043 to select the desired graphic and the position to be marked, and determines the selected graphic by the left control handle 1045 of the left operating arm 1043, and performs pinch selection (such as the function of the left mouse button) and drawing operation by the finger operation of the right control handle 1029 of the right operating arm 1044.

[0097] like Figure 3 and Figure 7 As shown, in one specific embodiment, the main interface G1 provides a drawing tool G2 and a main undo button G8. The drawing tool G2 of the main interface G1 provides various preset shapes, such as a rectangular preset main marker G3 and a circular preset main marker G4. Of course, the preset shapes are not limited to these examples and can be various other shapes, and the shapes are not limited to regular shapes. The drawing tool G2 of the main interface G1 can also provide a pen, allowing users to customize and draw shapes. Thus, users can operate the drawing tool with the mouse and use the drawing tool G2 to draw shapes on the initial medical image displayed on the main interface G1 to obtain the main marker.

[0098] exist Figure 7 In the example shown, the user can draw a circular main marker G5, a rectangular main marker G7, or a custom main marker G6 using the mouse and based on preset shapes provided by the drawing tool G2. Furthermore, the main undo button G8 provides a command to undo the currently drawn shape corresponding to the main marker; the user can simply click the main undo button G8 to undo the current shape. Of course, in addition to displaying markers and medical images, the main interface G1 also displays the initial medical image containing the markers.

[0099] Figure 5 The workflow for creating a master-end tag according to a preferred embodiment of the present invention is illustrated. For example... Figure 5 As shown, the workflow for creating a master tag includes the following steps:

[0100] Step A1: Start the master terminal marking instruction acquisition module 1012; Since the master terminal marking instruction acquisition module 1012 is embedded in the running program of the doctor's master console 101, the master terminal marking function is automatically started after the doctor's master console 101 program is run, but the marking mode needs to be started to perform marking operations.

[0101] Step A2: Determine whether the slave device is in a marking state; in one embodiment, the marking functions of the master and slave devices are mutually exclusive, meaning that only one device can perform a marking operation at any given time; if the slave device 200 is currently in a marking state, the master device 100 automatically enters step A10 to exit the master marking mode; if the slave device 200 is not currently in a marking state, the master device 100 can enter the master marking mode in step A3. Therefore, the master marking sharing device preferably also acquires the marking state of the slave device. When the marking state of the slave device is currently in a non-marking state, the master marking mode is activated, and marking is performed on the initial medical image to obtain the master marking instruction. Preferably, in step S4, the local current marking state corresponding to the master device is sent to the slave device 200 in real time.

[0102] Step A3: The master end enters the master end marking mode; after entering the master end marking mode, step A4 is executed: the master end 100 sends the local current marking status corresponding to the master end to the slave end 200 in real time, thereby sending the marking status of the master end 100 to the slave end 200, so that the slave end 200 can determine whether it can enter the marking mode based on the marking status of the master end.

[0103] After entering the main terminal marking mode, step A5 is executed: determine whether to select a preset marking graphic; in step A5, the user can select a system preset graphic to draw the main terminal mark, or select a custom graphic to draw the graphic.

[0104] Proceed to step A6: Draw the master marker according to the drawing method selected in the previous step A5 to generate the master marker instruction; the user can draw the master marker on the initial medical image using the left and right control handles.

[0105] After the main marker is drawn, step A7 can be used to further determine whether to cancel the drawn main marker; if yes, proceed to step A8 to delete the latest main marker; otherwise, the program automatically proceeds to step A9. Step A8: Delete the latest main marker, preferably deleting only the most recent main marker at a time. Step A9: Determine whether to exit the main marker; if yes, exit marker mode A10; if no, the process can continue looping from A5 to A9.

[0106] In a preferred embodiment, when the main terminal 100 enters the main terminal marking mode, its main terminal interaction interface G1 automatically displays a preset main terminal marking graphic and a main terminal undo button on the screen, and when the main terminal 100 exits the main terminal marking mode, the main terminal interaction interface G1 automatically hides the preset main terminal marking graphic and the main terminal undo button on the screen.

[0107] More specifically, in a particular operational method, such as Figure 6As shown, the master doctor triggers a specific foot pedal B1 to output a master marking command, thereby entering master marking mode A3. After entering master marking mode A3, two marking methods can be selected, including: the first marking method: the master doctor determines to draw a preset graphic B2; then, the master doctor selects the preset graphic B3 through the left operating arm 1043; then, the master doctor draws the preset graphic B4 through the right operating arm 1044; the second marking method: the master doctor determines to draw a custom graphic B5; then, the master doctor draws the custom graphic path B6 through the right operating arm 1044.

[0108] All of the above marking methods can ultimately be undoed (B7). If undoing is confirmed, the user can select the main undo button (B8) via the left operating arm (1043) to undo the current main-end marked graphic. After undoing, the user can cycle through and select preset graphics for drawing, or redraw a custom graphic path. For example, the user can move the cursor to the graphic selection window (i.e., drawing tool G2) of the main-end interactive interface G1 via the left operating arm (1043), pinch their left hand fingers to select a preset graphic, and release their left hand after selection. Then, the user can move the cursor to the position to be marked via the right operating arm (1044), pinch their right hand fingers and move the position to perform the marking operation, and release their fingers after marking is complete.

[0109] In one specific embodiment, without selecting a preset marker graphic, the main end 100 draws a custom marker graphic by default. The right operating arm 1044 is used to move the cursor to the desired marking position, and the finger is pinched to mark and move the cursor. Releasing the finger completes one marking operation. Alternatively, the left operating arm 1043 moves the cursor to the main end's undo button G8, and the finger is pinched to undo the operation.

[0110] The above describes a scenario where only one of the master and slave terminals can perform marking. However, in other embodiments, the master and slave terminals can also mark simultaneously and save their respective marking information. Furthermore, when only one of the master and slave terminals can mark, the surgical robot system can set a marking priority level. For example, if both the master and slave terminals issue marking commands simultaneously, the object to be marked is determined based on the priority level. If the marking priority level of the master terminal is higher than that of the slave terminal, the master terminal is allowed to perform the marking operation first.

[0111] Figure 8 The flowchart illustrates a process for a master end to share a master end tag with a slave end according to a preferred embodiment of the invention. For example... Figure 8 As shown, in one embodiment, the process of the master end sharing the master end tag with the slave end includes the following steps:

[0112] Step 1: The master terminal completes the master terminal marking to generate the master terminal marking instruction; specifically, the master terminal doctor completes the master terminal marking operation by manipulating the right operating arm 1044 and the left operating arm 1043 and pinching the fingers on the master terminal 100.

[0113] Step 2: The master end saves the position information of the master end marker, mainly the coordinate information; the master end marker instruction acquisition module 1012 will save the X and Y coordinate information of all master end marker points on the screen of the master end interactive interface in real time, and considering the difference in resolution between the master end display screen and the slave end display screen, it is preferable to also save the resolution information of the master end screen.

[0114] Step 3: Send the location information of the master marker to the slave; After the master marker is completed, the master information sharing module 1014 will send the location information of the master marker to the slave 200;

[0115] Step 4: Receive the position information of the master marker from the slave end; After receiving the position information of the master marker, the slave end 200 performs data parsing to obtain the coordinates of the master marker;

[0116] Step 5: The slave end reconstructs the master end marker based on the parsed data; during the drawing process, the slave end draws the master end marker proportionally according to the resolution of the slave end's display screen;

[0117] Step 6: After the master marker is drawn from the slave end, the master marker, the slave marker, and the medical image will be fused to obtain an initial medical image with the fused master marker and slave marker and then displayed.

[0118] Combination Figure 4 and Figure 11 In one specific embodiment, the slave display module 2025 includes a slave interactive interface H1, capable of displaying initial medical images and labeled medical images. Further, the slave interactive interface H1 provides a drawing tool H2 (i.e., a graphic selection window), a labeling button H9, and a slave undo button H8. The drawing tool H2 of the slave interactive interface H1 can also provide various preset graphics, such as a rectangular preset slave label H3, a circular preset slave label H4, etc. Of course, the preset graphics are not limited to these examples and can be various other shapes, and the shapes are not limited to regular graphics. The drawing tool H2 of the slave interactive interface H1 can also provide a pen, facilitating user-defined drawing of graphics; allowing users to operate the drawing tool H2 and use it to draw graphics corresponding to the slave labels on the initial medical images displayed on the slave interactive interface H1, thereby obtaining the slave labels.

[0119] exist Figure 11In the example shown, the user can draw a circular slave marker H5, a rectangular slave marker H7, or a custom slave marker H6 based on the preset graphics provided by the drawing tool H2 in the slave interface H1. Furthermore, the slave undo button H8 provides a command to undo the currently drawn graphic corresponding to the slave marker; the user can undo the current graphic by simply left-clicking the slave undo button H8. Of course, in addition to displaying markers and medical images, the slave interface H1 also displays the initial medical image containing the markers.

[0120] In some embodiments, the slave interface H1 also provides an exit marker button (not shown), which is used to output an exit marker command, and the slave 200 exits the marker mode according to the exit marker command of the exit marker button. In some embodiments, the marker button H9 is used to output an enable marker command, and the slave 200 starts the marker mode according to the enable marker command of the marker button H9. In another embodiment, the marker button H9 and the exit marker button are integrated into a single button. After entering the slave marker mode, the marker button H9 transforms into the exit marker button, and after exiting the marker mode, the exit marker button transforms back into the marker button H9.

[0121] like Figure 4 As shown, in an exemplary embodiment, the slave-end marker instruction acquisition module 2022 includes a keyboard 2071 and a mouse 2072; the keyboard 2071 allows the user to input keys and text to facilitate the generation of text-based slave-end markers, thereby achieving text annotation; the mouse 2072 allows the user to select and draw graphics to facilitate the generation of graphic-based slave-end markers. In a specific embodiment, the slave-end control device 202 is integrated with the image carriage 203, which can be configured with a partition 211 for placing the keyboard 2071 and the mouse 2072.

[0122] Figure 9 The workflow for creating a slave tag according to a preferred embodiment of the present invention is illustrated. For example... Figure 9 As shown, in a non-limiting embodiment, the process of creating the slave tag includes the following steps:

[0123] Step C1: Start the slave marking instruction acquisition module; Since the slave marking instruction acquisition module 2022 is embedded in the surgical robot program on the slave end, the slave marking function is automatically started after the surgical robot program is run. The slave marking mode needs to be entered to perform the slave marking operation.

[0124] Step C2: Determine whether the master terminal 100 is in a marking state; in one embodiment, the marking functions of the master and slave terminals are mutually exclusive, and only one terminal can perform the marking operation at any given time; if the master terminal 100 is currently in a marking state, the slave terminal 200 automatically enters step C10 to exit the slave marking mode; if the master terminal 100 is not currently in a marking state, the slave terminal 200 can enter the slave marking mode in step C3. Therefore, the slave control device 202 preferably determines whether to mark the initial medical image to obtain the slave marking based on the non-marking state of the master terminal, and preferably sends the local current marking state corresponding to the slave terminal to the master terminal 100 in step C4.

[0125] Step C3: The slave device enters the slave marking mode; after entering the slave marking mode, step C4 is executed: the slave device 200 sends the marking status corresponding to the slave device to the master device 100, thereby sending the marking status of the slave device 200 to the master device 100, so that the master device 100 can determine whether it can enter the slave marking mode based on the marking status of the slave device. For example, the slave marking mode can be entered by clicking the marking button on the screen of the slave device interaction interface using mouse 2072. Preferably, after entering the slave marking mode, the slave device interaction interface automatically displays the preset marking graphic and the slave device cancel button on the screen.

[0126] After entering the slave marking mode, step C5 is executed: determine whether to select a preset marking graphic; in step C5, the user can select a system preset graphic to draw the slave marking, or select a custom graphic to draw the graphic.

[0127] Proceed to step C6: Draw the end markers according to the drawing method selected in the previous step C5; the user can draw end markers on the medical image by moving the mouse.

[0128] After the slave marker is drawn, step C7 can be used to determine whether to cancel the drawn slave marker; if so, proceed to step C8 to delete the latest slave marker, otherwise the program will automatically proceed to step C9.

[0129] Step C8: Delete the latest slave tag, preferably only the most recent slave tag can be deleted at a time.

[0130] Step C9: Determine whether to exit the slave marking process; if yes, exit marking mode C10; if no, continue the loop from C5 to C9.

[0131] In a preferred embodiment, when the slave device 200 enters the slave marking mode, its slave interaction interface automatically displays a preset slave marking graphic and a slave undo button on the screen, and when the slave device 200 exits the slave marking mode, the slave interaction interface automatically hides the preset slave marking graphic and slave undo button on the screen.

[0132] More specifically, in a particular operational method, such as Figure 10 As shown, the slave-side assistant doctor enters slave-side marking mode C3 by clicking the marking button with the mouse to output slave-side marking instructions. After entering slave-side marking mode C3, similar to the master-side, two marking methods can be selected, including: The first marking method: the slave-side assistant doctor determines to draw a preset graphic D2; then, the slave-side assistant doctor selects the preset graphic D3 with the left mouse button; then, the slave-side assistant doctor draws the preset graphic D4 with the left mouse button; The second marking method: the slave-side assistant doctor determines to draw a custom graphic D5; then, the slave-side assistant can draw the custom graphic path D6 with the left mouse button.

[0133] All of the above marking methods can ultimately be undoed (D7). If undoing is confirmed, the current slave-side marking graphic can be undone by selecting the slave-side undo button with the left mouse button. Alternatively, preset graphics can be selected and drawn repeatedly, or a custom graphic path can be redrawn. Therefore, the slave-side assistant doctor enters slave-side marking mode (C3) by clicking the marking button on the slave-side interactive interface. When drawing preset marking graphics, first move the mouse cursor to the graphic selection window, left-click to select the preset marking graphic, and release the mouse button. Next, move the mouse cursor to the drawing area, left-click to move and draw, and release the mouse button after drawing.

[0134] In one specific embodiment, when no preset marker graphic is selected, the slave device 200 draws a custom marker graphic by default. The cursor is moved to the drawing area using the mouse, and the left mouse button is clicked to move and draw the graphic. The mouse button is released after drawing is complete. To undo an operation, the cursor is moved to the undo button in the drawing area, and the left mouse button is clicked to undo the operation.

[0135] Figure 12 The flowchart illustrates a process for sharing a slave tag from the slave end to the master end according to a preferred embodiment of the present invention. For example... Figure 12 As shown, in one embodiment, the process of sharing the slave tag from the slave end to the master end includes the following steps:

[0136] Step 11: The slave device completes the slave marking to generate a slave marking instruction; specifically, the slave device assists the doctor to complete the slave marking operation on the slave device 200 via mouse and / or keyboard;

[0137] Step 12: The slave end saves the position information of the slave end marker, mainly the coordinate information; the slave end marker instruction acquisition module 2022 will save the X and Y coordinate information of all slave end marker points on the screen of the slave end interactive interface in real time, and considering the difference in resolution between the master end display screen and the slave end display screen, it is preferable to also save the resolution information of the slave end screen.

[0138] Step 13: Send the location information of the slave tag to the master end; After the slave tag is completed, the slave information sharing module 2024 will send the location information of the slave tag to the master end 100;

[0139] Step 14: The master end receives the location information of the slave end tag; after receiving the location information of the slave end tag, the master end 100 performs data parsing to obtain the coordinates of the slave end tag.

[0140] Step 15: The master end reconstructs the slave end markers based on the parsed data. During the drawing process, the master end draws the slave end markers proportionally according to the resolution of the master end's display screen.

[0141] Step 16: After the master end completes the drawing of the slave end marker, the master end marker, the slave end marker and the initial medical image will be fused together to obtain the initial medical image fused with the master end marker and the slave end marker and then displayed.

[0142] More specifically, such as Figure 13 As shown, the master terminal 100 obtains the slave terminal tagging information Q1 shared by the slave terminal 200, and uses the saved master terminal tagging information and the established initial medical image Q2 to obtain the tagged medical image based on data fusion technology Q3 and display it to the master terminal Q4. Similarly, the slave terminal 200 obtains the master terminal tagging information Q5 shared by the master terminal 100, and uses the saved slave terminal tagging information and the established initial medical image Q2 to obtain the tagged initial medical image based on data fusion technology Q3 and display it to the slave terminal Q6.

[0143] Therefore, the master terminal 100 and the slave terminal 200 respectively perform data fusion on the data they acquire. The master and slave terminals first acquire medical image data, including but not limited to endoscopic video data and CT images. Then, they acquire slave-end marker information and master-end marker information. After the master terminal 100 and slave terminal 200 respectively acquire the position information of these markers, a data fusion algorithm is used to obtain a medical image with the marker information. Finally, the corresponding initial medical image containing the markers is displayed through the respective display modules of the master and slave terminals.

[0144] More in detail, such as Figure 14As shown, in a specific embodiment, the principle of data fusion is as follows: the custom master-end marker L2 in process L1, the preset slave-end marker L4 in process L3, and the initial medical image in process L5 are fused in process L6 to obtain the master-end fused marked medical image L7, which is displayed on the master-end display screen. Additionally, the slave-end fused marked medical image L8 is also obtained and displayed on the slave-end display screen. This allows the master-end physician to remotely view the surgical scene through the master-end display screen and communicate with the slave-end assistant physician. Furthermore, the slave-end assistant physician can receive timely surgical guidance from the master-end physician and communicate with the master-end physician in the operating room through the slave-end display screen, ensuring the smooth and safe conduct of the surgery.

[0145] Furthermore, a preferred embodiment of the present invention provides a readable storage medium storing a program that, when executed, performs a tag-sharing method as described above by the tag-sharing device. Additionally, a preferred embodiment of the present invention provides an electronic device for performing a tag-sharing method for a surgical robot, the electronic device including a processor and the readable storage medium described above, the processor being configured to execute the program stored on the readable storage medium. It should also be understood that any of the tag-sharing methods described above are equally applicable to the tag-sharing device provided by the present invention.

[0146] While the present invention has been disclosed above, it is not limited thereto. Those skilled in the art can make various modifications and variations to the present invention without departing from its spirit and scope. Therefore, if such modifications and variations fall within the scope of the claims and their equivalents, the present invention also intends to include such modifications and variations.

Claims

1. A doctor's main control console, used as the main terminal of a surgical robot system, characterized in that, The doctor's main control console integrates a first foot pedal, a second foot pedal, a main operating unit, and a main end marker sharing device. The main operating unit includes a first operating arm and a second operating arm. The first foot pedal is used to output an enable marking command, so that the master terminal starts the marking mode according to the enable marking command, so as to allow marking on the initial medical image and put the master terminal and / or the slave terminal of the surgical robot system into an operation lock state; The second foot pedal is used to output an exit marking command, so that the master terminal exits the marking mode according to the exit marking command, prohibits marking the initial medical image, and puts the master terminal and / or the slave terminal into the operation unlock state; The first operating arm is used to select the master end marker according to the operating instructions input by the user; The second manipulator is used to create the master marker on the initial medical image according to the operation command input by the user, using the selected master marker; The master-end marker sharing device is configured to: acquire the currently generated initial medical image; upon acquiring the master-end marker instruction input by the user, send the master-end marker information contained in the master-end marker instruction to the slave end in real time, and fuse the master-end marker information in the initial medical image; upon receiving the slave-end marker information sent in real time by the slave end, fuse the slave-end marker information in the initial medical image.

2. The doctor's main control console according to claim 1, characterized in that, After receiving the slave tag information sent in real time by the slave end, the master tag sharing device fuses the slave tag information into the initial medical image that fuses the master tag information; And / or, the initial medical image, which integrates the master-end marking information and / or the slave-end marking information, is sent to the slave end in real time.

3. The doctor's main control console according to claim 1, characterized in that, The master-end marker sharing device fuses the master-end marker information in the initial medical image, including: reconstructing a master-end marker corresponding to the master-end marker information on the initial medical image based on the position information in the master-end marker information; and fusing the slave-end marker information in the initial medical image, including: reconstructing a slave-end marker corresponding to the slave-end marker information on the initial medical image based on the position information in the slave-end marker information.

4. The doctor's main control console according to claim 1, characterized in that, After the master-end marker sharing device fuses the master-end marker information in the initial medical image and / or fuses the slave-end marker information in the initial medical image, it further includes: Obtain image change information of the currently generated initial medical image, and update the master-end labeling information and / or the slave-end labeling information according to the image change information of the initial medical image, so that the master-end labeling information and / or the slave-end labeling information follow the changes of the initial medical image in real time.

5. The doctor's main control console according to claim 1, characterized in that, The master-end tag sharing device is further configured to: acquire the tag status of the master end and / or the slave end, and output a master-slave control command or an enable tag command according to the tag status of the master end and / or the slave end; wherein the tagging functions of the master end and the slave end are mutually exclusive, and the master end and the slave end execute the tagging operation sequentially according to the priority level; The master-end tag sharing device is further configured to: acquire the tag status of the slave end, and when it is determined that the slave end is currently in a non-tag state, activate the tag mode and send the local current tag status to the slave end in real time.

6. The doctor's main control console according to claim 1, characterized in that, The master-end tag sharing device includes: The master-end image data acquisition module is used to acquire the currently generated initial medical images; The master-end tag instruction acquisition module is used to acquire the master-end tag instruction input by the user, and also to acquire the slave-end tag information sent by the slave end in real time; A master-end data fusion module is used to fuse the master-end labeling information and / or the slave-end labeling information in the initial medical image; The master-end information sharing module is used to send the initial medical image, which integrates the master-end marking information and / or the slave-end marking information, to the slave end in real time, and can also receive data sent by the slave end; The master display module is used to display the initial medical image that integrates the master marking information and / or the slave marking information.

7. The doctor's main control console according to claim 6, characterized in that, The main image data module is configured to process the real-time acquired image data using 3D modeling and artificial intelligence algorithms to generate the initial 3D medical image; the main display module is a 3D display screen to display the 3D medical image in stereoscopic form.

8. The doctor's main control console according to claim 6, characterized in that, The main display module includes a main interactive interface, which displays the initial medical image; the main interactive interface provides drawing tools, a pen, and a main undo button. The first operating arm is used to select a graphic and mark a position on the main interactive interface; the second operating arm is used to perform drawing operations according to the graphic selected by the first operating arm; the main undo button is used to provide an operation command to undo the currently drawn graphic corresponding to the main marker.

9. The doctor's main control console according to claim 8, characterized in that, The main terminal interaction interface is configured such that when the main terminal enters the marking mode, the main terminal interaction interface displays a preset main terminal marking graphic and a main terminal undo button on the screen; When the main device exits the marking mode, the main device's interactive interface hides the preset main device marking graphic and the main device undo button on the screen.

10. A surgical robot system, comprising a master end and a slave end connected in communication, the slave end comprising a surgical robot, characterized in that, The main terminal includes the doctor's main control console as described in any one of claims 1-9; The slave end is configured to: upon receiving master end marker information sent in real time by the master end, fuse the master end marker information into the initial medical image of the slave end, and then the slave end sends the initial medical image fused with the master end marker information and / or the slave end marker information to the master end in real time.

Images