Surgical robot system, control method and device thereof

Abstract

The surgical robot system, control method and device provided by the present application relate to the technical field of medical devices. It includes a surgically planning module and a surgical execution module connected to each other. The surgical planning module is used to obtain medical imaging data and determine a surgical plan based on the medical imaging data; the surgical execution module is used to perform surgical operations according to the surgical plan. The surgical execution module includes a surgical operation learning unit and a surgical robot. The surgical operation learning unit is used to learn the surgical operations of the surgical plan to obtain a learning result, and the surgical robot is used to perform surgical operations according to the surgical plan and the learning result. The present application can realize intelligent and automated diagnosis and treatment from imaging data to surgical operations. By enabling the surgical robot to autonomously learn surgical operations, the surgical robot gradually gains the ability to work independently, thereby performing surgical tasks on its own to reduce the burden on doctors, and can reduce human factors during the surgical process, thereby reducing the risk of surgical errors.

Description

Technical Field

[0001] This application relates to the technical field of medical devices, and particularly to a surgical robot system, its control method and device. Background Art

[0002] With the rapid development of artificial intelligence, more and more intelligent robots replace humans in industrial production, driving, medical treatment and other fields. Existing medical surgical robot systems include a master operation device, a central processor and a slave operation device.

[0003] In related technologies, the working mode of a surgical robot system is as follows: a doctor inputs operation parameters on the master operation device according to medical image instructions; correspondingly, the master operation device collects the doctor's operation parameters and transmits them to the central processor through an interface; the central processor converts the operation parameters into digital signals and transmits them to a stepping motor on the slave operation device through the interface, and the stepping motor drives the slave operation device to perform surgery based on the received digital signals.

[0004] In the above working mode, the entire surgical operation process of the surgical robot system requires a doctor to input operation parameters, with a low degree of intelligence and the possibility of operation errors. Summary of the Invention

[0005] This application provides a surgical robot system, its control method and device to solve the problems of low intelligence and easy operation errors.

[0006] In a first aspect, this application provides a surgical robot system, including: a surgical planning module and a surgical execution module connected to each other, where: the surgical planning module is used to obtain medical image data and determine a surgical plan based on the medical image data; the surgical execution module is used to perform surgical operations according to the surgical plan; the surgical execution module includes: a surgical operation learning unit and a surgical robot, where: the surgical operation learning unit is used to learn the surgical operations of the surgical plan to obtain a learning result; the surgical robot is used to perform surgical operations according to the surgical plan and the learning result.

[0007] Optionally, the surgical planning module includes: a data acquisition unit for obtaining medical image data; a modeling unit for performing three-dimensional reconstruction on the medical image data to obtain a target three-dimensional model; a film reading unit for determining lesion information based on the medical image data and / or the target three-dimensional model; a surgical plan generation unit for generating a surgical plan according to the lesion information, and the surgical plan includes operation steps and surgical tools for the entire surgical process.

[0008] Optionally, the film reading unit is specifically configured to: determine lesion information by comparing medical image data with standard image data, and / or comparing a target three-dimensional model with a standard three-dimensional model, where the lesion information includes at least one of the location characteristics, shape characteristics, CT value, and pathological result of the lesion.

[0009] Optionally, the film reading unit includes a pathological prediction subunit for predicting the pathological result of the lesion based on the medical image data and / or the target three-dimensional model, and the lesion information includes the pathological result.

[0010] Optionally, the modeling unit includes: an image segmentation subunit for performing image segmentation on the medical image data through a segmentation algorithm and extracting the image data of different tissues and / or organs from the segmentation result; a first modeling subunit for performing three-dimensional reconstruction on the image data of different tissues and / or organs respectively through a three-dimensional modeling algorithm to obtain corresponding tissue three-dimensional models and / or organ three-dimensional models; a second modeling subunit for fusing the tissue three-dimensional models and / or organ three-dimensional models to obtain a target three-dimensional model.

[0011] Optionally, the modeling unit further includes a segmentation subunit and / or an identification subunit, where: the segmentation subunit is used for segmenting at least part of the tissues and / or organs in the target three-dimensional model through a segmentation algorithm; the identification subunit is used for color-identifying at least part of the tissues and / or organs in the target three-dimensional model through an identification algorithm.

[0012] Optionally, the surgical planning module further includes: a naming unit for naming and segment-naming at least part of the different tissues and / or organs of the target three-dimensional model based on the lesion information; a surgical plan generation unit specifically configured to determine a surgical plan according to the naming result and the lesion information, and the surgical plan includes surgical operation plans for different named tissues and / or organs.

[0013] Optionally, the surgical planning module further includes: a surgical simulation unit for performing surgical simulation training according to the surgical plan to determine whether the surgical plan is reasonable.

[0014] Optionally, the surgical operation learning unit includes: a video acquisition subunit for acquiring the surgical video of the surgical plan; a video decomposition subunit for decomposing the surgical video to obtain surgical pictures and / or surgical video segments; a calibration subunit for calibrating the anatomical structures, surgical tools, and operation steps involved in the surgical pictures and / or surgical video segments through a calibration algorithm to obtain a calibration result; a learning subunit for learning the calibration result to obtain a learning result.

[0015] Optionally, a surgical robot includes: a surgical operation subunit for performing surgical operations; and a surgical assistance subunit for performing surgical assistance operations, where the surgical assistance operations assist in the execution of the surgical operations.

[0016] Optionally, the surgical execution module further includes a plan interpretation unit and a control unit, where: the plan interpretation unit is configured to interpret a surgical plan and obtain operation parameters based on the interpretation result, medical image data, and / or a target three-dimensional model, and the operation parameters include the robotic arm movement parameters of the surgical robot; the control unit is configured to control the surgical robot to perform surgical operations according to the operation parameters.

[0017] Optionally, the surgical assistance subunit includes an identification subunit for identifying anatomical structures during a surgical procedure and feeding back the identification result to the control unit; correspondingly, the control unit is further configured to obtain the identification result and control the surgical robot to perform surgical operations based on the identification result and the operation parameters.

[0018] Optionally, the surgical assistance subunit includes: a monitoring subunit for monitoring the surgical operations performed by the surgical robot during a surgical procedure and transmitting the monitoring result to a display.

[0019] In a second aspect, the present application provides a control method for a surgical robot system, where the surgical robot system includes a surgically planning module and a surgical execution module connected to each other, and the control method includes: obtaining medical image data through the surgically planning module and determining a surgical plan based on the medical image data; and performing surgical operations according to the surgical plan through the surgical execution module.

[0020] In a third aspect, the present application provides a control device for a surgical robot system, including: a memory, a processor; the memory is configured to store computer execution instructions; the processor is configured to execute the computer execution instructions to implement the control method provided in the second aspect as described above.

[0021] In a fourth aspect, the present application provides a computer-readable storage medium, where computer execution instructions are stored in the computer-readable storage medium, and when the computer execution instructions are executed, they are used to implement the control method provided in the second aspect as described above.

[0022] In a fifth aspect, the present application provides a computer program product, including a computer program; when the computer program is executed, it implements the control method provided in the second aspect as described above.

[0023] The surgical robot system provided by this application, its control method and device include a surgically planning module and a surgical execution module connected to each other. The surgical planning module is used to obtain medical image data and determine a surgical plan based on the medical image data. The surgical execution module is used to perform surgical operations according to the surgical plan. The surgical execution module includes a surgical operation learning unit and a surgical robot. The surgical operation learning unit is used to learn the surgical operations of the surgical plan to obtain a learning result, and the surgical robot is used to perform surgical operations according to the surgical plan and the learning result. This application can achieve intelligent and automated diagnosis and treatment from image data to surgical operations. By enabling the surgical robot to autonomously learn surgical operations, the surgical robot gradually gains the ability to work independently, thereby performing surgical tasks on its own, reducing the burden on doctors, and being able to reduce human factors during the surgical process, thus reducing the risk of surgical errors. In addition, complete automated surgical operations can avoid possible signal delays in the signal transmission process of the surgical robot system, further reducing the surgical risk. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] The drawings herein are incorporated into the specification and form a part of the specification, showing embodiments consistent with this application and, together with the specification, are used to explain the principles of this application.

[0025] Figure 1 Schematic structural diagram of the surgical robot system provided by an embodiment of this application Figure 1 ;

[0026] Figure 2 Schematic structural diagram of the surgical robot system provided by an embodiment of this application Figure 2 ;

[0027] Figure 3 Schematic structural diagram of the surgical robot system provided by an embodiment of this application Figure 3 ;

[0028] Figure 4 Schematic structural diagram of the surgical robot system provided by an embodiment of this application Figure 4 ;

[0029] Figures 5a - 5d Schematic diagram for naming the target three-dimensional model provided by an embodiment of this application;

[0030] Figure 6 Schematic diagram of the surgical guide provided by an embodiment of this application;

[0031] Figure 7 Schematic diagram V of the structural diagram of the surgical robot system provided by an embodiment of this application;

[0032] Figure 8 Schematic structural diagram of the surgical robot system provided by an embodiment of this application Figure 6 ;

[0033] Figure 9 Flowchart of the control method for the surgical robot system provided by the embodiments of this application.

[0034] Description of the reference numerals:

[0035] 110 - Surgical planning module;

[0036] 120 - Surgical execution module;

[0037] 200 - Surgical planning module;

[0038] 210 - Data acquisition unit;

[0039] 220 - Modeling unit;

[0040] 230 - Film reading unit;

[0041] 240 - Surgical plan generation unit;

[0042] 320 - Modeling unit;

[0043] 321 - Image segmentation sub-unit;

[0044] 322 - First modeling sub-unit;

[0045] 323 - Second modeling sub-unit;

[0046] 400 - Surgical robot system;

[0047] 410 - Data acquisition unit;

[0048] 420 - Modeling unit;

[0049] 421 - Image segmentation sub-unit;

[0050] 422 - First modeling sub-unit;

[0051] 423 - Second modeling sub-unit;

[0052] 424 - Segmentation sub-unit;

[0053] 425 - Identification sub-unit;

[0054] 430 - Film reading unit;

[0055] 431 - Pathology prediction sub-unit;

[0056] 440 - Surgical plan generation unit;

[0057] 450 - Naming unit;

[0058] 460 - Surgical simulation unit;

[0059] 700 - Surgical execution module;

[0060] 710 - Surgical operation learning unit;

[0061] 720 - Surgical robot;

[0062] 800 - Surgical execution module;

[0063] 810 - Surgical operation learning unit;

[0064] 811 - Video acquisition subunit;

[0065] 812 - Video decomposition subunit;

[0066] 813 - Calibration subunit;

[0067] 814 - Learning subunit;

[0068] 820 - Surgical robot;

[0069] 821 - Surgical operation subunit;

[0070] 822 - Surgical assistance subunit;

[0071] 830 - Solution interpretation unit;

[0072] 840 - Control unit.

[0073] Through the above - mentioned drawings, the specific embodiments of the present application have been shown, and there will be more detailed descriptions hereinafter. These drawings and written descriptions are not intended to limit the scope of the concept of the present application in any way, but to illustrate the concept of the present application to those skilled in the art by referring to specific embodiments. Detailed Description of the Embodiments

[0074] Here, the exemplary embodiments will be described in detail, and the examples are shown in the drawings. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present application. On the contrary, they are merely examples of devices and methods consistent with some aspects of the present application as detailed in the appended claims.

[0075] Figure 1 Schematic structure of the surgical robot system provided by the embodiments of the present application Figure 1 . As Figure 1As shown in the figure, the system includes: a surgial planning module 110 and a surgical execution module 120 which are interconnected. The surgical planning module 110 is used to obtain medical image data and determine a surgical plan based on the medical image data; the surgical execution module 120 is used to perform surgical operations according to the surgical plan; the surgical execution module 120 includes: a surgical operation learning unit and a surgical robot, where: the surgical operation learning unit is used to learn the surgical operations of the surgical plan to obtain a learning result; the surgical robot is used to perform surgical operations according to the surgical plan and the learning result.

[0076] Among them, the medical image data can be obtained by medical imaging devices S such as Computed-Tomography (CT) devices, Megnetic Resonace Imaging (MRI) devices, or ultrasonic devices, and then the surgical planning module 110 obtains it from these medical imaging devices S. According to the lesion information reflected by the medical image data, the surgical planning module 110 determines a complete surgical plan for performing surgical operations, and the surgical execution module 120 performs specific surgical operations according to this surgical plan. A data connection can be established between the surgical planning module 110 and the surgical execution module 120, that is, the surgical plan determined by the surgical planning module 110 can be sent to the surgical execution module 120, which can be a wired connection or a wireless connection. The surgical plan obtained by the surgical execution module 120 can be a simulated surgical video, or in the form of pictures, texts, etc. of the surgical operation steps, or a combination of one or more of its video, animation, pictures, and texts.

[0077] Exemplarily, the surgical operation learning unit can learn from the surgical operation video F. Specifically, the surgical operation video F refers to a real surgical video from the past that is similar to the current surgical plan. The surgical operation learning unit can match and learn from the accumulated video library. For the current surgical plan, the surgical operation learning unit can match one or more similar videos F for learning at the same time. It should be understood that the surgical operations of the surgical plan are not only in the form of video F, but also in the form of pictures or texts, or in the form of a combination of video, animation, pictures, or texts and any combination thereof.

[0078] This application can realize intelligent and automated diagnosis and treatment from image data to surgical operations. By enabling the surgical robot to autonomously learn surgical operations, the surgical robot gradually gains the ability to work independently, thereby performing surgical tasks on its own, reducing the burden on doctors, and being able to reduce human factors during the surgical process, thus reducing the risk of surgical errors. In addition, the complete automated surgical operation can avoid possible signal delays in the signal transmission process of the surgical robot system, further reducing the surgical risk.

[0079] Figure 2 Schematic diagram of the surgical robot system provided by the embodiment of the present application Figure 2 As Figure 2 shown, the surgical planning module 200 may include a data acquisition unit 210, a modeling unit 220, a film reading unit 230, and a surgical plan generation unit 240.

[0080] The data acquisition unit 210 is used to acquire medical image data. In some embodiments, the data acquisition unit 210 may be a computer device connected to the medical imaging device S, which can directly acquire the patient's medical image data from the medical imaging device S. In other embodiments, the data acquisition unit 210 may also be a computer device with a USB interface or other storage device interfaces. The medical image data acquired from the medical imaging device S can be stored in the storage device, and the medical image data can be acquired by connecting the storage device to the computer device. Among them, the data collected by the medical imaging device S can be named after the patient's name or other numbers to facilitate the computer device to acquire the corresponding patient's data. It should be understood that the "connection" in this embodiment can be any form of connection that can achieve data transmission, such as wired connection or wireless connection, etc., or data connection can also be achieved by means of email sending, etc. It should be understood that the data acquisition unit 210 can be connected to multiple medical imaging devices S, or can be connected to a remote medical imaging device S, or can also be directly connected to the hospital's picture archiving and communication system to acquire medical image data.

[0081] The modeling unit 220 of the surgical planning module 200 is used to perform three-dimensional reconstruction based on the medical image data to obtain a target three-dimensional model. Specifically, the modeling unit 220 can be implemented in the form of a computer device. Optionally, it can be a computer device connected to the data acquisition unit 210. The data acquisition unit 210 and the modeling unit 220 can also be implemented by the same computer device, or by different computer devices. For example, when the data acquisition unit 210 and the modeling unit 220 are implemented by the same computer device, data transmission can be achieved through data calling; when the data acquisition unit 210 and the modeling unit 220 are implemented by different computer devices, data transmission can be achieved through wired connection, wireless connection or network sending, etc. This embodiment does not make specific limitations, as long as the data acquisition unit 210 can be connected to the modeling unit 220 to achieve data transmission.

[0082] Further, after the data acquisition unit 210 acquires the image data, it can sequentially send the medical image data to the modeling unit 220 for modeling. Specifically, it can be sent in order according to a preset priority. For example, the medical image data can also be marked according to the urgency of the surgery: urgent, general, or non-urgent, etc. The data acquisition unit 210 can then send the medical image data to the modeling unit 220 for modeling in the order from urgent to general to non-urgent. In other embodiments, the medical image data can also be sorted and sent in the order of the time of data acquisition. Among them, there can also be multiple modeling units 220, that is, multiple modeling units 220 are connected to the data acquisition unit 210, and the data acquisition unit 210 can send the medical image data to different modeling units 220 for parallel processing according to a preset order. It should be understood that these multiple modeling units 220 can be implemented by the same computer device or by different computer devices, and this embodiment does not make specific limitations on this.

[0083] Figure 3 Schematic structural diagram of the surgical robot system provided by the embodiment of the present application Figure 3 。The following combines Figure 2 and Figure 3 to explain the modeling unit 320. Optionally, the modeling unit 320 may include an image segmentation subunit 321, a first modeling subunit 322, and a second modeling subunit 323. Among them, the image segmentation subunit 321 is used to perform image segmentation on the medical image data through a segmentation algorithm to extract the image data of different tissues and / or organs; the first modeling subunit 322 is used to perform three-dimensional reconstruction on the image data of different tissues and / or organs respectively through a three-dimensional modeling algorithm to obtain corresponding tissue three-dimensional models and / or organ three-dimensional models; the second modeling subunit 323 is used to fuse the tissue three-dimensional models and / or organ three-dimensional models to obtain a target three-dimensional model.

[0084] When the image segmentation subunit 321 performs image segmentation and extraction on the medical image data, it can automatically determine which tissues and / or organs to extract according to the scanned part in the medical image data. In some embodiments, in order to achieve automatic extraction of tissues and / or organs, the medical image data can be named head scan data, abdominal scan data, or chest scan data, etc., and the image segmentation subunit 321 can set the types of tissues and / or organs to be extracted for the scan data of these specific parts respectively. In other embodiments, corresponding features, such as CT values or shapes, can be further defined for these types of tissues and / or organs, so as to facilitate the image segmentation subunit 321 to automatically extract the corresponding tissues and / or organs according to the scanned part.

[0085] Exemplarily, the image segmentation algorithm can be a semantic segmentation (SS) algorithm, such as the U-NET algorithm, and the 3D modeling algorithm can be a marching cubes (MC) algorithm or a ray casting (RC) algorithm, etc. Before applying the above algorithms for image segmentation and modeling, the algorithms need to be trained to obtain more accurate segmentation results and modeling results. The training of the above algorithms can be performed using a dataset including original medical image data and the segmentation results and modeling results after manual segmentation and model reconstruction and fusion. Specifically, the image segmentation algorithm can be used to perform image segmentation on the medical image data, and the segmentation result can be compared with the manually segmented result, so as to correct the algorithm according to the comparison result and perform iteration to improve the accuracy of the algorithm calculation; the modeling algorithm can be used to perform modeling on the manually segmented result, and the modeling result can be compared with the manually modeled result, so as to correct the algorithm according to the comparison result and perform iteration to improve the accuracy of the algorithm calculation. Among them, the dataset can include image segmentation and modeling data of different types of tissues and / or organs. It should be understood that the larger the amount of data in the dataset, the higher the accuracy of the calculation result of the trained algorithm will be.

[0086] In the surgical planning of some specific diseases, it may be necessary to segment some tissues and / or organs. For example, in a lobectomy or hepatectomy, it is necessary to segment the lung or liver to facilitate the doctor to quickly and accurately find the segment where the lesion is located and perform surgical planning for the location where the lesion is located. Therefore, in some embodiments, the modeling unit 220 may further include a segmentation subunit for segmenting at least part of the tissues and / or organs in the target 3D model through a segmentation algorithm.

[0087] Exemplarily, the segmentation of the lung and the segmentation of the liver are both based on medical standards. To achieve the automated processing of the surgical planning process, the segmentation subunit can implement the segmentation of tissues and / or organs through a segmentation algorithm, where the segmentation algorithm can be a deep learning algorithm, such as neural networks (NN), convolutional neural networks (CNN), or recurrent neural networks (RNN), etc. Similarly, before applying the segmentation algorithm to perform the segmentation of tissues and / or organs, the segmentation algorithm also needs to be trained. Among them, the segmentation algorithm can learn based on the model that has been manually segmented and perform the segmentation of the tissues and / or organs in the target 3D model based on the learning result to achieve the automatic segmentation processing of the target 3D model.

[0088] In some other embodiments, the modeling unit 320 may further include an identification subunit, which is configured to perform color identification on at least some of the tissues and / or organs in the target three-dimensional model through an identification algorithm, so as to distinguish different tissues and / or organs in the target three-dimensional model. For example, different tissues are identified with different colors. Specifically, arteries can be identified with red, veins with blue, tracheas with gray, lungs with yellow, and lesions with green, etc., so as to facilitate doctors to view the model.

[0089] Still as Figure 2 shown, the surgical planning module 200 may further include a film reading unit 230, and the film reading unit 230 is configured to determine lesion information based on medical image data and / or the target three-dimensional model. Optionally, the film reading unit 230 determines the lesion information by comparing the medical image data with standard image data and / or comparing the target three-dimensional model with a standard three-dimensional model. The lesion information may include at least one of the location feature, shape feature, CT value, and pathological result of the lesion.

[0090] Among them, a lesion is usually a place where there is a difference from the standard data. For example, the standard CT value of the liver is 55±10HU, but by looking through the medical image data, it is found that the CT value of some parts of the liver is greater than or less than the standard CT value of the liver. In this way, the part with abnormal CT value can be determined as the lesion part. Another example is in hypertrophic obstructive cardiomyopathy, the standard thickness of the interventricular septum is about 12mm, but by looking through the medical image data and / or the target three-dimensional model, it is found that the thickness of some parts of the interventricular septum is greater than the standard thickness of the interventricular septum, and the part with abnormal thickness can be determined as the lesion part.

[0091] Still as Figure 2 shown, the surgical planning module 200 may further include a surgical plan generation unit, and the surgical plan generation unit is configured to generate a surgical plan according to the lesion information, and the surgical plan includes the operation steps and surgical tools for the entire surgical process. Specifically, the surgical plan can be generated based on the lesion information and in combination with surgical guidelines. For example, in a pulmonary lesion resection surgery, one or more of wedge resection, subsegmental resection, combined subsegmental resection, segmental resection, and lobectomy can be selected as the surgical plan according to the lesion information and in combination with surgical guidelines. Another example is that when a patient has multiple lesions, different surgical plans can be determined for different lesions respectively. Optionally, the surgical plan can be in the form of an animation of a virtual surgery on the target three-dimensional model.

[0092] In some embodiments, the surgical plan can be determined based only on information such as the location and shape features of the lesion. In other embodiments, the surgical plan also needs to consider the pathological results of the lesion, that is, if the pathological results of the lesion are different, there will be significant differences in the surgical plan. In the prior art, the surgical plan is determined by performing intraoperative pathological analysis during the operation, which requires sampling and test analysis during the operation. That is to say, sampling needs to be performed after the patient is under general anesthesia and the sample needs to be tested and analyzed. During the test analysis process, the patient needs to wait for about twenty minutes under general anesthesia. After the test doctor issues the pathological report, the surgeon then selects the corresponding surgical plan according to the pathological results for the operation. This will prolong the operation time, increase the surgical risk, and the accuracy of intraoperative pathological analysis depends to a large extent on the experience of the test doctor, which places high requirements on the test doctor.

[0093] Therefore, in order to more specifically plan the surgical plan, the film reading unit 230 may further include a pathological prediction subunit, which is used to predict the pathological results of the lesion based on the medical image data and / or the target three-dimensional model. Among them, the lesion information includes the pathological results, that is, the film reading unit 230 determines the pathological results through the pathological prediction subunit. The predicted pathological results can be specifically determined according to some features of the lesion. For example, they can be data such as the shape feature and / or CT value of the lesion. The shape feature of the lesion may include the shape, size, edge smoothness, etc. of the lesion, and the CT value of the lesion can be the CT value of each voxel in the lesion or the average CT value of the lesion. Among them, the shape feature and / or CT value of the lesion can be determined according to the medical image data and / or the target three-dimensional model.

[0094] In some embodiments, multiple groups of lesion information obtained from the medical image data and / or the target three-dimensional model and the pathological results of the lesion specimens obtained by actual surgical cutting can be pre-correspondingly stored in the database. When the pathological prediction subunit performs pathological prediction, it can match based on the lesion information / lesion features determined from the medical image data and / or the target three-dimensional model in the database to obtain the matching pathological results, and the matching pathological results are the predicted pathological results.

[0095] In some embodiments, in order to improve the accuracy of pathological prediction, lesion information determined from the medical image data and / or the target three-dimensional model and their corresponding actual pathological results can also be continuously collected to expand the database and help improve the accuracy of pathological prediction. In other embodiments, the pathological prediction subunit can also be implemented through a pathological prediction algorithm. Similarly, a data set including multiple groups of lesion information obtained from the medical image data and / or the target three-dimensional model and the pathological results of the lesion specimens obtained by actual surgical cutting can be used to train the pathological prediction algorithm to improve the accuracy of the prediction results of the pathological prediction subunit.

[0096] Exemplarily, the surgical plan generated by the surgical plan generation unit can also be output in the form of a structured report. Among them, the structured report can include information such as patient information, disease brief introduction, lesion information, model anatomical analysis, and surgical plan. In order to ensure that the automatically generated surgical plan is reasonable, the structured report can also be sent to the doctor for review. Since the structured report includes patient information, disease brief introduction, lesion information, model anatomical analysis, and surgical plan, the doctor can conveniently and quickly evaluate the surgical plan without viewing other materials. If the doctor approves the surgical plan, the surgical plan can be approved. If the doctor believes that the surgical plan needs to be modified, the surgical plan can be modified and then uploaded again.

[0097] In order to achieve the intelligence and automation of the surgical robot system, the surgical execution module needs to interpret the surgical plan output by the surgical planning module. Therefore, in order to facilitate the surgical robot to read the surgical plan output by the surgical planning module, the surgical planning module 200 can also include a naming unit. The naming unit is used to name and / or segment at least some different tissues and / or organs of the target three-dimensional model based on the lesion information. The surgical plan generation unit can determine the surgical plan according to the naming result and the lesion information. The surgical plan includes surgical operation plans for different named tissues and / or organs. Specifically, the naming unit can name the tissues or organs around the lesion and the tissues or organs associated with the lesion and other parts. Further, in order to achieve the automatic naming of tissues or organs, the naming unit can also adopt a naming algorithm, and the naming algorithm can be a deep learning algorithm, which is used to name each tissue and / or organ of the target three-dimensional model after learning based on the named model.

[0098] In some other embodiments, the surgical planning module 200 can also include a surgical simulation unit, which is used to perform surgical simulation training according to the surgical plan to determine whether the surgical plan is reasonable. Among them, the surgical simulation training can be virtual execution to help the doctor judge whether the surgical plan is reasonable, that is, directly simulate cutting the cutting part on the target three-dimensional model to determine whether the cutting is reasonable, etc., or control the surgical execution module to directly perform a simulated operation so that the doctor can check whether the surgical operation of the surgical execution module meets the expectations before the operation, further reducing the surgical risk.

[0099] The surgical planning module of the surgical robot system of the present application will be further explained below in combination with a specific pulmonary nodule resection operation.

[0100] Figure 4 Schematic diagram of the structure of the surgical robot system provided by the embodiment of the present application Figure 4 As Figure 4As shown, the surgical robot system 400 includes a data acquisition unit 410, a modeling unit 420, an image review unit 430, a surgical plan generation unit 440, a naming unit 450, and a surgical simulation unit 460. Among them, the modeling unit 420 includes an image segmentation subunit 421, a first modeling subunit 422, a second modeling subunit 423, a segmentation subunit 424, and an identification subunit 425. The image review unit 430 includes a pathological prediction subunit 431.

[0101] For the specific lung nodule resection surgery proposed in this embodiment, the diagnosis and treatment steps of the surgical robot system can be as follows:

[0102] First step: The data acquisition unit 410 acquires the medical image data of the patient and names the medical image data as chest scan data.

[0103] Second step: The modeling unit 420 extracts the lungs and their associated tissues from the medical image data, performs model reconstruction respectively, and then fuses the individual tissue models to obtain the final target three-dimensional model. The associated tissues include the trachea, blood vessels, and lesions, etc.

[0104] Third step: The image review unit 430 reviews the medical image data and / or the target three-dimensional model data to determine the lesion (the lesion is a lung nodule in this embodiment).

[0105] Fourth step: The modeling unit 420 performs segmentation processing on the lung model to obtain lung segmentation data.

[0106] Fifth step: The pathological prediction subunit 431 of the image review unit 430 determines the pathological result of the lung nodule. In this embodiment, the size of the lung nodule is determined to be 6×8 mm and the CT value is -350 HU through the medical image data and / or the target three-dimensional model. The lung nodule is predicted to be invasive adenocarcinoma through a pathological prediction algorithm or by matching in a pathological database.

[0107] Sixth step: The naming unit 450 names the different tissues and / or organs of the target three-dimensional model after the above processing. The naming can specifically include naming tissues such as lung segments, blood vessels, trachea, and lesions. When naming the segmented organs, at least the lung segment where the lesion is located needs to be named. For the convenience of generating a surgical plan, it is also necessary to name the blood vessels and / or the trachea. Naming the blood vessels and / or the trachea includes naming the types of blood vessels and also naming the segments of blood vessels.

[0108] In this embodiment, naming the target three-dimensional model specifically includes naming the lung segment where the nodule is located and the blood vessels and trachea associated with or around this lung segment.

[0109] Figures 5a - 5dSchematic diagram for naming the target three-dimensional model provided by the embodiment of the present application. Figures 5a - 5d Exemplarily, a specific method for naming the target three-dimensional model is given. For example, a lung segment can be named as S1+2 ( Figure 5a as shown), an artery can be named as A1+2a, A1+2b, A1+2c, A3, and A4+5 ( Figure 5b as shown), a vein can be named as V1+2a, V1+2b+c, V3b, V3c, and V4+5 ( Figure 5c as shown), and a trachea can be named as B1+2, B3, and B4+5 ( Figure 5d as shown), where S represents a lung segment, A represents an artery, V represents a vein, and B represents a trachea.

[0110] Step 7: The surgical plan generation unit 440 determines that the lung segment to be resected is the posterior segment of the upper left lung apex, i.e., S1+2, according to the above naming results and lesion information.

[0111] Further, in combination with the surgical guidelines, the surgical plan is determined as follows: blocking A1+2a by the watershed analysis method and precisely resected the nodule in the posterior segment of the upper left lung apex. The surgical guidelines can be classified according to the lesion information, and different surgical plans can be preset for different types of lesion information. For example, for lung nodules, the surgical guidelines can include regulations on the treatment strategies for nodules of different sizes and different pathologies, so that the surgical plan generation unit 440 can determine the surgical plan based on the surgical guidelines.

[0112] Figure 6 Schematic diagram of the surgical guidelines provided by the embodiment of the present application. As Figure 6 shown, the surgical guidelines show the corresponding pathological results (illustrated pathology) and surgical strategies (illustrated strategies) for different medical imaging data (illustrated images). Specifically, the surgical guidelines specifically stipulate that when the nodule size is less than 3 cm, different strategies can be respectively specified according to information such as the average CT value and the pathological result. When the average CT value of the nodule is -700 HU to -400 HU and the pathological result is atypical adenomatous hyperplasia or carcinoma in situ, the corresponding treatment strategy is follow-up observation, that is, there is no need for subsequent surgical plan planning; when the average CT value of the nodule is greater than 1 and the pathological result is invasive adenocarcinoma, the corresponding treatment strategy is lobectomy, and the surgical plan needs to be formulated according to the surgical strategy. The formulation process of the surgical plan can refer to the above embodiments and will not be elaborated here. For some other specific situations of nodules, the surgical strategy can also include operations such as designing the resection margin sphere to determine the surgical plan. The surgical strategy can also include specifying different blood vessel and trachea treatment methods for the resection of different segments. For example, the resection of the posterior segment of the upper left lung apex also corresponds to surgical strategies such as blocking the A1+2a blood vessel.

[0113] Optionally, the modeling unit 420 can also distinguish different tissues in the built target three-dimensional model through the identification subunit 425. For example, different tissues can be identified with different colors. Specifically, arteries can be identified with red, veins with blue, tracheas with gray, lungs with yellow, and lesions with green, etc., so as to facilitate the user to view the model.

[0114] The above embodiments introduce the surgical planning module of the surgical robot system provided by the embodiments of the present application. Next, the surgical execution module of the surgical robot system will be explained.

[0115] Figure 7 FIG. five is a schematic structural diagram of the surgical robot system provided by the embodiments of the present application. As Figure 7 shown, the surgical execution module 700 of the surgical robot system includes a surgical operation learning unit 710 and a surgical robot 720. The surgical operation learning unit is used to learn the surgical operations of the surgical plan to obtain a learning result; the surgical robot is used to perform surgical operations according to the surgical plan and the learning result.

[0116] In some embodiments, the method of learning the surgical operations of the surgical plan may include: obtaining a surgical video similar to the surgical plan from a past surgical operation video library (such as Figure 1 the video F shown), and learning the surgical operation sequence, surgical tools, anatomical structures, etc. from the surgical video. By enabling the surgical robot to autonomously learn surgical operations, the surgical robot gradually gains the ability to work independently, thereby performing surgical tasks on its own, reducing the burden on doctors, and being able to reduce human factors during the surgical process, thus reducing the risk of surgical errors.

[0117] Figure 8 FIG. is a schematic structural diagram of the surgical robot system provided by the embodiments of the present application Figure 6 . As Figure 8 shown, optionally, the surgical operation learning unit 810 may include a video acquisition subunit 811, a video decomposition subunit 812, a calibration subunit 813, and a learning subunit 814. The video acquisition subunit 811 acquires the surgical video of the surgical plan; the video decomposition subunit 812 is used to decompose the surgical video to obtain surgical pictures and / or surgical video segments; the calibration subunit 813 is used to calibrate the anatomical structures, surgical tools, and operation steps involved in the surgical pictures and / or surgical video segments to obtain a calibration result; the learning subunit 814 is used to learn the calibration result to obtain a learning result.

[0118] Specifically, the video acquisition subunit 811 can obtain the surgical video through forms such as a USB interface, or can also obtain the surgical video by retrieving from a network database. This embodiment does not make specific restrictions on this. Among them, the method for obtaining the surgical video from the surgical video database can specifically include: the surgical plan generated by the surgical planning module can include multiple tags. Similarly, the surgical videos in the surgical video database can also be respectively marked with multiple tags, so as to facilitate quickly retrieving the most matching surgical video in the surgical video database based on these tags. For example, in the surgical plan of "blocking A1+2a counterstaining and precisely excising the nodule in the posterior segment of the left upper pulmonary apex" provided in the above embodiment, the tags of this surgical plan can include "watershed analysis method", "blocking", "A1+2a", "counterstaining", "excision", "left upper lung", "posterior segment", and "nodule" and other tags, and can even include information tags related to the patient, such as "adult", "child", "infant", "male", or "female", etc. When obtaining the surgical video, the surgical video with the closest surgical plan can be retrieved and matched in the database according to the above tags.

[0119] The method for the video decomposition subunit 812 to decompose the surgical video can specifically be: extracting key frames from the surgical video as surgical pictures, or it can also be segmenting the surgical video to segment the surgical video into video segments corresponding to different surgical operations. Among them, the surgical pictures can be used to learn anatomical structures and surgical instruments, etc., and the surgical video segments can be used to learn surgical operations.

[0120] The method for the calibration subunit 813 to calibrate the surgical pictures can specifically include: calibrating the anatomical structure of the tissues or organs in the surgical pictures and calibrating the surgical instruments. Among them, the calibration of the tissues and / or organs in the surgical pictures can also be achieved through a calibration algorithm. Similarly, the calibration algorithm can be a deep learning algorithm. Before applying the calibration algorithm for calibration, the algorithm also needs to be trained to obtain a more accurate calibration result. Among them, the training of the calibration algorithm can be to obtain surgical pictures including anatomical structure calibration and surgical instrument calibration results, and train the calibration algorithm based on the surgical pictures including the calibration results. Among them, the surgical pictures including the calibration results used to train the calibration algorithm can be achieved through manual calibration.

[0121] The method for the calibration subunit 813 to calibrate the surgical video segment can specifically be: calibrating the surgical operation corresponding to the surgical video segment. For example, the calibration subunit 813 is also used to calibrate the specific process of the surgical operation, calibrate the surgical instruments used in the surgical operation, and calibrate the operation position and / or orientation, etc. of the surgical operation. The learning subunit 814 learns based on these calibration results to obtain a learning result.

[0122] Still as Figure 8 shown, optionally, the surgical execution module 800 may further include a surgical plan interpretation unit 830 and a control unit 840, where: the surgical plan interpretation unit 830 is configured to interpret the surgical plan and obtain operation parameters based on the interpretation result, medical image data, and / or the target three-dimensional model, and the operation parameters include the robotic arm movement parameters of the surgical robot; the control unit 840 is configured to control the surgical robot 820 to perform surgical operations according to the operation parameters. Specifically, the surgical plan interpretation unit 830's interpretation of the surgical plan specifically includes: determining specific surgical operations based on the learning result of the surgical video of the surgical plan (i.e., the learning result obtained by the learning subunit 714 in the above embodiment), and determining operation parameters in combination with medical image data and / or the target three-dimensional model on the basis of determining the surgical operations.

[0123] The control unit 840 controls the surgical robot 820 to perform surgical operations based on the operation parameters. Among them, the operation parameters may be parameters related to how the robotic arms of the surgical robot 820 move. Specifically, these operation parameters need to determine the positions of each organizational structure from the medical image data and / or the target three-dimensional model, and calculate and obtain them in combination with the positioning of the surgical robot 820 relative to the human body during the operation.

[0124] Still as Figure 8 shown, the surgical robot 820 specifically includes: a surgical operation subunit 821 and a surgical assistance subunit 822, where the surgical operation subunit 821 is configured to perform surgical operations; the surgical assistance subunit 822 is configured to perform surgical assistance operations, and the surgical assistance operations assist in the execution of surgical operations. Among them, the surgical operation subunit 821 may include surgical instruments such as scalpels, hemostatic forceps, tissue scissors, and tissue forceps, and the surgical assistance subunit 822 may include auxiliary instruments such as endoscopes, probes, and sensors. Both the surgical operation subunit 821 and the surgical assistance subunit 822 can be loaded on the robotic arms of the surgical robot 820.

[0125] Taking the da Vinci surgical robot as an example, the da Vinci surgical robot includes 4 robotic arms, which are respectively used to hold and move the surgical operation subunit 821 and the surgical assistance subunit 822. The instruments loaded on each robotic arm can be replaced according to specific surgical operations, and which instrument should be loaded for each surgical operation can be determined based on the learning result of the surgical operation learning unit 810. Further, in order to realize the automatic replacement of instruments by the robotic arms, the surgical robot 820 may further include an instrument rack, and the robotic arms can realize the automatic replacement of instruments on the instrument rack.

[0126] Optionally, the surgical assistance subunit 822 may further include an identification subunit for identifying anatomical structures during a surgical procedure and feeding back the identification result to the control unit. Correspondingly, the control unit 840 is further configured to obtain the identification result and control the surgical robot 820 to perform a surgical operation based on the identification result and the operation parameters.

[0127] Specifically, the identification subunit may include a sub-controller and an endoscope or an identification probe, etc. Taking the endoscope as an example, an image of a specific tissue at the surgical site can be obtained through the endoscope and fed back to the sub-controller for identification. When the sub-controller identifies that the tissue in the image obtained by the endoscope is the tissue to be cut in the surgical plan, the identification result is fed back to the control unit 840, and the control unit 840 controls the surgical robot 820 to cut the tissue using the surgical operation subunit 821. When the sub-controller identifies that the tissue in the image obtained by the endoscope is not the tissue to be cut in the surgical plan, the identification result is fed back to the control unit 840, and the control unit 840 controls the surgical robot 820 to move the endoscope to other positions to continue the identification until the identified tissue is the tissue for which the surgical operation is to be performed, and then controls the surgical robot 820 to perform the surgical operation using the corresponding surgical operation subunit 821.

[0128] In other embodiments, to ensure the safety of the surgical operation performed by the surgical robot 820, the surgical assistance subunit 822 may further include a monitoring subunit for monitoring the surgical operation performed by the surgical robot 820 during the surgical procedure and transmitting the monitoring result to the display. Specifically, the monitoring subunit may be composed of an external camera, an endoscope, and a display. When the surgical robot 820 performs a surgery, the external camera and the endoscope can be used to obtain the in-vivo and ex-vivo images during the surgical procedure in real time and display them on the display. Optionally, the operation process and operation parameters of the surgical robot 820 can also be displayed on the display, that is, the display includes an ex-vivo monitoring image display area, an in-vivo monitoring image display area, and an operation process and operation parameter display area. In this way, the doctor can judge whether the operation of the surgical robot 820 meets the expectations according to the images displayed on the display, and once an incorrect operation occurs, the operation of the surgical robot 820 can be stopped in time and the surgical process and / or operation parameters can be adjusted.

[0129] The above embodiments have explained in detail the structure of the surgical robot system provided by the present application. Next, the control method, control device, and readable storage medium of the surgical robot system provided by the present application will be described.

[0130] Figure 9 This is a flowchart of the control method of the surgical robot system provided by the embodiments of the present application. The surgical robot system includes a surgical planning module and a surgical execution module connected to each other. As Figure 9As shown, the control method includes:

[0131] S901: Obtain medical image data through a surgical planning module, and determine a surgical plan based on the medical image data;

[0132] S902: Execute surgical operations according to the surgical plan through a surgical execution module.

[0133] The embodiment of the present application also provides a control device for a surgical robot system, including: a memory, a processor, wherein the memory is used to store computer execution instructions; the processor is used to execute the computer execution instructions to implement the control method of the surgical robot system provided in the above embodiment.

[0134] The embodiment of the present application also provides a computer-readable storage medium, in which computer execution instructions are stored, and when the computer execution instructions are executed, they are used to implement the control method of the surgical robot system provided in the above embodiment.

[0135] The present application also provides a computer program product, including a computer program; when the computer program is executed, it implements the control method provided in the second aspect above.

[0136] Those skilled in the art will readily think of other implementation schemes of the present application after considering the specification and practicing the invention disclosed herein. The present application is intended to cover any variations, uses, or adaptations of the present application, which follow the general principles of the present application and include known common knowledge or conventional technical means in the technical field not disclosed in the present application. The specification and embodiments are only regarded as exemplary, and the true scope and spirit of the present application are pointed out by the following claims.

[0137] It should be understood that the present application is not limited to the exact structure already described and shown in the drawings, and various modifications and changes can be made without departing from its scope. The scope of the present application is only limited by the appended claims.

Claims

1. A surgical robot system, characterized in that, Including: A surgically planning module and a surgical execution module that are interconnected, where: The surgically planning module is used to obtain medical imaging data and determine a surgical plan based on the medical imaging data; The surgical execution module is used to perform surgical operations according to the surgical plan; The surgical execution module includes: a surgical operation learning unit and a surgical robot, where: The surgical operation learning unit is used to learn the surgical operations of the surgical plan to obtain a learning result; The surgical operation learning unit includes: A video acquisition sub-unit for acquiring the surgical video of the surgical plan; A video decomposition sub-unit for decomposing the surgical video to obtain surgical pictures and / or surgical video segments; A calibration sub-unit for calibrating the anatomical structures, surgical tools, and operation steps involved in the surgical pictures and / or surgical video segments through a calibration algorithm to obtain a calibration result; A learning sub-unit for learning the calibration result to obtain a learning result; The surgical robot is used to perform surgical operations according to the surgical plan and the learning result.

2. The surgical robot system according to claim 1, wherein The surgically planning module includes: A data acquisition unit for acquiring the medical imaging data; A modeling unit for performing three-dimensional reconstruction based on the medical imaging data to obtain a target three-dimensional model; An image reading unit for determining lesion information based on the medical imaging data and / or the target three-dimensional model; A surgical plan generation unit for generating a surgical plan according to the lesion information, and the surgical plan includes the operation steps and surgical tools for the entire surgical process.

3. The surgical robot system according to claim 2, wherein The image reading unit is specifically used to: determine lesion information by comparing the medical imaging data with standard imaging data, and / or by comparing the target three-dimensional model with a standard three-dimensional model, and the lesion information includes at least one of the location characteristics, shape characteristics, CT value, and pathological result of the lesion.

4. The surgical robot system according to claim 2, wherein, The image reading unit includes a pathological prediction sub-unit for predicting the pathological result of the lesion based on the medical imaging data and / or the target three-dimensional model, and the lesion information includes the pathological result.

5. The surgical robot system according to claim 2, wherein The modeling unit includes: An image segmentation sub-unit for performing image segmentation on the medical imaging data through a segmentation algorithm and extracting the imaging data of different tissues and / or organs from the segmentation result; A first modeling sub-unit for performing three-dimensional reconstruction on the imaging data of different tissues and / or organs respectively through a three-dimensional modeling algorithm to obtain corresponding tissue three-dimensional models and / or organ three-dimensional models; A second modeling sub-unit for fusing the tissue three-dimensional models and / or organ three-dimensional models to obtain the target three-dimensional model.

6. The surgical robot system according to claim 5, characterized in that, The modeling unit further includes a segmentation sub-unit and / or an identification sub-unit, where: The segmentation sub-unit is used to perform segmentation processing on at least part of the tissues and / or organs in the target three-dimensional model through a segmentation algorithm; The identification sub-unit is used to perform color identification on at least part of the tissues and / or organs in the target three-dimensional model through an identification algorithm.

7. The surgical robot system according to any one of claims 2 to 6, characterized in that, The surgically planning module further includes: A naming unit for naming and / or segmenting at least some different tissues and / or organs of the target three-dimensional model based on the lesion information; The surgical plan generation unit is specifically configured to determine a surgical plan according to the naming result and the lesion information, and the surgical plan includes surgical operation plans for different named tissues and / or organs.

8. The surgical robot system according to any one of claims 2 to 6, characterized in that, The surgical planning module further includes: A surgical simulation unit for performing surgical simulation training according to the surgical plan to determine whether the surgical plan is reasonable.

9. The surgical robot system according to any one of claims 2 to 6, characterized in that, The surgical robot includes: A surgical operation subunit for performing the surgical operation; A surgical assistance subunit for performing surgical assistance operations, and the surgical assistance operations assist in the execution of the surgical operation.

10. The surgical robot system according to claim 9, characterized in that, The surgical execution module further includes a plan interpretation unit and a control unit, wherein: The plan interpretation unit is configured to interpret the surgical plan and obtain operation parameters based on the interpretation result, the medical image data, and / or the target three-dimensional model, and the operation parameters include the robotic arm movement parameters of the surgical robot; The control unit is configured to control the surgical robot to perform the surgical operation according to the operation parameters.

11. The surgical robot system according to claim 10, wherein The surgical assistance subunit includes an identification subunit for identifying anatomical structures during the surgical process and feeding back the identification result to the control unit; Correspondingly, the control unit is further configured to obtain the identification result and control the surgical robot to perform the surgical operation based on the identification result and the operation parameters.

12. The surgical robot system according to claim 9, wherein The surgical assistance subunit includes: A monitoring subunit for monitoring the surgical operations performed by the surgical robot during the surgical process and transmitting the monitoring result to a display.

Images