Artificial intelligence surgery system and control method thereof

Abstract

An artificial intelligence surgery system and a control method thereof are provided. The artificial intelligence surgery system includes a recording device that records a surgery process as a video and inputs the recorded video to a learned artificial intelligence algorithm to determine a surgery step, and a server that updates the learned artificial intelligence algorithm of the recording device using a video received from the recording device.

Description

Technical Field

[0001] This invention relates to an artificial intelligence surgical system and its control method. Background Technology

[0002] There is a need to develop technologies that can provide information during surgery to assist surgeons. To provide this information, it is necessary to be able to identify surgical actions.

[0003] Therefore, there is a need to develop a technology that enables computers to recognize surgical actions from surgical images.

[0004] Furthermore, deep learning has recently been widely applied in the analysis of medical images. Deep learning is defined as a collection of machine learning algorithms that attempt to achieve high-level abstractions (the operations of summarizing core content or functions from large amounts of data or complex information) through a combination of various nonlinear transformation methods. Broadly speaking, deep learning can be seen as a field of machine learning that teaches computers how to think like humans. Summary of the Invention

[0005] Technical issues

[0006] The technical problem to be solved by the present invention is to provide an artificial intelligence surgical system and its control method that can use artificial intelligence models to provide surgical-related videos in an optimized manner.

[0007] The technical problems to be solved by this invention are not limited to those mentioned above. Those skilled in the art will be able to clearly understand other technical problems not mentioned from the following description.

[0008] Technical solution

[0009] The artificial intelligence surgical system according to the present invention for solving the above-mentioned technical problems includes: a recording device for recording the surgical process as a video and inputting the recorded video into a learned artificial intelligence algorithm to determine the surgical steps; and a server for updating the learned artificial intelligence algorithm of the recording device using the video received from the recording device.

[0010] At this point, the recording device can use the learned artificial intelligence algorithm to delete the video portion recording non-surgical conditions from the recorded video.

[0011] Furthermore, the recording device can selectively record angiographic images using the learned artificial intelligence algorithm.

[0012] Furthermore, the recording device can use the learned artificial intelligence algorithm to infer which surgical step the recorded video represents, and associate the recorded video with the inferred surgical step. In the case that the inferred surgical step is a surgical step that requires the integration of blood vessels and organs, the recorded video can be associated with the inferred surgical step.

[0013] Furthermore, the recording device can pause the recording of the video while associating the recorded video with the inferred surgical steps.

[0014] Furthermore, the recording device can segment the recorded video and transmit at least a portion of the segmented video to the server. The server can also use at least a portion of the segmented video received from the recording device to update the learned artificial intelligence algorithm and transmit the updated learned artificial intelligence algorithm to the recording device.

[0015] Furthermore, the control method of the artificial intelligence surgical system according to the present invention, which includes a recording device and a server, for solving the above-mentioned technical problems, may include the following steps: the recording device records the surgical procedure as a video; the recording device inputs the recorded video into a learned artificial intelligence algorithm to determine the surgical steps; if the recording device determines the determined surgical steps as preset surgical steps, the recorded video is associated with the determined surgical steps; the recording device transmits at least a portion of the recorded video to the server; and the server uses the video received from the recording device to update the learned artificial intelligence algorithm of the recording device.

[0016] A computer program according to an embodiment of the present invention is stored in a computer-readable storage medium for use in conjunction with a computer as hardware to execute a control method for an artificial intelligence surgical system.

[0017] Other specific aspects of this invention are detailed in the description and accompanying drawings.

[0018] Beneficial effects

[0019] According to the present invention, artificial intelligence algorithms can be used to associate (integrate) videos corresponding to specific surgical steps in recorded surgical videos with the specific surgical steps, thereby providing optimized surgical navigation information.

[0020] Furthermore, the present invention can provide an artificial intelligence surgical system and its control method that can use recorded video to update artificial intelligence algorithms to an optimized state.

[0021] The effects of the present invention are not limited to those mentioned above, and those skilled in the art can clearly understand other effects not mentioned below through the description below. Attached Figure Description

[0022] Figure 1 This is a conceptual diagram illustrating the artificial intelligence surgical system of the present invention.

[0023] Figure 2 This is a flowchart illustrating the control method of the artificial intelligence surgical system of the present invention.

[0024] Figure 3 This is a schematic diagram illustrating the structure of a device for implementing a control method for an artificial intelligence surgical system. Detailed Implementation

[0025] The advantages, features, and methods for implementing the invention will be readily understood by referring to the embodiments described in detail below with reference to the accompanying drawings. However, the invention is not limited to the embodiments disclosed below, which can be implemented in various different forms. These embodiments are provided merely to complete the disclosure of the invention and to fully inform those skilled in the art of the scope of the invention, which is defined only by the scope of the claims.

[0026] The terminology used in this specification is for illustrative purposes and is not intended to limit the invention. Unless specifically mentioned herein, singular forms include plural forms. The terms "comprises" and / or "comprising" as used in this specification do not exclude the presence or addition of more than one other constituent element besides those mentioned. Throughout the specification, the same reference numerals refer to the same constituent element, and "and / or" includes each of the mentioned constituent elements and all combinations thereof. Although "first," "second," etc., are used to describe multiple constituent elements, these constituent elements are clearly not limited by these terms. These terms are only used to distinguish one constituent element from other constituent elements. Therefore, the first constituent element mentioned below can obviously be a second constituent element within the technical concept of this invention.

[0027] Unless otherwise defined, all terms (including technical and scientific terms) used in this specification may be used in the sense that would be commonly understood by one of ordinary skill in the art to which this invention pertains. Furthermore, terms defined in commonly used dictionaries should not be interpreted ideally or excessively unless otherwise explicitly and specifically defined.

[0028] Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings.

[0029] Figure 1This is a conceptual diagram illustrating the artificial intelligence surgical system of the present invention.

[0030] according to Figure 1 The artificial intelligence surgical system (or robotic surgical system) of the present invention includes a medical image capturing device 10, a server 100, a recording device 30, a display 32, and a surgical robot 34, all located in the operating room. According to an embodiment, the medical image capturing device 10 may be omitted from the artificial intelligence surgical system according to the disclosed embodiment.

[0031] The surgical robot 34 includes a camera device 36 and surgical instruments 38.

[0032] The recording device 30 can perform robotic surgery by controlling the surgical robot 34.

[0033] Robotic surgery is performed by a user controlling a surgical robot 34 using a recording device 30. In one embodiment, robotic surgery can also be performed automatically by the recording device 30 without user control.

[0034] In this specification, although the recording device 30 and the surgical robot 34 are described as separate, this is not a limitation. The surgical robot 34 may also be included in the recording device 30, thereby being controlled as a component of the recording device 30.

[0035] Server 100 is a computing device that includes at least one processor and a communication unit.

[0036] The recording device 30 includes a computing device comprising at least one processor and a communication unit. In one embodiment, the recording device 30 includes hardware and a software interface for controlling the surgical robot 34.

[0037] The camera device 36 includes at least one image sensor. That is, the camera device 36 includes at least one camera device for photographing the subject (i.e., the surgical site). In one embodiment, the camera device 36 includes at least one camera coupled to the surgical arm of the surgical robot 34.

[0038] In one embodiment, the image captured by the camera device 36 is displayed on the display 32.

[0039] In one embodiment, the surgical robot 34 includes one or more surgical tools 38 capable of performing operations such as cutting, clamping, fixing, and grasping at the surgical site. The surgical tools 38 are used in conjunction with the surgical arms of the surgical robot 34.

[0040] The recording device 30 can receive or generate information required for the surgery from the server 100 and provide it to the user. For example, the recording device 30 displays the generated or received information required for the surgery on the display 32.

[0041] For example, a user can control the movement of the surgical robot 34 by operating the recording device 30 while viewing the display 32, thereby performing robotic surgery.

[0042] Server 100 uses medical image data of the object pre-captured by medical imaging device 10 to generate information required for robotic surgery and provides the generated information to recording device 30.

[0043] The recording device 30 provides information received from the server 100 to the user by displaying it on the monitor 32, or uses the information received from the server 100 to control the surgical robot 34.

[0044] In one embodiment, the unit that can be used in the medical image acquisition device 10 is not limited; for example, other medical image acquisition units such as CT, X-ray, PET, and MRI can be used.

[0045] In one embodiment, surgical images obtained from the camera device 36 are transmitted to the recording device 30.

[0046] In one embodiment, the recording device 30 can segment surgical images acquired during the surgery in real time.

[0047] In one embodiment, the recording device 30 transmits surgical images to the server 100 during or after the surgery.

[0048] Server 100 can receive and analyze surgical images.

[0049] Server 100 learns and stores at least one model for analyzing surgical images.

[0050] Server 100 uses learning data to learn at least one model, and the learning data includes, but is not limited to, surgical images and information about the surgical images.

[0051] The embodiments disclosed below are not only applicable to relevant Figure 1 The artificial intelligence surgical system shown can also be applied to all kinds of embodiments that use surgical images to perform learning and identify specific operations based on the results of the learning.

[0052] Furthermore, for ease of explanation, "computer" will be described below as performing a learning-based surgical procedure recognition method according to embodiments disclosed in this specification. "Computer" can refer to... Figure 1 The server 100 or recording device 30, but not limited thereto, can also be used to mean a device capable of performing computational processing.

[0053] Figure 2 This is a flowchart illustrating the control method of the artificial intelligence surgical system of the present invention.

[0054] The artificial intelligence surgical system of the present invention may include a recording device 30 and a server 100. The control method of the artificial intelligence surgical system of the present invention can be implemented with the recording device 30 and the server 100 as the main components.

[0055] The recording device 30 can record the surgical procedure as a video (or surgical image) (S210).

[0056] Surgical images can be actual surgical images or virtual images used for simulation.

[0057] Actual surgical images refer to data obtained as an actual medical team performs surgery; for example, they could be images of an actual surgical scene performed by a surgical robot 34.

[0058] That is, actual surgical images are data recorded for the surgical site and surgical procedures during the actual surgical process.

[0059] The virtual image used for simulation refers to a simulated image generated based on medical images captured by the medical image capturing device 10. For example, it may be a simulated model generated by three-dimensional modeling of medical images of actual patients.

[0060] At this point, by rehearsing or simulating the simulation model in virtual space, virtual surgical images can be generated. Therefore, virtual images can be data recorded on the surgical site and surgical procedures performed on the simulation model.

[0061] In addition, surgical images may include more than one image frame.

[0062] Each image frame may include a part of the patient's body, namely the surgical site.

[0063] Furthermore, each image frame can include not only the patient's surgical site, but also surgical instruments, consumables required during the operation, etc. In other words, surgical images refer to data composed of image frames of surgical operations recorded according to various scenes, based on the time of the surgical process.

[0064] Therefore, the data composed of such image frames can be called a surgical image sequence.

[0065] The recording device 30 can input the recorded video into a learned artificial intelligence algorithm to determine (or infer) the surgical steps (S220).

[0066] Learned artificial intelligence algorithms can refer to artificial intelligence models that have learned based on deep learning, such as algorithms that use convolutional neural networks (CNNs) to learn.

[0067] Furthermore, the learned artificial intelligence algorithms may include at least one of the following: Random Forest (RF), Support Vector Machine (SVC), eXtra Gradient Boost (XGB), Decision Tree (DC), K-nearest Neighbors (KNN), Gaussian Naive Bayes (GNB), Stochastic Gradient Descent (SGD), Linear Discriminant Analysis (LDA), Ridge Regression, Lasso Regression, and Elastic Net.

[0068] The learned artificial intelligence algorithm (or artificial intelligence model) can be an algorithm designed (learned) to take recorded video (or surgical images) as input values ​​and surgical steps as output values.

[0069] In this manual, surgical steps output from learned artificial intelligence algorithms may be named as judgment surgical steps or inference surgical steps.

[0070] If the determined (inferred) surgical step is determined to be a preset surgical step, the recording device 30 can associate the recorded video with the determined surgical step (S230).

[0071] Furthermore, the learned artificial intelligence algorithm of this invention can perform a variety of functions.

[0072] For example, the recording device 30 can use the learned artificial intelligence algorithm to delete the video portion recording non-surgical conditions from the recorded video.

[0073] Furthermore, the recording device 30 can also utilize the learned artificial intelligence algorithm without recording images corresponding to non-surgical conditions.

[0074] In addition, the recording device 30 can also selectively record only images containing specific surgical procedures using the learned artificial intelligence algorithm.

[0075] For example, the recording device 30 can use the learned artificial intelligence algorithm to selectively record angiographic images.

[0076] Furthermore, the recording device 30 can use the learned artificial intelligence algorithm to infer which surgical step the recorded video represents.

[0077] That is, the recording device 30 can input the recorded video into the learned artificial intelligence algorithm to determine (infer) the surgical steps, and can determine whether the determined (inferred) surgical steps are the preset surgical steps.

[0078] The recording device 30 can associate (or integrate) the recorded video with the inferred surgical steps (i.e., if the inferred surgical steps are preset surgical steps).

[0079] For example, if the inferred surgical procedure is a pre-defined surgical procedure (e.g., a surgical procedure requiring vascular and organ integration), the recording device 30 can associate (or integrate) the recorded video with the inferred surgical procedure.

[0080] The associated video can be a portion (or a part) of a recorded video corresponding to that surgical step.

[0081] Furthermore, the recording device 30 can pause video recording while associating the recorded video with the inferred surgical steps.

[0082] If the recorded video is correlated with the inferred surgical steps, the recording device 30 can resume recording.

[0083] Furthermore, at least one of the recording device 30 and the server 100 can use the recorded video to update the learned artificial intelligence algorithm.

[0084] Therefore, the recording device 30 can transmit at least a portion of the recorded video to the server (S240).

[0085] The recording device 30 can segment the recorded video and transmit at least a portion of the segmented video to the server 100.

[0086] Here, the recording device 30 can use learned artificial intelligence algorithms to segment the recorded video.

[0087] The recording device 30 can identify surgical steps from the recorded video and segment the recorded video based on the identified surgical steps. Then, the recording device 30 can transmit a portion of the segmented video to the server 100 for updating the artificial intelligence algorithm, according to the preset importance of the surgical steps.

[0088] Server 100 can use the video received from the recording device 30 to update the learned artificial intelligence algorithm of the recording device (S250).

[0089] Specifically, server 100 can use at least a portion of the segmented video received from the recording device to update the learned artificial intelligence algorithm.

[0090] For example, server 100 can set the received video as the input value and the surgical steps in the video as the output value, thereby performing machine learning. Server 100 can then update the artificial intelligence algorithm (artificial intelligence model) through machine learning.

[0091] The updated learned artificial intelligence algorithm is transmitted to the recording device.

[0092] Then, the server 100 can transmit the updated artificial intelligence algorithm to the recording device 30 to update the learned artificial intelligence algorithm of the recording device 30.

[0093] That is, the artificial intelligence surgical system of the present invention adds AI function to the recording device 30, thereby enabling selective recording or surgical navigation in the recording device.

[0094] Furthermore, the artificial intelligence surgical system of the present invention enables the recording device 30 to be linked with the server 100 so that the server 100 can update the artificial intelligence model (artificial intelligence algorithm) in the recording device, and the operating standards of the artificial intelligence model can be set separately.

[0095] Furthermore, the artificial intelligence surgical system of the present invention can provide the required video (surgical images) for surgical steps in real time by controlling the artificial intelligence model through the recording device 30, so as to perform surgical navigation, and can match (associate) the surgical steps and the recorded video (surgical images) and store them.

[0096] Figure 3 This is a schematic diagram illustrating the structure of a device for implementing a control method for an artificial intelligence surgical system.

[0097] The device may be equipped on the recording device 30 and / or the server 100.

[0098] Reference Figure 3 The processor 310 may include one or more cores (not shown) as well as an image processing unit (not shown) and / or connection paths (e.g., bus, etc.) for transmitting and receiving signals with other components.

[0099] According to one embodiment, the processor 310 can execute one or more instructions stored in the memory 320, thereby performing... Figure 2 The artificial intelligence surgical control method described herein.

[0100] For example, when the device 300 is equipped with the recording device 30, the processor 310 executes one or more instructions stored in the memory 320, so that the recording device can record the surgical procedure as a video. The recording device inputs the recorded video into a learned artificial intelligence algorithm to determine the surgical steps. If the recording device determines that the determined surgical steps are preset surgical steps, the recorded video is associated with the determined surgical steps, and the recording device can transmit at least a portion of the recorded video to a server.

[0101] Furthermore, when the device 300 is equipped with the server 100, the processor 310 executes one or more instructions stored in the memory 320, thereby enabling the server to update the learned artificial intelligence algorithm of the recording device using the video received from the recording device.

[0102] Additionally, the processor 310 may also include random access memory (RAM, not shown) and read-only memory (ROM, not shown) for temporarily and / or permanently storing signals (or data) processed within the processor. Furthermore, the processor 310 may be implemented as a system-on-chip (SoC) comprising at least one of a graphics processing unit, random access memory, and read-only memory.

[0103] The memory 320 can store programs (more than one instruction) for processing and control by the processor 310. The programs stored in the memory 320 are divided into multiple modules according to their functions.

[0104] The control method of the artificial intelligence surgical system according to an embodiment of the present invention described above can be implemented as a program (or application) executed in conjunction with a computer as hardware and stored in a medium.

[0105] The steps of the methods or algorithms described in relation to embodiments of the present invention can be implemented directly in hardware, or by software modules executed in hardware, or by a combination thereof. The software modules can reside in RAM, ROM, erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), flash memory, hard disk, removable disk, CD-ROM, or any computer-readable recording medium of any form known in the art to which this invention pertains.

[0106] The embodiments of the present invention have been described above with reference to the accompanying drawings. However, those skilled in the art will understand that the invention can be implemented in other specific forms without altering the technical concept or essential features of the invention. Therefore, the embodiments described above should be understood as exemplary in all respects, and not limiting.

Claims

1. An artificial intelligence surgical system, comprising: The recording device records the surgical procedure as a video and inputs the recorded video into a learned artificial intelligence algorithm to determine the surgical steps; as well as The server uses the video received from the recording device to update the learned artificial intelligence algorithm of the recording device. The recording device utilizes the learned artificial intelligence algorithm to infer which surgical step the recorded video represents, segments the recorded video based on the inferred surgical step, and transmits at least a portion of the segmented video to the server according to the inferred importance of the surgical step. The server updates the learned artificial intelligence algorithm using at least a portion of the segmented video received from the recording device, and transmits the updated learned artificial intelligence algorithm to the recording device. The recording device is configured to selectively record the surgical procedure using the updated, learned artificial intelligence algorithm and to perform surgical navigation to provide the video required during the surgical steps in real time.

2. The artificial intelligence surgical system as described in claim 1, characterized in that, The recording device uses the learned artificial intelligence algorithm to delete the video portion recording non-surgical conditions from the recorded video.

3. The artificial intelligence surgical system as described in claim 2, characterized in that, The recording device uses the learned artificial intelligence algorithm to selectively record angiographic images.

4. The artificial intelligence surgical system as described in claim 2, characterized in that, The recording device associates the recorded video with the inferred surgical steps, wherein the inferred surgical steps are those requiring vascular and organ integration.

5. The artificial intelligence surgical system as described in claim 4, characterized in that, The recording device pauses video recording while associating the recorded video with the inferred surgical steps.

6. A control method for an artificial intelligence surgical system, the artificial intelligence surgical system comprising a recording device and a server, comprising the following steps: The recording device records the surgical procedure as a video; The recording device inputs the recorded video into a learned artificial intelligence algorithm to determine the surgical steps; If the recording device determines that the determined surgical step is a preset surgical step, the recorded video is associated with the determined surgical step. The recording device segments the recorded video according to the determined surgical steps, and transmits at least a portion of the segmented video to the server according to the determined importance of the surgical steps. as well as The server uses the video received from the recording device to update the artificial intelligence algorithm that the recording device has learned. The recording device selectively records the surgical procedure using the updated, learned artificial intelligence algorithm and performs surgical navigation to provide the video required during the surgical steps in real time.

7. The control method for the artificial intelligence surgical system as described in claim 6, characterized in that, The recording device uses the learned artificial intelligence algorithm to delete the video portion recording non-surgical conditions from the recorded video.

8. The control method for the artificial intelligence surgical system as described in claim 7, characterized in that, The recording device uses the learned artificial intelligence algorithm to selectively record angiographic images.

9. The control method for the artificial intelligence surgical system as described in claim 7, characterized in that, If the determined surgical step is one that requires vascular and organ integration, the recorded video is associated with the determined surgical step.

10. The control method for the artificial intelligence surgical system as described in claim 9, characterized in that, The recording device pauses video recording while associating the recorded video with the determined surgical steps.

11. A computer-readable recording medium storing a computer program for use in conjunction with a computer as hardware to perform the method according to any one of claims 6 to 10.

Images