Method and system for predicting abdominal surgery-related information from medical image data

Abstract

The present invention relates to a method for training a model for predicting abdominal surgery information, which is performed by a computing device, the method comprising a step of training an artificial neural network model by using medical image data and masking data labeled with the degree of adhesion between the peritoneum and organs such that, when a medical image is input, the model outputs abdominal surgery-related information of the input medical image.

Description

Method and system for predicting abdominal surgery-related information from medical image data

[0001] The present invention relates to a method and system for predicting abdominal surgery-related information from medical image data.

[0002]

[0003] AI technology being used in clinical settings is primarily being applied to provide solutions that improve the accuracy of tumor diagnosis. Because tumor diagnosis relies on the experience and expertise of clinicians using imaging tests such as X-rays, CT, MRI, and PET-CT, differences among radiologists and the resulting potential for diagnostic errors can arise. Applying AI technology to AI-based diagnostic solution programs that can improve accuracy can lead to cost-effective screening and increased accuracy, rapidly expanding the ripple effect in the healthcare market.

[0004] However, surgery remains a field of clinical practice that relies heavily on the surgeon's experience and skill. Among various surgical fields, particularly abdominal surgery, minimally invasive surgery (MIS) using laparoscopy or robotics can be categorized as open surgery. MIS is increasingly preferred over open surgery due to its smaller scars and faster recovery and return to normal daily life.

[0005] However, severe peritoneal adhesions, which can cause organ damage, can occur due to various reasons such as previous surgical history and number of surgeries, radiation therapy, tumors, and peritonitis. These severe peritoneal adhesions can block the access route to the abdominal cavity when MIS is planned, and repeated access attempts can cause serious surgical complications such as intestinal and bladder rupture, bleeding, and vascular damage. Even if these problems are surgically repaired, intestinal obstruction can occur in up to 40% of cases, which can significantly delay the return to daily life. This risk occurs frequently, especially among young doctors with relatively little surgical experience and technical skills, and is linked to a decrease in applications to the surgical field due to burnout caused by coping with various medical accidents and lawsuits, resulting in social problems.

[0006] The clinical need for techniques to predict severe peritoneal adhesions before surgery continues to grow with the increase in MIS, patient demand, and lawsuits due to surgical complications. However, there is no research on predicting possible routes for abdominal access that avoid severe peritoneal adhesions. Some CT and MRI studies on the radiological findings of visceral peritoneal adhesions, which are long-term adhesions in the abdominal cavity, have been published intermittently. However, clear radiological findings on parietal peritoneal adhesions, which are severe adhesions between the parietal peritoneum and the visceral peritoneum surrounding the abdominal organs, that can suggest a safe peritoneal access route for surgery, have not been reported.

[0007] Accordingly, the inventors of the present invention compared and analyzed preoperative computed tomography (CT) scans and surgical images based on image data on the location and degree of adhesion during surgery in patients undergoing abdominal surgery, defined radiological findings corresponding to severe peritoneal adhesions, and developed a CT radiomics-based algorithm to predict and suggest a route for accessing the abdominal cavity while avoiding severe peritoneal adhesions.

[0008]

[0009] In order to solve the above-mentioned problem, the present invention provides a method for training an artificial neural network model to output information related to abdominal surgery of an input medical image when a medical image is input, using medical image data and masking data in which the degree of adhesion between the peritoneum and organs is labeled.

[0010] In addition, the present invention provides a prediction method performed by a computing device, the prediction method including a step of training an artificial neural network model to output information related to abdominal surgery of an input medical image using medical image data and masking data in which the degree of adhesion between the peritoneum and an organ is labeled, when a medical image is input, and a step of inputting a medical image to the artificial neural network model and outputting information related to abdominal surgery of the input medical image.

[0011] In addition, the present invention provides a computing device comprising at least one processor and a memory, wherein the at least one processor performs an operation of training an artificial neural network model to output information related to abdominal surgery of an input medical image using medical image data and masking data in which the degree of adhesion between the peritoneum and an organ is labeled when a medical image is input.

[0012] In addition, the present invention provides a computer program stored in a computer-readable recording medium, wherein the computer program, when executed by a computing device, causes the computing device to perform an operation of learning a prediction model, and the operation includes an operation of learning an artificial neural network model to output information related to abdominal surgery of an input medical image using medical image data and masking data in which the degree of adhesion between the peritoneum and an organ is labeled, when a medical image is input, the computer program stored in a computer-readable recording medium has the purpose of providing a computer program.

[0013]

[0014] One aspect of the present invention for achieving the above-described object provides a method for learning an abdominal surgery information prediction model, which is performed by a computing device, comprising the step of learning an artificial neural network model to output abdominal surgery-related information of an input medical image when the medical image is input, using medical image data and masking data in which the degree of adhesion between the peritoneum and an organ is labeled.

[0015] In another aspect of the present invention, the medical image is a CT image or an MRI image, and the degree of adhesion between the peritoneum and the organ may be the degree of adhesion between the parietal peritoneum and the visceral peritoneum surrounding the abdominal organs.

[0016] In another aspect of the present invention, the degree of adhesion is divided into at least three degrees of adhesion, and each degree of adhesion is labeled differently, and the at least three degrees of adhesion may include low adhesion including non-adhesion, intermediate adhesion which is a degree of adhesion higher than the low degree of adhesion, and severe adhesion which is a degree of adhesion higher than the intermediate degree of adhesion.

[0017] In another aspect of the present invention, the training step may include a step of training a first artificial neural network model to output organs included in an input medical image by using medical image data, when the medical image is input, and a step of training a second artificial neural network model to output the degree of adhesion for each organ region by using information output from the first artificial neural network model, when the medical image is input.

[0018] In another aspect of the present invention, a step of calculating an abdominal surgery path using information output from the second artificial neural network model may be further included.

[0019] In another aspect of the present invention, the abdominal surgery path can be determined based on the degree of adhesion in each organ region and the surgical target region output from the second artificial neural network model.

[0020] In addition, one aspect of the present invention provides a prediction method performed by a computing device, the prediction method including the steps of training an artificial neural network model to output information related to abdominal surgery of an input medical image using medical image data and masking data in which the degree of adhesion between the peritoneum and an organ is labeled when a medical image is input, and the steps of inputting a medical image to the artificial neural network model and outputting information related to abdominal surgery of the input medical image.

[0021] In addition, one aspect of the present invention provides a computing device comprising at least one processor and a memory, wherein the at least one processor performs an operation of training an artificial neural network model to output information related to abdominal surgery of an input medical image using medical image data and masking data in which the degree of adhesion between the peritoneum and an organ is labeled, when a medical image is input, the computing device.

[0022] In another aspect of the present invention, the at least one processor can perform an operation of inputting a medical image into the artificial neural network model and deriving abdominal surgery-related information from the input medical image.

[0023] In addition, one aspect of the present invention provides a computer program stored in a computer-readable recording medium, wherein the computer program, when executed by a computing device, causes the computing device to perform an operation of learning a prediction model, the operation including an operation of learning an artificial neural network model to output information related to abdominal surgery of an input medical image using medical image data and masking data in which the degree of adhesion between the peritoneum and an organ is labeled, when the medical image is input, the computer program stored in a computer-readable recording medium.

[0024] In another aspect of the present invention, the operation may include an operation of inputting a medical image into the artificial neural network model and deriving abdominal surgery-related information from the input medical image.

[0025]

[0026] According to the present invention, when a medical image of a surgical target is input, abdominal surgery-related information related to the input medical image can be output, thereby improving the convenience of surgery by the operating surgeon.

[0027] Additionally, since the degree of adhesion between the peritoneum and organs is included in the information related to abdominal surgery, the abdominal surgery route can be established in a way that minimizes postoperative complications, thereby minimizing side effects for patients.

[0028] Additionally, since information on abdominal surgery includes an abdominal surgical route based on the degree of adhesion between the peritoneum and organs, even clinicians with less surgical experience can perform surgery with minimal side effects.

[0029]

[0030] Figure 1 is a drawing to explain visceral peritoneal adhesions and parietal peritoneal adhesions observed in medical images.

[0031] FIG. 2 is a schematic block diagram illustrating a computing device according to an embodiment of the present invention.

[0032] Figure 3 shows CT segmented images labeled with different colors according to the severity of adhesions between the anterior peritoneum and organs.

[0033] Figure 4 illustrates data in which 3D masking data labeled with different colors according to the severity of adhesion between the anterior wall peritoneum and organs is drawn on a 3D CT image.

[0034] Figures 5 to 7 are drawings for explaining the first artificial neural network model that classifies (segments) and outputs organs included in medical images.

[0035] Figures 8 to 10 are diagrams for explaining a second artificial neural network model that outputs the degree of adhesion for each organ region included in a medical image using information output from a first artificial neural network model.

[0036] FIG. 11 illustrates one aspect of information output from a prediction model according to an embodiment of the present invention.

[0037]

[0038]

[0039] Hereinafter, the present invention will be described in detail with reference to the attached drawings.

[0040]

[0041] Hereinafter, the term "medical imaging" refers to images that can confirm the organs in the abdominal cavity, and is a concept that includes CT (Computed Tomography) images, PET (Positron Emission Tomography) images, ultrasound images, MRI (Magnetic Resonance Imaging) images, etc.

[0042] Hereinafter, the term "parietal peritoneum" refers to the peritoneum lining the inner wall of the abdominal cavity.

[0043] Hereinafter, the term "visceral peritoneum" refers to the peritoneum that surrounds the organs and structures inside the abdominal cavity.

[0044] Hereinafter, the term "information related to abdominal surgery" may mean information required for minimally invasive surgery such as laparoscopic surgery, and is a concept including visceral peritoneal adhesions, parietal peritoneal adhesions, and abdominal surgery routes.

[0045]

[0046] Referring to FIG. 2, a computing device (100) according to an embodiment of the present invention is specifically described.

[0047]

[0048] A computing device (100) according to an embodiment of the present invention is configured to generate a prediction model used in the present invention, and is configured to derive abdominal surgery-related information corresponding to an input medical image using the generated prediction model.

[0049] Referring to FIG. 2, the computing device (100) includes a communication unit (110), an input unit (120), a processor (130), a running processor (140), a control unit (150), a memory (160), and an output unit (170). That is, the computing device (100) may have, for example, communication / input / computation / output functions, and may be implemented as various electronic devices such as, for example, a server, a desktop PC, a notebook PC, a tablet PC, etc.

[0050] In one embodiment, the communication unit (110) is configured for communication with an external device, and data transmission and reception with the external device may be possible through the communication unit (110).

[0051] In one embodiment, the input unit (120) may receive commands or data to be used by components (e.g., a processor, a learning processor) of the computing device (100) from an external source (e.g., a user) of the computing device (100). The input unit (120) may include, for example, a microphone, a mouse, a keyboard, keys (e.g., buttons), or a digital pen (e.g., a stylus pen).

[0052] In one embodiment, the control unit (150) may control one or more other components (e.g., hardware or software components) of the computing device (100) connected to the processor, for example, by executing software.

[0053] In one embodiment, the processor (130) and the learning processor (140) may perform data processing or calculation functions, and as at least part of the data processing or calculation functions, may store commands or data received from other components in volatile memory, process the commands or data stored in the volatile memory, and store result data in non-volatile memory. In addition, the learning processor (140) may include a hardware structure specialized for processing artificial intelligence models. Hereinafter, the processor (130) and the learning processor (140) are described separately, but the present invention is not particularly limited thereto, and a category in which the processor (130) and the learning processor (140) are integrated into a single processor to perform data processing, calculation, or learning is also included in the present invention.

[0054] The memory (160) can store various data used in the computing device (100). The data can include, for example, input data or output data for software and commands related thereto.

[0055] The output unit (170) can output information processed or calculated in the computing device (100) to the outside, and a display, speaker, etc. can be included here.

[0056]

[0057] The learning processor (140) trains an artificial neural network model using learning data to derive abdominal surgery-related information from a queried medical image when a medical image is queried. Below, the structure of the learning data used to train the artificial neural network model is described in detail.

[0058]

[0059] First, a step may be performed in which a computing device (100) collects medical images to be used as training data. The collected medical image data is preprocessed by a processor (130), and the preprocessed data is used to train an artificial neural network model.

[0060]

[0061] Surgical images that can confirm the severity and anatomical location of adhesions are collected from patients undergoing open surgery or MIS surgery, and the degree of adhesion between the anterior peritoneum and organs corresponding to the MIS access route, more specifically, between the anterior peritoneum and the visceral peritoneum surrounding the organs, is compared with the surgical images, and masking data is generated in which each area is labeled differently according to the degree of adhesion (see Figures 3 and 4).

[0062] The generated masking data can be in three-dimensional form, and the degree of adhesion is divided into at least three degrees of adhesion (see Table 1).

[0063]

[0064] Adhesion Degree Description Labeling Type Low adhesions Adhesions that are non-adhesive, minimal, or membranous and can be easily separated by blunt dissection Green Intermediate adhesions Adhesions that can be dissected by blunt dissection but require sharp dissection due to the start of vascularization, or adhesions that can be separated by sharp dissection only because blood vessels have clearly formed Yellow Severe adhesions Adhesions that are so severe that the organs are strongly attached that they can only be separated by sharp dissection (preventing organ damage is impossible) Red

[0065] Specifically, the masking data generated by the processor (130) can be covered over a medical image, and in the present invention, data with the masking data drawn on the medical image is used to train an artificial neural network model (see FIG. 4).

[0066]

[0067] To explain this in detail, it is as follows.

[0068] First, a learning processor (140) trains a first artificial neural network model using medical image data so that, when a medical image is input, the organs included in the input medical image are output in a distinguishable manner. The results derived by the first artificial neural network are illustrated in Fig. 7.

[0069] Specifically, the present invention developed an abdominal organ segmentation model (AOS model) using the Whole Abdominal Organ Dataset (WORD) and TransUNet, which achieved state-of-the-art (SOTA) performance in multi-organ CT segmentation (see Figs. 5 and 6). Furthermore, the AOS model was enhanced using TotalSegmentor, a deep learning segmentation model capable of automatically segmenting major anatomical structures of the body based on CT (see Fig. 7).

[0070] Next, the learning processor (140) trains a second artificial neural network model to output the degree of adhesion for each organ region when a medical image is input, using the results derived by the first artificial neural network (i.e., medical image data in which organs are distinguished from each other) and the medical image data covered with masking data. FIG. 11 illustrates the results derived by the second artificial neural network model, in which areas predicted to have low adhesion are shown in green, areas predicted to have moderate adhesion are shown in yellow, and areas predicted to have severe adhesion are shown in red. In addition, adhesion severity, access potential, and complication rates are each expressed as numerical values ​​for each area, which may provide useful information related to abdominal surgery to the operating surgeon.

[0071] Specifically, in the present invention, an initial model for predicting adhesion severity (ASG initial model) was developed by selecting ROIs and extracting and standardizing Hounsfield unit (HU) values ​​from the AOS model (see Fig. 8). In addition, unnecessary features were removed from CT radiomics features suggesting severe peritoneal adhesions, and the ASG initial model was enhanced by utilizing machine learning-based algorithms such as logistic regression, support vector machine, and random forest (see Fig. 9). In addition, deep learning in the form of a 3D global CNN was applied to abdominal CT images to identify the locations of organs, determine the severity of adhesions in each region, and identify an abdominal access route to prevent future organ damage based on the severity of adhesions in each organ region. Key CT radiomics features were extracted from CT images and 3D CT, and a multi-modal fusion algorithm was developed by combining multiple modalities to enhance the ASG model (see Fig. 10).

[0072] In another embodiment of the present invention, a Clinical Decision Support System (CDSS) that outputs an abdominal surgical path that allows for minimally invasive surgery when a medical image (CT image) is input can be provided. This CDSS system can also be implemented through an artificial neural network model, and the parameters installed therein are as follows.

[0073]

[0074] Shape features, Histogram features, Texture features, Filter-based, Deep learning, Shape, boundary clarity, surface homogeneity, surface nodularity, Mean, standard deviation, maximum, minimum, randomness, flatness, bias, Gray-level co-occurrence matrix (GLCM), Gray-level run-length matrix (GLRLM), Box counting, Wavelet analysis, Fractal analysis, Heatmap (CAM) Accuracy, Probability

[0075] Meanwhile, in another embodiment of the present invention, the processor (130) may use information derived from the second artificial neural network model, and when the surgical target area is determined, the processor (130) may calculate an abdominal surgical path. For example, if the surgical target area is input to the computing device (100) as the uterus, the abdominal surgical path including an area predicted to have a low degree of adhesion may be calculated. In other words, areas predicted to have a moderate degree of adhesion and areas predicted to have a severe degree of adhesion may be calculated not to be included in the abdominal surgical path, thereby minimizing the probability of side effects occurring during actual abdominal surgery. Meanwhile, if there is no area predicted to have a low degree of adhesion throughout the abdominal region, the abdominal surgical path including an area predicted to have a moderate degree of adhesion may be calculated. In any case, the abdominal surgical path including an area predicted to have a severe degree of adhesion may not be calculated.

[0076]

[0077] The method according to one embodiment of the present invention described above may be implemented in the form of program commands that can be executed through various computer means and recorded on a computer-readable medium. The computer-readable medium may include program commands, data files, data structures, etc., alone or in combination. The program commands recorded on the medium may be those specially designed and configured for the present invention or may be those known and available to those skilled in the art of computer software. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks, and magnetic tapes, optical media such as CD-ROMs and DVDs, magneto-optical media such as floptical disks, and hardware devices specially configured to store and execute program commands, such as ROMs, RAMs, and flash memories. Examples of program commands include not only machine language codes generated by a compiler, but also high-level language codes that can be executed by a computer using an interpreter, etc. The hardware devices described above may be configured to operate as one or more software modules to perform the operations of the present invention, and vice versa.

[0078]

[0079] While the present invention has been described with reference to embodiments illustrated in the drawings to facilitate understanding and reproduction by those skilled in the art, these are merely exemplary, and those skilled in the art will understand that various modifications and equivalent embodiments are possible based on the embodiments of the present invention. Therefore, the scope of protection of the present invention should be defined by the claims.

[0080]

[0081] (Explanation of symbols)

[0082] 100: Computing Device

[0083] 110: Communications Department

[0084] 120: Input section

[0085] 130: Processor

[0086] 140: Running Processor

[0087] 150: Control unit

[0088] 160: Memory

[0089] 170: Output section

Claims

1. A method for learning a prediction model for abdominal surgery information performed by a computing device, A method for producing a medical image, comprising: a step of training an artificial neural network model to output information related to abdominal surgery of an input medical image using medical image data and masking data labeled with the degree of adhesion between the peritoneum and organs, when a medical image is input; method.

2. In paragraph 1, The above medical images are CT images or MRI images. The degree of adhesion between the peritoneum and the organs is the degree of adhesion between the parietal peritoneum and the visceral peritoneum surrounding the abdominal organs. method.

3. In paragraph 2, The above adhesion levels are divided into at least three adhesion levels, and each adhesion level is labeled differently. At least three degrees of adhesion above, Low adhesion including non-adhesion; Intermediate adhesion, which is a degree of adhesion higher than the above low degree of adhesion; and Severe adhesions, which are adhesions of a higher degree than the above moderate adhesions; method.

4. In paragraph 1, The above learning steps are: A step of training a first artificial neural network model to distinguish and output organs included in an input medical image by using medical image data when the medical image is input; and A step of training a second artificial neural network model to output the degree of adhesion by organ region when a medical image is input, using information output from the first artificial neural network model; including; method.

5. In paragraph 4, A step of calculating an abdominal surgery path using information output from the second artificial neural network model is further included. method.

6. In paragraph 5, The above abdominal surgical path is determined based on the degree of adhesion in each organ region and the surgical target region output from the second artificial neural network model. method.

7. A prediction method performed by a computing device, A step of training an artificial neural network model to output information related to abdominal surgery of an input medical image by using medical image data and masking data in which the degree of adhesion between the peritoneum and organs is labeled; A step of inputting a medical image into the artificial neural network model and outputting information related to abdominal surgery of the input medical image; How to predict.

8. As a computing device, at least one processor; and memory; including; At least one processor of the above, An operation of training an artificial neural network model to output information related to abdominal surgery of an input medical image by using medical image data and masking data labeled with the degree of adhesion between the peritoneum and organs when a medical image is input; Computing device.

9. In paragraph 8, At least one processor of the above, An operation of inputting a medical image into the above artificial neural network model and deriving information related to abdominal surgery from the input medical image; Computing device.

10. A computer program stored in a computer-readable recording medium, wherein the computer program, when executed by a computing device, causes the computing device to perform an operation of learning a predictive model, the operation comprising: An operation of training an artificial neural network model to output information related to abdominal surgery of an input medical image by using medical image data and masking data labeled with the degree of adhesion between the peritoneum and organs when a medical image is input; including; A computer program stored on a computer-readable recording medium.

11. In paragraph 10, The above actions are: An operation including inputting a medical image into the artificial neural network model and deriving information related to abdominal surgery from the input medical image; A computer program stored on a computer-readable recording medium.

Images