Information processing device, method, and program

Abstract

In endoscopic submucosal dissection, the blood vessels contained within the submucosa are visualized. [Solution] An information processing device according to one embodiment of the present invention includes a highlighting unit that highlights the region of blood vessels contained in the submucosa in the image of an endoscopic system used in endoscopic submucosal dissection for early gastrointestinal malignant tumors.

Description

Technical Field

[0004] , , , , ,

[0006] , , , ,

[0005] , , , , , , , ,

[0001] The present invention relates to an information processing apparatus, method, and program.

Background Art

[0002] Endoscopic submucosal dissection (ESD) is widely performed as a treatment for early gastrointestinal cancer (mainly the esophagus, stomach, duodenum, and large intestine), and it is showing an increasing trend.

[0003] Bleeding is the most common complication during ESD treatment, and it can be particularly lethal. In particular, bleeding is likely to occur when dissecting the submucosal layer, and it is important to avoid inadvertently damaging blood vessels or the muscular layer during submucosal layer dissection for smooth and safe ESD.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

[0007] Therefore, the present invention aims to visualize blood vessels contained within the submucosa during endoscopic submucosal dissection. [Means for solving the problem]

[0008] An information processing device according to one embodiment of the present invention includes a highlighting unit that highlights the region of blood vessels contained within the submucosa in the image of an endoscopic system used in endoscopic submucosal dissection for early gastrointestinal malignant tumors.

[0009] According to the present invention, it is possible to visualize blood vessels contained within the submucosa during endoscopic submucosal dissection. [Brief explanation of the drawing]

[0010] [Figure 1] This is a diagram illustrating the outline of the present invention. [Figure 2] This diagram shows the overall configuration of one embodiment of the present invention. [Figure 3] This is a flowchart of a process for highlighting the blood vessel region contained within the submucosa according to one embodiment of the present invention. [Figure 4] This is a diagram illustrating the steps of endoscopic submucosal dissection according to one embodiment of the present invention. [Figure 5] This is a functional block diagram of an information processing device according to one embodiment of the present invention. [Figure 6] This diagram illustrates the process for highlighting the blood vessel region contained within the submucosa according to one embodiment of the present invention. [Figure 7] This is a diagram illustrating the extraction of a vascular region according to one embodiment of the present invention. [Figure 8] This is a flowchart of a process for suppressing false detection of vascular regions according to one embodiment of the present invention. [Figure 9] This figure illustrates the suppression of false detection of the vascular region according to one embodiment of the present invention. [Figure 10]This is a diagram for explaining suppression of false detection of a blood vessel region according to an embodiment of the present invention. [Figure 11] This is a diagram for explaining suppression of false detection of a blood vessel region according to an embodiment of the present invention. [Figure 12] This is a diagram for explaining suppression of false detection of a blood vessel region according to an embodiment of the present invention. [Figure 13] This is an example of highlighting a blood vessel region according to an embodiment of the present invention. [Figure 14] This is a diagram for explaining display during bleeding according to an embodiment of the present invention. [Figure 15] This is a diagram for explaining notification of the necessity of coagulation hemostasis according to an embodiment of the present invention. [Figure 16] This is a diagram for explaining notification of a treatment according to an embodiment of the present invention. [Figure 17] This is a diagram for explaining notification of the risk of bleeding according to an embodiment of the present invention. [Figure 18] This is a hardware configuration diagram of an information processing apparatus according to an embodiment of the present invention.

Mode for Carrying Out the Invention

[0011] Hereinafter, embodiments of the invention will be described based on the drawings.

[0012] <Overview> FIG. 1 is a diagram for explaining the overview of the present invention. The present invention targets endoscopic submucosal dissection for early gastrointestinal malignancies. Note that it may be endoscopic submucosal dissection performed only by a human, endoscopic submucosal dissection performed by a human operating a robot, or endoscopic submucosal dissection performed only by a robot.

[0013] In this invention, the information processing device 10 (which will be described in detail later with reference to Figure 2) highlights the blood vessel region contained within the submucosa in the image of the endoscope system. The [original image] in Figure 1 shows the surgical field region of the image of the endoscope system, and the [highlighted image] in Figure 1 highlights the blood vessel region contained within the submucosa in the [original image] of Figure 1.

[0014] Specifically, the information processing device 10 estimates the area of ​​blood vessels within the surgical field region of the endoscopic system's image using a rule-based approach (or a pre-trained machine learning model), and also estimates the area of ​​the submucosa within the surgical field region of the endoscopic system's image using the pre-trained machine learning model. Based on the estimation results for the blood vessel area and the submucosa area, the information processing device 10 extracts and highlights the areas of blood vessels contained within the submucosa.

[0015] <System Configuration> Figure 2 shows the overall configuration according to one embodiment of the present invention. The information processing system 1 includes an information processing device 10, an endoscopy system 20, and a display device 30. The case in which a physician 11 performs endoscopic submucosal dissection for an early-stage gastrointestinal malignant tumor on a patient 12 will be described below. Each of these will be described below.

[0016] <<Information Processing Device>> The information processing device 10 highlights the areas of blood vessels contained within the submucosal layer of the patient 12 in the image from the endoscope system 20. The information processing device 10 consists of one or more computers.

[0017] Specifically, the information processing device 10 acquires images from the endoscope system 20 (specifically, images of the inside of the patient's body 12 captured by the endoscope system 20). The information processing device 10 also extracts the areas of blood vessels contained within the submucosa in the surgical field region of the images acquired from the endoscope system 20. The information processing device 10 then displays the images from the endoscope system 20, with the areas of blood vessels contained within the submucosa highlighted, on the display device 30.

[0018] Furthermore, as shown in Figure 1, the information processing system 1 may be used not only during endoscopic submucosal dissection but also for educational and confirmation purposes after the procedure (i.e., it may highlight the areas of blood vessels contained within the submucosa in the endoscopic system images from past endoscopic submucosal dissections).

[0019] <<Endoscopy System>> The endoscope system 20 images the inside of the patient's body 12 and sends the images of the inside of the body to the information processing device 10 and the display device 30.

[0020] <<Display device>> The display device 30 is a monitor, display, etc., that displays data acquired from the information processing device 10 and the endoscopy system 20. The physician 11 can recognize the image from the endoscopy system 20 in which the area of ​​blood vessels contained in the submucosal layer of the patient 12 is highlighted.

[0021] Furthermore, at least two of the information processing device 10, the endoscope system 20, and the display device 30 may be implemented in a single device.

[0022] <Processing method> Figure 3 is a flowchart of a process for highlighting the region of blood vessels contained within the submucosa according to one embodiment of the present invention.

[0023] In step 101 (S101), the information processing device 10 acquires images from the endoscope system 20 (specifically, images of the inside of the patient's body captured by the endoscope system 20).

[0024] In step 102 (S102), the information processing device 10 acquires the surgical field region of each image included in the video from the endoscope system 20 acquired in S102 (that is, the image of the part showing the inside of the patient's body 12).

[0025] In step 103 (S103), the information processing device 10 estimates which step of the endoscopic submucosal dissection procedure is being performed.

[0026] In step 104 (S104), the information processing device 10 determines whether the process estimated in S103 is a "mucosal incision" or a "submucosal dissection" process (which will be described in detail later with reference to Figure 4). If it is a "mucosal incision" or a "submucosal dissection" process, the device proceeds to steps 105 and 106. If it is neither a "mucosal incision" nor a "submucosal dissection" process, the process is terminated.

[0027] Steps 103 and 104 may be omitted.

[0028] In step 105 (S105), the information processing device 10 estimates the submucosal region within the surgical field area acquired in S102.

[0029] In step 106 (S106), the information processing device 10 estimates the area of ​​blood vessels within the surgical field area acquired in S102.

[0030] In step 107 (S107), the information processing device 10 extracts the region where the submucosal region estimated in S105 and the vascular region estimated in S106 overlap as the vascular region.

[0031] In step 108 (S108), the information processing device 10 highlights the blood vessel region extracted in S107.

[0032] Figure 4 is a diagram illustrating the steps of endoscopic submucosal dissection according to one embodiment of the present invention. Endoscopic submucosal dissection is performed in the following order: (0) after preparation, (1) marking (specifically, marking the area to be resected (i.e., around the malignant tumor)), (2) local injection (specifically, injecting saline solution to raise the malignant tumor), (3) mucosal incision (specifically, incising the marked mucosa) and submucosal dissection (specifically, dissecting the malignant tumor together with the submucosa), and (4) after the resection is completed, the specimen is submitted.

[0033] The timing at which physicians 11 etc. want to highlight the vascular area is limited to the above-mentioned (3) mucosal incision and submucosal dissection steps, and highlighting the vascular area at any other time would be disruptive. Therefore, as shown in steps 103 and 104 of Figure 3, it may be possible to estimate which step of endoscopic submucosal dissection is being performed and highlight the vascular area only when it is either the "mucosal incision" or "submucosal dissection" step.

[0034] For example, when the surgical field area of ​​an endoscopic submucosal dissection (CSM) is input to the information processing device 10, it can estimate which step of the CSM is being performed by inputting the surgical field area acquired by the information processing device 10 into a pre-trained model that has been machine-learned (for example, using a Convolutional Neural Network (CNN) method) to output which step of the CSM is being performed.

[0035] <Function Block> Figure 5 is a functional block diagram of an information processing device 10 according to one embodiment of the present invention. The information processing device 10 comprises an image acquisition unit 101, a surgical field area acquisition unit 102, a submucosal layer area estimation unit 103, a learned model storage unit 104, a vascular region estimation unit 105, a vascular region extraction unit within the submucosal layer 106, and a highlighting unit 107. By executing a program, the information processing device 10 functions as the image acquisition unit 101, the surgical field area acquisition unit 102, the submucosal layer area estimation unit 103, the vascular region estimation unit 105, the vascular region extraction unit within the submucosal layer 106, and the highlighting unit 107. Each of these will be described below.

[0036] The image acquisition unit 101 acquires images from the endoscope system 20 (specifically, images of the inside of the patient's body 12 captured by the endoscope system 20).

[0037] The surgical field area acquisition unit 102 acquires the surgical field area of ​​each image included in the video of the endoscope system 20 acquired by the video acquisition unit 101 (that is, the image of the part of the patient's body that shows the inside of the human body 12).

[0038] The submucosal region estimation unit 103 estimates the submucosal region included in the surgical field region acquired by the surgical field region acquisition unit 102. Specifically, the submucosal region estimation unit 103 estimates the submucosal region included in the surgical field region by inputting the surgical field region acquired by the surgical field region acquisition unit 102 into a pre-trained model that has been machine-learned to output the submucosal region included in the surgical field region when the surgical field region is input.

[0039] The trained model memory unit 104 stores a trained model that, when a surgical field region is input, outputs the submucosal region included in that surgical field region (for example, using a semantic segmentation method).

[0040] The vascular region estimation unit 105 estimates the vascular regions included in the surgical field region acquired by the surgical field region acquisition unit 102. Specifically, the vascular region estimation unit 105 estimates the vascular regions included in the surgical field region using a rule-based approach. Alternatively, the vascular region estimation unit 105 may estimate the vascular regions included in the surgical field region by inputting the surgical field region acquired by the surgical field region acquisition unit 102 into a trained model that has been machine-learned (for example, supervised machine learning where the vascular region is the true value) so that when the surgical field region is input, the vascular regions included in the surgical field region are output.

[0041] [Example of rule-based vascular region estimation] As shown below, the blood vessel region estimation unit 105 can estimate that pixels with a blood vessel-like color are blood vessel regions. For example, the blood vessel region estimation unit 105 defines a blood vessel region as a pixel where the luminance value of the R component of the RGB color channel is greater than or equal to a threshold. For example, the blood vessel region estimation unit 105 defines a blood vessel region as a pixel whose H component (hue) in the HSV color space is within a certain range. For example, the vascular region estimation unit 105 calculates statistical values ​​(e.g., mean, variance) of the R component of the RGB color channel (or around 0 degrees in the HSV color space), determines the range of R component values ​​for vascular regions from these statistical values, and during estimation, identifies pixels whose R component values ​​fall within this range as vascular regions. Note that statistical values ​​may be calculated using multiple images of one patient (e.g., the patient being estimated), or using multiple images of multiple patients. For example, the vascular region estimation unit 105 uses the nearest neighbor method (k-nearest neighbor method). Specifically, the vascular region estimation unit 105 extracts vascular regions by clustering using edges and color as features, and collects the feature quantities of the vascular regions (features used for judgment during estimation; called representative features). During estimation, the vascular region estimation unit 105 clusters using similar features (i.e., edges and color), determines whether the feature quantities of the patient image being estimated are close to the representative features, and identifies pixels in the vicinity of the representative features as vascular regions.

[0042] The submucosal vascular region extraction unit (also simply called the vascular region extraction unit) 106 extracts the region where the submucosal region estimated by the submucosal region estimation unit 103 and the vascular region estimated by the vascular region estimation unit 105 overlap as the vascular region.

[0043] [Suppression of false detections in the vascular region (in case of bleeding)] As shown below, the vascular region extraction unit 106 can suppress the misidentification of bleeding blood as a vascular region.

[0044] For example, the vascular region extraction unit 106 extracts regions from the region estimated by the vascular region estimation unit 105 where the shorter side is less than or equal to the diameter of the tip of the electrosurgical unit as vascular regions (this will be described in detail later with reference to Figure 9).

[0045] For example, the vascular region extraction unit 106 extracts regions from the region estimated by the vascular region estimation unit 105 where the aspect ratio is greater than or equal to a threshold as vascular regions (this will be described in detail later with reference to Figure 10).

[0046] For example, the vascular region extraction unit 106 extracts regions with an area below a threshold from the regions estimated by the vascular region estimation unit 105 as vascular regions (this will be described in detail later with reference to Figure 11).

[0047] [Suppression of false detections in the vascular region (in the case of the muscle layer)] As described below, the vascular region extraction unit 106 can suppress the misidentification of non-vascular regions (for example, the muscular layer) contained within the submucosa as vascular regions.

[0048] For example, the vascular region extraction unit 106 determines that a region is not a blood vessel if the area of ​​each region estimated by the vascular region estimation unit 105 is greater than a threshold, and performs a closing process on regions whose area is less than or equal to the threshold. Through the closing process, multiple regions estimated by the vascular region estimation unit 105 are combined. If the area of ​​the combined region after the closing process is greater than or equal to a threshold, the vascular region extraction unit 106 determines that it is not a blood vessel region contained within the submucosa (for example, a region of the muscular layer) (this will be described in detail later with reference to Figure 12).

[0049] For example, the vascular region extraction unit 106 performs an opening process on regions estimated by the vascular region estimation unit 105 whose area is below a threshold, and removes non-vascular regions contained within the submucosa (this will be described in detail later with reference to Figure 12).

[0050] For example, the vascular region extraction unit 106 detects a first region within the surgical field as a region that is not a blood vessel within the submucosa (for example, a region of the muscular layer), detects a second region that is smaller in area than the first region, which has been estimated by the vascular region estimation unit 105, as a vascular region, and extracts the second region that does not overlap with the first region as a vascular region (this will be described in detail later with reference to Figure 12).

[0051] The highlighting unit 107 highlights the areas of blood vessels contained within the submucosa, which have been extracted by the blood vessel area extraction unit 106. The highlighting unit 107 may also highlight areas within the submucosa.

[0052] [Examples of other displays]

[0053] For example, the highlighting unit 107 estimates the bleeding situation and adjusts the highlighting according to the bleeding situation (this will be described in detail later with reference to Figure 14).

[0054] For example, the highlighting unit 107 determines whether or not to perform coagulation or hemostasis and notifies (for example, by displaying) this (this will be described in detail later with reference to Figure 15).

[0055] For example, the highlighting unit 107 determines and notifies (e.g., displays) the appropriate action for a blood vessel or bleeding (this will be described in detail later with reference to Figure 16).

[0056] For example, the highlighting unit 107 determines and notifies (e.g., displays) the risk of bleeding based on the area of ​​blood vessels contained within the submucosa (this will be described in detail later with reference to Figure 17).

[0057] Figure 6 is a diagram illustrating the process for highlighting the region of blood vessels contained within the submucosa according to one embodiment of the present invention.

[0058] First, as shown in [Image Acquisition], images from the endoscope system 20 are acquired. Each image from the endoscope system 20 includes the surgical field area (the image showing the inside of the patient's body 12) and a thumbnail of the immediately preceding surgical field area.

[0059] Next, as shown in [Acquisition of Surgical Field Area], the surgical field area (the part of the image showing the inside of the patient's body 12) is acquired for each image of the video from the endoscopic system 20.

[0060] Next, as shown in [Estimation of the submucosal region], the submucosal region included in the surgical field is estimated, and as shown in [Estimation of the vascular region], the vascular region included in the surgical field is estimated.

[0061] Next, as shown in [Extraction of Vascular Regions], the region where the estimated submucosal region and the estimated vascular region overlap is extracted as the vascular region.

[0062] Subsequently, as indicated by [Highlighting of Vascular Regions], the image from the endoscope system 20 is displayed with the vascular regions within the submucosa highlighted.

[0063] Figure 7 is a diagram illustrating the extraction of vascular regions according to one embodiment of the present invention. As shown in [Estimated result of submucosal region + Estimated result of vascular region] in Figure 7, the vascular region estimation unit 105 also estimates the regions of blood vessels other than blood vessels contained within the submucosa (the regions circled in Figure 7). The vascular region extraction unit 106 filters (excludes) the regions of blood vessels that do not overlap with the submucosal region, as shown in [Extraction of vascular region] in Figure 7, so as not to extract such regions of blood vessels other than blood vessels contained within the submucosa, and extracts only the regions of blood vessels that overlap with the submucosal region.

[0064] Figure 8 is a flowchart of the process for suppressing false detection of vascular regions according to one embodiment of the present invention.

[0065] In step 1 (S1), the vascular region extraction unit 106 determines whether the shorter end of the region estimated by the vascular region estimation unit 105 is less than or equal to the diameter of the tip of the electrosurgical unit of the endoscope. If the shorter end is less than or equal to the diameter of the tip of the electrosurgical unit, the process proceeds to step 5; if the shorter end is longer than the diameter of the tip of the electrosurgical unit, the process proceeds to step 2.

[0066] In step 2 (S2), the vascular region extraction unit 106 determines whether the aspect ratio of the region estimated by the vascular region estimation unit 105 is 1:5 or greater. If the aspect ratio is 1:5 or greater, the process proceeds to step 5; otherwise, the process proceeds to step 3.

[0067] In step 3 (S3), the vascular region extraction unit 106 determines whether the area of ​​the region estimated by the vascular region estimation unit 105 is below a threshold. If the area is below the threshold, the process proceeds to step 5; if the area is greater than the threshold, the process proceeds to step 4.

[0068] In step 4 (S4), the vascular region extraction unit 106 determines that the region estimated by the vascular region estimation unit 105 is not a vascular region, and therefore does not extract the region estimated by the vascular region estimation unit 105 as a vascular region.

[0069] In step 5 (S5), the vascular region extraction unit 106 determines that the region estimated by the vascular region estimation unit 105 is a vascular region, and extracts the region estimated by the vascular region estimation unit 105 as a vascular region.

[0070] Note that the decisions of S1, S2, and S3 may be made in an order other than that shown in Figure 8, or at least one of the decisions of S1, S2, and S3 may be made.

[0071] Figure 9 is a diagram illustrating the suppression of false detection of vascular regions (Step 1 in Figure 8) according to one embodiment of the present invention. If the region estimated by the vascular region estimation unit 105 is absolutely large, it is highly likely that the region is not a vascular region but a region of bleeding blood. However, since it is not possible to determine the absolute size of a vascular region in the surgical field, the relative size of the vascular region (specifically, the thickness of the vascular region) is determined by comparing it with the diameter of the tip of the electrosurgical unit of the endoscope. The vascular region extraction unit 106 extracts regions where the shorter side is less than or equal to the diameter of the tip of the electrosurgical unit as vascular regions.

[0072] Figure 10 is a diagram illustrating the suppression of false detection of vascular regions (step 2 in Figure 8) according to one embodiment of the present invention. If the region estimated by the vascular region estimation unit 105 is not an elongated region, it is highly likely that the region is a region of bleeding blood rather than a vascular region. The vascular region extraction unit 106 extracts regions as vascular regions where the aspect ratio (short side:long side) is greater than or equal to a threshold (e.g., 1:5) (i.e., the long side / short side ratio is greater than or equal to the threshold) so as not to mistakenly extract regions of bleeding blood as vascular regions. The aspect ratio threshold may be set according to how wide of a vascular region you want to extract.

[0073] Figure 11 is a diagram illustrating the suppression of false detection of vascular regions (step 3 in Figure 8) according to one embodiment of the present invention. If the area of ​​the region estimated by the vascular region estimation unit 105 is large, it is highly likely that the region is an area of ​​bleeding blood rather than a vascular region. The vascular region extraction unit 106 extracts regions with an area below a threshold as vascular regions to prevent the extraction of a region of bleeding blood as a vascular region by mistake.

[0074] Figure 12 is a diagram illustrating the suppression of false detection of vascular regions (false detection of non-vascular regions (e.g., muscular layer regions) contained within the submucosa) according to one embodiment of the present invention. For example, the muscular layer is contained within the submucosa, and there is a possibility that the muscular layer itself or the regions of blood vessels within the muscular layer may be falsely detected as blood vessels contained within the submucosa. In addition, red noise that appears in the image (for example, red noise that appears in the submucosa due to light reflection) may be falsely detected as blood vessels contained within the submucosa. Since these muscular layers themselves, blood vessels within the muscular layer, and red noise appear on the image as densely packed small red regions, closing and opening processes are performed as follows.

[0075] [Closing process] For example, the vascular region extraction unit 106 determines that a region is not a blood vessel if the area of ​​each region estimated by the vascular region estimation unit 105 is greater than a threshold. Next, the vascular region extraction unit 106 performs a closing process on regions whose area is less than or equal to the threshold (specifically, all regions estimated by the vascular region estimation unit 105 whose area is less than or equal to the threshold (this may also apply to some regions)). The closing process combines the multiple regions estimated by the vascular region estimation unit 105. If the area of ​​the combined region after the closing process is greater than or equal to the threshold, the vascular region extraction unit 106 determines that it is not a blood vessel region contained within the submucosa (for example, a region of the muscular layer) and does not extract it as a blood vessel region.

[0076] In other words, by connecting small red areas on the surface of the muscle layer (i.e., areas that are easily mistaken for blood vessels), if the resulting area is large, it can be determined to be the muscle layer.

[0077] [Opening process] For example, the vascular region extraction unit 106 determines that a region is not a blood vessel if the area of ​​each region estimated by the vascular region estimation unit 105 is greater than a threshold. Next, the vascular region extraction unit 106 performs an opening process on regions whose area is less than or equal to the threshold (specifically, all regions estimated by the vascular region estimation unit 105 whose area is less than or equal to the threshold (this may also be only some regions)) to remove non-vascular regions (e.g., noise) contained within the submucosa.

[0078] In other words, it is possible to prevent the false detection of non-vascular areas (e.g., noise) within the submucosa as blood vessels within the submucosa.

[0079] Furthermore, it is possible to perform only the closing process, only the opening process, or both the closing and opening processes.

[0080] For example, the vascular region extraction unit 106 detects a first region within the surgical field as a region that is not a blood vessel within the submucosa (e.g., a region of the muscular layer), detects a second region that is smaller in area than the first region and estimated by the vascular region estimation unit 105 as a vascular region, and extracts the second region that does not overlap with the first region as a vascular region. For example, the vascular region extraction unit 106 shrinks the surgical field and performs hue conversion (for example, converting from RGB to HSV to convert Hue, Saturation, and Value), and detects red and pink regions (for example, since the area around 0 degrees on the Hue hue circle represents red, regions with Hue values ​​at angles near that (355-5 degrees)) as regions that are not blood vessels within the submucosa (e.g., a region of the muscular layer).

[0081] In other words, a mask image is generated of non-vascular areas within the submucosa (large red or pink areas, for example, the muscular layer), and the vascular areas that overlap with this mask image are filtered (removed), allowing only the vascular areas that do not overlap with the mask image to be extracted.

[0082] In this way, by estimating the blood vessel regions based on pixels that appear to be the color of blood vessels within the image, it is possible to extract the blood vessel regions while excluding areas that are not blood vessels, such as bleeding blood or non-vascular areas (e.g., the muscle layer) contained within the submucosa.

[0083] Figure 13 shows an example of highlighting a vascular region according to one embodiment of the present invention. For example, the highlighting involves enhancing the contrast only within the brightness band (range) of the vascular region contained in the submucosa (for example, by converting the image to HSV and increasing the S component of the vascular region to enhance saturation).

[0084] Furthermore, as shown in [Display Example 1] in Figure 13, only the image with the blood vessel region highlighted may be displayed, or both the original image and the image with the blood vessel region highlighted may be displayed for comparison, as shown in [Display Example 2] in Figure 13.

[0085] Figure 14 is a diagram illustrating the indication of bleeding according to one embodiment of the present invention.

[0086] In situations where bleeding is occurring or has occurred, as shown in Figure 14 [Bleeding], the entire surgical field area becomes red, and if the vascular area is highlighted as shown in Figure 14 [Highlighting of Vascular Area], it can actually hinder the recognition of the blood vessels. The highlighting unit 107 can estimate the bleeding situation and reduce the degree of highlighting, as shown in Figure 14 [Highlighting of Vascular Area (Weak)], or disable the highlighting process altogether, as shown in [Highlighting of Vascular Area (None)].

[0087] For example, the highlighting unit 107 may estimate the bleeding situation based on the area of ​​the region estimated by the vascular region estimation unit 105, or it may estimate the bleeding situation using a pre-trained model that has been trained by machine learning so that the bleeding situation is output when a surgical field region is input.

[0088] For example, the highlighting unit 107 reduces the degree of highlighting as the amount of bleeding increases (i.e., the area of ​​the region estimated by the vascular region estimation unit 105 is larger).

[0089] For example, the highlighting unit 107 determines that it is difficult to recognize small blood vessels when there is a lot of bleeding (i.e., the area of ​​the region estimated by the vascular region estimation unit 105 is large), and therefore refrains from performing the highlighting process.

[0090] [Notifications based on the size of the blood vessel or the extent of bleeding] The following describes the notification system based on the size of the blood vessel or the bleeding situation, with reference to Figures 15 and 16. First, the highlighting unit 107 distinguishes whether the region estimated by the blood vessel region estimation unit 105 represents a blood vessel or bleeding blood. For example, the highlighting unit 107 may distinguish using a machine learning-based (two-class classification task) trained model, or it may distinguish using a rule-based system (e.g., color (blood vessels have low saturation, bleeding has high saturation), area, texture (blood vessels have high granularity, bleeding has low granularity), etc.).

[0091] Figure 15 is a diagram illustrating the notification of the necessity of coagulation hemostasis according to one embodiment of the present invention. It may be unclear whether coagulation is necessary when a blood vessel area is highlighted, or whether hemostasis (coagulation hemostasis or compression hemostasis) is necessary when bleeding occurs. The highlighting unit 107 estimates the size of the blood vessel to determine and notify whether coagulation is necessary, and estimates the bleeding situation to determine and notify whether hemostasis (coagulation hemostasis or compression hemostasis) is necessary.

[0092] For example, the highlighting unit 107 may estimate the size of a blood vessel based on the area of ​​the region estimated by the blood vessel region estimation unit 105, or it may estimate the size of a blood vessel using a trained model that has been machine-learned to output the size of a blood vessel when a surgical field region is input. The highlighting unit 107 determines that coagulation is necessary if the size of the blood vessel is above a threshold (for example, the coagulation mode of the electrosurgical unit should be used), and determines that coagulation is unnecessary if the size of the blood vessel is below a threshold (for example, the incision mode of the electrosurgical unit should be used).

[0093] For example, the highlighting unit 107 may estimate the bleeding situation based on the area of ​​the region estimated by the vascular region estimation unit 105, or it may estimate the bleeding situation using a trained model that has been machine-learned to output the bleeding situation when a surgical field region is input. The highlighting unit 107 determines that hemostasis (e.g., coagulation hemostasis, compression hemostasis) is necessary if the bleeding is above a threshold, and determines that hemostasis is unnecessary if the bleeding is below the threshold.

[0094] Figure 16 is a diagram illustrating the notification of treatment according to one embodiment of the present invention. When a blood vessel area is highlighted, or when bleeding occurs, it may be unclear what treatment is appropriate. The highlighting unit 107 estimates the size of the blood vessel or the bleeding situation, determines the appropriate treatment, and notifies the user.

[0095] For example, the highlighting unit 107 may estimate the size of the blood vessel based on the area of ​​the region estimated by the blood vessel region estimation unit 105, or it may estimate the size of the blood vessel using a pre-trained model that has been machine-learned to output the size of the blood vessel when a surgical field region is input. The highlighting unit 107 determines the appropriate treatment according to the size of the blood vessel.

[0096] For example, the highlighting unit 107 may estimate the bleeding situation based on the area of ​​the region estimated by the vascular region estimation unit 105, or it may estimate the bleeding situation using a trained model that has been machine-learned to output the bleeding situation when a surgical field region is input. The highlighting unit 107 determines the appropriate treatment according to the bleeding situation.

[0097] Figure 17 is a diagram illustrating the notification of bleeding risk according to one embodiment of the present invention. The highlighting unit 107 determines and notifies (e.g., displays) the bleeding risk based on the size and distribution of blood vessels (i.e., the area and distribution of the blood vessel region contained within the submucosa). For example, the highlighting unit 107 estimates the bleeding risk from the area and distribution of the blood vessel region contained within the submucosa using a rule-based approach or a machine learning-trained model. The bleeding risk may be expressed as a percentage (e.g., a bleeding probability of 30%) or as a gauge (bar graph).

[0098] <Effects> In one embodiment of the present invention, it is possible to notify the user of structures that cause bleeding and the associated risks. By recognizing the area of ​​the submucosa that can be peeled away while maintaining distance from the muscle layer, and the area of ​​blood vessels within it, bleeding and perforation can be prevented.

[0099] <Hardware Configuration> Figure 18 is a hardware configuration diagram of an information processing device 10 according to one embodiment of the present invention. The information processing device 10 may include a control unit 1001, a main memory unit 1002, an auxiliary memory unit 1003, an input unit 1004, an output unit 1005, and an interface unit 1006. Each of these will be described below.

[0100] The control unit 1001 is a processor (for example, a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), etc.) that executes various programs installed in the auxiliary storage unit 1003.

[0101] The main memory unit 1002 includes non-volatile memory (ROM (Read Only Memory)) and volatile memory (RAM (Random Access Memory)). The ROM stores various programs, data, etc., necessary for the control unit 1001 to execute various programs installed in the auxiliary memory unit 1003. The RAM provides a work area that is expanded when the various programs installed in the auxiliary memory unit 1003 are executed by the control unit 1001.

[0102] The auxiliary storage unit 1003 is an auxiliary storage device that stores various programs and information used when various programs are executed.

[0103] The input unit 1004 is an input device that allows the operator of the information processing device 10 to input various instructions to the information processing device 10.

[0104] The output unit 1005 is an output device that outputs the internal state of the information processing device 10, etc.

[0105] The interface unit 1006 is a communication device for connecting to a network and communicating with other devices.

[0106] Although embodiments of the present invention have been described in detail above, the present invention is not limited to the specific embodiments described above, and various modifications and changes are possible within the scope of the gist of the present invention as described in the claims. [Explanation of Symbols]

[0107] 1. Information Processing System 10 Information Processing Devices 20 Endoscopy Systems 30 Display device 11 Doctors 12 patients 101 Video Acquisition Unit 102 Surgical field area acquisition department 103 Submucosal layer area estimation section 104 Pre-trained model memory 105 Blood vessel region estimation section 106 Blood vessel region extraction part in submucosal layer region 107 Highlight section 1001 Control Unit 1002 Main memory 1003 Auxiliary storage unit 1004 Input section 1005 Output section 1006 Interface section

Claims

1. In endoscopic submucosal dissection for early gastrointestinal malignancies, the image from the endoscopic system highlights the area of ​​blood vessels contained within the submucosa. Equipped with, information processing device.

2. The information processing apparatus according to claim 1, wherein the highlighting unit highlights the region of the submucosa.

3. A video acquisition unit that acquires images from the endoscope system, A surgical field area acquisition unit that acquires the surgical field area of ​​the image from the endoscopic system acquired, A submucosal region estimation unit estimates the submucosal region included in the acquired surgical field region using a pre-trained model that has been trained to output the submucosal region when the surgical field region is input, A vascular region estimation unit that estimates the vascular region included in the acquired surgical field region, A submucosal region vascular region extraction unit extracts the region where the submucosal region estimated by the submucosal region estimation unit and the vascular region estimated by the vascular region estimation unit overlap. The information processing apparatus according to claim 1, further comprising the above.

4. The information processing apparatus according to claim 3, wherein the submucosal vascular region extraction unit extracts as vascular regions regions regions from the regions estimated by the vascular region estimation unit regions in which the shorter side is less than or equal to the diameter of the tip of the electrosurgical unit.

5. The information processing apparatus according to claim 3, wherein the submucosal vascular region extraction unit extracts regions from the regions estimated by the vascular region estimation unit in which the aspect ratio is equal to or greater than a threshold as vascular regions.

6. The information processing apparatus according to claim 3, wherein the submucosal vascular region extraction unit extracts regions with an area less than or equal to a threshold from the regions estimated by the vascular region estimation unit as vascular regions.

7. The information processing apparatus according to claim 3, wherein the submucosal vascular region extraction unit performs a closing process on regions whose area is less than or equal to a threshold among the regions estimated by the vascular region estimation unit, and if the area of ​​the region after the closing process is greater than or equal to a threshold, the unit determines that the region is not a vascular region included in the submucosal layer and does not extract it as a vascular region.

8. The information processing apparatus according to claim 3, wherein the submucosal vascular region extraction unit performs an opening process on regions whose area is less than or equal to a threshold among the regions estimated by the vascular region estimation unit, and removes regions that are not blood vessels contained in the submucosal layer.

9. The information processing apparatus according to claim 3, wherein the submucosal vascular region extraction unit detects a first region within the surgical field region as a region that is not a blood vessel within the submucosa, detects a second region estimated by the vascular region estimation unit and having a smaller area than the first region as a blood vessel region, and extracts the second region that does not overlap with the first region as a blood vessel region.

10. The information processing apparatus according to claim 3, wherein the highlighting unit, when the surgical field region of the image of the endoscope system is input, highlights the vascular region when the estimated step, based on a trained model that has been trained to output the steps of endoscopic submucosal dissection, is mucosal incision or submucosal dissection.

11. The information processing apparatus according to claim 10, wherein the steps are preparation, marking, local injection, mucosal incision and submucosal dissection, and completion of excision.

12. The information processing apparatus according to claim 1, wherein the highlighting unit estimates the bleeding situation and adjusts the highlighting according to the bleeding situation.

13. The information processing apparatus according to claim 1, wherein the highlighting unit determines and notifies whether or not coagulation or hemostasis should be performed.

14. The information processing device according to claim 1, wherein the highlighting unit determines and notifies the appropriate treatment for blood vessels or bleeding.

15. The information processing device according to claim 1, wherein the highlighting unit determines and notifies of the risk of bleeding based on the region of blood vessels contained in the submucosal layer.

16. A method executed by an information processing device, To highlight the vascular region within the submucosa in endoscopic images from an endoscopic system during endoscopic submucosal dissection for early gastrointestinal malignancies. A method that includes this.

17. In an information processing device, To highlight the vascular region within the submucosa in endoscopic images from an endoscopic system during endoscopic submucosal dissection for early gastrointestinal malignancies. A program to execute.

Images