Apparatus and method for guiding inspection of large intestine by using endoscope
The colon examination guide device uses an endoscope with a deep learning model to analyze colon folds, providing visual cues for complete visualization, improving polyp detection in colonoscopies.
Patent Information
- Application Number
- JP2024151419
- Authority / Receiving Office
- JP · JP
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2021-01-14
- Filing Date
- 2024-09-03
- Publication Date
- 2024-11-08
- Estimated Expiration
- 2041-01-22
Smart Images

Figure 2024160013000001_ABST
Abstract
Description
[Technical field]
[0001] The present invention relates to a colon examination guide, and more particularly to a colon examination guide device using an endoscope and and methods for providing the same. [Background technology]
[0002] In general, colonoscopy is used to detect small raised lesions on the inner wall of the colon that are precursors of colon cancer. By detecting and removing polyps, which are part of the affected tissue, colon cancer can be eradicated. Specifically, the purpose of the device is to detect and remove polyps through a colonoscope. This can prevent the occurrence of interval cancer. However, In nearly 30% of cases, colonoscopy fails to detect polyps. It is known.
[0003] This is because there are many folds in the large intestine, and colon polyps hidden behind the folds can be found. In reality, the only way to assess this is to rely on the concentration of the doctor who performed the test. The only method currently available is an indirect method, such as whether the test took more than six minutes. Even the most observant physicians will recognize that different people have different ways of searching for treasure during a treasure hunt. Similarly, depending on the person's inspection style, they may repeat what they have seen, but not what they have seen. There is a high chance that Ro will not watch it.
[0004] Therefore, when performing a colonoscopy, it is necessary to observe each of the multiple folds in the subject's colon one by one. What is needed is a method that allows for confirmation through an endoscope without missing anything. Summary of the Invention [Problem to be solved by the invention]
[0005] The present invention has been made in view of the above circumstances, and its object is to provide a method for performing a colonoscopy. ,to recognize colonic folds in colonoscopic images, and provide them to ,professionals performing colonoscopy. The point is...
[0006] Specifically, the object of the present invention is to provide a method for identifying the folds of the large intestine by size, which is recognized from a colonoscopic image. , and to display the visual effect with different magnitudes.
[0007] Another object of the present invention is to provide a method for imaging the posterior portion of the folds of the large intestine by using an endoscope. If not, provide visual effects for the folds and have an expert verify the folds. The object of the present invention is to eliminate the visual effect of the fold when the surface is photographed using an endoscope.
[0008] The problems to be solved by the present invention are not limited to the problems mentioned above, and may include any other problems not mentioned above. Other problems will be clearly understood by those skilled in the art from the following description. [Means for solving the problem]
[0009] In a method performed by the device according to the present invention for solving the above-mentioned problems, receiving an image taken from an endoscope inserted into the large intestine in real time; identifying an image of each section of the large intestine that includes at least one fold within the image; and displaying a first visual effect representing the pleats in the image of each of the sections. determining whether the back surface of the fold is captured for each image; and Among the images of each section, at least one section in which the back surface of the pleat is not photographed displaying a second visual effect in the image, which indicates that the underside of the fold is not photographed; and
[0010] Here, the recognition step is based on light irradiated into the inside of the large intestine by the endoscope. This can be done.
[0011] The recognizing step further includes recognizing the image of each section using a deep learning model; The deep learning model uses colon images of multiple subjects obtained from external annotators. Fold data in the image, the amount of change in the shadow caused by the irradiated light in the large intestine, and blood vessels It can be a machine-learned model based on
[0012] Here, the first visual effect is to provide a corresponding fold in the image of each section, The visual effect includes a marker being displayed, and the size of each marker is determined based on the size of the corresponding fold. The distance between the first and second electrodes may be determined based on the distance between the first and second electrodes.
[0013] In addition, when the back side of the corresponding fold is photographed, the step of deleting the second visual effect is further performed. can be included in.
[0014] Other methods, systems and methods for implementing the present invention are also provided. A computer-readable recording medium for recording a computer program for performing the method is further provided. can be provided. Effect of the Invention
[0015] According to the present invention as described above, when performing colonoscopy, it is possible to obtain a colon histology image from the colonoscope image. By identifying the folds of the colon and providing them to specialists performing colonoscopy, This has the effect of making it easier to detect polyps that may be located in the area.
[0016] Specifically, according to the present invention, the folds of the large intestine are classified according to the size of the folds recognized from the images of the large intestine endoscope. By displaying the visual effect with different sizes, when the folds are large, some of the folds The effect is to guide the user to check the whole thing, rather than just checking the whole thing and then passing it. There is fruit.
[0017] In addition, according to the present invention, the rear surface of the folds of the large intestine is not photographed by an endoscope. In this case, a visual effect is provided for the folds, and an expert examines the back of the folds. When photographed using an endoscope, the visual effect of the folds is eliminated, making it possible to see the folds clearly. The effect of making it possible to clearly distinguish between the parts of the back of the There is fruit.
[0018] The effects of the present invention are not limited to those mentioned above, and further effects not mentioned are as follows: Those skilled in the art should be able to clearly understand the above description. [Brief description of the drawings]
[0019] [Figure 1] 1 is a block diagram that illustrates a schematic diagram of an apparatus for providing endoscopic colon examination guidance in accordance with the present invention; [Diagram 2] FIG. 1 is a block diagram that illustrates, in schematic form, how a deep learning model is trained for use in guiding colon examinations according to the present invention. [Diagram 3] 1 is an exemplary diagram showing a process in which a processor of the device according to the present invention provides guidance for colon examination using an endoscope. [Figure 4a] FIG. 1 is an illustrative diagram (1) showing a first visual effect representing folds displayed on an image captured from an endoscope inserted into a subject's large intestine according to the present invention. [Figure 4b]FIG. 2 is an illustrative diagram (2) showing a first visual effect representing folds displayed on an image captured from an endoscope inserted into a subject's large intestine according to the present invention. [Figure 4c] FIG. 3 is an illustrative diagram showing a first visual effect representing folds displayed on an image captured from an endoscope inserted into a subject's large intestine according to the present invention. [Figure 4d] FIG. 4 is an illustrative diagram showing a first visual effect representing folds displayed on an image captured from an endoscope inserted into a subject's colon according to the present invention (4). [Figure 5a] FIG. 1 is an illustrative diagram (1) showing a second visual effect being displayed, representing a fold in which an image of the back side of the fold is not captured, in accordance with the present invention. [Figure 5b] FIG. 2 is an illustrative diagram (2) showing a second visual effect being displayed, representing a fold in which an image of the back side of the fold in which the first visual effect is displayed is not captured, according to the present invention. [Figure 5c] FIG. 3 is an illustrative diagram (3) showing a second visual effect being displayed, representing a fold in which an image of the back side of the fold in which the first visual effect is displayed is not captured, according to the present invention. [Figure 5d] FIG. 4 is an illustrative diagram (4) showing a second visual effect being displayed, representing a fold in which an image of the back side of the fold is not captured, in accordance with the present invention, in which a first visual effect is displayed. [Figure 6] 4 is a flowchart showing a process in which a processor of the device according to the present invention guides a colon examination using an endoscope. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0020] The advantages and features of the present invention and the method of achieving them are described in detail below with the accompanying drawings. However, the present invention will become more apparent from reference to the embodiments disclosed below. However, the present invention is not limited to the above and may be embodied in many different forms. The embodiments will be described in detail below with reference to the accompanying drawings, in which: The present invention is not limited to the above-mentioned embodiments, and is not limited to the above-mentioned embodiments. It is only defined by
[0021] The terms used in this specification are intended to describe the embodiments and are not intended to limit the present invention. In this specification, the singular includes the plural unless specifically stated otherwise. As used in the specification, the words "comprises" and / or "comprises" mean "rising" refers to the presence or addition of one or more other elements beyond the element being mentioned The same reference numerals refer to the same elements throughout the specification, and the like, including the following: "and / or "includes" includes each and every combination of one or more of the recited elements. Although "first," "second," etc. are used to describe various components, Of course, the components are not limited by these terms. , are used merely to distinguish one component from another. Therefore, It goes without saying that the first component may also be the second component within the technical concept of the present invention. stomach.
[0022] Unless otherwise defined, all terms (including technical and scientific terms) used herein are are used in a manner commonly understood by those skilled in the art to which the present invention pertains. In addition, terms defined in commonly used dictionaries are not expressly and specifically defined. Unless otherwise stated, it should not be interpreted ideally or excessively.
[0023] Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings.
[0024] FIG. 1 shows a schematic diagram of an apparatus 10 for providing endoscopic colonic examination guidance in accordance with the present invention. FIG.
[0025] FIG. 2 shows a deep learning model used for colon examination guidance according to the present invention. FIG. 1 is a block diagram illustrating the learning process.
[0026] FIG. 3 shows a processor 140 of the device 10 according to the present invention for endoscopic colon examination guidance. 1 is an exemplary diagram showing a process of providing
[0027] 4a to 4d are images taken from an endoscope inserted into the large intestine of a subject according to the present invention. FIG. 13 is an illustrative diagram showing a first visual effect representing folds being displayed on an image.
[0028] 5a to 5d are images of the back side of the pleats showing the first visual effect according to the present invention. FIG. 13 is an example diagram showing a second visual effect being displayed, representing unphotographed folds.
[0029] Hereinafter, a colon examination guide using an endoscope according to the present invention will be provided with reference to Figs. 1 to 5d. A device 10 will now be described. Here, device 10 is a local computing device. The present invention can also be embodied in a server device.
[0030] The device 10 recognizes the folds of the large intestine from an image of the large intestine endoscopy and detects the inside of the large intestine. By providing endoscopy specialists with the This has the effect of making it easier to find.
[0031] Specifically, the device 10 classifies the folds of the large intestine according to the size of the folds recognized from the colonoscope image. By displaying the effect with different sizes, when the folds are large, The effect is to guide the user to check the whole picture, rather than checking only a part of it and then passing it. may have the following structure:
[0032] The device 10 also uses the endoscope 20 to capture images of the back surface of the folds of the large intestine. If the folds are not visible, a visual effect is provided for the folds. When the back surface is photographed using the endoscope 20, the visual effect of the folds is eliminated. This allows you to clearly distinguish between areas where the backside of the folds has been checked and areas where it has not been checked. Here, the endoscope 20 is inserted into the large intestine. The apparatus for observing biological tissue in the large intestine includes a camera 210, a light 220, etc. It can be done.
[0033] First, referring to FIG. 1, the device 10 includes a communication unit 110, a display unit 120, a memory unit 130, and a 30 and a processor 140. Here, the device 10 includes fewer components than those shown in FIG. Fewer or more components may be included.
[0034] The communication unit 110 communicates between the device 10 and a wireless communication system, and between the device 10 and an endoscope 20. One or more interfaces that enable wireless communication between the device 10 and an external device (not shown), or between the device 10 and an external device (not shown). The communication unit 110 may include the above modules. It may include one or more modules that connect to a network.
[0035] The communication unit 110 transmits in real time images captured by an endoscope inserted into the large intestine of a subject. The communication unit 110 can receive signals from endoscopes inserted into the large intestines of multiple subjects. The images are received from the endoscope, which is inserted into the colon multiple times for one patient. You can receive the captured images.
[0036] Alternatively, the communication unit 110 may communicate with the endoscope for learning the deep learning model. As an example of annotators, the medical staff provided annotations for the images of the colon of the examinee. You can receive the folded data.
[0037] The display unit 120 and the touch sensor may be layered together or may be integrally formed. By using the touch screen, a touch panel can be realized. 1 and a user, and provides an input interface between the device 10 and the user. The input / output interface can be provided.
[0038] The display unit 120 displays various information generated by the processor 140 to the user. In addition to providing the user with various information, the system can also receive various information from the user.
[0039] In more detail, the display unit 120 displays a first visual effect for each section of the image to represent the folds. The display unit 120 can display the results of the folds in the images of each section. The back surface of the fold is photographed in at least one section where the back surface is not photographed. A second visual effect may be displayed to indicate that the
[0040] Memory 130 may store information supporting various functions of device 10. 0 indicates a number of application programs (application programs) that are run on the device 10. ion program or application, the device It can store data and instructions for up to 10 operations. These application programs At least a part of the above is downloaded from an external server (not shown) via wireless communication. In addition, at least a part of such application programs may be The application programs may exist for the basic functions of the device 10. The memory 130 is installed on the device 10 and is read / written by the processor 140. The device 10 can be driven to perform the operations (or functions) described above.
[0041] The memory 130 stores a fold recognition image captured by an endoscope inserted into the large intestine of a subject. Here, you can store a deep learning model for identifying The model is a convolutional neural network (CNN). nal network (CNN), but this does not necessarily mean that The neural network may be formed in a variety of structures, without being limited to the above.
[0042] CNN applies multiple filters to each region of an image and creates a feature map. A convolution layer that generates a feature map (eMap) and By spatially integrating the two, it becomes possible to extract features that are invariant to changes in position and rotation. The Pooling Layer is formed by repeating the Pooling Layer multiple times. This allows the visualization of low-level features such as points, lines, and surfaces into complex, meaningful levels of information. It is possible to extract features at various levels, from low to high level features.
[0043] The convolutional layer applies a filter and a local receptive field (Local R) to each patch of the input image. The nonlinear activation function (Activation F By taking the function, we obtain a feature map. Compared to other network structures, CNNs have the advantage of sparsity (Spars) e Connectivity and Shared Weights Such a connection structure can be characterized by using a filter that has By reducing the number of parameters to be used and making the backpropagation algorithm more efficient, As a result, prediction performance can be improved.
[0044] Pooling Layer (Pooling Layer or Sub-sampling Layer) ) leverages the regional information of the feature maps obtained from the previous convolutional layer to generate new feature maps. In general, the newly generated feature map by the pooling layer is The size of the feature map is reduced to a smaller size, but the most typical integration method is to Max Pooling, which selects the maximum value of the corresponding region, and feature matching Average Pooling, which calculates the average value of the corresponding area in a group. The feature maps of a pooling layer are generally closer to the input than the feature maps of the previous layer. It is less affected by the position of any structures or patterns present in the image. That is, the pooling layer removes noise and distortions in the input image or previous feature maps. It is now possible to extract stronger features for the regional changes that have been observed, and these features have a higher classification ability. In addition, the role of other pooling layers is to improve the performance of deep structures. The higher the learning layer, the wider the range of characteristics that can be reflected. As the extraction layers accumulate, the lower layers reflect regional characteristics, while the higher layers reflect It is possible to generate features that reflect more abstract overall image characteristics.
[0045] The final features extracted by repeating the convolutional and pooling layers in this way are Multi-Layer Perceptron (MLP) and support Support Vector Machine (SVM) The classification models are connected in the form of a fully-connected layer, It can be used for training and prediction of class models.
[0046] The memory 130 can store the image acquired by the communication unit 110. The image 130 may be images taken from endoscopes inserted into the colons of multiple subjects, or from a single subject. Images taken from an endoscope inserted into the colon of a subject multiple times can be stored.
[0047] Alternatively, the memory 130 may store a memory for storing the deep learning model. As an example of annotators, medical staff will The fold data obtained from the above can be stored.
[0048] In addition to the operations associated with the application programs, the processor 140 typically performs The processor 140 controls the overall operation of the device 10. The signal, data, information, etc. input or output from the memory 130 is processed or stored in the memory 130. By running the application program, the appropriate information or function is provided to the user. Can be provided or processed.
[0049] The processor 140 runs application programs stored in the memory 130. In order to perform the above-described processing, at least some of the components shown in FIG. The processor 140 is included in the device 10 for running the application program. At least two or more of the components can be operated in combination with each other. do.
[0050] The operation of the processor 140 will be described in detail below with reference to FIGS.
[0051] The processor 140 detects depths in an image taken from an endoscope inserted into the colon of a subject. Based on the learning model, each segment in the colon that contains at least one fold is That is, the processor 140 can recognize the at least one of the images in each of the sections. The system can also recognize one fold and identify the size of the fold.
[0052] Referring now to FIG. 2, the processor 140 executes the deep learning model The deep learning model recognizes the image of each section by Fold data in colon images of a plurality of subjects obtained from an annotator and the illumination data in the colon are This is a machine-learned model based on the amount of shadow change caused by the light and the blood vessel pattern. obtain.
[0053] Specifically, the processor 140 collects at least one of the following signals from an endoscope inserted into the colon of a plurality of subjects: and acquiring at least one image taken one or more times, and for each of the plurality of images The annotator can then obtain fold data from the colon. Multiple images can be taken multiple times on a single subject, and the subjects can be identified as experts. Images from endoscopies performed on multiple subjects or images from endoscopies performed on multiple subjects It could be.
[0054] The processor 140 then calculates the shadow variation amount of the inside of the colon, the blood vessels and the fold data. The first model can be machine-learned based on
[0055] Here, the amount of shading change is calculated by the processor 140 detecting the amount of shading change from the illumination 220 of the endoscope 20. The shadows generated by the light irradiated on the folds are changed by a threshold value or more. Specifically, the processor 140 determines whether the shadow is to be generated based on the already set shadow. If the image becomes darker than the threshold value, it can be determined that there is a fold.
[0056] The processor 140 also detects a blood vessel pattern formed by blood vessels in an endoscope image of the large intestine. Based on the recognized blood vessel pattern, a region having folds is detected in an endoscope image of the large intestine. The area can be recognized.
[0057] In detail, the processor 140 may perform a colonoscopy based on the recognized blood vessel pattern. At least one vascular pattern in the image is detected from the illumination 220 of the endoscope 20 to the fold. Whether the connection of the blood vessel pattern is interrupted by the shadows generated by the irradiated light This is the situation seen in the colonoscopic image, so it looks as if the blood vessels are interrupted. The part that appears as a fold is recognized as a boundary line of the fold, and the fold region is detected in the endoscopic image of the large intestine. can be recognized.
[0058] The processor 140 also recognizes the shape of blood vessels in the endoscope image of the large intestine, Based on the shape of the blood vessels, areas with folds can be recognized in endoscopic images of the large intestine.
[0059] In detail, the processor 140 generates an endoscopic image of the large intestine based on the recognized shape of the blood vessel. If the shape of at least one blood vessel in the fold is recognized as a hook, this indicates that the blood vessel is a fold. In the colonoscopic image, it appears as if the bow has passed through a curved area. Therefore, by recognizing the curved hook-like parts of the blood vessels as folds, the large intestine The fold region can be recognized in the endoscopic image.
[0060] The processor 140 also recognizes the color of at least one blood vessel in the endoscopy image of the large intestine. Based on the recognized blood vessel color, a region having folds can be recognized in an endoscope image of the large intestine. Cut.
[0061] In detail, the processor 140 determines whether the change in the color intensity of the recognized blood vessel is greater than the previously set value. When it is recognized that the change is equal to or greater than the amount of change that was determined, it is determined that the blood vessel passes between the fold and the back surface of the fold. Since the intensity changes when passing through the blood vessel, the change in intensity in the blood vessel is already set. A portion of a blood vessel where the change is equal to or greater than the above-mentioned amount is recognized as a boundary line of a fold, and The fold regions can be recognized in the image.
[0062] The processor 140 is provided in the endoscope 20 from the point where the endoscope 20 is inserted. The illumination 220 illuminates the inside of the colon until a point is identified by the light. That is, the processor 140 divides the large intestine into n sections. The adult large intestine is about 150cm to 170cm long. cm, and is identified by the light emitted from the illumination 220 provided in the endoscope 20. The distance to the point may be approximately 10 cm to 15 cm. Therefore, the processor 140 The large intestine is divided into approximately 10 to 15 sections, and the folds can be recognized in each section. do.
[0063] Specifically, the processor 140 determines the point at which the endoscope 20 is inserted and the A point identified by light irradiated into the inside of the colon by the provided lighting 220 can be generated based on the deep learning model.
[0064] As an example, referring to FIG. 3, the processor 140 may include a processor 200 a first point P1 where the endoscope 20 first enters, and a second point P2 where the endoscope 20 first enters the first point P1. The first section is a section up to a second point P2 identified by the light irradiated inside the large intestine. You can set it as follows.
[0065] The processor 140 also detects in the image that the endoscope 20 is at the end point of the first section. When the endoscope 20 is located at the second point P2, A third point P3 is identified by the light irradiated inside the large intestine by the light 220. It can be set as the second section.
[0066] Furthermore, the processor 140 determines whether the endoscope 20 is at the end point of the n-1th section in the image. When the vehicle is located at the n-1-th point Pn-1, the vehicle is located at the n-1-th point Pn-1 and the vehicle is located at the n-1-th point Pn-1. The colon is identified by the light irradiated into the inside of the colon by the illumination 220 of the endoscope 20. The section up to point Pn, which is the start point of the last section that can be entered inside, is set as section n. can.
[0067] Here, a first point P1 is a point where the endoscope 20 first enters the large intestine, and a second point P2 is a point where the endoscope 20 enters the large intestine. The second point P2~, excluding the nth point Pn, which is the starting point of the last section that 20 can enter, The n-1th point Pn-1 may overlap with the previous section. However, the section to be confirmed by the endoscope 20 in the large intestine is displayed through the display unit 120 without omission. By displaying the entire colon, it is possible to examine the entire colon without missing any sections. may have the following structure:
[0068] In addition, the lengths of the first to nth sections are almost the same or slightly different. This makes little difference since the intensity of the light emitted by the illumination 220 of the mirror 20 is the same.
[0069] Referring to section introduction frame A (301) of FIG. 3, the processor 140 The endoscope 20 is illuminated from the illumination 220 of the endoscope 20 at the entry point where the endoscope 20 enters. The last point that can be identified by light can be set as one section.
[0070] More specifically, the processor 140 determines the first entry point of the endoscope 20 in the first section. A first point P1 at which the large intestine is observed by the illumination 220 of the endoscope 20 from the first point P1. The first section can be set up to a second point P2 identified by light irradiated inside.
[0071] Referring to the section intermediate frame 302 of FIG. 3, the processor 140 A first visual effect representing the pleats can be displayed within the image of each of the sections. The visual effect may include a visual effect in which each marker is displayed on the corresponding fold in the front. The size of each marker can be determined based on the size of the corresponding fold.
[0072] Here, the processor 140 performs a small amount of processing in the introductory portion of each image of the section by the method described above. Recognizing at least one fold, recognizing a vascular pattern for each of the at least one fold, The endoscope 20 captures images while passing through the introduction sections of the respective sections, and thereby the recognition is performed. If a recognized vascular pattern is present, it can be determined that there is a fold at the position of the corresponding vascular pattern. Then, the processor 140 displays a first visual effect for the determined pleat. This makes it easier for specialists to check the folds.
[0073] More specifically, the processor 140 applies a first visual effect representing the pleats to the first section. Here, the endoscope 20 can display the following in the first section according to the operation of the specialist. According to the first visual effect, the patient enters the large intestine while checking the folds in the large intestine one by one. Can be moved.
[0074] As an example, referring to FIG. 4a, the processor 140 may The first visual effect can be displayed by an arrow-shaped marker for each size of the fold. The arrow-shaped size of the first pleat 401, which is larger, is smaller than the arrow-shaped size of the second pleat 402, which is smaller. It may be smaller than the size of the arrow.
[0075] Next, referring to FIG. 4b, the processor 140 calculates the size of each of the pleats in the first section. The first visual effect can be displayed by a message-shaped marker according to the size. The size of the message shape for the first pleat 401 having a larger pleat size is It may be smaller than the message-like size for two pleats 402 .
[0076] Next, referring to FIG. 4c, the processor 140 determines the size of each pleat in the first section. By blinking separately, you can display the first visual effect. Therefore, the blinking magnitude of the first pleat 401, which has a larger pleat size, is smaller than that of the first pleat 402. The blinking magnitude of the second pleat 402 may be smaller than the blinking magnitude of the second pleat 402 .
[0077] Then, referring to FIG. 4d, the processor 140 calculates the thickness of each of the pleats in the first section. The first visual effect can be displayed by a circular marker according to the size of the fold. The circular size of the first pleat 401 is larger than that of the second pleat 402, which is smaller in size. It may be smaller than the circular size for .
[0078] Here, the specialist sees the first view displayed by the processor 140 via the display unit 120. Based on the visual effect, at least one fold can be easily recognized in the image of each section, and the endoscope can 20, the underside of each of the folds can be identified and the underside can be photographed.
[0079] Therefore, the processor 140 captures the back surface of the pleats for each image of the section. You can determine whether or not it is there.
[0080] Then, referring to section introduction frame B (303) in FIG. 3, processor 140 As a result of the above judgment, at least one of the images of each section in which the back surface of the fold is not photographed is selected. A second visual effect indicating that the underside of the fold is not captured in any of the images of the other sections. The second visual effect can be displayed on the back surface of the corresponding pleat in the image of each section. , a visual effect in which each marker is displayed. The size can be determined based on the size of the back surface of the corresponding pleat.
[0081] Here, the endoscope 20 detects the inside of the large intestine by the second visual effect in the first section. Specifically, the endoscope 20 can move into the large intestine while checking the tubes one by one. After recognizing the back surface of each of the pleats for each region, a start point of the region is identified for each region. After returning to the starting point, for each of the regions, at least one hi that has not been photographed in the back surface portion is photographed. The second visual effect allows you to check all of the Da.
[0082] First, referring to FIG. 5a, the processor 140 photographs the back surface of the pleats in the first section. The second visual effect is displayed by arrow-shaped markers for each size of the unshaded folds. Therefore, the size of the arrow for the first pleat 501, which is the larger pleat, is , the size of the pleat is smaller than the size of the arrow for the second pleat 502, Ugh.
[0083] Next, referring to FIG. 5b, the processor 140 captures the back surface of the pleats in the first section. The second visual field is displayed by message-shaped markers for each size of the unshaded folds. Therefore, the message about the first pleat 501 having a large pleat size can be displayed. The size of the message shape is larger than the size of the message shape for the second pleat 502 having a smaller pleat size. It is possible that the difference is smaller than the
[0084] Then, referring to FIG. 5c, the processor 140 processes the back surface of the pleats in the first section. Blinking ) to display the second visual effect. The blinking magnitude for the first pleat 502 is smaller than that for the second pleat 502. It may be smaller than the blinking magnitude.
[0085] Then, referring to FIG. 5d, the processor 140 processes the back surface of the pleats of the first section. The second visual effect is created by using circular markers for each size of the corresponding folds that are not photographed. Therefore, the size of the circle for the first pleat 501, which has a large pleat size, can be displayed as Therefore, the size of the pleat may be smaller than the circular size for the second pleat 502. do.
[0086] The processor 140 then determines whether the corresponding fold is to be visualized by the endoscope 20 based on the second visual effect. When the back side of the camera is photographed, the second visual effect can be deleted. 40 is a first comparison of the folds that the specialist confirmed using the endoscope 20 and the folds that he did not confirm. By providing a 1st or 2nd visual effect, the folds that pass without checking are eliminated. This allows all folds to be seen without any need for a microscope, improving the accuracy of colon examinations. .
[0087] FIG. 6 shows a process in which the processor 140 of the device 10 according to the present invention guides a colon examination using an endoscope. 1 is a flowchart showing a process of performing the above-mentioned operations of the processor 140 in the device 20. It is feasible.
[0088] First, the processor 140 receives the subject's In an image taken from an endoscope inserted into the large intestine, at least one fold in the large intestine is identified. The image of each section containing the above can be recognized (S601).
[0089] Here, the processor 140 detects the amount of light emitted by the endoscope to the inside of the large intestine. The processor 140 can recognize the folds based on the deep learning model. The deep learning model can recognize the image of each section by Fold data in colon images of a plurality of subjects obtained from the annotator, and This model is machine-learned based on the amount of shadow change caused by the irradiated light and the blood vessel pattern. It is possible.
[0090] The processor 140 can display a first visual effect representing the fold within the image of each of the sections. (S602).
[0091] The first visual effect is that a marker is displayed on the corresponding fold in the image of each section. The size of each of the markers is determined based on the size of the corresponding fold. It can be determined.
[0092] The processor 140 determines whether the back surface of the fold is captured for each image of the section. It can be determined whether or not the
[0093] The processor 140 determines, as a result of the determination, that the back surface of the fold is The back surface of the relevant fold is not photographed in at least one unphotographed section. A second visual effect may be displayed to indicate that the
[0094] The processor 140 may be configured to remove the second visual effect when the back surface of the corresponding fold is photographed. This can be removed (S605).
[0095] When the back surface of the corresponding fold is photographed, the processor 140 The second visual effect was removed, and the specialist confirmed that the imaging of the back side of the fold was completed. It can be provided in an easily understandable manner.
[0096] FIG. 6 shows that steps S601 to S605 are executed in sequence. The above is merely an illustrative description of the technical ideas of the embodiments, and the present invention is not limited to the above. A person having ordinary skill in the art would be able to make the above-mentioned modifications without departing from the essential characteristics of this embodiment. 6. The procedure shown in FIG. 6 may be modified to a lesser extent, or steps S601 to S605 may be performed. Any one of these steps can be applied in parallel by various modifications and variations. Therefore, FIG. 6 is not limited to a chronological order.
[0097] The above-mentioned method according to the present invention is executed in combination with a server, which is hardware. The program (or application) can be stored on a medium. .
[0098] The above-mentioned program is implemented by the computer by reading the program. In order to execute the method, the processor (CPU) of the computer C, C++, JAVA, read through the computer's device interface, It can contain code coded in a computer language such as machine code. Such code may include associated functions that define the functionality required to perform the method. The functional code may include a function that is The execution procedure related constraints necessary for the processor of the computer to execute the specified procedure Such code may include control code for implementing the functions on the computer. Additional information or media necessary for the processor to execute the program may be included within said computer or This is a memory reference code that indicates which location (address) in external memory should be referenced. Further, the processor of the computer may further include a program for executing the functions. In order to do this, communication with any other remote computers or servers is required. If necessary, the code can communicate with any other computer remotely using the communication module of said computer. How should it communicate with computers and servers? What information or media should it use? It may further include communication related code for whether to send or receive, etc.
[0099] The storage medium is a register, a cache, a memory, etc. It is not a medium for storing data, but a medium for storing data semi-permanently and reading it by equipment. g) A medium capable of storing data. Specifically, examples of the medium on which data is stored include ROM, AM, CD-ROM, magnetic tape, floppy disk, optical data storage device, etc. However, the present invention is not limited to this. That is, the program may be used to communicate with various servers to which the computer can be connected. The information may be stored on various recording media on the server or on various recording media on the user's computer. The medium is distributed among computer systems connected by a network, and the distribution method A computer readable code may be stored in the formula.
[0100] The steps of the methods or algorithms described in connection with the embodiments of the present invention may be implemented in hardware. Whether it is embodied directly or as a software module executed by hardware, The software module may be implemented in RAM (Ra ndom Access Memory), ROM(Read Only Memory), EPROM (Erasable Programmable ROM), EEPROM (E lectrically Erasable Programmable ROM), Flash Memory, Hard Disk, Removable Disk, CD-R OM, or any form of computer readable medium known in the art to which the present invention pertains. The information may always be present on a readable recording medium.
[0101] The embodiments of the present invention have been described above with reference to the accompanying drawings. A person skilled in the art would understand that the present invention can be applied to other products without changing the technical idea or essential features of the present invention. It will be understood that the present invention can be embodied in specific forms. The examples should be considered in all respects as illustrative and not restrictive.
Claims
1. 1. A method performed by an apparatus comprising: receiving images taken in real time from the endoscope inserted into the subject's large intestine; 、 Recognizing an image of each section in the image that includes at least one fold in the large intestine. and displaying a first visual effect representing the pleats within the image of each of the sections; determining whether or not the back surface of the fold is captured for each image of the section; As a result of the judgment, among the images of each section, there are a few images in which the back surface of the fold is not photographed. A second visual field indicating that the back surface of the corresponding pleat is not photographed in at least one image of the corresponding pleat. displaying the effect; A method for guiding colon examination using an endoscope, comprising:
2. The recognition step comprises: The method is characterized in that the method is performed based on light irradiated to the inside of the large intestine by the endoscope. A method for guiding colon examination using an endoscope according to claim 1.
3. The recognizing step recognizes the image of each section using a deep learning model; The deep learning model uses external annotators to generate multiple subject images. The fold data in the colon image and the shadow change amount and blood pressure change amount in the colon due to the irradiated light are The method for using an endoscope according to claim 1, characterized in that the model is a machine-learned model based on a tube. A colon examination guide method.
4. The first visual effect is to display a marker on the corresponding fold in the image of each section. Includes the visual effects displayed, The size of each marker is determined based on the size of the corresponding fold. The method for guiding colon examination using an endoscope according to claim 1 .
5. The method further includes removing the second visual effect when the back side of the corresponding fold is photographed.
2. The method for guiding a large intestine examination using an endoscope according to claim 1.
Citation Information
Patent Citations
Image processor, method, and program
JP2012024509A
Biological navigation device
US20080183038A1
Endoscope system, reporting method, and program
WO2019087969A1
Image processing apparatus, image processing method, program, and endoscope system
WO2019088008A1
Information processing device, information processing method, and information processing system
WO2020174862A1
