Endoscope-based colon examination guide device and method

Abstract

To enable multiple wrinkles in a large intestine of a subject to be checked through an endoscope without missing each of them.SOLUTION: A method of recognizing wrinkles in a large intestine using the visual effect based on a deep learning model performed by a device includes the steps of: recognizing a blood vessel pattern in an image obtained by imaging the large intestine through a processor of the device; recognizing an area where there are wrinkles in the image on the basis of the recognized blood vessel pattern; and displaying the visual effect on rear surfaces of the wrinkles that are not imaged in the image when it is determined that the rear surfaces of the recognized wrinkles are not imaged through the processor of the device.SELECTED DRAWING: Figure 5a

Description

[Technical Field]

[0001] The present invention relates to a colon examination guide, and more particularly to a colon examination guide device using an endoscope and and methods thereof. [Background technology]

[0002] In general, colonoscopy is used to detect small raised lesions on the inner wall of the colon, which are precursors of colon cancer. By detecting and removing polyps, which are parts of the colon, colon cancer can be eradicated. Specifically, it is used to detect and remove polyps through a colonoscope. This can prevent the occurrence of interval cancer. Colonoscopy fails to detect polyps in nearly 30% of cases. It is known.

[0003] This is because there are many folds in the large intestine, and colon polyps hidden behind the folds can be found. In reality, the only way to assess this is to rely on the concentration of the doctor who performed the test. Currently, the only method available is an indirect method, such as whether the test took more than six minutes. Even doctors who observe closely will find that different people have different ways of finding treasure when they are treasure hunting. Similarly, depending on the person's examination style, they may repeatedly check what they have seen, but may also check what they have not seen. There's a good chance Roha won't watch it.

[0004] Therefore, when performing a colonoscopy, it is necessary to observe each of the multiple folds in the examinee's colon one by one. A method is needed that allows for confirmation through an endoscope without missing anything. Summary of the Invention [Problem to be solved by the invention]

[0005] The present invention has been made in view of the above circumstances, and its object is to provide a method for performing colonoscopy. ,recognizes colonic folds in colonoscopy images and provides them to specialists performing colonoscopy The reason is that.

[0006] Specifically, the object of the present invention is to provide a method for identifying the folds of the large intestine by size, which are recognized from colonoscopic images. The purpose of the present invention is to display the visual effect with different magnitudes.

[0007] Another object of the present invention is to provide a method for detecting the posterior surface of the folds of the large intestine by an endoscope. If not, provide a visual effect for the folds and have an expert inspect the folds. To eliminate the visual effect of the folds when the surface is photographed using an endoscope.

[0008] The problems to be solved by the present invention are not limited to the problems mentioned above, but may include other problems not mentioned above. Other problems will be clearly understood by those skilled in the art from the following description. [Means for solving the problem]

[0009] In a method performed by the device according to the present invention for solving the above-mentioned problems, receiving in real time an image taken from an endoscope inserted into the large intestine; identifying an image of each section of the large intestine that includes at least one fold within the image; and displaying a first visual effect representing the pleats in the image of each of the sections. A step of determining whether or not the back surface of the pleats is photographed for each image, and as a result of the determination, Among the images of the respective sections, at least one section in which the back surface of the pleats is not photographed displaying a second visual effect in the image indicating that the backside of the corresponding fold is not photographed; and

[0010] Here, the recognition step is based on light irradiated into the inside of the large intestine by the endoscope. This can be done.

[0011] The recognizing step recognizes the image of each section using a deep learning model; The deep learning model uses colon images of multiple subjects obtained from external annotators. Fold data in the image, the amount of change in shadow caused by the irradiated light in the large intestine, and blood vessels It can be a machine-learned model based on

[0012] Here, the first visual effect is to display a corresponding fold in the image of each section. The visual effect includes markers being displayed, and the size of each marker is determined by the size of the corresponding pleat. The determination may be based on the size of the

[0013] In addition, when the back side of the corresponding pleat is photographed, the step of deleting the second visual effect is further performed. can be included in.

[0014] Other methods, systems, and methods for implementing the present invention are also provided. A computer-readable recording medium for recording a computer program for It can be provided. [Effects of the Invention]

[0015] According to the present invention as described above, when performing colonoscopy, it is possible to identify the large intestine from the image of the colonoscope. By recognizing the folds and providing them to the specialist performing the colonoscopy, the posterior This has the effect of making it possible to easily detect polyps that may be located in the area.

[0016] Specifically, according to the present invention, the folds of the large intestine are classified by size as recognized from a colonoscope image. By displaying different sizes of visual effects, when the size of the folds is large, a part of the folds is This is effective in guiding users to check the whole thing, rather than just checking it and then passing. There is fruit.

[0017] Furthermore, according to the present invention, the rear surface of the folds of the large intestine is not photographed by an endoscope. In this case, a visual effect is provided for the folds, and an expert examines the back of the folds. When photographed using an endoscope, the visual effect of the folds can be removed to reveal the folds. The effect is to clearly distinguish between the checked and unchecked parts of the back of the There is fruit.

[0018] The effects of the present invention are not limited to those mentioned above, and further effects not mentioned are as follows: The above description should be clearly understandable to a person skilled in the art. [Brief explanation of the drawings]

[0019] [Figure 1] 1 is a block diagram illustrating a schematic diagram of an apparatus for providing a guided endoscopic colon examination in accordance with the present invention; [Figure 2] 1 is a block diagram that shows in schematic form how a deep learning model used to guide colon examinations is trained, in accordance with the present invention; FIG. [Figure 3] 10 is an exemplary diagram showing a process in which a processor of the device according to the present invention provides guidance for colon examination using an endoscope. FIG. [Figure 4a] FIG. 1 is an illustrative view (1) showing a first visual effect representing folds displayed on an image captured from an endoscope inserted into the colon of a subject according to the present invention. [Figure 4b]FIG. 2 is an illustrative view (2) showing a first visual effect representing folds displayed on an image captured from an endoscope inserted into the colon of a subject according to the present invention. [Figure 4c] FIG. 3 is an illustrative view (3) showing a first visual effect representing folds displayed on an image captured from an endoscope inserted into the colon of a subject according to the present invention. [Figure 4d] FIG. 4 is an illustrative view (4) showing a first visual effect representing folds displayed on an image captured from an endoscope inserted into the colon of a subject according to the present invention. [Figure 5a] FIG. 1 is an illustrative diagram (1) showing a second visual effect being displayed, representing a pleat in which an image of the backside of the pleat in which the first visual effect is displayed is not captured, according to the present invention. [Figure 5b] FIG. 2 is an illustrative diagram (2) showing a second visual effect being displayed, representing a pleat in which an image of the backside of the pleat in which the first visual effect is displayed is not captured, according to the present invention. [Figure 5c] FIG. 3 is an illustrative diagram (3) showing a second visual effect being displayed, representing a pleat in which an image of the backside of the pleat in which the first visual effect is displayed is not captured, according to the present invention. [Figure 5d] FIG. 4 is an illustrative diagram (4) showing a second visual effect being displayed, representing a pleat in which an image of the backside of the pleat in which the first visual effect is displayed is not captured, according to the present invention. [Figure 6] 1 is a flowchart showing a process in which a processor of the device according to the present invention guides a colon examination using an endoscope. DETAILED DESCRIPTION OF THE INVENTION

[0020] The advantages and features of the present invention and the manner in which they are achieved are described in detail below in conjunction with the accompanying drawings. However, the present invention will be elucidated with reference to the embodiments disclosed below. However, the present invention is not limited to the above and may be embodied in various different forms. The examples will be provided so that the disclosure of the present invention will be complete and understandable to those of ordinary skill in the art to which the present invention pertains. The present invention is not limited to the above-described embodiments, and is not limited to the embodiments set forth in the claims. It is only defined by

[0021] The terms used in this specification are intended to describe the embodiments and are not intended to limit the present invention. In this specification, the singular includes the plural unless specifically stated otherwise. As used in the specification, "comprises" and / or "comprises" "rising" refers to the presence or addition of one or more other elements in addition to the element being mentioned. The same reference numerals refer to the same elements throughout the specification, and the like are used interchangeably with "and / or "includes each and every combination of one or more of the listed elements. Although "first," "second," etc. are used to describe various components, Of course, the components are not limited by these terms. , are used merely to distinguish one component from another. It goes without saying that the first component may also be the second component within the technical concept of the present invention. stomach.

[0022] Unless otherwise defined, all terms (including technical and scientific terms) used herein are used in a way that can be commonly understood by those skilled in the art to which the present invention pertains. In addition, terms defined in commonly used dictionaries are not expressly defined. Unless otherwise stated, it should not be interpreted ideally or excessively.

[0023] Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings.

[0024] FIG. 1 shows a schematic diagram of an apparatus 10 for providing endoscopic colonic guidance in accordance with the present invention. FIG.

[0025] FIG. 2 shows a deep learning model used for colon examination guidance according to the present invention. FIG. 1 is a block diagram illustrating the learning process.

[0026] FIG. 3 shows a processor 140 of the device 10 according to the present invention for endoscopic colonic examination guidance. 10 is an exemplary diagram illustrating a process of providing the

[0027] 4a to 4d are images taken from an endoscope inserted into the colon of a subject according to the present invention. FIG. 10 is an illustrative diagram showing a first visual effect representing folds being displayed on an image.

[0028] 5a to 5d show images of the backside of the pleats according to the present invention, which show the first visual effect. FIG. 10 is an exemplary diagram showing a second visual effect being displayed, representing unphotographed folds.

[0029] Hereinafter, a colon examination guide using an endoscope according to the present invention will be provided with reference to FIGS. 1 to 5d. A device 10 will now be described. Here, device 10 is a local computing device. It can also be implemented in a server device.

[0030] The device 10 recognizes the folds of the large intestine from an image of the large intestine during colonoscopy, and By providing endoscopy specialists with the necessary tools, polyps that may be located behind the folds of the colon can be easily identified. This has the effect of making it easier to find.

[0031] Specifically, the device 10 visually analyzes the folds of the large intestine according to the size of the folds recognized from the colonoscope image. By displaying different sizes of the visual effect, when the size of the folds is large, it is possible to The effect is to guide the user to check the whole picture, rather than just checking a few things and then passing. can have:

[0032] The device 10 also captures images of the backside of the folds of the large intestine using the endoscope 20. If the folds are not visible, a visual effect is provided for the folds. When the back surface is photographed using the endoscope 20, the visual effect of the folds can be eliminated. This allows you to clearly distinguish between areas where the backside of the pleats has been checked and areas where it has not been checked. Here, the endoscope 20 is inserted into the large intestine. The device for observing biological tissue in the large intestine includes a camera 210, a light 220, etc. It can be done.

[0033] First, referring to FIG. 1, the device 10 includes a communication unit 110, a display unit 120, a memory 111, and a 30 and processor 140. Here, device 10 includes more components than those shown in FIG. It may contain fewer or more components.

[0034] The communication unit 110 communicates between the device 10 and a wireless communication system, and between the device 10 and an endoscope 20. one or more devices that enable wireless communication between the device 10 and an external device (not shown) The communication unit 110 may include the above modules. It may include one or more modules that connect to a network.

[0035] The communication unit 110 transmits in real time images captured by an endoscope inserted into the large intestine of a subject. The communication unit 110 can receive signals from endoscopes inserted into the large intestines of multiple subjects. The images are received from the endoscope, which is inserted into the colon multiple times for one patient. You can receive the captured images.

[0036] Alternatively, the communication unit 110 may communicate with the endoscope for learning the deep learning model. As an example of annotators, the images of the examinee's colon were annotated using annotations obtained from medical staff. You can receive the data.

[0037] The display unit 120 and the touch sensor may be layered together or may be integrally formed. By using the touch screen, a touch panel can be realized. 10 and a user, and provides an input interface between the device 10 and a user. It can provide a force interface.

[0038] The display unit 120 displays various information generated by the processor 140 to the user. It is also possible to provide various information and receive input of various information from the user.

[0039] More specifically, the display unit 120 displays a first visual effect for each section of the image to represent the folds. The display unit 120 can also display the results of the folds in the images of each section. The back side of the fold is photographed in at least one section where the back side is not photographed. A second visual effect can be displayed to indicate that the device is not connected.

[0040] Memory 130 may store information that supports various functions of device 10. 0 represents a number of application programs (applications) that are run on the device 10. ion program or application), the device It can store data and commands for up to 10 operations. These application programs At least some of the above are downloaded from an external server (not shown) via wireless communication. At least some of such application programs may be The application programs may exist for the basic functionality of the device 10. stored in memory 130, installed on the device 10, and pre-programmed by processor 140. The device 10 can be driven to perform the operations (or functions) described above.

[0041] The memory 130 stores a fold-detecting image taken by an endoscope inserted into the large intestine of a subject. Here, you can store deep learning models to recognize The model is a convolutional neural network (CNN). nal network (CNN), but this does not necessarily mean that It can be formed by neural networks of various structures, without being limited to the above.

[0042] CNN applies multiple filters to each region of an image to generate a feature map. Convolution Layer that generates feature map (e Map) and By spatially integrating the two groups, we can extract features that are invariant to changes in position and rotation. The pooling layer is formed by repeating the layer multiple times. This allows us to extract low-level features such as points, lines, and surfaces into complex, meaningful levels. It is possible to extract features at various levels, from high-level features to low-level features.

[0043] The convolutional layer applies a filter and a local receptive field (Local R) to each patch in the input image. The inner product of the nonlinear activation function (Activation F Feature Map is obtained by taking Compared to other network structures, CNNs have the advantage of sparse connectivity (SPARS). e Connectivity and Shared Weights Such a connection structure can be characterized by using a filter with By reducing the number of parameters to be used and making learning by the backpropagation algorithm more efficient, As a result, prediction performance can be improved.

[0044] Pooling Layer (or Sub-sampling Layer) ) utilizes the regional information of the feature maps obtained from the previous convolutional layer to generate new feature maps. In general, the newly generated feature map by the pooling layer is Although the size is reduced to a smaller size than the feature map, a typical integration method is to Max Pooling, which selects the maximum value of the corresponding region, and feature matching Average pooling, which calculates the average value of the corresponding region within a group. The feature maps of the pooling layer are generally closer to the input than the feature maps of the previous layer. The effect of the position of any structure or pattern present in the image can be reduced. That is, the pooling layer removes noise and distortions in the input image or previous feature maps. This allows us to extract stronger features for regional variations, and these features are more classifiable. In addition, the role of other pooling layers is to improve the performance of the deep structure. The higher the learning level, the wider the range of features that can be reflected. As the extraction layers accumulate, the lower layers reflect regional characteristics, while the higher layers reflect It is possible to generate features that reflect more abstract overall image characteristics.

[0045] The final features extracted by repeating the convolutional layer and pooling layer in this way are Multi-Layer Perceptron (MLP) and Support Vector Machine (SVM) The classification model is connected in the form of a fully connected layer, and It can be used for training and prediction of class models.

[0046] The memory 130 can store the image acquired by the communication unit 110. The image 130 may be images taken from endoscopes inserted into the colons of multiple subjects, or images taken from a single subject. Images taken from an endoscope inserted into the colon of a subject multiple times can be stored.

[0047] Alternatively, the memory 130 may store an endoscope-based image for training the deep learning model. As an example of annotators, medical staff will make annotations on the images of the examinee's colon taken by the The resulting fold data can be stored.

[0048] In addition to operations associated with the application programs, processor 140 typically The processor 140 controls the overall operation of the device 10. The signal, data, information, etc. input or output from the memory 130 is processed or stored in the memory 130. By running the application program, the appropriate information or function is provided to the user. Can be provided or processed.

[0049] The processor 140 runs an application program stored in the memory 130. In order to do so, at least some of the components shown in FIG. The processor 140 is included in the device 10 for running the application program. At least two or more of the components can be operated in combination with each other. do.

[0050] The operation of the processor 140 will be described in detail below with reference to FIGS.

[0051] The processor 140 detects defects in an image taken from an endoscope inserted into the colon of a subject. Based on a learning model, each segment in the colon that contains at least one fold is That is, the processor 140 can recognize the at least one image from the image of each section. The system can also recognize one fold and identify the size of the fold.

[0052] Referring now first to FIG. 2, the processor 140 executes the deep learning model The deep learning model recognizes the image of each section by The fold data in the colon images of a plurality of subjects obtained from the annotator and the reference data in the colon are This is a machine-learned model based on the amount of shadow change caused by the light and the blood vessel pattern. obtain.

[0053] Specifically, the processor 140 collects at least one image from an endoscope inserted into the colon of a plurality of subjects. and acquiring at least one image taken one or more times, and for each of the plurality of images The annotator then accurately identifies the folds of the large intestine. Multiple images can be taken multiple times on a single subject. images from endoscopies performed on multiple subjects or endoscopies performed on multiple subjects It could be.

[0054] The processor 140 then calculates the shadow variation amount, blood vessel and fold data of the inside of the colon. The first model can be machine-learned based on the above.

[0055] Here, the amount of change in shading is determined by the processor 140 detecting the amount of change in shading from the illumination 220 of the endoscope 20. The shadows generated by the light irradiated on the folds are changed by a predetermined threshold or more. Specifically, the processor 140 determines whether the shadow is generated based on the already set shadow. If the image darkens to a level equal to or greater than the threshold value, it can be determined that there are folds.

[0056] The processor 140 also detects a blood vessel pattern formed by blood vessels in an endoscope image of the large intestine. and based on the recognized blood vessel pattern, detecting a region having folds in an endoscope image of the large intestine. The area can be recognized.

[0057] In detail, the processor 140 may perform a colonoscopy based on the recognized blood vessel pattern. At least one vascular pattern in the image is detected from the illumination 220 of the endoscope 20 to the folds. Whether the connection of the blood vessel pattern is interrupted by the shadows generated by the irradiated light This is the situation seen in the colonoscopy image, so it looks as if the blood vessels are interrupted. The part that appears as a fold is recognized as a boundary line of the fold, and the fold region is detected in the endoscopic image of the large intestine. can be recognized.

[0058] The processor 140 also recognizes the shape of blood vessels in the endoscope image of the large intestine, and Based on the shape of the blood vessels, areas with folds can be recognized in endoscopic images of the large intestine.

[0059] In detail, the processor 140 generates an endoscopic image of the large intestine based on the recognized shape of the blood vessel. If the shape of at least one blood vessel in the fold is recognized as a hook, this means that the blood vessel is a fold of the fold. In the endoscopic image of the colon, it appears as if it is bending as it passes through the curved part. Therefore, by recognizing the curved hook-shaped parts of the blood vessels as folds, the The fold region can be recognized in the endoscopic image.

[0060] The processor 140 also recognizes the color of at least one blood vessel in the endoscopic image of the large intestine. Then, based on the color of the recognized blood vessels, it is possible to recognize an area with folds in an endoscopic image of the large intestine. Cut.

[0061] In detail, the processor 140 determines whether the change in the color intensity of the recognized blood vessels is in accordance with the previously set value. If it is recognized that the change is equal to or greater than the amount of change that was determined, it is determined that the blood vessel is passing between the fold and the back surface of the fold. The intensity changes when the blood vessel passes through, so the change in intensity in the blood vessel is A portion of a blood vessel where the change is equal to or greater than the above amount is recognized as a boundary line of a fold, and the endoscope of the large intestine The fold region can be recognized in the image.

[0062] The processor 140 is provided in the endoscope 20 from the point where the endoscope 20 is inserted. The illumination 220 illuminates the inside of the colon to a point identified by the light. That is, the processor 140 divides the large intestine into n sections. The adult large intestine is about 150cm to 170cm long. cm, and is identified by the light emitted from the illumination 220 provided in the endoscope 20. The distance to the point may be approximately 10 cm to 15 cm. The large intestine is divided into approximately 10 to 15 sections, and the folds can be recognized in each section. do.

[0063] Specifically, the processor 140 determines the point at which the endoscope 20 is inserted and the A point identified by light irradiated into the interior of the colon by the provided illumination 220 can be generated based on the deep learning model.

[0064] As an example, referring to FIG. 3, the processor 140 may include a a first point P1 where the endoscope 20 first enters, and a second point P2 where the endoscope 20 first enters the first point P1 where the endoscope 20 first enters the first point P2. The first section is the section up to the second point P2 identified by the light irradiated inside the large intestine. You can set it as follows.

[0065] The processor 140 also detects in the image that the endoscope 20 is at the end point of the first section. When the endoscope 20 is located at the second point P2, the second point P2 and the The third point P3 is identified by the light irradiated into the inside of the large intestine by the light 220. It can be set as the second section.

[0066] Furthermore, the processor 140 determines whether the endoscope 20 is at the end point of the n-1th section in the image. When the vehicle is located at the n-1th point Pn-1, the vehicle is located at the n-1th point Pn-1 and the vehicle is located at the n-1th point Pn-1. The colon is identified by the light irradiated into the inside of the colon by the illumination 220 of the endoscope 20. The section up to point Pn, which is the start point of the last section that can be entered inside, is set as the nth section. can.

[0067] Here, a first point P1 is the initial point where the endoscope 20 enters the large intestine, and a second point P2 is the point where the endoscope 20 enters the large intestine. The second point P2~, excluding the nth point Pn, which is the starting point of the last section that 20 can enter, The n-1th point Pn-1 may overlap with the previous section. However, the section in the large intestine that needs to be checked by the endoscope 20 can be displayed without omission through the display unit 120. By displaying the entire colon, it is possible to examine the entire colon without missing any sections. can have:

[0068] The lengths of the first to nth sections are almost the same or slightly different. There is not much difference since the intensity of the light emitted by the illumination 220 of the mirror 20 is the same.

[0069] Referring to section introduction frame A (301) of FIG. 3, processor 140 The endoscope 20 is illuminated from the entrance point where the endoscope 20 enters. The last point that can be identified by light can be set as one section.

[0070] More specifically, the processor 140 determines the first section where the endoscope 20 first enters. a first point P1 at which the large intestine is observed by the illumination 220 of the endoscope 20 from the first point P1; The first section can be set up to a second point P2 that is identified by the light irradiated inside.

[0071] Referring to the section intermediate frame 302 of FIG. 3, the processor 140 A first visual effect representing the pleats can be displayed within the image of each section. The visual effect of each marker being displayed on the corresponding fold in the front The size of each marker can be determined based on the size of the corresponding fold.

[0072] Here, the processor 140 performs a small amount of processing in the introductory portion of each section of the image by the method described above. and recognizing at least one fold, recognizing a blood vessel pattern for each of the at least one fold, and The endoscope 20 captures images while passing through the introduction sections of the respective sections, and thereby the recognition is performed. If there is a recognized vascular pattern, it can be determined that there is a fold at the position of the corresponding vascular pattern. The processor 140 then displays a first visual effect for the determined pleats. This makes it easier for specialists to check the folds.

[0073] More specifically, the processor 140 applies a first visual effect representing the pleats to the first section. Here, the endoscope 20 can be displayed in the first section by the operation of the specialist. According to the first visual effect, the user can see the folds in the large intestine one by one and enter the inside of the large intestine. It can be moved.

[0074] As an example, referring to FIG. 4a, the processor 140 may For each size of the folds, a first visual effect can be displayed using an arrow-shaped marker. The arrow-shaped size of the first pleat 401, which is large, is smaller than the arrow-shaped size of the second pleat 402, which is small. It may be smaller than the arrow-shaped size.

[0075] Next, referring to FIG. 4b, the processor 140 calculates the size of each pleat in the first section. The first visual effect can be displayed by a message-like marker according to the size. The message size for the first pleat 401 having a larger pleat size is It may be smaller than the message-like size for two pleats 402.

[0076] Next, referring to FIG. 4c, the processor 140 calculates the size of each pleat in the first section. Bliking separately to show the first visual effect Therefore, the blinking magnitude of the first pleat 401, which has a large pleat size, is The blinking magnitude of the second pleats 402 may be smaller than the blinking magnitude of the second pleats 402 .

[0077] Next, referring to FIG. 4d, processor 140 calculates the thickness of each pleat in the first section. The first visual effect can be displayed by a circular marker according to the size of the fold. The circular size of the large first pleat 401 is smaller than that of the small second pleat 402. It may be smaller than the circular size for

[0078] Here, the specialist may view the first image displayed by the processor 140 via the display unit 120. Based on the visual effect, at least one fold can be easily recognized in the image of each section, and the endoscope 20, the back surface of each of the pleats can be confirmed and photographed.

[0079] Therefore, the processor 140 captures the back surface of the pleats for each image of the respective sections. You can determine whether or not it exists.

[0080] Then, referring to section introduction frame B (303) in FIG. 3, processor 140 As a result of the above determination, at least one of the images of each section in which the back surface of the pleats is not photographed is A second visual effect indicates that the underside of the fold is not captured in the image of the other section. The second visual effect can be displayed on the back side of the corresponding pleat in the image of each section. , and a visual effect in which each marker is displayed. The size can be determined based on the size of the back surface of the corresponding pleat.

[0081] Here, the endoscope 20 detects the inside of the large intestine by the second visual effect in the first section. Specifically, the endoscope 20 can move inside the large intestine while checking the After recognizing the back surface of each of the pleats for each region, the opening where the region starts is identified for each region. After returning to the starting point, for each of the areas, at least one image of the back surface is not captured. The second visual effect allows you to check all the Das without missing a thing.

[0082] First, referring to FIG. 5a, the processor 140 photographs the back surface of the pleats in the first section. A second visual effect is provided by arrow-shaped markers for each size of the corresponding unshaded folds. Therefore, the size of the arrow for the first pleat 501, which has a large pleat size, can be displayed as , the size of the pleat is smaller than the size of the arrow for the second pleat 502, which is smaller. Ugh.

[0083] Next, referring to FIG. 5b, the processor 140 photographs the back surface of the pleats in the first section. The message-shaped markers are used to display the second visual field for each size of the unshaded folds. Therefore, the message about the first pleat 501 having a large pleat size can be displayed. The size of the message shape is smaller than the size of the message shape for the second pleat 502 having a smaller pleat size. It is possible that this is smaller than the

[0084] Next, referring to FIG. 5c, the processor 140 processes the first section of the backside of the pleats. Blinking ) can display a second visual effect. The blinking magnitude for the first pleat is smaller than that for the second pleat 502, It may be smaller than the blinking magnitude.

[0085] Next, referring to FIG. 5d, the processor 140 processes the first section of the backside of the pleats. The second visual effect is displayed by circular markers for each size of the corresponding folds that are not photographed. Therefore, the circular size of the first pleat 501, which has a large pleat size, can be displayed as follows. , the pleat size may be smaller than the circular size for the second pleat 502. do.

[0086] The processor 140 then determines whether the corresponding fold is visible by the endoscope 20 based on the second visual effect. When the backside of the camera is photographed, the second visual effect can be removed. 40 is a diagram showing the first comparison of the folds that the specialist confirmed using the endoscope 20 and the folds that he did not confirm. By providing a clear visual effect through the first or second visual effect, the folds that pass without checking can be avoided. This allows all folds to be seen, improving the accuracy of colon examinations. .

[0087] FIG. 6 shows a process in which the processor 140 of the device 10 according to the present invention guides a colonoscopy. 1 is a flowchart showing the process of the processor 140 in the device 20. It is feasible.

[0088] First, the processor 140 receives the subject's In an image taken from an endoscope inserted into the large intestine, at least one fold in the large intestine is The image of each section containing the above can be recognized (S601).

[0089] Here, the processor 140 detects the light emitted by the endoscope into the inside of the large intestine. The processor 140 can recognize the folds based on the deep learning model. The deep learning model can recognize the image of each section by The fold data in the colon images of a plurality of subjects obtained from the annotators and the anterior fold data in the colon This model is machine-learned based on the amount of shadow change caused by the irradiated light and the blood vessel pattern. It is possible.

[0090] The processor 140 can display a first visual effect representing the pleats within the image of each of the sections. (S602).

[0091] The first visual effect is that a marker is displayed on the corresponding fold in the image of each section. The size of each marker is determined based on the size of the corresponding fold. It can be determined.

[0092] The processor 140 determines whether the back surface of the pleats is photographed for each image of the respective sections. It can be determined whether or not the

[0093] As a result of the determination, the processor 140 determines whether the back surface of the pleats is The back surface of the relevant pleat is not photographed in at least one unphotographed section. A second visual effect may be displayed to indicate that the user has made a mistake (S604).

[0094] The processor 140 may remove the second visual effect when the back surface of the corresponding pleat is photographed. This can be removed (S605).

[0095] When the back side of the corresponding fold is photographed, the processor 140 The second visual effect was removed, and the specialist confirmed that the imaging of the back side of the fold was completed. It can be provided in an easily understandable manner.

[0096] FIG. 6 shows that steps S601 to S605 are executed in sequence. The present invention is merely an example of the technical concept of the embodiment. A person skilled in the art would be able to easily implement the invention without departing from the essential characteristics of this embodiment. 6. The procedure described in FIG. 6 may be modified to a small extent, or steps S601 to S605 may be performed. Any one of these steps can be applied in parallel by various modifications and variations. Therefore, FIG. 6 is not limited to a chronological order.

[0097] The method according to the present invention described above is executed in combination with a server, which is hardware. The program (or application) can be stored on a medium. .

[0098] The above-mentioned program is implemented by the computer after being read. In order to execute the method, the processor (CPU) of the computer C, C++, JAVA, readable through a computer's device interface It can contain code coded in a computer language such as machine code. Such code may be associated with functions, etc., that define the functionality required to perform the method. It may contain functional code for implementing said function. The execution procedure related constraints necessary for the processor of the computer to execute the predetermined procedure Such code may include control code for performing the functions on the computer. additional information or media necessary for the processor to execute the program The memory reference code indicates the location (address) of the external memory where the reference should be made. Furthermore, the computer processor may further include a In order to do this, communication with any other remote computers or servers is required. If necessary, the code can be used to communicate with any other computer at a remote location using the computer's communication module. How to communicate with computers and servers, and what information and media to use during communication. It may also include communication related code for whether to send or receive, etc.

[0099] The storage medium is a register, a cache, a memory, etc. It is not a storage medium, but a semi-permanent storage medium that can be read by a device. g) Means a medium capable of storing data. Specifically, examples of the storage medium include ROM, There are AM, CD-ROM, magnetic tape, floppy disk, optical data storage devices, etc. In other words, the program can be used to access various services that the computer can connect to. The information may be stored on various storage media on the server or on the user's computer. The medium may be distributed among computer systems connected via a network, and the distributed method may A computer readable code may be stored in the formula.

[0100] The steps of the methods or algorithms described in connection with the embodiments of the present invention may be implemented in hardware. It may be implemented directly or as a software module executed by hardware. The software module may be implemented in RAM (Ra ndom Access Memory), ROM(Read Only Memory), EPROM (Erasable Programmable ROM), EEPROM (E lectrically Erasable Programmable ROM), Flash Memory, Hard Disk, Removable Disk, CD-R OM, or any form of computer readable medium known in the art to which the present invention pertains. It may always be present on a readable recording medium.

[0101] Although the embodiments of the present invention have been described above with reference to the accompanying drawings, A person skilled in the art would understand that the present invention can be applied to other devices without changing the technical idea or essential features thereof. It will be understood that the present invention may be embodied in various specific forms. The examples are to be considered in all respects as illustrative and not restrictive.

Claims

1. A method for recognizing folds in the large intestine using a deep learning model-based visual effect performed by a device, Recognizing, through a processor of said device, a vascular pattern in an image of the colon; recognizing regions of folds in the image based on the recognized vascular patterns; displaying a visual effect on a fold corresponding to the backside of the recognized fold when it is determined through the processor of the device that the backside of the recognized fold has not been photographed; A method comprising:

2. The step of recognizing a blood vessel pattern includes:

10. The method of claim 1, wherein the method is performed through the processor based on light projected into the interior of the colon by an endoscope.

3. The step of recognizing the vascular pattern includes recognizing the image of each section output from the deep learning model through the processor; The method according to claim 2, characterized in that the deep learning model is a model machine-learned based on fold data in colon images of multiple subjects obtained from an external annotator, and the amount of shading change due to light irradiated inside the colon and blood vessels.

4. the visual effect includes a visual effect in which a respective marker is displayed on the fold in the image of each section; The method of claim 3 , wherein the size of each marker is determined based on the size of the fold.

5. The method of claim 1, further comprising a step of removing the visual effect if it is determined that the back side of the fold has been photographed.

6. A device for recognizing folds in the large intestine using visual effects based on a deep learning model. a display unit that displays an image of the large intestine; a processor for controlling operations associated with displaying the image; The processor: Recognizing a vascular pattern within the image; recognizing a region having folds in the image based on the recognized vascular pattern; The device is characterized in that, when it is determined that the back side of the fold has not been photographed, the display unit is controlled to display a visual effect on the fold corresponding to the back side of the recognized fold.

7. The processor:

7. The device according to claim 6, wherein the imaging is performed based on light irradiated into the interior of the large intestine by an endoscope.

8. The processor: The deep learning model recognizes the images of each section, The deep learning model is The device described in claim 7, characterized in that the model is machine-learned based on fold data in colon images of multiple subjects obtained from an external annotator, and the amount of shading change due to light irradiated inside the colon and blood vessels.

9. the visual effect includes a visual effect in which a marker is displayed on a corresponding fold in the image of each section; The apparatus according to claim 8, wherein the size of each marker is determined based on the size of the corresponding fold.

10. The device described in Claim 6, characterized in that the visual effect is further removed when the back side of the fold is photographed.

Images