Endoscope device, control method for the endoscope device, and computer program

Abstract

To provide an endoscope device, a method of controlling the endoscope device, and a computer program.SOLUTION: A method of controlling the endoscope device includes: the steps of: acquiring an image of the inside of the body from an image sensor; inputting the image into a trained artificial neural network model for classification of the acquired image; the artificial neural network model classifying the image into an insufflation-state image, a water-supply-state image, and / or a suction-state image, and outputting the classification result; and controlling the endoscope device such that an insufflation unit, a water-supply unit and / or a suction unit of the endoscope device is driven according to the output classification result.SELECTED DRAWING: Figure 2

Description

Technical Field

[0001] The present invention relates to a method for controlling an endoscope device using an artificial neural network model, the endoscope device, and a computer program.

Background Art

[0002] An endoscope is a general term for a medical instrument that observes organs by inserting a scope into the body without performing surgery or autopsy (pathological autopsy). The endoscope inserts a scope into the human body, irradiates light, and visualizes the light reflected from the inner wall surface. Depending on the purpose and body part, the types of endoscopes are classified, and roughly speaking, they can be classified into a rigid endoscope in which the endoscope tube is formed of metal and a flexible endoscope typified by a gastrointestinal endoscope.

[0003] Endoscopy specialists perform endoscopy examinations. When entering the esophagus, when the esophagus is stenosed, they must press the air button; when there is a foreign object on the screen, they must press the water button; when gastric juice can be seen on the screen, they must finish the medical treatment and press the suction button when withdrawing from the esophagus. At this time, many button operations are required for a single endoscopy diagnosis, and difficulties are encountered.

Summary of the Invention

Problems to be Solved by the Invention

[0004] The present disclosure is for solving the problems of the aforementioned prior art, and provides a method for controlling an endoscope device, the endoscope device, and a computer program, in which the air supply status, water supply status, and / or suction status of the endoscope are identified without user operation, the identified results are transmitted to the endoscope system, and air supply, water supply, and / or suction are automatically executed.

[0005] However, the technical problems to be solved by the present embodiment are not limited to the technical problems as described above, and other technical problems may exist. [Means for solving the problem]

[0006] A control method for an endoscope device according to an embodiment of the present disclosure, comprising one or more processors, includes the steps of: acquiring an image of the inside of a body from an image sensor; inputting the acquired image to a trained artificial neural network model for classification of the acquired image; the artificial neural network model classifying and outputting the image as an air supply image, a water supply image, and / or an air intake image; and controlling the endoscope device such that the air supply unit, water supply unit, and / or air intake unit of the endoscope device are driven based on the outputted classification results.

[0007] The artificial neural network model includes a plurality of artificial neural network models, and the plurality of artificial neural network models can classify and output at least one situational image from among the air supply situational image, the water supply situational image, and / or the air intake situational image.

[0008] In the step where the artificial neural network model classifies and outputs the images as air supply images, water supply images, and / or inhalation images, the artificial neural network model may be trained to determine an image of an organ being narrowed as the endoscopic device enters the human body as an air supply image.

[0009] In the step where the artificial neural network model classifies and outputs the images as air supply image, water supply image, and / or inhalation image, the artificial neural network model may be trained to determine that an image of a foreign object attached to the lens of the endoscope device is a water supply image, an air supply image, and / or inhalation image.

[0010] In the step where the artificial neural network model classifies and outputs the images as air supply images, water supply images, and / or inhalation images, the artificial neural network model may be trained to determine that an image of the endoscope device withdrawing from an organ after completing medical treatment, or an image in which substances that need to be removed from inside an organ are identified, is an inhalation image.

[0011] The aforementioned artificial neural network model also includes CNNs (convolutional neural networks).

[0012] The artificial neural network model includes a convolution hierarchy and a fully connected hierarchy, the convolution hierarchy extracts features of internal body images input via a convolution operation, and the fully connected hierarchy can output a classification result relating to which of the following situational images—the air supply situational image, the water supply situational image, and / or the inhalation situational image—the extracted image features ultimately correspond to.

[0013] The step in which the artificial neural network model classifies and outputs the images into air supply status images, water supply status images, and / or air intake status images also further includes a step of applying critical values ​​to the images classified by the artificial neural network model.

[0014] The step in which the artificial neural network model classifies and outputs the images as air supply image, water supply image, and / or inhalation image further includes a step of confirming the degree of agreement between the classified images and the classification results of the images before and after the classification.

[0015] The step in which the artificial neural network model classifies and outputs the images as air supply image, water supply image, and / or inhalation image further includes labeling each air supply image, each water supply image, and / or inhalation image, in which case the labeled label is also a multi-label in which the image has additional labeling related to information about internal body parts.

[0016] An endoscope according to one embodiment of the present disclosure includes a memory for storing internal body images taken from the endoscope, and a processor that inputs the internal body images to a learned artificial neural network model, classifies and outputs the images as air supply images, water supply images, and / or suction images, and controls the endoscope so that the air supply unit, water supply unit, and / or suction unit are driven according to the output classification results.

[0017] The aforementioned processor can acquire images of the inside of the body from an image sensor via an image acquisition unit.

[0018] The processor transmits a control signal from the control unit to the drive unit, and the drive unit may, based on the control signal, open or close a valve connected to the air supply pump of the air supply unit, open or close a valve connected to the water supply pump of the water supply unit, or open or close a valve connected to the air intake pump of the air intake unit.

[0019] The processor may perform post-processing on the images output, which are classified into the air supply status image, the water supply status image, and / or the intake status image, via a post-processing unit.

[0020] The post-processing unit also includes a critical value comparison unit that applies critical values ​​to the classified image, and a degree of agreement confirmation unit that confirms the degree of agreement between the classified image and the classification results of the images before and after the classified image.

[0021] According to an aspect of the present invention, a computer program stored in a recording medium is provided for causing a computer to execute the above-described method.

Advantages of the Invention

[0022] According to an embodiment of the present disclosure, during an endoscopic procedure, when the esophagus is stenosed, when a foreign object appears on the screen, when gastric juice is visible on the screen, and when the endoscopic device is withdrawn from the esophagus after completing a medical treatment via the endoscopic device, without the operation of a doctor, through an image classification process of an artificial neural network model via an image acquired through an image sensor, the air supply situation, the water supply situation, and the air intake situation are automatically classified, and based on the classification result, the driving of the air supply unit, the water supply unit, and the air intake unit can be automatically performed.

[0023] The effects of the present invention are not limited to the effects mentioned above.

Brief Description of the Drawings

[0024] [Figure 1] It is a diagram showing an endoscopic device according to an embodiment of the present disclosure. [Figure 2] It is a diagram showing a control method of an endoscopic device according to an embodiment of the present disclosure. [Figure 3] It is a block diagram of an endoscopic system according to an embodiment of the present disclosure. [Figure 4] It is a conceptual diagram specifically showing an endoscopic system according to an embodiment of the present disclosure. [Figure 5] It is a conceptual diagram showing an artificial neural network model according to an embodiment of the present disclosure. [Figure 6] (a) and (b) are conceptual diagrams showing an artificial neural network model according to other embodiments of the present disclosure. [Figure 7] It is a flowchart showing a detailed step of a stage in which an artificial intelligence model classifies and outputs into an air supply image, a water supply image, and an air intake image in a control method of an endoscopic device according to an embodiment of the present disclosure. [Figure 8]This is a conceptual diagram illustrating how critical values ​​are applied to images classified by an artificial neural network, according to one embodiment of the present disclosure. [Figure 9] This is a conceptual diagram illustrating how the degree of agreement between the classification results of the preceding and succeeding images of a classified image is checked according to one embodiment of the present disclosure. [Figure 10] This is a conceptual diagram showing how multiple labels are displayed on the screen of an endoscope device according to one embodiment of the present disclosure. [Modes for carrying out the invention]

[0025] The terms used in this invention are used solely to describe specific embodiments and are not intended to limit the scope of other embodiments. Singular expressions also include plural expressions unless explicitly stated in the context. Terms used herein, including technical and scientific terms, may have the same meaning as those generally understood by a person of ordinary skill in the art described herein. General, predefined terms used herein are to be interpreted as having the same or similar meaning as they do in the context of the relevant art, and not as to be idealistic or overly formal unless explicitly defined herein. Where applicable, a term defined herein is not to be construed as excluding embodiments of the invention.

[0026] In the following, various embodiments will be described in detail with reference to the accompanying drawings, so as to be easily implemented by a person with ordinary skill in the art to which the present invention pertains. However, the technical idea of ​​the present invention can be embodied in various forms and is not limited to the embodiments described herein. In the description of embodiments disclosed herein, if it is determined that specifically describing related prior art would obscure the gist of the technical idea of ​​the present invention, then specific descriptions relating to such prior art will be omitted. Identical or similar components will be given the same reference numeral, and redundant descriptions relating thereto will be omitted.

[0027] Here, the term "~part" as used in this embodiment refers to a component that performs a specific function, whether by software or hardware such as an FPGA (field programmable gate array) or an application-specific integrated circuit (ASIC). However, "~part" is not limited to those performed by software or hardware. "~part" may also exist as data stored on an addressable recording medium, be embodied by instructions, and be configured so that one or more processors perform a specific function.

[0028] Software may include computer programs, code, instructions, or a combination of one or more of these, which can configure a processing unit to operate as desired, or independently or collectively, instruct the processing unit. Software and / or data may be permanently or temporarily embodied in a type of machine, component, physical device, virtual device, computer recording medium or device, or transmitted signal wave, in order to be interpreted by a processing unit or to provide instructions or data to a processing unit. The software may also be distributed on a network of computer systems, stored in a distributed manner, or executed. The software and data may be stored on a recording medium readable by one or more computers. The software may be read into main memory from other computer-readable media, such as data storage devices, or from other devices via a communication interface. Software instructions stored in main memory may cause the processor to perform processes or steps described in detail below. Alternatively, fixed wiring circuits may be used in place of, or in combination with, software instructions to perform processes consistent with the principles of the present invention. Therefore, embodiments consistent with the principles of the present invention are not limited to any particular combination of hardware circuits and software.

[0029] The terms used in this application are used solely to describe specific embodiments and are not intended to limit the invention. A singular expression includes plural expressions unless the context explicitly limits it to that singular expression. In this application, terms such as “includes” or “having” should be understood to indicate the existence of features, numbers, stages, operations, components, parts, or combinations thereof described in the specification, and not to preemptively exclude the possibility of the existence or addition of one or more other features, numbers, stages, operations, components, parts, or combinations thereof. Terms such as the first and second may be used to describe a variety of components, but such components are not limited by the terms. The terms are used solely for the purpose of distinguishing one component from others.

[0030] The “learning model” as used in this invention also includes all forms of algorithms or methodologies used to learn or understand specific patterns or structures from data. That is, the learning model includes not only machine learning models such as regression models, decision trees, random forests, support vector machines, K-nearest neighbors, naive phases, and clustering algorithms, but also deep learning models such as neural networks, convolutional neural networks, circulatory neural networks, Transformer-based neural networks, GANs (Generative Adversarial Networks), and autoencoders. The “learning model” specifies a set of learned parameters or weights used to predict or classify an output for a given input, and the model can be learned through methods such as directed learning, undirected learning, semi-directed learning, and reinforcement learning. It also includes not only single models, but also diverse learning methods and structures such as ensemble models, multimodal models, and models via transfer learning. Such learning models may also be pre-trained on a separate computer device from the computer device that predicts the output for a given input and used on the other computer device.

[0031] Figure 1 is a drawing showing an endoscope device according to one embodiment of the present disclosure.

[0032] Referring to Figure 1, the endoscopic device 100 according to one embodiment of the present disclosure is also a flexible endoscope, specifically a gastrointestinal endoscope. The endoscopic device 100 also includes a configuration that can acquire medical images of the inside of the digestive tract, and, if necessary, a configuration that allows for the insertion of tools and the performance of treatment or procedures while viewing the medical images.

[0033] The endoscope device 100 also includes an output unit 110, a drive unit 130, a scope 150, and a control unit 400.

[0034] The output unit 110 also includes a display that shows medical images. The output unit 110 also includes a display module that can output visualized information or implement a touch screen, such as a liquid crystal display (LCD), thin-film transistor-liquid crystal display (TFT LCD), organic light-emitting diode (OLED), flexible display, or three-dimensional display (3D display).

[0035] The output unit 110 also includes various means of providing medical images, that is, information related to images of the inside of the body. For example, it also includes a speaker that provides information related to images of the inside of the body audibly.

[0036] The output unit 110 can display images of the inside of the body acquired by the scope 150, or images of the inside of the body processed by the control unit 400. In addition to displaying images of the inside of the body, the output unit 110 can further display the results of the artificial neural network model classifying and analyzing the images of the inside of the body.

[0037] For example, the output unit 110 may display information indicating that the frame of the internal body image requires inhalation, or information indicating that inhalation is currently being performed.

[0038] Although Figure 1 shows one output unit 110, there can be multiple output units 110. In that case, the output unit displaying the internal body image acquired by the scope 150 and the output unit displaying the information processed by the control unit 400 may be separated. Furthermore, on a single output unit, the first screen displaying the internal body image acquired by the scope 150 and the second screen displaying the internal body image with information processed by the control unit 400 may also be separated.

[0039] The control unit 400 can control the overall operation of the endoscope device 100. The control unit 400 also includes all types of devices capable of processing data. For example, the control unit 400 is a hardware-integrated data processing device that has physically structured circuits to perform functions expressed by code or instructions contained within a program. Examples of hardware-integrated data processing devices may include, but are not limited to, microprocessors, central processing units (CPUs), processor cores, multiprocessors, application-specific integrated circuits (ASICs), and field-programmable gate arrays (FPGAs).

[0040] The control unit 400 can control the movement of the scope 150 via the drive unit 130, which is connected to the scope 150. The control unit 400 can perform various control operations via the scope 150 for imaging the inside of the body, specifically the inside of the digestive tract.

[0041] The control unit 400 can perform various processing operations on medical images acquired through the scope 150. The control unit 400 can control the endoscope device 100 based on control signals received from the computing device. For example, the control unit 400 can open or shut off the suction pump based on an open / close signal for a valve connected to the suction pump.

[0042] The drive unit 130 can provide the power necessary for the scope 150 to be inserted into or move within the body. For example, the drive unit 130 may also include a plurality of motors connected to wires inside the scope 150, and a tension adjustment unit for adjusting the tension of the wires. The drive unit 130 can control the power of each of the plurality of motors and control the scope 150 in various directions.

[0043] In Figure 1, the control unit 400 and the drive unit 130 are shown as separate and distinct hardware components, but this is not the only way to represent them. For example, the control unit 400 and the drive unit 130 may be provided as a single piece of hardware, or as two or more separate pieces of hardware. When provided as two or more separate pieces of hardware, parts of the configuration of the control unit 400 and parts of the configuration of the drive unit 130 may be physically separated.

[0044] The scope 150 also includes an insertion section 150a, a curved section 150b, and a tip section 150c. Inside the insertion section 150a, the curved section 150b, and the tip section 150c are also at least one of the following: an air supply pump for injecting air into the body through the scope 150; an air intake pump for providing negative pressure or vacuum and drawing air from the body through the scope 150; and a water supply pump for injecting cleaning water into the body through the scope 150. Each pump also includes a valve for controlling the flow of fluid. The valves of each pump can be opened and closed by the control unit 400.

[0045] According to one embodiment of the present disclosure, the intake pump valve may be opened and closed based on a control signal from a computing device or control from the control unit 400. In one embodiment, the control unit 400 may open the intake valve based on a time for opening the intake valve calculated by the computing device. The control unit 400 may shut off the intake valve based on an intake valve shutoff signal from the computing device or after a time calculated by the computing device has elapsed.

[0046] The scope 150 also includes an insertion part 150a that is inserted into the body, specifically into the digestive tract, and an operating unit 160 that receives input from the user to control the movement of the insertion part 150a and perform various procedures inside the digestive tract.

[0047] The insertion section 150a is configured to bend flexibly, and one end is connected to the drive unit 130, so that the degree or direction of bending can be determined by the drive unit 130. Since internal body imaging and procedures are performed at the end of the insertion section 150a, the scope 150 also includes various cables and tubes that extend to the end of the insertion section 150a.

[0048] The end of the insertion portion 150a may have a curved portion 150b and a tip portion 150c. The tip portion 150c may be equipped with an image sensor 151 capable of acquiring images related to the inside of the body. The rotation angle of the tip portion 150c is adjusted by a drive unit 130 that receives a control signal from the control unit 400, allowing for the confirmation of various parts inside the body, the capture of images of the inside of the body, and the advancement of procedures inside the body.

[0049] The curved portion 150b can be connected to the tip portion 150c. The drive unit 130 receives a control signal from the control unit 400 and can adjust the degree of curvature of the curved portion 150b. The degree of curvature of the curved portion 150c can adjust the rotation angle of the tip portion 150c.

[0050] According to this embodiment, when the first curved steering unit 161 and the second curved steering unit 162 are operated, a signal is transmitted to the control unit 400 via a signal transmission system attached to the operating unit 160. The signal processed by the control unit 400 is transmitted to the drive unit 130, and the curved unit 150b can be controlled by driving the motor. The first curved steering unit 161 can control the vertical movement of the curved unit 150b and the tip 150c, and the second curved steering unit 162 can control the horizontal movement of the curved unit 150b and the tip 150c.

[0051] Referring to Figure 1, the tip 150c of the scope 150 contains an image sensor 151 internally, and the end portion contains a light source 153, a lens 154, a first working channel 155, and a second working channel 156. The internal body image captured through the lens 154 can be sensed through the image sensor 151. The light source 153 emits light at the part adjacent to the lens 154, illuminating the dark interior of the body, thereby making the internal body imaging through the lens 154 clearer and more precise.

[0052] Through the first working channel 155, tools for treating and managing lesions may be inserted during the endoscopic procedure. For example, flushing water for water delivery may be supplied to the inside of the body through the operating channel 163 via the first working channel 155. Also, for example, the first working channel 155 may draw in fluid from inside the body, specifically from the digestive tract, through the valve of the suction pump in the suction section 430 (Figure 4).

[0053] In Figure 1, in addition to the first working channel 155, a second working channel 156 is also illustrated. The second working channel 156 can perform different types of operations than those performed by the first working channel 155. For example, while the first working channel 155 is performing the operation of drawing in fluid, the second working channel 156 may be performing the operation of injecting air for delivery into the body.

[0054] The control unit 160 also includes multiple input buttons that provide various functions for the endoscopist to control the maneuvering of the insertion section 150a and perform the procedure via the working channel 155. The endoscopist can perform suction inside the digestive tract via the control unit 160, independently of the control of the computing device.

[0055] Furthermore, the operation of the control unit 400 is performed by the drive unit 130, and for this purpose, the drive unit 130 may be equipped with necessary processing devices such as a microprocessor and a CPU (central processing unit). In that case, the control unit 400 processes the internal body images acquired through the scope 150, and the drive unit 130 can control the endoscope device 100 based on control signals received from the computing device.

[0056] Figure 2 is a diagram showing a control method for an endoscope device according to one embodiment of the present disclosure. Figure 3 is a block diagram of an endoscope system according to one embodiment of the present disclosure. Figure 4 is a conceptual diagram specifically showing an endoscope system according to one embodiment of the present disclosure.

[0057] Referring to Figures 1 to 4, a control method for an endoscope device according to one embodiment of the present disclosure includes the steps of: acquiring an image of the inside of the body from an image sensor 151 (S100); inputting the acquired image into a learned artificial neural network model 200 for classification of the acquired image (S200); having the artificial neural network model 200 classify the image into an air supply image, a water supply image, and / or an air intake image and output them (S300); and controlling the endoscope device 100 so that the air supply unit 410, the water supply unit 420, and / or the air intake unit 430 of the endoscope device 100 are driven based on the outputted classification results (S400).

[0058] An endoscope device 100 according to one embodiment of the present disclosure includes a hardware device or a part of a hardware device that performs comprehensive data processing and calculations, and also includes a software-based computing environment connected to a communication network. For example, the endoscope device 100 also includes a server that performs an intensive data processing function and is the entity that shares resources, and also includes a client that shares resources through interaction with the server.

[0059] Furthermore, the endoscope device 100 also includes a cloud system in which multiple servers and clients interact to process data comprehensively. The above description is merely one example of the configuration of the endoscope device 100, and the configuration of the endoscope device 100 can be configured in a variety of ways, within the scope that a person skilled in the art can understand based on the contents of this disclosure.

[0060] Referring to Figure 3, one embodiment of the endoscope device 100 of the present disclosure also includes a processor 101, a memory 102, an artificial neural network model 200, an image acquisition unit 300, a control unit 400, and a post-processing unit 500.

[0061] A processor 101 according to one embodiment of the present disclosure can be understood as a component unit including hardware and / or software for performing computing operations. For example, the processor 101 may interpret computer programs and perform data processing for machine learning. The processor 101 may handle computational processes such as processing input data for machine learning, feature extraction for machine learning, and error calculation based on backpropagation. A processor 101 for performing such data processing may also include a central processing unit (CPU), a general-purpose graphics processing unit (GPGPU), a tensor processing unit (TPU), an application-specific integrated circuit (ASIC), or a field-programmable gate array (FPGA). The above-mentioned types of processor 101 are merely examples, and the types of processor 101 can be configured in a variety of ways, within the scope understandable to those skilled in the art based on the content of the present disclosure.

[0062] The processor 101 can use images of the inside of the body, such as endoscopic images, as training data to train the artificial neural network model 200. Specifically, the processor 101 can train the artificial neural network model 200 to determine in real time whether or not air, water, and / or suction are necessary inside the body, based on endoscopic images of the inside of the body. In this case, the endoscopic images also include endoscopic video footage.

[0063] Furthermore, the trained artificial neural network model 200 can be input with images of the inside of the body, i.e., endoscopic images, and these images can be classified and output as air supply status images, water supply status images, and / or suction status images. The endoscopic device can then be controlled so that the air supply unit, water supply unit, and / or suction unit are driven based on the output classification results.

[0064] In this specification, "internal body" refers to organs such as the stomach, duodenum, small intestine, and large intestine, and more specifically, it may refer to parts of organs, such as the cardia, gastric angle, and pylorus of the stomach. The following explanation will use the digestive organs as examples.

[0065] During the endoscopic procedure, the endoscopist performs endoscopic examinations, but must also press the air button when entering the esophagus, the water button when a foreign object is attached to the lens 154, and the suction button when gastric fluid is visible on the screen, or when the medical treatment is completed and the endoscope is withdrawn from the esophagus. In this process, many button operations are required for a single endoscopic diagnosis, which can be difficult.

[0066] Specifically, when performing an endoscopic procedure, a certain distance must be maintained between the scope 150 equipped with the lens 154 and the digestive organs. Therefore, medical personnel may inject air into the digestive organs during the endoscopic procedure to inflate them. If the air injection is insufficient, lesions located between the folds of the organs or depressed lesions may not be observed. Furthermore, different amounts of air injection result in different organ morphologies, and consequently, different lesion morphologies. Therefore, appropriate air injection, judged in real time during the endoscopic procedure, is essential for accurate diagnosis. In addition, if a foreign object is attached to the lens 154, the accuracy of the medical image will decrease, so air may be sprayed onto the lens to remove the foreign object.

[0067] Furthermore, when performing an endoscopic procedure, it is necessary to clean the lesion and the surrounding area, and the lens 154 also needs to be cleaned in case foreign matter adheres to it and the image is not clearly visible. At this time, cleaning water may be sprayed onto the affected area or the lens.

[0068] Furthermore, during endoscopic procedures, inspiratory actions are frequently performed to aspirate and expel fluids or foreign objects present inside the digestive tract in order to secure a clear field of view and accurately detect lesions during the endoscopic procedure. For example, if foreign objects are attached to the area around the affected area or fluid accumulates, inspiratory action is performed because it becomes difficult to obtain accurate images. Inspiratory action is also performed to remove debris generated when a biopsy is performed for the purpose of treatment or tissue examination. The inspiratory unit 430 appropriately aspirates any excessive air or gas that may be injected during the process of the organ being observed expanding.

[0069] Thus, water supply, air supply, and suction are necessary in a variety of situations, and according to embodiments of this disclosure, the endoscope device 100 can determine, based on an internal body image, when water supply, air supply, and / or suction are necessary inside the body, and if necessary, control the air supply unit 410, water supply unit 420, and suction unit 430 of the endoscope device 100.

[0070] For example, the processor 101 may train the artificial neural network model 200 to determine that inhalation is necessary when there is a foreign object around the affected area shown in the internal body image, or when the inside of the body is overinflated.

[0071] At this time, the processor 101 may train one artificial neural network model 200 that can determine whether the lens 154 that captured the internal body image needs to be cleaned, whether there is a foreign object around the affected area shown in the internal body image, or whether the inside of the body is over-inflated, or it may train multiple artificial neural network models 200 that correspond to each of these cases.

[0072] The processor 101 can train the artificial neural network model 200 through supervised learning, which uses training data as input values. Alternatively, it can train the artificial neural network model 200 through unsupervised learning, which finds criteria for data recognition by learning the types of data necessary for data recognition on its own without any guidance. Alternatively, it can train the artificial neural network model 200 through reinforcement learning, which utilizes feedback regarding whether the results of data recognition obtained through learning are correct or not.

[0073] The artificial neural network model 200 disclosed herein also includes CNNs (convolutional neural networks). However, it is not limited to CNNs; the artificial neural network model 200 disclosed herein, and neural networks in general, also include network models such as DNNs (deep neural networks), RNNs (recurrent neural networks), BRDNNs (bidirectional recurrent deep neural networks), MLPs (multilayer perceptrons), and transformers.

[0074] The processor 101 inputs images of the inside of the body into the trained artificial neural network model 200 and can determine from the images whether air insufflation, water insufflation, and suction are necessary within the digestive tract. Based on the output values ​​of the artificial neural network model 200, the processor 101 can generate signals to control the endoscope device 100.

[0075] Specifically, the processor 101 can generate control signals for performing air supply, water supply, and / or inhalation. The processor 101 can generate control signals to selectively open and close the valves of the air supply pump, water supply pump, and / or inhalation pump connected to the working channels 155 and 156 so that inhalation is performed through the working channels 155 and 156 of the scope 150 of the endoscope device 100 inserted into the digestive tract. Based on the output results of the artificial neural network model 200, the processor 101 can either directly generate control signals to control the endoscope device 100, or instruct the endoscope device 100 to generate control signals.

[0076] For example, the processor 101 can distinguish between cases where cleaning of the lens used to capture images of the inside of the body is necessary, where there is a foreign object around the affected area shown in the image of the inside of the body, and where the inside of the digestive tract is excessively distended, and control the valve of the suction pump of the endoscope device 100.

[0077] The processor 101 receives internal body images frame by frame while the endoscopic procedure is being performed, and for each frame, it may determine whether or not air insufflation, water insufflation, and / or suction are necessary. The above operations may be repeated until the endoscopic procedure is completed.

[0078] A memory 102 according to one embodiment of the present disclosure may be understood as a component unit including hardware and / or software for storing and managing data processed by the endoscope device 100. That is, the memory 102 may store data in any form generated or determined by the processor 101, and data in any form received by the network unit (not shown). For example, the memory 102 may also include at least one type of recording medium from among flash memory type, hard disk type, multimedia card micro type, card type memory, RAM (random access memory), SRAM (static random access memory), ROM (read-only memory), EEPROM (electrically erasable programmable read-only memory), PROM (programmable read-only memory), magnetic memory, magnetic disk, or optical disk. The memory 102 may also include a database system for controlling and managing the data in a predetermined manner. The aforementioned types of memory 102 are merely examples; therefore, the types of memory 102 can be configured in a variety of ways, within the scope understandable to those skilled in the art, based on the contents of this disclosure.

[0079] Memory 102 can structure and organize and manage data, data combinations, and program code (code) that can be executed by the processor 101, which are necessary for the processor 101 to perform calculations. Memory 102 can also store program code that causes the processor 101 to operate in order to generate training data.

[0080] According to this disclosure, the memory 102 can store internal body images taken from the endoscope device 100 and, based on these images, can store control signals generated by the processor 101.

[0081] A network unit according to one embodiment of this disclosure can be understood as a component that transmits and receives data via any known wired wireless communication system. For example, the network unit may use a wired wireless communication system such as a local area network (LAN), wideband code division multiple access (WCDMA), LTE (long term evolution), WiBro (wireless broadband internet), 5G, ultrawide-band wireless communication, Zigbee, radio frequency (RF) communication, wireless LAN, Wi-Fi (wireless fidelity), near field communication (NFC), or Bluetooth to transmit and receive data. The aforementioned communication systems are merely examples, and a variety of wired wireless communication systems other than those described above can be applied to transmit and receive data in the network unit.

[0082] The network unit can receive data necessary for the processor 101 to perform calculations via wired or wireless communication with any system or client. The network unit can also transmit data generated through the calculations of the processor 101 via wired or wireless communication with any system or client. For example, the network unit can receive medical images via communication with a medical image storage and transmission system, a cloud server performing tasks such as medical data standardization, or a separate computing device. The network unit can also transmit various types of data generated through the calculations of the processor 101 via communication with the aforementioned systems, servers, or separate computing devices.

[0083] The processor 101 inputs images of the inside of the body to the learned artificial neural network model 200, classifies the images into air supply status images, water supply status images, and / or suction status images and outputs them, and controls the endoscope device so that the air supply unit, water supply unit, and / or suction unit are driven based on the output classification results.

[0084] The processor 101 can acquire images of the inside of the body from the image sensor via the image acquisition unit 300. The processor 101 also transmits control signals from the control unit 400 to the drive unit 130, and the drive unit 130 can open and close a valve connected to the air pump of the air supply unit 410, open and close a valve connected to the water pump of the water supply unit 420, or open and close a valve connected to the air intake pump of the air intake unit 430 based on the control signals. The air supply unit 410, water supply unit 420, and air intake unit 430 are located inside the insertion part 150a of the scope 150, and can supply air, supply water, or inhale air into the body based on drive information controlled via the control unit 400.

[0085] The processor 101 can perform post-processing on the images output via the post-processing unit 500, which are classified into air supply status images, water supply status images, and / or intake status images.

[0086] According to this disclosure, the post-processing unit 500 also includes a critical value comparison unit 510 (Figure 4) that applies critical values ​​to the classified image, and a degree of agreement confirmation unit 520 (Figure 4) that confirms the degree of agreement between the classified image and the classification results of the images before and after the classified image, as will be described later.

[0087] Referring to Figure 4, the image acquisition unit 300 can acquire images of the inside of the body. For example, it can acquire situational images such as image A of entering a narrowed esophagus E, image B of a foreign object S attached to the lens L, and image C of gastric juice GJ visible around the esophagus E.

[0088] Images acquired via the image acquisition unit 300 can be input to the artificial neural network model 200. The artificial neural network model 200 can classify the various internal body images it receives and categorize them into air supply status images 301, water supply status images 302, and / or inhalation status images 303. Images that do not fall into the three aforementioned status image categories can also be classified separately.

[0089] The artificial neural network model 200 can output three classified situational images. In this case, the outputted image can be transmitted to one of the air supply unit 410, water supply unit 420, or intake unit 430 of the control unit 400. If the image does not fall under the aforementioned three types of situational images, it is not transmitted to the control unit 400.

[0090] For example, suppose an endoscope device 100 located inside the body captures a specific image at a specific point in time. In this case, if the image sensor 151 of the endoscope device 100 located inside the body detects an image A entering the narrowed esophagus E, the image acquisition unit 300 can acquire the image A entering the narrowed esophagus E via the image sensor 151. The image A entering the esophagus E acquired by the image acquisition unit 300 is input to the artificial neural network model 200, and the artificial neural network model 200 can determine that the image is an air supply status image 301 based on its learned information.

[0091] If the image is determined to be the air supply status image 301, the artificial neural network model 200 may output information to the control unit 400 indicating that the image is the air supply status image 301. Based on the information transmitted from the artificial neural network model 200, the control unit 400 may open or close a valve connected to the air supply pump of the air supply unit 410 via the drive unit 130. The air supply unit 410 injects air supplied from the air supply pump through the working channels 155 and 156, injecting air into the narrowed esophagus E, thereby widening the esophagus E and allowing the tip 150c of the endoscope device 100 to be easily inserted into the esophagus E.

[0092] According to one embodiment of the present disclosure, in the step (S300) in which the artificial neural network model 200 classifies and outputs images as air supply status images, water supply status images and / or insufflation status images, the artificial neural network model 200 may be learned to determine an image in which an organ is narrowed as the endoscope device 100 enters the human body as an air supply status image.

[0093] For example, the artificial neural network model 200 can be trained to interpret an image of esophageal stenosis as an image of air insufflation when the endoscope device 100 enters the esophagus.

[0094] Furthermore, according to one embodiment of the present disclosure, in the step (S300) in which the artificial neural network model 200 classifies and outputs images as air supply status images, water supply status images and / or inhalation status images, the artificial neural network model 200 may determine that an image of a foreign object attached to the lens of the endoscope device 100 is a water supply status image, an air supply status image and / or inhalation status image.

[0095] To remove foreign matter attached to the lens, one or more of the following actions can be activated: water supply, air supply, and / or suction, thereby removing the foreign matter attached to the lens.

[0096] Furthermore, according to one embodiment of the present disclosure, in the step (S300) in which the artificial neural network model 200 classifies and outputs images as air supply status images, water supply status images and / or inhalation status images, the artificial neural network model 200 may be learned to determine an image in which the endoscope device 100 has finished medical treatment and is withdrawn from the organ, or an image in which a foreign body inside the organ is confirmed, as an inhalation status image.

[0097] For example, the artificial neural network model 200 can be trained to interpret the image of the endoscope device 100 exiting the esophagus after completing medical treatment as an image of inhalation. Furthermore, if substances that need to be removed from inside an organ, such as excess gastric juice, digested food, nasal mucus, or lesions, are identified, the artificial neural network model 200 can be trained to interpret the image of the endoscope device 100 identifying foreign objects inside an organ as an image of inhalation, in order to inhale and remove these substances from inside the organ via the inhalation action.

[0098] In Figure 4, one artificial neural network model 200 is illustrated as determining whether air supply, water supply, and / or inhalation are necessary in various cases. However, the artificial neural network model 200 also includes multiple artificial neural network models that make decisions for each case.

[0099] Thus, by utilizing the artificial neural network model 200 provided in this disclosure, depending on the situation, it is possible to dramatically reduce the physical burden on endoscopists and significantly improve the convenience of endoscopic diagnosis by using an automated water supply, air supply, and / or suction execution solution, compared to conventional technologies in which water supply, air supply, and / or suction operations had to be performed through complex operations.

[0100] Figure 5 is a conceptual diagram showing an artificial neural network model according to one embodiment of the present disclosure.

[0101] Referring to Figure 5, the artificial neural network model 200 according to one embodiment of the present disclosure also includes a CNN (convolutional neural network). Images transmitted from the image acquisition unit 300 can be input to the artificial neural network model 200. At this time, the convolution layer (CONV) 210 of the artificial neural network model 200 can analyze the image via convolution operations. The convolution layer 210 can perform operations on various layers and extract feature parts of the image. The extracted feature parts can be processed via a pooling layer 220 to reduce the dimensionality of the image data and improve the computational efficiency of the neural network. The convolution layer 210 and the pooling layer 20 can be iteratively constructed to materialize the feature parts of the image.

[0102] The features extracted through the process described above can be transmitted to a fully connected (FC) hierarchy (230). The fully connected hierarchy 230 can output a classification result for the input extracted features, indicating which class (water supply status image, air supply status image, air intake status image) each image belongs to. The output can be transmitted to the control unit 400.

[0103] Figures 6(a) and 6(b) are conceptual diagrams showing artificial neural network models according to other embodiments of the present disclosure. Figure 6(a) is a conceptual diagram showing an artificial neural network model according to another embodiment of the present disclosure. Figure 6(b) is a conceptual diagram showing an artificial neural network model according to yet another embodiment of the present disclosure.

[0104] Referring to Figure 6(a), the artificial neural network model 200' also includes multiple artificial neural network models 201', 202'. In this case, the multiple artificial neural network models 201', 202' can classify and output at least one situational image from among the air supply situation image, the water supply situation image, and / or the inhalation situation image.

[0105] For example, the first artificial neural network model 201' can classify the input image into an air supply status image and a water supply status image and output it, while the second artificial neural network model 202' can classify the input image into an inspiratory status image and output it.

[0106] Referring to Figure 6(b), the artificial neural network model 200" also includes multiple artificial neural network models 201", 202", and 203". In this case, the multiple artificial neural network models 201", 202", and 203" can classify and output at least one situational image from among the air supply situation image, the water supply situation image, and / or the inhalation situation image.

[0107] For example, the first artificial neural network model 201" can classify the input image into an air supply status image and output it, the second artificial neural network model 202" can classify the input image into a water supply status image and output it, and the third artificial neural network model 203" can classify the input image into an inhalation status image and output it.

[0108] The multiple artificial neural network models disclosed herein are not limited to those disclosed in Figures 6(a) and 6(b), and the number of artificial neural network models and the range of situational images classified by each artificial neural network model can be realized in a variety of ways.

[0109] Figure 7 is a flowchart showing the detailed steps in a control method for an endoscope device according to one embodiment of the present disclosure, in which an artificial intelligence model classifies and outputs images as air supply images, water supply images, and inhalation images.

[0110] Referring to Figure 7, in one embodiment of the present disclosure, in a method for controlling an endoscope device, the step (S300) in which the artificial neural network model 200 classifies and outputs images into air supply status images, water supply status images, and / or insulation status images also includes the step (S310) of applying a threshold value to the images classified by the artificial neural network model 200, and the step (S320) of confirming the degree of agreement between the classified images and the classification results of the images before and after the classified images.

[0111] Figure 8 is a conceptual diagram showing how critical values ​​are applied to images classified by an artificial neural network, according to one embodiment of the present disclosure.

[0112] Referring to Figure 8, critical values ​​can be applied as a post-processing step to improve the accuracy of classification of air supply status images, water supply status images, and / or inhalation status images determined using the artificial neural network model 200. In this case, critical values ​​related to specific criteria can be applied to the classification criteria of air supply status images, water supply status images, and / or inhalation status images classified via the artificial neural network model 200, and only images that satisfy these critical values ​​can be determined to correspond to specific status images.

[0113] In this case, the critical value can be determined based on data learned from images of the air supply status, water supply status, and / or the inhalation status. Furthermore, as illustrated in Figure 7, the critical values ​​based on classification can be set in various ways, such as the first critical value 301a, the second critical value 301b, and the third critical value 301c. During endoscopic procedures, these critical values ​​can be applied differently depending on the patient's condition, allowing for different water supply control, air supply control, and inhalation control depending on the situation.

[0114] Figure 9 is a conceptual diagram showing how the degree of agreement between the classification results of the preceding and succeeding images of a classified image is checked according to one embodiment of the present disclosure.

[0115] Referring to Figure 9, the degree of agreement can be checked as a post-processing step to improve the classification accuracy of the air supply status image, water supply status image, and / or inspiratory status image determined using the artificial neural network model 200. That is, although internal body image acquisition by the endoscopic device is done on a frame-by-frame basis, the degree of output agreement between images can be checked by comparing the image acquired in the previous image frame acquired immediately before the currently acquired frame and the image acquired in the subsequent image frame acquired immediately after the currently acquired frame with the image acquired in the current frame. In this case, the degree of output agreement checks the degree of agreement between the adjacent frame image and the image output after the image corresponding to each frame has been input to the artificial neural network and classified, and this can improve the classification accuracy of the images classified via the artificial neural network model 200.

[0116] Figure 10 is a conceptual diagram showing how a multi-label display appears on the screen of an endoscope device according to one embodiment of the present disclosure.

[0117] The artificial neural network model 200 can be input with labels related to endoscopic images as training data. These endoscopic images are also endoscopic images taken via an endoscopic device. The labels also include information indicating that the internal body image is in a state where air, water, and / or inhalation are necessary. For example, the state requiring inhalation may include a variety of conditions, such as the presence of a foreign body around the affected area shown in the internal body image, and the over-inflation of the digestive tract, in which case the condition may be labeled as requiring inhalation.

[0118] Furthermore, for example, if the internal body image is sharply focused, or if there are no foreign objects around the affected area shown in the internal body image, it may be labeled as a normal state where inhalation is not required. The labeling process may be performed by medical staff, or a separate artificial neural network model may be used for labeling.

[0119] According to one embodiment of the present disclosure, the step (S300) in which the artificial neural network model 200 classifies and outputs the images as air supply status images, water supply status images, and / or inhalation status images further includes the step (S330) of labeling each image according to the air supply status image, water supply status image, and / or inhalation status image. In this case, the labeled label is also a multi-label in which the image has additional labeling related to information about internal body parts.

[0120] Figure 10 illustrates an example of an image related to multi-labeling. As illustrated in Figure 10, when labeling a specific part inside the body, in addition to labeling information related to air supply, water supply, and inhalation, information related to whether the location is, for example, the esophagus, duodenum, or stomach may be added to the label and output. That is, when classifying an image, the artificial neural network model 200 can classify and determine the location inside the body where the water supply, air supply, or inhalation is occurring, and transmit this information to the control unit 400.

[0121] The control unit 400 can also determine information relating to which organ the tip 150c of the endoscope device is currently located inside, and whether the tip 150c is currently inserted into or withdrawn from the body. Through this, when controlling the air supply, water supply, and suction operations of the endoscope device, the degree of air supply, water supply, and suction can be controlled more precisely based on the position and direction of movement of the tip 150c.

[0122] Memory 102 can store numerous application programs, data for computer device operation, and instruction words. Memory 102 may be implemented in internal memory such as ROM or RAM included in the processor 101, or in memory separate from the processor 101. In one embodiment, memory 102 can store neural networks and learning data.

[0123] Specifically, the processor 101 can control the operation of the computer device by utilizing various programs stored in the computer device's memory 102. The processor 101 also includes the CPU, RAM, ROM, system bus, etc. The processor 101 can be embodied by a single CPU or multiple CPUs (or a DSP (digital signal processor), SoC (system-on-chip)). In one embodiment, the processor 101 can be embodied by a digital signal processor (DSP), a microprocessor, or a TCON (time controller) that processes digital signals. However, it is not limited to these, and may include or be defined by one or more of the following: a central processing unit (CPU), an MCU (microcontroller unit), an MPU (microprocessing unit), a controller, an application processor (AP), a communication processor (CP), or an ARM processor. Furthermore, the processor 101 can also be implemented as an SoC or LSI (large-scale integration) with a built-in processing algorithm, and also as an FPGA (field-programmable gate array).

[0124] The apparatus described above may be embodied by hardware components, software components, and / or combinations of hardware and software components. For example, the apparatus and components described in this embodiment may be embodied using one or more general-purpose or special-purpose computers, such as a processor, controller, ALU (arithmetic logic unit), digital signal processor, microcomputer, FPGA, PLU (programmable logic unit), microprocessor, or any other device capable of executing and responding to instructions. The processing device may execute an operating system (OS) and one or more software applications executed on the operating system (OS). The processing device may also access, store, manipulate, process, and generate data in response to the execution of software. For convenience of understanding, the processing device has been described as being used as a single unit, but a person with ordinary skill in the art will know that the processing device may also include multiple processing elements and / or multiple types of processing elements. For example, the processing unit may include multiple processors, or one processor and one controller. Other processing configurations, such as a parallel processor, are also possible.

[0125] Software may include computer programs, code, instructions, or a combination of one or more of these, which can configure a processing unit to operate as desired, or independently or collectively instruct the processing unit. Such software and / or data may be permanently or temporarily embodied in a type of machine, component, physical device, virtual equipment, computer recording medium or device, or transmitted signal wave, in order to be interpreted by the processing unit or to provide instructions or data to the processing unit. Such software may also be distributed across a network of computer systems, stored in a distributed manner, or executed. Such software and data may be stored on a possible recording medium readable by one or more computers.

[0126] The method according to this embodiment is embodied in a program instruction form that can be executed via various computer means and can be recorded on a computer-readable medium. The computer-readable medium may include program instructions, data files, data structures, etc., individually or in combination. The program instructions recorded on the medium may be specially designed and configured for this embodiment, or they may be publicly known and usable by those skilled in the computer software art. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks, and magnetic tapes; optical media such as CD-ROMs (compact disc read-only memory), DVDs (digital versatile discs), and magneto-optical media such as floptical disks; and hardware devices specially configured to store and execute program instructions, such as ROMs, RAMs, and flash memory. Examples of program instructions include not only machine code generated by compilers, but also high-level language code that can be executed by a computer using an interpreter or the like. The aforementioned hardware device is configured to operate as one or more software modules in order to perform the operation of this embodiment, and vice versa.

[0127] As described above, even though this embodiment is described by limited embodiments and drawings, a person with ordinary skill in the art can make various modifications and variations from the above description. For example, the described technology may be performed in a different order than described, and / or components such as described systems, structures, devices, and circuits may be combined or combined in a different manner than described, or replaced or substituted by other components or equivalents, and the appropriate results may be achieved. Thereafter, other embodiments, other embodiments, and equivalents of the claims also fall within the scope of the claims. [Explanation of Symbols]

[0128] 100 Endoscopes 101 Processors 102 memory 110 Output section 130 Drive unit 150 Scope 151 Image Sensor 153 Light source 154 lenses 155,156 Working Channels 160 Operation section 200 Artificial Neural Network Models 300 Image Acquisition Section 400 Control Unit 410 Air supply unit 420 Water supply section 430 Intake section 500 Post-processing 510 Critical Value Comparison Section 520 Matching degree verification unit

Claims

1. In a control method for an endoscope device including one or more processors, The process involves one or more processors acquiring an image of the inside of the body from an image sensor. The step of inputting the internal body images into an artificial neural network model configured to classify the internal body images for each frame into at least one of the following: an air supply image, a water supply image, and an inhalation image; (i) applying classification-specific critical values ​​to the classified image, and (ii) determining the degree of agreement between the classification results of the previous and subsequent frames in the classified image, one or more processors perform this post-classification processing. A method for controlling an endoscope, comprising the step of controlling the endoscope by having one or more processors control the endoscope so that it performs at least one of air supply, water supply, and suction via the working channels of the endoscope by automatically opening and closing a valve corresponding to at least one of the air supply unit, water supply unit, and air intake unit of the endoscope in response to the classification satisfying a classification-specific critical value and the degree of agreement exceeding a criterion.

2. The aforementioned artificial neural network model includes multiple artificial neural network models, The control method for an endoscope device according to claim 1, wherein the plurality of artificial neural network models classify and output at least one situational image from among the air supply situational image, the water supply situational image, and / or the inhalation situational image.

3. The method for controlling an endoscope according to Claim 1, wherein the artificial neural network model is trained to determine an image of an organ being narrowed when the endoscope enters the human body as an image of the air supply situation.

4. The method for controlling an endoscope according to Claim 1, wherein the artificial neural network model is trained to determine an image of a foreign object attached to the lens of the endoscope as an image of a water supply situation, an air supply situation, and / or an air intake situation.

5. The control method for an endoscope according to Claim 1, wherein the artificial neural network model is trained to determine an image of the endoscope device withdrawing from an organ after completing medical treatment, or an image of substances that need to be removed from inside an organ being identified, as an inhalation status image.

6. The control method for an endoscope device according to claim 1, wherein the artificial neural network model includes a CNN (convolutional neural network).

7. The aforementioned artificial neural network model includes a convolution hierarchy and a fully connected hierarchy. The aforementioned convolution hierarchy extracts features of internal body images input via convolution operations. The control method for an endoscope device according to claim 6, wherein the fully connected hierarchy outputs a classification result relating to which of the following situation images—the air supply situation image, the water supply situation image, and / or the air intake situation image—the extracted image features ultimately correspond to.

8. Further comprising the step of labeling according to the air supply status image, the water supply status image and / or the intake status image, The control method for an endoscope device according to claim 1, wherein the labeled label is a multi-label in which the image is further labeled with information relating to a part inside the body.

9. A memory that stores images of the inside of the body taken from an endoscope, The images of the inside of the body are input into the trained artificial neural network model. For each frame, the image of the inside of the body is classified into at least one of the following: an image of the air supply situation, an image of the water supply situation, and an image of the inhalation situation. (i) apply classification-specific critical values ​​to the classified image, and (ii) determine the degree of agreement between the classification results of the previous and subsequent frames in the classified image. An endoscope device comprising a processor that controls at least one of an air supply unit, a water supply unit, and an air intake unit by automatically opening and closing a corresponding valve via a drive unit in response to the classification satisfying a classification-specific critical value and the degree of agreement exceeding a criterion, and performs at least one of the air supply, water supply, and air intake via a working channel of the endoscope device.

10. The aforementioned processor, The endoscopic device according to claim 9, which acquires an image of the inside of the body from an image sensor via an image acquisition unit.

11. The aforementioned processor, The control unit transmits the control signal to the drive unit. The endoscope apparatus according to claim 9, wherein the drive unit opens and closes a valve connected to the air supply pump of the air supply unit, opens and closes a valve connected to the water supply pump of the water supply unit, or opens and closes a valve connected to the air intake pump of the air intake unit, based on the control signal.

12. A computer program stored on a recording medium for using a computer to perform the method described in any one of claims 1 to 8.

Images