Endoscope device, method of controlling that endoscope device, and computer program

Abstract

To provide an endoscope device, a method of controlling the endoscope device, and a computer program.SOLUTION: A method of controlling the endoscope device includes: the steps of: acquiring an image of the inside of the body from an image sensor; inputting the image into a trained artificial neural network model for classification of the acquired image; the artificial neural network model classifying the image into an insufflation-state image, a water-supply-state image, and / or a suction-state image, and outputting the classification result; and controlling the endoscope device such that an insufflation unit, a water-supply unit and / or a suction unit of the endoscope device is driven according to the output classification result.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] The present invention relates to a control method for an endoscope apparatus using an artificial neural network model, the endoscope apparatus, and a computer program. [Background technology]

[0002] An endoscope is a medical instrument that is inserted into the body to observe organs without performing surgery or autopsies (pathological anatomy). An endoscope is inserted into the human body, irradiates it with light, and visualizes the light reflected from the surface of the internal wall. Endoscopes are classified according to their purpose and the body part they are used in. Broadly speaking, they can be divided into rigid endoscopes, in which the endoscopic tube is made of metal, and flexible endoscopes, such as those used in gastrointestinal endoscopes.

[0003] When endoscopists perform endoscopic examinations, they must press the air button when entering the esophagus, if the esophagus is narrowed, the water button when a foreign object appears on the screen, the suction button when gastric juice is visible on the screen, and the suction button when the medical treatment is completed and the object is removed from the esophagus. This requires multiple button operations for a single endoscopic diagnosis, which can be difficult. Summary of the Invention [Problem to be solved by the invention]

[0004] The present disclosure is intended to solve the problems of the prior art described above, and provides a control method for an endoscopic device, an endoscopic device, and a computer program in which the air supply status, water supply status, and / or suction status of an endoscope are identified without user operation, the identified results are transmitted to an endoscopic system, and air supply, water supply, and / or suction is automatically performed.

[0005] However, the technical problem that this embodiment aims to solve is not limited to the above-mentioned technical problem, and other technical problems may exist. [Means for solving the problem]

[0006] A control method for an endoscopic device according to an embodiment of the present disclosure is a control method for an endoscopic device including one or more processors, and includes the steps of acquiring an image of the inside of a body from an image sensor, inputting the image to a trained artificial neural network model for classifying the acquired image, the artificial neural network model classifying the image into an air supply status image, a water supply status image, and / or an intake status image and outputting the image, and controlling the endoscopic device so that the air supply unit, water supply unit, and / or intake unit of the endoscopic device are driven based on the output classification result.

[0007] The artificial neural network model may include a plurality of artificial neural network models, and the plurality of artificial neural network models may classify and output at least one of the air supply situation image, the water supply situation image, and / or the inhalation situation image.

[0008] In the step of classifying and outputting the image into an air supply status image, a water supply status image, and / or an inhalation status image by the artificial neural network model, the artificial neural network model may be trained to determine an image of an organ being constricted when the endoscope device enters the human body as an air supply status image.

[0009] In the step in which the artificial neural network model classifies and outputs the image into an air supply status image, a water supply status image, and / or an inhalation status image, the artificial neural network model can be trained to determine that an image in which a foreign object is attached to the lens of the endoscope device is a water supply status image, an air supply status image, and / or an inhalation status image.

[0010] In the step where the artificial neural network model classifies and outputs the images into an air supply status image, a water supply status image, and / or an inhalation status image, the artificial neural network model can be trained to determine that an image in which the endoscopic device has finished medical treatment and is emerging from an organ, or an image in which a substance that needs to be removed from inside an organ is confirmed, is an inhalation status image.

[0011] The artificial neural network model also includes a convolutional neural network (CNN).

[0012] The artificial neural network model includes a convolution layer and a fully connected layer, and the convolution layer extracts features of the input internal body image through a convolution operation, and the fully connected layer can output a classification result regarding which of the air supply situation image, the water supply situation image, and / or the inhalation situation image the extracted image features ultimately correspond to.

[0013] The step of the artificial neural network model classifying and outputting the image into an air supply status image, a water supply status image, and / or an inhalation status image may further include a step of applying a threshold value to the image classified by the artificial neural network model.

[0014] The step of the artificial neural network model classifying and outputting the images into an air supply status image, a water supply status image, and / or an inhalation status image may further include a step of checking the degree of agreement between the classified images and the classification results of previous and subsequent images.

[0015] The step of the artificial neural network model classifying and outputting the images into an air supply status image, a water supply status image, and / or an inhalation status image further includes a step of labeling the images by the air supply status image, the water supply status image, and / or the inhalation status image, and in this case, the labeled images are also multi-labeled, in which labeling related to information on internal body parts is added.

[0016] An endoscopic device according to one embodiment of the present disclosure includes a memory that stores an internal body image captured by the endoscopic device, and a processor that inputs the internal body image into a trained artificial neural network model, classifies the image into an air supply status image, a water supply status image, and / or an intake status image, and outputs the classified image, and controls the endoscopic device so that an air supply unit, a water supply unit, and / or an intake unit is driven based on the output classification result.

[0017] The processor may acquire an image of the interior of the body from an image sensor via an image acquisition unit.

[0018] The processor transmits a control signal from the control unit to the drive unit, and the drive unit can open or close a valve connected to the air supply pump of the air supply unit, open or close a valve connected to the water supply pump of the water supply unit, or open or close a valve connected to the intake pump of the intake unit, based on the control signal.

[0019] The processor may perform post-processing on the images classified into the air supply status image, the water supply status image, and / or the inhalation status image and output via a post-processing unit.

[0020] The post-processing unit may also include a threshold value comparison unit that applies a threshold value to the classified image, and a consistency check unit that checks the consistency between the classified image and the classification results of previous and subsequent images.

[0021] According to an aspect of the present invention, there is provided a computer program stored on a recording medium for causing a computer to execute the above-described method. [Effects of the Invention]

[0022] According to an embodiment of the present disclosure, during an endoscopic procedure, without the operation of a physician, when the esophagus is constricted, when a foreign object is found on the screen, when gastric juice is visible on the screen, and when medical treatment via the endoscopic device is completed and the patient emerges from the esophagus, the air supply status, water supply status, and intake status can be automatically classified through an image classification process of an artificial neural network model using images acquired via an image sensor, and the air supply unit, water supply unit, and intake unit can be automatically driven based on the classification results.

[0023] The effects of the present invention are not limited to those mentioned above. [Brief explanation of the drawings]

[0024] [Figure 1] 1 is a diagram illustrating an endoscope apparatus according to an embodiment of the present disclosure. [Figure 2] 10A and 10B are diagrams illustrating a control method for an endoscopic device according to an embodiment of the present disclosure. [Figure 3] FIG. 1 is a block diagram of an endoscopic system according to an embodiment of the present disclosure. [Figure 4] FIG. 1 is a conceptual diagram specifically illustrating an endoscope system according to an embodiment of the present disclosure. [Figure 5] FIG. 1 is a conceptual diagram illustrating an artificial neural network model according to one embodiment of the present disclosure. [Figure 6] 10(a) and 10(b) are conceptual diagrams illustrating an artificial neural network model according to another embodiment of the present disclosure. [Figure 7] 10 is a flowchart showing detailed steps in a control method for an endoscopic device according to an embodiment of the present disclosure, in which an artificial intelligence model classifies and outputs images into an air supply image, a water supply image, and an intake image. [Figure 8]FIG. 1 is a conceptual diagram illustrating the application of a threshold value to an image classified by an artificial neural network, according to one embodiment of the present disclosure. [Figure 9] FIG. 10 is a conceptual diagram illustrating a process of checking the degree of agreement between classification results of previous and next images of a classified image according to an embodiment of the present disclosure. [Figure 10] FIG. 10 is a conceptual diagram illustrating a multi-label display on a screen of an endoscope device according to an embodiment of the present disclosure. DETAILED DESCRIPTION OF THE INVENTION

[0025] The terms used in the present invention are merely used to describe specific embodiments and are not intended to limit the scope of other embodiments. A singular expression includes a plural expression unless the context clearly dictates otherwise. Terms used herein, including technical or scientific terms, may have the same meaning as commonly understood by a person of ordinary skill in the technical field described in the present invention. Among the terms used in the present invention, commonly defined terms should be interpreted as meanings that are identical to or similar to the meanings they have in the context of the related art, and should not be interpreted as ideal or overly formal unless explicitly defined in the present invention. In some cases, even if a term is defined in the present invention, it should not be interpreted as excluding embodiments of the present invention.

[0026] Hereinafter, various embodiments will be described in detail with reference to the accompanying drawings so that those skilled in the art can easily implement the present invention. However, the technical concept of the present invention may be embodied in various forms and is not limited to the embodiments described herein. In describing the embodiments disclosed herein, if a detailed description of related known technology is deemed to obscure the gist of the technical concept of the present invention, the detailed description of the known technology will be omitted. Identical or similar elements will be designated by the same reference numerals, and redundant description thereof will be omitted.

[0027] Here, the term "module" used in this embodiment refers to a component that performs a specific function, which is performed by software or hardware such as a field programmable gate array (FPGA) or an application-specific integrated circuit (ASIC). However, the "module" is not limited to being performed by software or hardware. The "module" may exist in the form of data stored on an addressable recording medium, or may be embodied by an instruction word, and configured to cause one or more processors to execute a specific function.

[0028] Software may include computer programs, code, instructions, or a combination of one or more of these, which may configure a processing device to operate as desired or may instruct the processing device, either independently or collectively. The software and / or data may be permanently or temporarily embodied in some type of machine, component, physical device, virtual device, computer storage medium or device, or transmitted signal wave to be interpreted by the processing device or to provide instructions or data to the processing device. The software may also be distributed across network-coupled computer systems, stored or executed in a distributed manner. The software and data may be stored in one or more computer-readable storage media. The software may be read into main memory from other computer-readable media, such as a data storage device, or from another device via a communication interface. The software instructions stored in main memory may cause a processor to perform the processes or steps described in detail below. Alternatively, hard-wired circuitry may be used in place of or in combination with software instructions to implement processes consistent with the principles of the invention. Thus, embodiments consistent with the principles of the invention are not limited to any specific combination of hardware circuitry and software.

[0029] The terms used in this application are merely used to describe specific embodiments and are not intended to limit the present invention. The singular terms include the plural terms unless the context clearly dictates otherwise. In this application, terms such as "comprise" or "have" specify the presence of a specified feature, number, step, operation, component, part, or combination thereof, but should be understood not to preclude the presence or addition of one or more other features, numbers, steps, operations, components, parts, or combinations thereof. Terms such as "first" and "second" may be used to describe various components, but the components are not limited by these terms. These terms are used solely to distinguish one component from another.

[0030] The term "learning model" as used herein includes all types of algorithms or methodologies used to learn or understand specific patterns or structures from data. That is, the term "learning model" includes not only machine learning models such as regression models, decision trees, random forests, support vector machines, k-nearest neighbors, naive phase, and clustering algorithms, but also deep learning models such as neural networks, convolutional neural networks, recurrent neural networks, Transformer-based neural networks, generative adversarial networks (GANs), and autoencoders. A "learning model" refers to a set of learned parameters or weights used to predict or classify an output for a specific input. The model can be trained through methods such as supervised learning, unsupervised learning, semi-supervised learning, and reinforcement learning. It also includes not only single models but also various learning methods and structures such as ensemble models, multimodal models, and models via transfer learning. Such learning models can be pre-trained on a computer device separate from the computer device that predicts the output for the input and used on another computer device.

[0031] FIG. 1 is a diagram illustrating an endoscope apparatus according to an embodiment of the present disclosure.

[0032] 1, an endoscopic device 100 according to an embodiment of the present disclosure is a flexible endoscope, specifically a gastrointestinal endoscope. The endoscopic device 100 includes a configuration for acquiring medical images of the inside of the gastrointestinal tract and, if necessary, a configuration for inserting a tool and performing treatment or therapy while viewing the medical images.

[0033] The endoscope device 100 also includes an output unit 110 , a drive unit 130 , a scope 150 , and a control unit 400 .

[0034] The output unit 110 may include a display that displays medical images, a liquid crystal display (LCD), a thin film transistor-liquid crystal display (TFT LCD), an organic light-emitting diode (OLED), a flexible display, a 3D display, or a display module that outputs visualized information or implements a touch screen.

[0035] The output unit 110 may include various means for providing information related to medical images, i.e., internal body images, such as a speaker for providing audible information related to the internal body images.

[0036] The output unit 110 may display the internal body image acquired by the scope 150 or the internal body image processed by the control unit 400. The output unit 110 may display the internal body image and may further display the results of classification and analysis of the internal body image by the artificial neural network model.

[0037] For example, the output unit 110 may display information indicating that the corresponding frame of the image of the inside of the body requires inhalation, information indicating that inhalation is currently being performed, or the like.

[0038] 1, one output unit 110 is illustrated, but there may be multiple output units 110. In this case, an output unit that displays an internal body image acquired by the scope 150 and an output unit that displays information processed by the control unit 400 may be separated. Also, one output unit may be separated into a first screen that displays an internal body image acquired by the scope 150 and a second screen that displays an internal body image and information processed by the control unit 400.

[0039] The control unit 400 can control the overall operation of the endoscopic device 100. The control unit 400 can also include all types of devices capable of processing data. For example, the control unit 400 can be a data processing device built into hardware having physically structured circuits to perform functions expressed by codes or instructions contained in a program. Examples of data processing devices built into hardware include processing devices such as a microprocessor, a central processing unit (CPU), a processor core, a multiprocessor, an application specific integrated circuit (ASIC), and a field programmable gate array (FPGA), but the technical concept of the present disclosure is not limited thereto.

[0040] The control unit 400 may control the movement of the scope 150 through the driving unit 130 connected to the scope 150. The control unit 400 may perform various control operations for imaging the inside of the body, specifically, the inside of the digestive tract, through the scope 150.

[0041] The control unit 400 may perform various processes on the medical image acquired through the scope 150. The control unit 400 may control the endoscope device 100 based on a control signal received from a computing device. For example, the control unit 400 may open or shut off the suction pump based on an open / close signal for a valve connected to the suction pump.

[0042] The driving unit 130 may provide the power necessary for the scope 150 to be inserted into or move within the body. For example, the driving unit 130 may include a plurality of motors connected to wires inside the scope 150 and a tension adjustment unit that adjusts the tension of the wires. The driving unit 130 may control the power of each of the plurality of motors to control the scope 150 in various directions.

[0043] 1, the control unit 400 and the driving unit 130 are illustrated as separate and distinct pieces of hardware, but are not limited thereto. For example, the control unit 400 and the driving unit 130 may be implemented as a single piece of hardware, or as two or more pieces of hardware. When implemented as two or more pieces of hardware, part of the configuration of the control unit 400 and part of the configuration of the driving unit 130 may be physically separated.

[0044] The scope 150 also includes an insertion section 150a, a bending section 150b, and a tip section 150c. The insertion section 150a, the bending section 150b, and the tip section 150c include at least one of an air pump that injects air into the body through the scope 150, an intake pump that provides negative pressure or vacuum and draws air from the body through the scope 150, and a water pump that injects cleansing water into the body through the scope 150. Each pump also includes a valve for controlling the flow of fluid. The valve of each pump can be opened and closed by the control unit 400.

[0045] According to one embodiment of the present disclosure, the valve of the intake pump may be opened or closed based on a control signal from a computing device or based on control by the control unit 400. In one embodiment, the control unit 400 may open the intake valve based on an intake valve opening time calculated by the computing device. The control unit 400 may close the intake valve based on an intake valve shutoff signal from the computing device or after a time calculated by the computing device has elapsed.

[0046] The scope 150 also includes an insertion section 150a inserted inside the body, specifically inside the digestive tract, and an operation section 160 that receives input from the user to control the movement of the insertion section 150a and perform various treatments inside the digestive tract.

[0047] The insertion section 150a is configured to bend flexibly, and one end of the insertion section 150a is connected to the driving section 130, so that the degree or direction of bending can be determined by the driving section 130. Since imaging of the inside of the body and treatment are performed at the end of the insertion section 150a, the scope 150 also includes various cables and tubes extending to the end of the insertion section 150a.

[0048] A curved portion 150b and a tip portion 150c may be formed at the distal end of the insertion portion 150a. The tip portion 150c may include an image sensor 151 that can acquire images related to the inside of the body. The rotation angle of the tip portion 150c is adjusted by the driver 130 that receives a control signal from the controller 400, and the tip portion 150c can check various parts inside the body, capture images of the inside of the body, and perform treatment inside the body.

[0049] The bending portion 150b may be connected to the distal end portion 150c. The driver 130 may adjust the degree of bending of the bending portion 150b in response to a control signal from the controller 400. The rotation angle of the distal end portion 150c may be adjusted depending on the degree of bending of the bending portion 150c.

[0050] According to this embodiment, when the first bending and steering unit 161 and the second bending and steering unit 162 are operated, a signal is transmitted to the control unit 400 via a signal transmission system attached to the operation unit 160. The signal processed by the control unit 400 is transmitted to the drive unit 130 to drive the motor, thereby controlling the bending unit 150b. The first bending and steering unit 161 can control the up and down movement of the bending unit 150b and the distal end portion 150c, and the second bending and steering unit 162 can control the left and right movement of the bending unit 150b and the distal end portion 150c.

[0051] 1, the distal end 150c of the scope 150 includes an image sensor 151 therein and a light source 153, a lens 154, a first working channel 155, and a second working channel 156 at its end. An image of the inside of the body captured through the lens 154 can be sensed through the image sensor 151. The light source 153 emits light adjacent to the lens 154 to brighten the dark inside of the body, thereby enabling clearer and more precise imaging of the inside of the body through the lens 154.

[0052] During endoscopic surgery, tools for treating and managing lesions can be inserted through first working channel 155. For example, irrigation water can be supplied to the inside of the body through operation unit working channel 163 through first working channel 155. Furthermore, for example, first working channel 155 can suck fluid from the inside of the body, specifically, the inside of the digestive tract, through the valve of the suction pump of suction unit 430 (FIG. 4).

[0053] 1 illustrates a second working channel 156 in addition to the first working channel 155. The second working channel 156 can perform an operation different from that performed in the first working channel 155. For example, when the first working channel 155 performs an operation of aspirating fluid, the second working channel 156 can also perform an operation of injecting air into the body for insufflation.

[0054] The operation unit 160 also includes a plurality of input buttons that provide various functions so that the endoscopist can control the steering of the insertion unit 150a and perform surgery through the working channel 155. The endoscopist can also perform suction inside the digestive tract via the operation unit 160, separately from controlling the computing device.

[0055] The above-described operations of the control unit 400 are performed by the driving unit 130, which may include a necessary processing device such as a microprocessor or a central processing unit (CPU). In this case, the control unit 400 processes the internal body image acquired through the scope 150, and the driving unit 130 controls the endoscope device 100 based on a control signal received from the computing device.

[0056] Fig. 2 is a diagram illustrating a control method for an endoscope device according to an embodiment of the present disclosure. Fig. 3 is a block diagram of an endoscope system according to an embodiment of the present disclosure. Fig. 4 is a conceptual diagram specifically illustrating an endoscope system according to an embodiment of the present disclosure.

[0057] 1 to 4, a method for controlling an endoscope device according to one embodiment of the present disclosure includes a step of acquiring an image of the inside of a body from an image sensor 151 (S100), a step of inputting the image to a trained artificial neural network model 200 for classification of the acquired image (S200), a step of the artificial neural network model 200 classifying the image into an air supply status image, a water supply status image, and / or an intake status image and outputting the image (S300), and a step of controlling the endoscope device 100 so that the air supply unit 410, the water supply unit 420, and / or the intake unit 430 of the endoscope device 100 are driven based on the output classification result (S400).

[0058] The endoscopic device 100 according to one embodiment of the present disclosure may include a hardware device or part of a hardware device that performs comprehensive data processing and calculations, and may also include a software-based computing environment connected to a communication network. For example, the endoscopic device 100 may include a server that performs intensive data processing functions and is an entity that shares resources, and may also include clients that share resources through interaction with the server.

[0059] The endoscope device 100 also includes a cloud system in which multiple servers and clients interact with each other and process data comprehensively. The above description is only one example related to the configuration of the endoscope device 100, and the configuration of the endoscope device 100 may be configured in various ways within the scope that can be understood by those skilled in the art based on the contents of this disclosure.

[0060] Referring to FIG. 3, an endoscope apparatus 100 according to an embodiment of the present disclosure also includes a processor 101, a memory 102, an artificial neural network model 200, an image acquisition unit 300, a control unit 400, and a post-processing unit 500.

[0061] The processor 101 according to an embodiment of the present disclosure may be understood as a component including hardware and / or software for performing computing operations. For example, the processor 101 may read a computer program and perform data processing for machine learning. The processor 101 may process input data for machine learning, feature extraction for machine learning, and error calculation based on backpropagation. The processor 101 for performing such data processing may also include a central processing unit (CPU), a general-purpose graphics processing unit (GPGPU), a tensor processing unit (TPU), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA), etc. The type of the processor 101 described above is merely an example, and the type of the processor 101 may be configured in various ways within the scope of what one skilled in the art would understand based on the contents of the present disclosure.

[0062] The processor 101 may use images of the inside of the body, such as endoscopic images, as training data to train the artificial neural network model 200. Specifically, the processor 101 may train the artificial neural network model 200 to determine in real time whether or not insufflation, water supply, and / or suction is required inside the body based on the endoscopic images of the inside of the body. In this case, the endoscopic images may include the endoscopic images.

[0063] In addition, an image of the inside of the body, i.e., an endoscopic image, can be input to the trained artificial neural network model 200, and the image can be classified into an air supply status image, a water supply status image, and / or an inhalation status image and output, and the endoscopic device can be controlled so that the air supply unit, the water supply unit, and / or the inhalation unit are driven based on the output classification result.

[0064] In this specification, the term "inside the body" refers to organs such as the stomach, duodenum, small intestine, and large intestine, and more specifically, to parts of organs, such as the cardia, gastric angle, and pylorus of the stomach. In the following description, the digestive organs are taken as an example.

[0065] During endoscopic procedures, endoscopists must perform the following operations when entering the esophagus; when the esophagus is constricted, they must press the air button; when a foreign object is attached to the lens 154, they must press the water button; when gastric juice is visible on the lens 154, i.e., the screen, and when the lens is removed from the esophagus after medical treatment, they must press the suction button. This requires operating many buttons for a single endoscopic diagnosis, which can be difficult.

[0066] Specifically, during endoscopic surgery, a certain distance must be maintained between the scope 150 equipped with the lens 154 and the digestive organs. Therefore, medical staff may inject air into the digestive organs during endoscopic surgery to expand the organs. If the amount of air injected is insufficient, lesions located between the folds of the organs or sunken lesions may not be observed. Furthermore, since different amounts of air injected result in different organ shapes and therefore different lesion shapes, proper air injection, determined in real time during endoscopic surgery, is essential for accurate diagnosis. Furthermore, if a foreign object is attached to the lens 154, the accuracy of medical images may be reduced, so air may be sprayed onto the lens to remove the foreign object.

[0067] Furthermore, when performing endoscopic surgery, it is necessary to clean the affected area, which is the lesion and its surrounding area, and it is also necessary to clean the lens 154 in case foreign matter gets on the lens 154 and the image becomes difficult to see. At this time, cleaning water can be sprayed onto the affected area or the lens.

[0068] Furthermore, during endoscopic surgery, a suction operation is frequently performed to suck in and expel liquids or foreign objects present inside the digestive tract to ensure a clear field of view and accurately detect lesions. For example, suction is performed when foreign objects or liquids accumulate around the affected area, making it difficult to obtain an accurate image. Suction is also performed to remove debris generated during a biopsy for surgery or tissue examination. The suction unit 430 appropriately sucks in excessive air or gas when the organ being observed is inflated.

[0069] As such, water supply, air supply, and suction are required in a variety of situations, and according to an embodiment of the present disclosure, the endoscopic device 100 can determine when water supply, air supply, and / or suction are required inside the body based on an image of the inside of the body, and if necessary, control the air supply unit 410, water supply unit 420, and suction unit 430 of the endoscopic device 100.

[0070] For example, the processor 101 may train the artificial neural network model 200 to determine when inspiration is necessary if there is a foreign object near the affected area shown on the internal body image or if the internal body is over-inflated.

[0071] At this time, the processor 101 may train one artificial neural network model 200 that can determine whether the lens 154 that captured the internal body image needs to be cleaned, whether there is a foreign object around the affected area shown in the internal body image, and whether the internal body is over-expanded, or may train multiple artificial neural network models 200 that correspond to each of these cases.

[0072] The processor 101 may train the artificial neural network model 200 through supervised learning, which uses training data as input values. Alternatively, the processor 101 may train the artificial neural network model 200 through unsupervised learning, which finds criteria for data recognition by learning the types of data required for data recognition on its own without any guidance. Alternatively, the processor 101 may train the artificial neural network model 200 through reinforcement learning, which uses feedback related to whether the results of data recognition through learning are correct.

[0073] The artificial neural network model 200 according to the present disclosure may include a convolutional neural network (CNN), but is not limited thereto, and the artificial neural network model 200 according to the present disclosure may also include network models such as a deep neural network (DNN), a recurrent neural network (RNN), a bidirectional recurrent deep neural network (BRDNN), a multilayer perceptron (MLP), and a transformer.

[0074] The processor 101 can input an image of the inside of the body into the trained artificial neural network model 200 and determine, from the image of the inside of the body, when insufflation of air, water, and suction are required inside the digestive tract. The processor 101 can generate a signal for controlling the endoscope device 100 based on the output value of the artificial neural network model 200.

[0075] Specifically, the processor 101 may generate control signals for performing air supply, water supply, and / or suction. The processor 101 may generate control signals for selectively opening and closing the valves of the air supply pump, water supply pump, and / or suction pump connected to the working channels 155, 156 of the scope 150 of the endoscopic device 100 inserted into the digestive tract, so that suction is performed through the working channels 155, 156. The processor 101 may directly generate control signals for controlling the endoscopic device 100 based on the output result of the artificial neural network model 200, or may instruct the endoscopic device 100 to generate control signals.

[0076] For example, the processor 101 may distinguish between cases where the lens used to capture the image of the inside of the body needs to be cleaned, where there is a foreign object around the affected area shown in the image of the inside of the body, and where the inside of the digestive tract is overly distended, and control the valve of the suction pump of the endoscopic device 100.

[0077] The processor 101 may receive the internal body image frame by frame while the endoscopic procedure is being performed, and may determine for each frame whether air insufflation, water insufflation, and / or suction is required. The above operations may be repeated until the endoscopic procedure is completed.

[0078] The memory 102 according to an embodiment of the present disclosure may be understood as a component including hardware and / or software for storing and managing data processed by the endoscope device 100. That is, the memory 102 may store any type of data generated or determined by the processor 101 and any type of data received by a network unit (not shown). For example, the memory 102 may include at least one type of recording medium selected from the group consisting of flash memory, hard disk, multimedia card micro, card-type memory, random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, magnetic disk, and optical disk. The memory 102 may also include a database system for managing data in a predetermined manner. The type of memory 102 described above is merely an example, and the type of memory 102 can be configured in a variety of ways within the scope of what would be understandable to a person skilled in the art based on the contents of this disclosure.

[0079] The memory 102 may structure, organize, and manage data, data combinations, and program code executable by the processor 101, which are necessary for the processor 101 to perform calculations. The memory 102 may also store program code that causes the processor 101 to generate training data.

[0080] According to the present disclosure, the memory 102 may store internal body images captured by the endoscope device 100 and may store control signals generated by the processor 101 based on the internal body images.

[0081] A network unit according to an embodiment of the present disclosure may be understood as a component that transmits and receives data via any known wired or wireless communication system. For example, the network unit may transmit and receive data using a wired or wireless communication system such as a local area network (LAN), wideband code division multiple access (WCDMA), long term evolution (LTE), wireless broadband internet (WiBro), fifth generation mobile communication (5G), ultra wide-band, Zigbee, radio frequency (RF) communication, wireless LAN, wireless fidelity (Wi-Fi), near field communication (NFC), or Bluetooth. The above communication system is merely an example, and various wired or wireless communication systems for transmitting and receiving data by the network unit may be applied in addition to the above examples.

[0082] The network unit may receive data necessary for the processor 101 to perform calculations via wired or wireless communication with any system or any client, etc. The network unit may also transmit data generated through calculations by the processor 101 via wired or wireless communication with any system or any client, etc. For example, the network unit may receive medical images via communication with a medical image storage and transmission system, a cloud server that performs tasks such as medical data standardization, or a separate computing device, etc. The network unit may transmit various data generated through calculations by the processor 101 via communication with the above-mentioned system, server, or separate computing device, etc.

[0083] The processor 101 inputs an image of the inside of the body into the trained artificial neural network model 200, classifies the image into an air supply situation image, a water supply situation image, and / or an inhalation situation image, and outputs the image, and controls the endoscope device so that the air supply unit, water supply unit, and / or inhalation unit are driven based on the output classification result.

[0084] The processor 101 may acquire an image of the inside of the body from the image sensor via the image acquisition unit 300. In addition, the processor 101 transmits a control signal from the control unit 400 to the driving unit 130, and the driving unit 130 may open or close a valve connected to an air pump of the air supply unit 410, open or close a valve connected to a water pump of the water supply unit 420, or open or close a valve connected to an intake pump of the intake unit 430, according to the control signal. The air supply unit 410, the water supply unit 420, and the intake unit 430 are disposed inside the insertion unit 150a of the scope 150, and may supply air, supply water, or inhale air toward the inside of the body based on drive information controlled via the control unit 400.

[0085] The processor 101 may perform post-processing on the images classified into the air supply status image, the water supply status image, and / or the inhalation status image and output via the post-processing unit 500.

[0086] According to the present disclosure, the post-processing unit 500 also includes a threshold value comparison unit 510 (FIG. 4) that applies a threshold value to the classified image, and a consistency check unit 520 (FIG. 4) that checks the consistency of the classified image with the classification results of previous and next images, as described below.

[0087] 4, the image acquiring unit 300 may acquire images of the inside of the body. For example, the image acquiring unit 300 may acquire images of the situation, such as image A showing a narrowed esophagus E, image B showing a foreign body S attached to a lens L, and image C showing gastric juice GJ around the esophagus E.

[0088] The images acquired through the image acquisition unit 300 can be input to the artificial neural network model 200. The artificial neural network model 200 can classify the various input internal body images and classify the images into an air supply status image 301, a water supply status image 302, and / or an inhalation status image 303. Images that do not fall into the above three types of status images can also be classified separately.

[0089] The artificial neural network model 200 can output the three classified situation images. At this time, the output image can be transmitted to one of the air supply unit 410, the water supply unit 420, and the air intake unit 430 of the control unit 400. If the image does not correspond to one of the three situation images, it is not transmitted to the control unit 400.

[0090] For example, assume that the endoscope device 100 located inside the body captures a specific image at a specific time point. At this time, if the image sensor 151 of the endoscope device 100 located inside the body detects image A entering the narrowed esophagus E, the image acquiring unit 300 may acquire image A entering the narrowed esophagus E via the image sensor 151. Image A entering the esophagus E acquired by the image acquiring unit 300 is input to the artificial neural network model 200, which may determine that the image is an air supply status image 301 based on the learned content.

[0091] If the image is determined to be gas supply status image 301, artificial neural network model 200 may output information that the image is gas supply status image 301 to control unit 400. Based on the information transmitted from artificial neural network model 200, control unit 400 may open or close a valve connected to the gas supply pump of gas supply unit 410 via driving unit 130. Gas supply unit 410 injects air supplied from the gas supply pump through working channels 155 and 156, and the air is injected into the narrowed esophagus E, widening the esophagus E, thereby allowing distal end portion 150c of endoscopic device 100 to be easily inserted into the esophagus E.

[0092] According to one embodiment of the present disclosure, in the step (S300) in which the artificial neural network model 200 classifies and outputs an image into an air supply status image, a water supply status image, and / or an inhalation status image, the artificial neural network model 200 can be trained to determine an image in which an organ is constricted when the endoscopic device 100 enters the human body as an air supply status image.

[0093] For example, the artificial neural network model 200 can be trained to determine an image of the esophagus being constricted when the endoscope device 100 enters the esophagus as an image of the insufflation situation.

[0094] Furthermore, according to one embodiment of the present disclosure, in the step (S300) in which the artificial neural network model 200 classifies and outputs an image into an air supply status image, a water supply status image, and / or an inhalation status image, the artificial neural network model 200 may determine that an image in which a foreign object is attached to the lens of the endoscopic device 100 is a water supply status image, an air supply status image, and / or an inhalation status image.

[0095] In order to remove foreign matter from the lens, one or more of water supply, air supply and / or suction may be activated to remove the foreign matter from the lens.

[0096] Furthermore, according to one embodiment of the present disclosure, in the step (S300) in which the artificial neural network model 200 classifies and outputs images into an air supply status image, a water supply status image, and / or an inhalation status image, the artificial neural network model 200 can be trained to determine an image in which the endoscopic device 100 has finished medical treatment and is emerging from an organ, or an image in which a foreign object is confirmed inside an organ, as an inhalation status image.

[0097] For example, the artificial neural network model 200 may be trained to determine an image of the endoscope device 100 completing medical treatment and emerging from the esophagus as an inhalation image. In addition, when a substance that needs to be removed, such as excess gastric juice, digestive matter, mucus, or a lesion, is found inside an organ, the artificial neural network model 200 may be trained to determine an image of the endoscope device 100 confirming a foreign object inside the organ as an inhalation image in order to inhale and remove the substance that needs to be removed from the organ through an inhalation operation.

[0098] Although FIG. 4 shows one artificial neural network model 200 determining whether air insufflation, water insufflation, and / or inspiration is required for various cases, the artificial neural network model 200 may include multiple artificial neural network models that determine each case.

[0099] In this way, by using the artificial neural network model 200 according to the present disclosure, the physical burden on endoscopists can be dramatically reduced and the convenience of endoscopic diagnosis can be significantly improved, compared to the prior art, in which water supply, air supply, and / or air suction operations had to be performed through complex operations, through an automated solution for executing water supply, air supply, and / or air suction operations depending on the situation.

[0100] FIG. 5 is a conceptual diagram illustrating an artificial neural network model according to one embodiment of the present disclosure.

[0101] Referring to FIG. 5, an artificial neural network model 200 according to an embodiment of the present disclosure may also include a convolutional neural network (CNN). An image transmitted from an image acquisition unit 300 may be input to the artificial neural network model 200. At this time, a convolution layer (CONV) 210 of the artificial neural network model 200 may analyze the image through a convolution operation. The convolution layer 210 may operate various layers to extract image features. The extracted features may be passed through a pooling layer 220 to reduce the dimensionality of the image data and improve the computational efficiency of the neural network. The convolution layer 210 and the pooling layer 220 may be configured iteratively to realize image features.

[0102] The features extracted through the above process can be transmitted to the fully connected (FC) layer 230. The fully connected layer 230 can output classification results for the input extracted features, indicating which class (water supply status image, air supply status image, or inhalation status image) each image belongs to. The output results can be transmitted to the control unit 400.

[0103] Figures 6(a) and 6(b) are conceptual diagrams illustrating an artificial neural network model according to another embodiment of the present disclosure. Figure 6(a) is a conceptual diagram illustrating an artificial neural network model according to another embodiment of the present disclosure. Figure 6(b) is a conceptual diagram illustrating an artificial neural network model according to yet another embodiment of the present disclosure.

[0104] 6(a), the artificial neural network model 200' also includes a plurality of artificial neural network models 201' and 202'. In this case, the plurality of artificial neural network models 201' and 202' can classify and output at least one of an air supply situation image, a water supply situation image, and / or an inhalation situation image.

[0105] For example, the first artificial neural network model 201' may classify an input image into an air supply status image and a water supply status image and output the same, and the second artificial neural network model 202' may classify an input image into an inhalation status image and output the same.

[0106] Referring to FIG. 6(b), the artificial neural network model 200″ also includes a plurality of artificial neural network models 201″, 202″, and 203″. In this case, the plurality of artificial neural network models 201″, 202″, and 203″ can classify and output at least one situation image from among an air supply situation image, a water supply situation image, and / or an inhalation situation image.

[0107] For example, the first artificial neural network model 201'' may classify an input image into an air supply status image and output it, the second artificial neural network model 202'' may classify an input image into a water supply status image and output it, and the third artificial neural network model 203'' may classify an input image into an inhalation status image and output it.

[0108] The multiple artificial neural network models according to the present disclosure are not limited to those disclosed in Figures 6(a) and 6(b), and the number of artificial neural network models and the range of situation images classified by each artificial neural network model can be embodied in various ways.

[0109] FIG. 7 is a flowchart showing detailed steps of a step in which an artificial intelligence model classifies and outputs images into an air supply image, a water supply image, and an intake image in a control method for an endoscope device according to an embodiment of the present disclosure.

[0110] Referring to FIG. 7, in a method for controlling an endoscopic device according to one embodiment of the present disclosure, the step (S300) in which the artificial neural network model 200 classifies and outputs an image into an air supply status image, a water supply status image, and / or an inhalation status image also includes the step (S310) of applying a thresholding value to the image classified by the artificial neural network model 200, and the step (S320) of checking the degree of agreement between the classified image and the classification results of previous and subsequent images.

[0111] FIG. 8 is a conceptual diagram illustrating the application of a threshold value to an image classified by an artificial neural network, according to one embodiment of the present disclosure.

[0112] 8, a threshold value may be applied as a post-processing step to improve the classification accuracy of the air supply situation image, water supply situation image, and / or inhalation situation image determined using the artificial neural network model 200. In this case, a threshold value related to a specific criterion may be applied to the classification criteria of the air supply situation image, water supply situation image, and / or inhalation situation image classified through the artificial neural network model 200, and only images that satisfy the threshold value may be determined to correspond to the specific situation image.

[0113] In this case, the critical value may be determined based on data learned from the image of the air supply situation, the image of the water supply situation, and / or the image of the inhalation situation. As shown in Fig. 7, the critical values ​​according to the classification are variously set as a first critical value 301a, a second critical value 301b, and a third critical value 301c, and the critical values ​​may be varied and applied depending on the state of the patient during the endoscopic procedure, so that the water supply control, the air supply control, and the inhalation control may be performed differently depending on the situation.

[0114] FIG. 9 is a conceptual diagram illustrating a process of checking the degree of agreement between classification results of previous and next images of a classified image according to an embodiment of the present disclosure.

[0115] 9, the degree of coincidence may be confirmed as a post-processing step to improve the classification accuracy of the air supply status image, water supply status image, and / or inhalation status image determined using the artificial neural network model 200. That is, the internal body image of the endoscope device is captured frame by frame, and the output coincidence between the images may be confirmed by comparing the image captured in the previous image frame captured immediately before the currently captured frame and the image captured in the next image frame captured immediately after the currently captured frame with the image of the currently captured frame. In this case, the output coincidence confirms the degree of coincidence between the adjacent frame images and the image output after the image corresponding to each frame is input and classified into the artificial neural network, thereby improving the classification accuracy of the images classified through the artificial neural network model 200.

[0116] FIG. 10 is a conceptual diagram illustrating a state in which multiple labels are displayed on a screen of an endoscope device according to an embodiment of the present disclosure.

[0117] The artificial neural network model 200 may receive, as training data, labels associated with endoscopic images. The endoscopic images may be images captured through an endoscope. The labels may include information indicating whether the internal body image requires air insufflation, water insufflation, and / or suction. For example, the state requiring suction may include various states, such as a foreign object near an affected area shown in the internal body image or excessive distension of the digestive tract, and these may be labeled as states requiring suction.

[0118] Also, for example, if the focus of the internal body image is clear or if there is no foreign object around the affected area shown in the internal body image, it may be labeled as a normal state in which inhalation is not required. The labeling work may be performed by medical staff, or a separate artificial neural network model for labeling may be used.

[0119] According to an embodiment of the present disclosure, the step S300 of classifying and outputting the images into an air supply status image, a water supply status image, and / or an inhalation status image by the artificial neural network model 200 may further include a step S330 of labeling the images according to the air supply status image, the water supply status image, and / or the inhalation status image. At this time, the labeled images may be multi-labeled, in which labeling related to information on internal body parts is added.

[0120] 10 exemplarily illustrates an image related to multi-labeling. As illustrated in FIG. 10, when labeling a specific part inside the body, in addition to labeling information related to air supply, water supply, and inhalation, information related to whether the corresponding position is, for example, the esophagus, duodenum, or stomach, may be added to the label and output. That is, when classifying an image, the artificial neural network model 200 may add and determine whether the position inside the body is in the water supply state, the air supply state, or the inhalation state, and transmit the information to the control unit 400.

[0121] The control unit 400 can also determine information related to which organ in the body the distal end portion 150c of the endoscope device is currently located inside, and whether the distal end portion 150c is currently inserted into or removed from the body, thereby enabling more precise control of the level of air supply, water supply, and air intake depending on the position and direction of the distal end portion 150c when controlling the air supply, water supply, and air intake operations of the endoscope device.

[0122] The memory 102 may store a number of application programs to be run, as well as data and instructions for operating the computer device. The memory 102 may be implemented as an internal memory such as a ROM or RAM included in the processor 101, or may be implemented as a memory separate from the processor 101. According to one embodiment, the memory 102 may store a neural network and training data.

[0123] Specifically, the processor 101 may control the operation of the computer device using various programs stored in the memory 102 of the computer device. The processor 101 may also include a CPU, RAM, ROM, a system bus, etc. The processor 101 may be embodied by a single CPU or multiple CPUs (or a digital signal processor (DSP), a system-on-chip (SoC)). In one embodiment, the processor 101 may be embodied by a digital signal processor (DSP) that processes digital signals, a microprocessor, or a time controller (TCON). However, the processor 101 may include, but is not limited to, one or more of a central processing unit (CPU), a microcontroller unit (MCU), a micro processing unit (MPU), a controller, an application processor (AP), a communication processor (CP), or an ARM processor, or may be defined by these terms. The processor 101 may also be implemented as an SoC or LSI (large scale integration) incorporating a processing algorithm, or may be implemented as an FPGA (field programmable gate array).

[0124] The devices described above may be implemented using hardware components, software components, and / or a combination of hardware and software components. For example, the devices and components described herein may be implemented using one or more general-purpose or special-purpose computers, such as a processor, controller, arithmetic logic unit (ALU), digital signal processor, microcomputer, FPGA, programmable logic unit (PLU), microprocessor, or any other device capable of executing and responding to instructions. The processing device may execute an operating system (OS) and one or more software applications running on the operating system (OS). The processing device may also access, store, manipulate, process, and generate data in response to the execution of software. For ease of understanding, the processing device is described as being a single processing element; however, those skilled in the art will recognize that the processing device may include multiple processing elements and / or multiple types of processing elements. For example, the processing unit may include multiple processors, or one processor and a controller. Other processing configurations are also possible, such as parallel processors.

[0125] Software may also include computer programs, code, instructions, or a combination of one or more of these, which may configure or, independently or collectively, instruct a processing device to operate in a desired manner. The software and / or data may be permanently or temporarily embodied in some type of machine, component, physical device, virtual device, computer storage medium or device, or transmitted signal wave to be interpreted by or provide instructions or data to a processing device. The software may also be distributed across network-coupled computer systems, stored or executed in a distributed manner. The software and data may be stored on one or more computer-readable storage media.

[0126] The method according to the present invention may be embodied in the form of program instructions that can be executed by various computer means and stored on a computer-readable medium. The computer-readable medium may include program instructions, data files, data structures, and the like, alone or in combination. The program instructions stored on the medium may be specially designed and constructed for the present invention, or may be well known and available to those skilled in the art of computer software. Examples of computer-readable medium include magnetic media such as hard disks, floppy disks, and magnetic tape; optical media such as CD-ROMs (compact disc read only memories) and DVDs (digital versatile discs), and magneto-optical media such as floptical disks; and hardware devices specially configured to store and execute program instructions, such as ROMs, RAMs, and flash memories. Examples of the program instructions include not only machine language code, such as that produced by a compiler, but also high-level language code that can be executed by a computer using an interpreter, etc. The hardware devices described above may be configured to operate as one or more software modules to perform the operations of the present invention, and vice versa.

[0127] Although the present embodiment has been described above with reference to limited embodiments and drawings, those skilled in the art will appreciate that various modifications and variations may be made from the foregoing description. For example, the described techniques may be performed in a different order than described, and / or the described components, such as systems, structures, devices, and circuits, may be combined or combined in a different manner than described, or may be substituted or replaced by other components or equivalents, and still achieve suitable results. Therefore, other implementations, other embodiments, and equivalents to the claims are intended to be within the scope of the claims. [Explanation of symbols]

[0128] 100 Endoscopic device 101 processors 102 memory 110 Output section 130 Drive unit 150 Scope 151 Image Sensor 153 Light source 154 Lens 155,156 working channels 160 Operation section 200 Artificial Neural Network Model 300 Image Acquisition Department 400 control section 410 Air supply unit 420 Water supply section 430 Intake section 500 Post-processing section 510 Critical Value Comparison Unit 520 Matching confirmation unit

Claims

1. 1. A method for controlling an endoscope apparatus including one or more processors, comprising: acquiring an image of the interior of the body from an image sensor; inputting the acquired image into a trained artificial neural network model for classification of the image; the artificial neural network model classifying the image into an air supply status image, a water supply status image, and / or an inhalation status image and outputting the image; a step of controlling the endoscopic device so that the air supply unit, water supply unit and / or suction unit of the endoscopic device are driven based on the output classification result.

2. the artificial neural network model includes a plurality of artificial neural network models; The method for controlling an endoscope apparatus according to claim 1 , wherein the plurality of artificial neural network models classify and output at least one of the air supply situation image, the water supply situation image, and / or the intake situation image.

3. the artificial neural network model classifies the image into an air supply status image, a water supply status image, and / or an inhalation status image, and outputs the classified image; The control method for an endoscope device according to claim 1, wherein the artificial neural network model is trained to judge an image of an organ being constricted when the endoscope device enters the human body as an image of an air supply situation.

4. the artificial neural network model classifies the image into an air supply status image, a water supply status image, and / or an inhalation status image, and outputs the classified image; The control method for an endoscopic device according to claim 1, wherein the artificial neural network model is trained to judge an image in which a foreign object is attached to the lens of the endoscopic device as a water supply status image, an air supply status image, and / or an air intake status image.

5. the artificial neural network model classifies the image into an air supply status image, a water supply status image, and / or an inhalation status image, and outputs the classified image; The control method for an endoscopic device according to claim 1, wherein the artificial neural network model is trained to determine an image of the endoscopic device exiting an organ after medical treatment has been completed, or an image of a substance inside the organ that needs to be removed being confirmed, as an inhalation situation image.

6. The method for controlling an endoscope apparatus according to claim 1 , wherein the artificial neural network model includes a convolutional neural network (CNN).

7. the artificial neural network model includes a convolutional layer and a fully connected layer; The convolution layer extracts features of the input internal body image through a convolution operation; The control method for an endoscopic device according to claim 6, wherein the fully connected hierarchy ultimately outputs a classification result regarding which of the air supply status image, the water supply status image and / or the inhalation status image the extracted image features correspond to.

8. The step of classifying the image into an air supply status image, a water supply status image, and / or an inhalation status image and outputting the image by the artificial neural network model comprises: The method of claim 1 , further comprising applying a threshold value to the image classified by the artificial neural network model.

9. The step of classifying the image into an air supply status image, a water supply status image, and / or an inhalation status image and outputting the image by the artificial neural network model comprises: The method of claim 1 , further comprising checking a degree of agreement between the classified image and classification results of the front and rear images.

10. The step of classifying the image into an air supply status image, a water supply status image, and / or an inhalation status image and outputting the image by the artificial neural network model comprises: further comprising labeling the images by the air supply status image, the water supply status image, and / or the inhalation status image; 2. The method for controlling an endoscope apparatus according to claim 1, wherein the image is a multi-label image to which labeling relating to information about internal body parts is added.

11. a memory for storing an internal body image captured by the endoscope device; an endoscope device comprising: a processor that inputs an image of the inside of the body into a trained artificial neural network model, classifies the image into an air supply situation image, a water supply situation image, and / or an inhalation situation image, and outputs the image; and controls the endoscope device so that the air supply unit, the water supply unit, and / or the inhalation unit are driven based on the output classification result.

12. The processor: The endoscope apparatus according to claim 11, wherein an image of the inside of the body is acquired from an image sensor via an image acquisition unit.

13. The processor: A control signal from the control unit is transmitted to the drive unit. The endoscope device according to claim 11, wherein the drive unit opens and closes a valve connected to an air pump of the air supply unit, opens and closes a valve connected to a water pump of the water supply unit, or opens and closes a valve connected to an intake pump of the intake unit, based on the control signal.

14. The processor: performing post-processing on the images classified into the air supply status image, the water supply status image, and / or the intake status image and output via a post-processing unit; The post-processing unit includes: a threshold comparator for applying a threshold to the classified image; The endoscope apparatus according to claim 11, further comprising a coincidence checking unit that checks the coincidence of the classified image with the classification result of the front and rear images.

15. A computer program stored on a recording medium for causing a computer to execute the method according to any one of claims 1 to 10.

Images