Methods for analyzing endoscopic videos, computing devices, and computer programs

Abstract

This invention provides a method for analyzing endoscopic video, a computing device, and a program that determine the final action that an endoscopic device should perform based on the endoscopic video. [Solution] A method for analyzing endoscopic video performed by a computing device is disclosed. The method includes the steps of: determining a frame-specific class corresponding to an operation to be performed by the endoscopic device for each frame of an endoscopic video composed of multiple frames; and determining a final class corresponding to a final operation to be performed by the endoscopic device for the endoscopic video based on the frame-specific class, wherein the class includes at least one of a first class for operating the air pump of the endoscopic device, a second class for operating the water pump of the endoscopic device, a third class for operating the suction pump of the endoscopic device, and a fourth class corresponding to classes other than the first to third classes.

Description

Technical Field

[0001] The present disclosure relates to a technique for analyzing endoscopic videos, and specifically, to a method, a computing device, and a computer program for analyzing an endoscopic video that determines a final operation to be performed by an endoscopic device based on the endoscopic video.

Background Art

[0002] An endoscope is a general term for a medical instrument that performs surgery or inserts a scope into the body of a human or animal without dissection to observe the inside of the body. The endoscope inserts a scope into the human body, irradiates light, and visualizes the light reflected on the surface of the inner wall. The types of endoscopes are classified according to the purpose and body part, and broadly classified into a rigid endoscope in which the endoscope tube is formed of metal and a flexible endoscope typified by a digestive endoscope.

[0003] On the other hand, during endoscopic surgery, medical staff use various pumps provided in the endoscopic device to secure the field of view and perform the surgery safely. Exemplarily, an air pump can be used to inject air to expand the digestive tract so that the scope can easily pass through, and to stretch the wrinkles inside the organ to secure the field of view. Alternatively, a water pump can be used to inject water to wash the inside of the organ or the lens of the scope during endoscopic surgery, or to remove obstacles that obstruct the field of view to maintain a clean field of view, and to hydrate the tissue to facilitate incision or biopsy in a specific surgery. Alternatively, a suction pump can be used to inhale foreign substances such as blood, mucus, and water generated during endoscopic surgery, or to enhance the safety of the surgery and prevent postoperative complications.

[0004] Such pumps are frequently used in endoscopic procedures, and medical staff repeatedly perform these actions during the procedure, inevitably leading to accumulated fatigue. Therefore, technology to assist the operation of the endoscopic pump is required. Furthermore, given that medical staff must perform endoscopic procedures in real time while viewing endoscopic video, the technology assisting the operation of the endoscopic pump must provide stable and consistent results. [Prior art documents] [Patent Documents]

[0005] [Patent Document 1] Patent No. 5566340 specification [Overview of the Initiative] [Problems that the invention aims to solve]

[0006] This disclosure aims to solve the problems of the prior art described above and relates to a method for analyzing endoscopic video, a computing device, and a computer program that determine the final action that an endoscopic device should perform based on the endoscopic video.

[0007] However, the technical challenges that this embodiment aims to address are not limited to those mentioned above; other technical challenges may also exist. [Means for solving the problem]

[0008] An embodiment of the present disclosure for achieving the aforementioned problems is disclosed, which provides a method for analyzing endoscopic video using a computing device. The method includes the steps of: determining a frame-specific class corresponding to an operation to be performed by the endoscopic device for each frame of an endoscopic video composed of multiple frames; and determining a final class corresponding to a final operation to be performed by the endoscopic device for the endoscopic video based on the frame-specific class, wherein the class includes at least one of a first class for operating the air pump of the endoscopic device, a second class for operating the water pump of the endoscopic device, a third class for operating the suction pump of the endoscopic device, and a fourth class corresponding to classes other than the first to third classes.

[0009] Alternatively, the step of determining the final class may include a step of determining the final class based on a class determined for at least one previous frame earlier than the first frame or a class determined for at least one subsequent frame later than the first frame.

[0010] Alternatively, the step of determining the final class may include a step of determining the final class based on the types and number of frame-specific classes determined for the plurality of frames.

[0011] Alternatively, the step of determining the frame-specific class may include saving the class determined for at least one previous frame before the class determined for the first frame, and saving the class determined for at least one next frame after the class determined for the first frame.

[0012] Alternatively, the step of determining the final class may include the step of determining the final class to be the first class if the first class is stored in a first consecutive number of instances; the step of determining the final class to be the second class if the second class is stored in a second consecutive number of instances; and the step of determining the final class to be the third class if the third class is stored in a third consecutive number of instances.

[0013] Alternatively, the first or second number may be greater than the third number.

[0014] Alternatively, the process may further include a step of controlling at least one of the air pump, the water pump, and the suction pump to perform the operation corresponding to the final class.

[0015] Alternatively, the step of determining the frame-specific class may further include determining the third class to be an image containing secretions inside the body into which the scope of the endoscope is inserted.

[0016] Alternatively, the step of determining the frame-specific class may further include the step of determining the image corresponding to the case where the scope of the endoscope device contacts the inner wall of the body to be the fourth class.

[0017] Alternatively, the step of determining the frame-specific class may further include a step of determining an image containing noise generated by the operation of the scope to be in the fourth class.

[0018] Alternatively, the noise may include a blurred image or motion blur.

[0019] A computing device for analyzing endoscopic video is disclosed by an embodiment of the present disclosure that addresses the aforementioned issues. The device includes a memory for storing endoscopic video composed of multiple frames, and a processor that determines a frame-specific class corresponding to an operation to be performed by the endoscopic device for each frame, and determines a final class corresponding to the final operation to be performed by the endoscopic device for the endoscopic video based on the frame-specific class, wherein the class includes at least one of a first class for operating the air pump of the endoscopic device, a second class for operating the water pump of the endoscopic device, a third class for operating the suction pump of the endoscopic device, and a fourth class corresponding to classes other than the first to third classes.

[0020] One embodiment of the present disclosure for achieving the aforementioned problems is disclosed, which is a computer program stored on a computer-readable storage medium. The computer program, when executed on one or more processors, performs operations for analyzing an endoscopic video, the operations including determining a frame-specific class corresponding to an operation to be performed by the endoscopic device for each frame of an endoscopic video composed of multiple frames, and determining a final class corresponding to a final operation to be performed by the endoscopic device for the endoscopic video based on the frame-specific class, the classes including at least one of a first class for operating the air pump of the endoscopic device, a second class for operating the water pump of the endoscopic device, a third class for operating the suction pump of the endoscopic device, and a fourth class corresponding to classes other than the first to third classes. [Effects of the Invention]

[0021] According to the embodiments of this disclosure, instability that may occur when using a model that individually judges each class can be prevented by controlling the pump of the endoscope device using the output of a single artificial neural network model.

[0022] Further, according to an embodiment of the present disclosure, the endoscope apparatus can be stably controlled in consideration of the similarity or coincidence degree between classes determined for each frame.

Brief Description of Drawings

[0023] [Figure 1] It is a block diagram of an endoscope system including an endoscope apparatus and a computing apparatus according to an embodiment of the present disclosure. [Figure 2] It is a drawing of an endoscope apparatus according to an embodiment of the present disclosure. [Figure 3] It is an exemplary diagram showing an artificial neural network according to an embodiment of the present disclosure. [Figure 4] It is an exemplary diagram showing an operation method of a computing apparatus according to an embodiment of the present disclosure. [Figure 5] It is a flowchart showing a method by which a computing apparatus controls an endoscope apparatus according to an embodiment of the present disclosure.

Modes for Carrying Out the Invention

[0024] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the accompanying drawings so that those having ordinary knowledge in the technical field of the present disclosure (hereinafter referred to as those skilled in the art) can easily implement them. The embodiments presented in the present disclosure are provided so that those skilled in the art can use or implement the content of the present disclosure. Therefore, various modifications to the embodiments of the present disclosure will be apparent to those skilled in the art. That is, the present disclosure can be embodied in various different forms and is not limited to the following embodiments.

[0025] Throughout the specification of the present disclosure, the same or similar reference numerals refer to the same or similar components. Also, for the purpose of clearly explaining the present disclosure, the reference numerals of the parts not related to the explanation of the present disclosure can be omitted from the drawings. <� <�

[0026] <� As used in this disclosure, the term "or" is intended to mean an implicational "or" rather than an exclusive "or". That is, wherever not otherwise specified or where its meaning is not clear from the context, "X uses A or B" should be understood to mean one of the natural implicational substitutions. For example, wherever not otherwise specified or where its meaning is not clear from the context, "X uses A or B" can be interpreted as X using A, X using B, or X using both A and B.

[0027] The terms "and / or" as used in this disclosure should be understood to include all possible combinations of one or more of the related concepts listed.

[0028] The terms “contains” and / or “contains” as used in this disclosure should be understood to mean the presence of certain features and / or components. However, the terms “contains” and / or “contains” should be understood not to exclude the presence or addition of one or more other features, other components and / or combinations thereof.

[0029] Wherever the context does not clearly indicate otherwise or singular form in this disclosure, singular should generally be interpreted as including "one or more."

[0030] The term “nth (where n is a natural number)” as used in this disclosure can be understood as an expression used to distinguish components of this disclosure from one another based on predetermined criteria such as functional, structural, or ease of explanation. For example, components in this disclosure that perform different functional roles may be distinguished as either a first component or a second component. However, components that are substantially identical within the technical concept of this disclosure but must be distinguished for ease of explanation may also be distinguished as either a first component or a second component.

[0031] On the other hand, the terms "module" or "unit" as used in this disclosure can be understood as referring to an independent functional unit that processes computing resources, such as a computer-related entity, firmware, software or a part thereof, hardware or a part thereof, or a combination of software and hardware. Here, a "module" or "unit" may be a unit composed of a single element, or it may be a unit expressed as a combination or set of multiple elements. For example, as a concept, a "module" or "unit" may refer to a hardware element or set thereof of a computing device, an application program that performs a specific function of software, a processing procedure embodied by the execution of software, or a set of instructions for executing a program. Furthermore, as a broader concept, a "module" or "unit" may refer to the computing device itself that constitutes a system, or an application executed on a computing device. However, the above concepts are merely examples, and the concepts of "module" or "unit" can be defined in various ways to the extent that a person skilled in the art can understand them based on the content of this disclosure.

[0032] As used in this disclosure, the term "model" can be understood as a system embodied using mathematical concepts and language to solve a particular problem, a set of software units to solve a particular problem, or an abstract model of a processing step to solve a particular problem. For example, a neural network "model" can refer to any system embodied as a neural network that has problem-solving capabilities through learning. Here, a neural network can have problem-solving capabilities by optimizing the parameters connecting nodes or neurons through learning. A neural network "model" may include a single neural network or a set of neural networks that are combinations of multiple neural networks.

[0033] As used in this disclosure, the term "image" refers to multidimensional data composed of discrete image elements. In other words, "image" can be understood as a digital representation of an object that can be seen by the human eye. For example, "image" can refer to multidimensional data composed of elements corresponding to pixels in a two-dimensional image. "Image" can also refer to multidimensional data composed of elements corresponding to voxels in a three-dimensional image.

[0034] As used in this disclosure, the term "video" refers to multidimensional data composed of multiple temporally consecutive "images." In other words, "video" can be understood as a term that describes a digital representation that changes over time. For example, "video" refers to data composed of a large number of two-dimensional images (frames) taken at regular time intervals, each of which may be multidimensional data consisting of pixels. "Video" can also include data generated by arranging three-dimensional images continuously along the time axis, in which case each frame may be multidimensional data consisting of voxels.

[0035] The definitions of the terms mentioned above are provided to aid in understanding this disclosure. Therefore, unless the terms mentioned above are explicitly stated to limit the content of this disclosure, it should be noted that the content of this disclosure is not intended to be used to limit the technical ideas.

[0036] Figure 1 is a block diagram of an endoscope system including an endoscope and a computing device according to one embodiment of the present disclosure, and Figure 2 is a configuration diagram of the endoscope device according to one embodiment of the present disclosure.

[0037] Referring to Figure 1, the endoscope system 10 includes an endoscope device 100 that acquires various information, including medical images of the inside of the body, transmits the medical images to a computing device 200, and receives control signals from the computing device 200, and a computing device 200 that acquires medical images from the endoscope device 100 and generates signals for controlling the endoscope device based on the medical images.

[0038] In this specification, the endoscopic device 100 may be a rigid endoscope or a flexible endoscope. A rigid endoscope includes a straight tube made of high-strength metal or plastic, while a flexible endoscope may include a tube made of a flexible material. Such a tube may be called a scope 150. Inside the scope 150 of rigid and flexible endoscopes, optical fiber or lens systems may be provided for transmitting illumination and images. They may also include channels for spraying air or water, channels into which instruments for biopsy are inserted, and so on. The operations and configurations described herein may be for controlling a rigid endoscope or a flexible endoscope. However, the types of endoscopic devices 100 to which this disclosure applies are not limited thereto.

[0039] Furthermore, as used herein, the endoscope device 100 can be used on animals or humans. That is, the body referred to herein may be an animal body or a human body. The configuration of the endoscope device 100 can be modified to take into account the size and anatomical characteristics of the animal or human body, and the operation and configuration described herein can also be appropriately modified according to the characteristics of the body.

[0040] The endoscope device 100 may include a configuration that allows for the acquisition of medical images of the inside of the body, and a configuration that allows for the insertion of instruments and the performance of treatment or procedures while viewing the medical images, if necessary. The endoscope device 100 may also include a control unit 120 that controls the overall operation of the endoscope device 100, a scope 150 that is inserted into the body, and a drive unit 130 that provides the power necessary for the operation of the scope 150.

[0041] Referring to Figure 2, the endoscope device 100 may include an output unit 110, a control unit 120, a drive unit 130, a pump unit 140, and a scope 150, and may further include a light source unit (not shown).

[0042] The output unit 110 may include a display for displaying medical images. The output unit 110 may also include a display module that can output visualized information, such as a liquid crystal display (LCD), thin-film transistor-liquid crystal display (TFTLCD), organic light-emitting diode (OLED), flexible display, or 3D display, or that can implement a touch screen.

[0043] The output unit 110 may include various means for providing medical images or information about medical images. The output unit 110 can display the medical images acquired by the scope 150 as they are, or display the medical images processed by the control unit 120. Alternatively, the output unit 110 can output information received via the computing device 200 to an external source.

[0044] The output unit 110 can provide information via auditory means in addition to visual means, and may include, for example, a speaker that provides an audible alarm for medical images. On the other hand, although one output unit 110 is shown in Figure 2, there may be multiple output units 110. In this case, some of the output units can display medical images acquired by the scope 150, while others can display information processed by the control unit 120 or the computing device 200.

[0045] The control unit 120 can control the overall operation of the endoscope device 100. For example, the control unit 120 can perform operations such as medical image acquisition via the scope 150, processing of acquired medical images, control operations for performing medical operations such as spraying cleaning water and suction, and a series of calculations for controlling the operation of the scope 150.

[0046] In this specification, the control unit 120 can process or modify information acquired by the endoscope device 100 for provision to the computing device 200. The control unit 120 can process or modify information received from the computing device 200 and generate signals for controlling the endoscope device 100. Exemplarily, the control unit 120 can generate a control signal to operate at least one of the air pump, water pump, and suction pump of the endoscope device 100 based on a class determination result generated by the computing device 200.

[0047] The control unit 120 may include any type of device capable of processing data. According to an exemplary embodiment, the control unit 120 may be a hardware-integrated data processing device having physically structured circuitry to perform functions expressed by code or instructions contained within a program. Examples of hardware-integrated data processing devices include, but are not limited to, microprocessors, central processing units (CPUs), processor cores, multiprocessors, ASICs (application-specific integrated circuits), and FPGAs (field programmable gate arrays).

[0048] The control unit 120 can control the operation of the scope 150 via the drive unit 130, which is connected to the scope 150. That is, the control unit 120 can generate control signals to be provided to the drive unit 130 in order to control the operation of the scope 150.

[0049] Exemplary, a series of operations by the endoscope device 100 of this disclosure to control the scope 150 can be performed as follows: The user can input the degree or direction of curvature of the scope 150 via the operating unit 151. The input information is transmitted to the control unit 120, which can process the input information and generate a signal to be provided to the drive unit 130. For example, the control unit 120 can calculate and provide to the drive unit 130 the motor position, angle, angular velocity, etc., corresponding to the degree or direction of curvature set by the user. The drive unit 130 can generate power based on the signal from the control unit 120 and transmit it to the scope 150. Thus, the scope 150 can move or curve in accordance with the value input by the user.

[0050] The drive unit 130 can provide the necessary power as the scope 150 is inserted into the body or as it bends or moves within the body. For example, the drive unit 130 may include a motor connected to a wire inside the scope 150 and a tension adjuster that adjusts the tension of the wire.

[0051] The drive unit 130 can control the scope 150 in various directions by controlling the power of the motors. For example, the motors may consist of multiple motors corresponding to the direction in which the insertion portion 152 at the end of the scope 150 is to bend, or multiple motors corresponding to the wires inside the scope 150. Specifically, the drive unit 130 may include a first motor that determines the x-axis motion of the scope 150 and a second motor that determines the y-axis motion of the scope 150. The drive unit 130 can control the x-axis position, y-axis position, z-axis position, roll, pitch, and yaw values ​​at the end of the scope 150, but the configuration of the drive unit 130 is not limited to these.

[0052] The tension adjustment unit 330 receives power from the motor and can pull the wires inside the scope 150 to generate tension. This allows the scope 150 to bend. The tension adjustment unit 330 can adjust the tension acting on multiple wires inside the scope 150 so that the scope 150 is bent by a determined amount and direction of bending.

[0053] The pump unit 140 may include at least one of the following: an air pump for injecting air into the body through the scope 150; a suction pump for providing negative pressure or vacuum to draw in one of gas, liquid, or foreign matter from the body through the scope 150; and a water pump for injecting cleaning water into the body through the scope 150. Each pump may include a valve for controlling the flow of fluid. The pump unit 140 may be opened and closed by the control unit 120. At least one of the suction pump, water pump, and air pump may be opened and closed by a control signal from the computing device 200 or by the control unit 120.

[0054] The scope 150 may include an insertion section 152 that is inserted into the digestive tract and an operating section 151 that receives input from the user to control the operation of the insertion section 152 and perform various operations.

[0055] The insertion section 152 is configured to be flexible and one end is connected to the drive unit 130, so that the degree or direction of curvature can be determined by the drive unit 130. Since medical imaging and surgery are performed at the end of the insertion section 152, the scope 150 may include a number of cables and tubes extending to the end of the insertion section 152. Inside the scope 150, a light source lens 153, an objective lens 154, a working channel 155, and an air and water channel 156 may be provided. Through the working channel 155, instruments for treating and manipulating lesions during endoscopic procedures may be inserted. Air can be injected and irrigation water can be supplied through the air and water channel 156. At least one of gas, liquid, and foreign matter drawn into the suction pump through the working channel 156 or a separately provided channel may be drawn in. On the other hand, in Figure 2, the air and water channel 156 is shown as a passage for supplying irrigation water, but is not limited to this. For example, the scope 150 may be equipped with a separate water set channel (not shown) inside, and cleaning water can also be supplied through the water set channel.

[0056] On the other hand, in this specification, expressions indicating that the scope 150 is bent by the control unit 120 or the drive unit 130 may mean that at least a portion of the scope 150, for example, the insertion portion 152, is bent.

[0057] The control unit 151 may include a number of input buttons that provide various functions (such as image acquisition and irrigation water spray) to allow the endoscopist to control the maneuvering of the insertion unit 152 and perform the procedure via the working channel 155 and the air and water channels 156. For example, the control unit 151 may include a number of buttons or joystick-type input devices that indicate the direction of the scope 150.

[0058] The light source unit may include a light source that illuminates the inside of the body through the endoscope scope 150. The light source unit may include an illumination device that generates white light, or may include multiple illumination devices that generate light with different wavelength bands. The type of light source, light intensity, white balance, etc., can be set via the light source unit. Alternatively, the aforementioned settings can also be set via the control unit 120. The light generated by the light source unit may be transmitted to the scope 150 via a path such as an optical fiber.

[0059] The computing device 200 can receive information from the endoscope device 100 and generate signals to control the endoscope device 100 based on the received information, and provide these signals to the endoscope device 100.

[0060] On the other hand, although Figure 1 shows the computing device 200 as being located inside the endoscope device 100, the computing device 200 may be located outside the endoscope device 100. In this case, the endoscope device 100 and the computing device 200 can be connected via a wired or wireless network to send and receive data. Furthermore, the endoscope system 10 may be operated in a cloud environment. In this case, the endoscope system 10 may further include the endoscope device 100, the computing device 200, and a cloud server.

[0061] A computing device 200 according to one embodiment of this disclosure may be a hardware device or part of a hardware device that performs comprehensive data processing and calculations, or it may be a software-based computing environment connected via a communication network. For example, the computing device 200 may be a server that performs intensive data processing functions and is the entity that shares resources, or it may be a client that shares resources through interaction with a server. Furthermore, the computing device 200 may be a cloud system that enables multiple servers and clients to interact with each other to comprehensively process data. The above description is merely an example related to the types of computing devices 200, and the types of computing devices 200 can be configured in a variety of ways within the scope that can be understood by a person skilled in the art based on the content of this disclosure.

[0062] Referring to Figure 1, a computing device 200 according to one embodiment of the present disclosure may include a processor 210, memory 220, and a network unit 230. However, since Figure 1 is merely an example, the computing device 200 may include other configurations to embody a computer environment. Furthermore, the computing device 200 may include only a portion of the disclosed configurations.

[0063] A processor 210 according to one embodiment of the present disclosure can be understood as a component unit including hardware and / or software for performing computing operations. For example, the processor 210 can read a computer program and perform data processing for machine learning. The processor 210 can handle computational processes such as processing input data for machine learning, feature extraction for machine learning, and error calculation based on backpropagation. A processor 210 for performing such data processing may include a central processing unit (CPU), a general-purpose graphics processing unit (GPGPU), a tensor processing unit (TPU), an application-specific integrated circuit (ASIC), or a field-programmable gate array (FPGA). The types of processors 210 described above are merely examples, and the types of processors 210 can be configured in a variety of ways that are understandable to those skilled in the art based on the content of the present disclosure.

[0064] According to this disclosure, the processor 210 can generate signals to control the endoscope device 100 based on endoscopic video of the inside of the body acquired via the endoscope device 100. The signals may be signals to activate or deactivate the pumps of the endoscope device 100. Based on the endoscopic image, the processor 210 can generate signals to activate or deactivate at least one of the air pump, water pump, and suction pump. For such operations, the processor 210 can train an artificial neural network model 300 and use the trained artificial neural network model 300.

[0065] In this specification, a single endoscopic image in a series of endoscopic videos acquired sequentially by the endoscope device 100 is referred to as a frame. A group of endoscopic images corresponding to a type of operation of the endoscope device 100 is referred to as a class. Illustratively, the first class may represent a group of endoscopic images in which the air pump of the endoscope device 100 must be operated; the second class may represent a group of endoscopic images in which the water pump of the endoscope device 100 must be operated; and the third class may represent a group of endoscopic images in which the suction pump of the endoscope device 100 must be operated. A fourth class may represent any class that does not fall under the first to third classes.

[0066] In this disclosure, the class of endoscopic images can be categorized by the type of pump of the endoscope device 100, as an example. However, the above criteria are illustrative, and the criteria for class classification may vary depending on the controllable configuration of the endoscope device. That is, the class can be categorized in accordance with the configuration to be operated by the endoscope device.

[0067] The processor 210 can train an artificial neural network model 300 that determines which of the many pumps in the endoscope device 100 should operate based on the endoscope image. The endoscope image can then be input to the trained artificial neural network model 300 to generate signals to operate or stop the pumps. In other words, the trained artificial neural network model 300 can determine the class corresponding to the endoscope image and generate a class determination result.

[0068] The processor 210 can generate training data by labeling endoscopic images into first, second, third, and fourth classes. Using this training data, an artificial neural network model 300 can be trained to infer the actions that the endoscopic device 100 should perform based on the endoscopic images.

[0069] The processor 210 can determine the class corresponding to at least one endoscopic image obtained from the endoscopic device 100, using an artificial neural network model 300 that has been trained to infer the actions that the endoscopic device 100 should perform based on the endoscopic image.

[0070] The processor 210 can determine the final action that the endoscope device 100 should perform on the endoscopic video based on the class result for individual frames. That is, it can determine the final class based on the frame-by-frame class. The endoscope device 100 can be controlled by the final class.

[0071] The processor 210 can continuously determine the class of each frame for each frame that is acquired sequentially. In other words, it does not generate a control signal to control the endoscope device 100 based on the class of a single frame. Instead, it considers how continuously the class of each frame is distributed across a series of endoscope videos. This is to ensure that the endoscope device 100 can operate stably by considering the similarity or degree of agreement between the classes determined for each frame.

[0072] On the other hand, unlike in this disclosure, if multiple models are used to individually determine each class, a situation may arise where an arbitrary endoscopic image is determined to be both a Class 1 and a Class 3 image. If the endoscopic device 100 is controlled using such results, post-processing work will be essential for operational safety, which may reduce the operating speed of the endoscopic device 100, where real-time operation is crucial. Therefore, according to this disclosure, the computing device 200 can control the pumps of the endoscopic device 100 using the results output from a single artificial neural network model 300, thereby preventing a situation where the pumps are used redundantly, which may occur when each class is individually determined.

[0073] A memory 220 according to one embodiment of the present disclosure can be understood as a component unit including hardware and / or software for storing and managing data processed by the computing device 200. That is, the memory 220 can store any form of data generated or determined by the processor 210 and any form of data received by the network unit 230. For example, the memory 220 may include at least one type of storage medium from among flash memory type, hard disk type, multimedia card micro type, card type memory, RAM (random access memory), SRAM (static random access memory), ROM (read-only memory), EEPROM (electrically erasable programmable read-only memory), PROM (programmable read-only memory), magnetic memory, magnetic disk, or optical disk. The memory 220 may also include a database system for controlling and managing data in a predetermined manner. The types of memory 220 described above are merely examples, and the types of memory 220 can be configured in a variety of ways within the scope understandable to those skilled in the art based on the content of the present disclosure.

[0074] Memory 220 can structure and organize the data, data combinations, and program code (code) that the processor 210 can execute, which are necessary for the processor 210 to perform calculations. Memory 220 can also store program code that causes the processor 210 to generate training data.

[0075] Memory 220 can store endoscopic videos acquired from the endoscope device 100. Memory 220 can also store parameters generated during the process of the processor 210 training the artificial neural network model 300.

[0076] Memory 220 can store information about each frame that makes up the endoscopic video used as training data and the corresponding label. For example, memory 220 can store images labeled in Class 1 that require cleaning of the lens of the scope 150 due to blurred image focus or foreign matter adhering to the lens, or images that require operation of the air pump, such as when the observation area formed on the inner wall of the digestive tract is narrow. Memory 220 can store images labeled in Class 2 that require cleaning of the lens of the scope 150 because water or foreign matter is attached to both edges of the lens, or images that require operation of the water pump, such as when there is foreign matter, blood, etc. in the lesion area and the affected area needs to be cleaned. Memory 220 can store images labeled in Class 3 that require operation of the suction pump because secretions are present inside the body into which the scope 150 of the endoscope device 100 is inserted. The memory 220 can store images labeled as Class 4, which correspond to images where the scope 150 of the endoscope device 100 comes into contact with the inner wall of the body, or images that include noise generated by the operation of the scope 150.

[0077] A network unit 230 according to one embodiment of the present disclosure can be understood as a component unit that transmits and receives data via any known wired wireless communication system. For example, the network unit 230 can transmit and receive data using a wired wireless communication system such as a local area network (LAN), wideband code division multiple access (WCDMA), LTE (long term evolution), WiBro (wireless broadband internet), 5th generation mobile communication (5G), ultrawide-band wireless communication (ultrawide-band), ZigBee, radio frequency (RF) communication, wireless LAN (wireless LAN), Wi-Fi (wireless fidelity), near field communication (NFC), or Bluetooth®. The above-mentioned communication systems are merely examples, and a wide variety of wired wireless communication systems for data transmission and reception of the network unit 230 are applicable beyond the examples given above.

[0078] The network unit 230 can receive data necessary for the processor 210 to perform calculations via wired or wireless communication with any system or client. The network unit 230 can also transmit data generated by the processor 210's calculations via wired or wireless communication with any system or client. For example, the network unit 230 can receive data including endoscopic images through communication with a medical image storage and transmission system, a cloud server that performs tasks such as medical data standardization, or an endoscope device 100. The network unit 230 can also transmit various types of data generated by the processor 210's calculations through communication with the aforementioned systems, servers, or the endoscope device 100.

[0079] The network unit 230 can acquire a series of endoscopic videos from the endoscope device 100. Then, once the processor 210 determines the final class for the endoscopic videos and generates control signals to control the endoscope device 100 based on this, the network unit 230 can transmit such control signals to the endoscope device 100.

[0080] Figure 3 is an illustrative diagram showing an artificial neural network model according to one embodiment of the present disclosure.

[0081] Referring to Figure 3, the artificial neural network model 300 can be trained to receive an endoscopic image as input and output a class corresponding to the endoscopic image. Here, the artificial neural network model 300 can perform training and inference operations using the computing device 200 shown in Figure 1. Here, the endoscopic image may represent each frame that makes up the endoscopic video.

[0082] Labeled endoscopic images can be used as training data to train the artificial neural network model 300. The endoscopic images can be labeled into four classes. The first class may be image labels corresponding to situations where it is determined that the air pump of the endoscopic device 100 needs to operate; the second class may be image labels corresponding to situations where it is determined that the water pump of the endoscopic device 100 needs to operate; the third class may be image labels corresponding to situations where it is determined that the suction pump of the endoscopic device 100 needs to operate; and the fourth class may be image labels corresponding to situations where it is determined that none of the air pump, water pump, or suction pump needs to operate.

[0083] The reason for the existence of a fourth class in this disclosure is that if only classes 1 through 3, in which the pump operates, exist, the pump may unintentionally operate unnecessarily during endoscopic procedures, interfering with the medical procedure or endangering safety. In other words, the existence of a fourth class, which corresponds to situations where the pump does not operate, ensures the safety of endoscopic procedures.

[0084] Class 1 may include cases where a foreign object is attached to the lens of the scope 150, causing the image to be blurred; where water or foreign objects are attached to the lens; or where the observation area formed on the inner wall of the digestive tract is narrow. Class 2 may include cases where water or foreign objects are attached to the lens and cleaning of the lens is necessary; or where foreign objects, blood, etc. are present in the lesion area and cleaning of the affected area is necessary. Class 3 may include cases where secretions (e.g., gastric juice) are present in front of the lens of the scope 150, causing the area in contact with the secretions to appear in a different color on the image. Class 4 may include cases where instruments for endoscopic procedures are present on the endoscopic image; where the endoscope scope 150 hits the inner wall of the body; motion blur due to the movement of the endoscope scope 150; or cases where the image is blurred due to lack of focus. Class 4 may include cases that are difficult to classify into Class 1 through Class 3.

[0085] The reason why the endoscope scope 150 being judged as Class 4 is that, for safety reasons, the air pump, water pump, and suction pump should not operate. Furthermore, if the endoscope image is blurred or foreign matter is attached to the lens, it corresponds to Class 1 or Class 2, and if the operation of the scope 150 or the lens itself is out of focus, it may correspond to Class 3.

[0086] On the other hand, the computing device 200 can apply various enhancement techniques to the endoscopic image in order to secure training data. Here, enhancement techniques that do not cause class changes can be used, and enhancement techniques based on cropping, which may cause foreign objects to disappear from the image, or enhancement techniques based on blur, which may generate information that may be confused with foreign objects, can be excluded.

[0087] The artificial neural network model 300 may include a classification model. The classification model may be a lightweight model to ensure real-time performance of endoscopic procedures. For example, the artificial neural network model 300 may include multiple convolutional layers, batch normalization blocks, activation functions, and fully connected layers.

[0088] Specifically, for training the artificial neural network model 300, the computing device 200 can extract features from the endoscopic images included in the training data through the processes of convolution, batch normalization, and activation function calculation. The computing device 200 can then output classes and their probabilities via a fully connected layer. Based on the correct label for one of the four classes included in the training data, the difference between the predicted value of the artificial neural network model 300 and the actual correct label can be calculated using a cross-entropy loss function. Using a backpropagation algorithm, the gradient of each weight with respect to the loss function value can be calculated, and the weights can be updated using gradient descent with the calculated gradients. This process can be repeated to adjust the weights of the loss function.

[0089] Here, the computing device 200 can apply corresponding weights to the first to fourth classes in the process of adjusting the weights of the loss function. Here, the computing device 200 can set the weight for the first or second class labels to be larger than the weight for the third class labels. This is because the image patterns corresponding to the first or second class are much more complex and diverse than the image patterns corresponding to the third class, and therefore, the accuracy of learning is improved by focusing on class 1 or 2 classification.

[0090] On the other hand, the artificial neural network model 300 can also be trained using methods other than the supervised learning method described above, such as semi-supervised learning, which uses training data in which only a portion of the endoscopic image is labeled; unsupervised learning, which learns the structure or pattern of the endoscopic image itself without correct labels for the training data; or self-supervised learning, which learns using labels automatically generated from the endoscopic image.

[0091] The trained artificial neural network model 300 can receive endoscopic image input and output a class corresponding to the endoscopic image. The artificial neural network model 300 can generate the results of class determination for each frame by performing inference on the endoscopic image frames.

[0092] Figure 4 is an illustrative diagram showing how a computing device operates according to one embodiment of the present disclosure.

[0093] Referring to Figures 4(a) through 4(c), the computing device 200 can determine the final class for operating the pump of the endoscope device 100 using the frame-by-frame class results output from the artificial neural network model 300. In other words, the computing device 200 can determine the final class by combining multiple frame-by-frame classes. Therefore, rather than operating based on the frame-by-frame class results for each frame, the endoscope device 100 will operate based on the accumulated judgment results after all the judgment results for multiple frames have been determined. That is, the computing device 200 can make a final decision by considering the degree of agreement or trend among multiple frame-by-frame class results.

[0094] Exemplary, the computing device 200 can store frame-by-frame class results in a memory 220 having a preset size. Here, the computing device 200 can store frame-by-frame class results in the order of the acquired endoscopic images. The memory 220 may consist of a portion of the memory 220 shown in Figure 1.

[0095] For example, memory 220 may have n spaces for storing n frame-specific class results, and the initial state may be that all n spaces are filled with the third class. Memory 220 is configured in a queue form, where information entered earlier is output first, and then information entered later is output later.

[0096] Then, the frame-specific class results output from the artificial neural network model 300 can fill the first space, Q1. And, once all the frame-specific class results for the nth frame have been saved, Qn will store the class result for the most recently input frame, and Q1 may store the class result for the most recent frame.

[0097] Figures 4(a) to 4(c) show the class results stored in memory 220 at a specific point in time. In this case, after a reference unit of time, for example, one frame has passed, the frame-specific class results located in n spaces can move one space to the right. Thus, the class result for the most recent frame may fill the position of Q1.

[0098] The computing device 200 can determine the final class based on the continuity and number of judgment results for each class across n spaces in the memory 220.

[0099] For example, as shown in Figure 4(c), if all n spaces in memory 220 have a reference number, for example, a first number, all of which are allocated to the first class, the computing device 200 can ultimately determine that they are in the first class. Here, the first number may be greater than the third number, which will be described later. For example, the first number may be n. The same determination can be made for the second class. That is, if all n spaces in memory 220 have a second number, all of which are allocated to the second class, the computing device 200 can ultimately determine that they are in the second class. Here, the second number may be greater than the third number, which will be described later.

[0100] For example, as shown in Figure 4(b), if the third class of a reference number, for example, a third number, is arranged consecutively in n spaces of memory 220, the computing device 200 can ultimately determine the third class. Here, the third number may be smaller than n.

[0101] Figure 5 is a flowchart showing how a computing device according to one embodiment of the present disclosure controls an endoscope device.

[0102] Referring to Figure 5, the computing device 200 can determine a frame-specific class corresponding to the operation that the endoscope device 100 should perform for each frame in an endoscope video composed of multiple frames (S110). Here, the class may include at least one of the following: a first class for operating the air pump of the endoscope device 100, a second class for operating the water pump of the endoscope device 100, a third class for operating the suction pump of the endoscope device 100, and a fourth class that does not correspond to the first to third classes.

[0103] The computing device 200 can classify images containing secretions inside the body into which the endoscope scope is inserted as a third class. The computing device 200 can classify images corresponding to the case where the endoscope scope is in contact with the inner wall of the body as a fourth class. The computing device 200 can classify images containing noise generated by the movement of the scope as a fourth class. Here, the noise may include a blurred image or motion blur.

[0104] The computing device 200 can determine the final class corresponding to the final operation that the endoscope should perform on the endoscope video, based on the frame-specific class (S120). The computing device 200 can determine the final class based on the types and number of frame-specific classes determined for multiple frames.

[0105] The computing device 200 can determine the class for each of the frames in a series, including the first frame that constitutes the endoscopic video. For example, the class of the first frame corresponds to one of Q1 through Qn in Figure 4.

[0106] The computing device 200 can then determine the final class based on the class determined for at least one previous frame earlier than the first frame, or the class determined for at least one subsequent frame later than the first frame.

[0107] Here, a class determined for at least one previous frame may be saved before a class determined for the first frame, and a class determined for at least one subsequent frame may be saved after a class determined for the first frame. For example, in Figure 4, if the class for the first frame is saved in Q1, and then a class is determined for the second frame, which is the frame following the first frame, the class for the first frame may be saved in Q2, and the class for the second frame may be saved in Q1.

[0108] The computing device 200 can store the frame-specific class determination results in the memory 220. The computing device 200 can determine the final class based on the types and number of classes stored. Specifically, if the first class is stored consecutively in a first number of instances, the computing device 200 can determine the final class to be the first class. If the second class is stored consecutively in a second number of instances, the computing device 200 can determine the final class to be the second class. And if the third class is stored consecutively in a third number of instances, the computing device 200 can determine the final class to be the third class. Here, the first or second number may be greater than the third number.

[0109] The computing device 200 can control the endoscope device 100 to perform an operation corresponding to the final class. That is, the computing device can control at least one of the air pump, water pump, and suction pump to perform an operation corresponding to the final class. The endoscope device 100 can operate the air pump if the final class is the first class. The endoscope device 100 can operate the water pump if the final class is the second class. The endoscope device 100 can operate the suction pump if the final class is the third class.

[0110] For example, the endoscope device 100 may operate the water pump and then sequentially operate the air pump depending on how long each frame class lasts. That is, depending on the intensity of the cleaning performed, the endoscope device 100 may operate the water pump for a predetermined time and then operate the air pump for a predetermined time.

[0111] For example, if it is determined to be Class 2 within 30 frames, the endoscope device 100 can operate the water pump for a time corresponding to 30 frames, and then operate the air pump for a specific time regardless of the frame-by-frame class result.

[0112] The endoscope device 100 can determine that high-intensity cleaning is necessary because it determines that the water pump still needs to be operated even if the water pump is operated in the second class during a specific frame (e.g., 20 frames). Therefore, the air pump can be operated additionally to improve the cleaning power.

[0113] When controlling the endoscope device 100 on a frame-by-frame basis by generating class results on a frame-by-frame basis, the operation may change from frame to frame. This disclosure controls the endoscope device 100 considering the continuity of multiple frame-by-frame classes for a series of endoscopic videos for safe operation. An example illustrates a situation where the endoscope device 100 is controlled solely by frame-by-frame classes. For example, in an endoscopic video consisting of three frames, if the class of each frame is class 3, class 4, or class 3, the suction pump must be operated during one frame, then the operation of the suction pump must be suspended for the following frame, and then the suction pump must be operated for another frame. In actual endoscopic procedures, the range of endoscopic image changes is large, so the judgment of the artificial neural network model 300 may change rapidly from frame to frame. Therefore, in this disclosure, for consistent and robust operation, the pump of the endoscope device 100 can be controlled when a series of class judgment results are output consecutively.

[0114] The descriptions of this disclosure set forth herein are illustrative, and a person with ordinary skill in the art to which this disclosure pertains will understand that it can be easily adapted to other specific forms without altering the technical idea or essential features of this disclosure. Therefore, the embodiments of this disclosure described above should be understood to be illustrative and not limiting in all respects. For example, each component described as a single type can be implemented in a distributed manner, and similarly, components described as distributed can be implemented in a combined form.

[0115] The scope of this disclosure is determined by the claims set forth below rather than by the detailed description above, and all forms of modification or alteration derived from the meaning, scope, and equivalents of the claims should be interpreted as being included within the scope of this disclosure. [Explanation of Symbols]

[0116] 10 Endoscopy Systems 100 Endoscopes 110 Output section 120 Control Unit 130 Drive unit 140 Pump section 150 Scope 200 computing devices 210 processors 220 memory 230 Network Department 300 Artificial Neural Network Models

Claims

1. A method for analyzing endoscopic video, performed by a computing device including at least one processor, In an endoscopic video composed of multiple frames, the step of determining the frame-specific class corresponding to the action that the endoscopic device should perform for each frame, A step of determining the final class corresponding to the final action that the endoscopic device should perform on the endoscopic video, based on the frame-specific class mentioned above. Includes, The aforementioned classes include at least one of the following: a first class corresponding to a control command for operating the air pump of the endoscope device; a second class corresponding to a control command for operating the water pump of the endoscope device; a third class corresponding to a control command for operating the suction pump of the endoscope device; and a fourth class that falls outside of the first to third classes. method.

2. The step of determining the final class includes determining the final class based on a class determined for at least one earlier frame that is earlier than the first frame or a class determined for at least one subsequent frame that is later than the first frame. The method according to claim 1.

3. The step of determining the final class includes a step of determining the final class based on the type and number of frame-specific classes determined for the plurality of frames. The method according to claim 2.

4. The step of determining the frame-specific class includes saving the class determined for at least one previous frame before the class determined for the first frame, and saving the class determined for at least one next frame after the class determined for the first frame. The method according to claim 3.

5. The stage in determining the final class is, If the first class is stored consecutively for a first number of times, the final class is determined to be the first class. If the second class is stored a second time in succession, the final class is determined to be the second class, If the third class is stored a third time in succession, the final class is determined to be the third class, and the process includes these steps. The method according to claim 4.

6. The first number or the second number is greater than the third number. The method according to claim 5.

7. The further step includes controlling at least one of the air pump, the water pump, and the suction pump to perform the operation corresponding to the final class, The method according to claim 1.

8. The step of determining the frame-specific class includes determining the third class to be an image containing secretions inside the body into which the scope of the endoscope is inserted. The method according to claim 1.

9. The step of determining the frame-specific class includes determining the fourth class to be an image corresponding to the case where the scope of the endoscope device comes into contact with the inner wall of the body. The method according to claim 1.

10. The step of determining the frame-specific class includes determining the image containing noise generated by the operation of the endoscope scope to be in the fourth class. The method according to claim 1.

11. The aforementioned noise includes a blurred image or motion blur. The method according to claim 10.

12. A computing device for analyzing endoscopic videos, A memory for storing endoscopic videos composed of multiple frames, A processor that determines a frame-specific class corresponding to the operation that the endoscope device should perform for each frame, and determines a final class corresponding to the final operation that the endoscope device should perform for the endoscope video based on the frame-specific class, Includes, The aforementioned classes include at least one of the following: a first class corresponding to a control command for operating the air pump of the endoscope device; a second class corresponding to a control command for operating the water pump of the endoscope device; a third class corresponding to a control command for operating the suction pump of the endoscope device; and a fourth class that falls outside of the first to third classes. Device.

13. A computer program stored on a computer-readable storage medium, The aforementioned computer program, when executed on one or more processors, is configured to perform operations for analyzing endoscopic video. The aforementioned operation is, In an endoscopic video composed of multiple frames, the operation of determining the frame-specific class corresponding to the operation that the endoscopic device should perform for each frame, Based on the frame-specific class, the operation of determining the final class corresponding to the final operation that the endoscope device should perform on the endoscope video, Includes, The aforementioned classes include at least one of the following: a first class corresponding to a control command for operating the air pump of the endoscope device; a second class corresponding to a control command for operating the water pump of the endoscope device; a third class corresponding to a control command for operating the suction pump of the endoscope device; and a fourth class that falls outside of the first to third classes. Computer program.

Images