Endoscope video analysis method, computing device, and computer program

Abstract

One embodiment of this disclosure provides an endoscopic video analysis method executed by a computing device including at least one processor. The method includes the following steps: in an endoscopic video consisting of multiple frames, determining a frame-by-frame category corresponding to an action that the endoscopic device needs to perform for each frame; and, based on the frame-by-frame categories, determining a final category corresponding to a final action that the endoscopic device needs to perform for the endoscopic video, the category including at least one of a first category for driving a gas pump operation of the endoscopic device, a second category for driving a water pump operation of the endoscopic device, a third category for driving a suction pump operation of the endoscopic device, and a fourth category not belonging to the first to the third categories.

Description

Technical Field

[0001] This disclosure relates to a technique for analyzing endoscopic videos, specifically, to an endoscopic video analysis method, computing device, and computer program for determining the final actions that an endoscopic device needs to perform based on endoscopic videos. Background Technology

[0002] An endoscope is a general term for medical instruments that allow the insertion of a scope into the human or animal body to observe internal structures without surgery or dissection. An endoscope visualizes the light reflected from the internal surfaces by inserting the scope into the body and shining light into them. Endoscopes can be classified into several types based on their purpose and the body part they are used in. Broadly speaking, they can be divided into rigid endoscopes (with metal scopes) and flexible endoscopes, represented by endoscopes for digestive organs.

[0003] On the other hand, during endoscopic medical procedures, the medical team uses various pumps installed in the endoscope to ensure a clear view and safe operation. For example, an air pump can be used to inflate digestive organs, allowing the endoscope to pass through easily and smoothing out internal organ folds to ensure a clear view. Alternatively, during endoscopic procedures, a water pump can be used to flush organs or the endoscope lens, or remove obstructions to maintain a clear view. In certain procedures, a water pump is used to hydrate tissue for easier incision or biopsy. Furthermore, a suction pump can be used to remove blood, mucus, or other fluids generated during the procedure, improving safety, and to remove foreign bodies to prevent postoperative complications.

[0004] The use of these pumps is so frequent in endoscopic procedures that medical teams inevitably experience fatigue due to the repetitive movements during these procedures. Therefore, a technology to assist in the movement of endoscopic pumps is needed. Furthermore, given that medical teams need to view endoscopic video while performing endoscopic procedures in real time, the technology to assist in the movement of endoscopic pumps needs to provide stable and consistent results.

[0005] Existing technical documents

[0006] Patent documents

[0007] Patent document 0001: Japanese Patent Publication No. 5566340 (June 27, 2014) Summary of the Invention

[0008] This disclosure aims to solve the aforementioned problems of the prior art. The problem to be solved by this disclosure is to provide an endoscopic video analysis method, computing device, and computer program that determines the final actions to be performed by an endoscopic device based on endoscopic video.

[0009] However, the technical problems to be solved in this embodiment are not limited to those mentioned above, and other technical problems may also exist.

[0010] Based on an embodiment of this disclosure for achieving the aforementioned problem, a method for analyzing endoscopic video executed by a computing device is disclosed. The method includes the following steps: in an endoscopic video consisting of multiple frames, determining a frame-by-frame category corresponding to an action that the endoscopic device needs to perform for each frame; and, based on the frame-by-frame category, determining a final category corresponding to a final action that the endoscopic device needs to perform for the endoscopic video, the category including at least one of a first category for driving an air pump operation of the endoscopic device, a second category for driving a water pump operation of the endoscopic device, a third category for driving a suction pump operation of the endoscopic device, and a fourth category not belonging to the first to the third categories.

[0011] Alternatively, the step of determining the final category may include determining the final category based on a category determined for at least one previous frame earlier than the first frame or a category determined for at least one subsequent frame later than the first frame.

[0012] Alternatively, the step of determining the final category may include the following steps: determining the final category based on the type and number of frame-by-frame categories determined for the plurality of frames.

[0013] As an alternative, the step of determining the frame-by-frame category may include the following steps: the category determined for the at least one previous frame is stored earlier than the category determined for the first frame, and the category determined for the at least one subsequent frame is stored later than the category determined for the first frame.

[0014] As an alternative, the step of determining the final category includes the following steps: when the first category is stored continuously and reaches a first quantity, the final category is determined as the first category; when the second category is stored continuously and reaches a second quantity, the final category is determined as the second category; and when the third category is stored continuously and reaches a third quantity, the final category is determined as the third category.

[0015] Alternatively, the first or second quantity may be greater than the third quantity.

[0016] Alternatively, the procedure may include controlling at least one of the gas pump, the water pump, and the suction pump to perform an action corresponding to the final category.

[0017] Alternatively, the step of determining the frame-by-frame category may also include the step of identifying images containing secretions inside the body through which the endoscope tube is inserted as the third category.

[0018] Alternatively, the step of determining the frame-by-frame category may also include the step of identifying images corresponding to situations where the endoscope tube of the endoscope contacts the inner wall of the body as the fourth category.

[0019] Alternatively, the step of determining the frame-by-frame category may also include the step of identifying the image containing noise caused by the lens tube motion as the fourth category.

[0020] Alternatively, the noise may include a focus-blurred image or motion blur.

[0021] According to one embodiment of this disclosure for achieving the aforementioned problem, a computing device for analyzing endoscopic video is disclosed. The device includes: a memory storing an endoscopic video consisting of multiple frames; and a processor determining a frame-by-frame category corresponding to an action to be performed by the endoscopic device for each frame, and based on the frame-by-frame category, determining a final category corresponding to a final action to be performed by the endoscopic device for the endoscopic video, the category including at least one of a first category for driving an air pump operation of the endoscopic device, a second category for driving a water pump operation of the endoscopic device, a third category for driving a suction pump operation of the endoscopic device, and a fourth category not belonging to the first to the third categories.

[0022] According to one embodiment of this disclosure for achieving the aforementioned objectives, a computer program is disclosed, stored in a computer-readable storage medium. When executed on one or more processors, the computer program performs multiple actions for analyzing endoscopic video, the multiple actions including: determining, in the endoscopic video consisting of multiple frames, a frame-by-frame category corresponding to an action that the endoscopic device needs to perform for each frame; and, based on the frame-by-frame category, determining a final category corresponding to a final action that the endoscopic device needs to perform for the endoscopic video, the category including at least one of a first category for driving the gas pump action of the endoscopic device, a second category for driving the water pump action of the endoscopic device, a third category for driving the suction pump action of the endoscopic device, and a fourth category not belonging to the first to the third categories.

[0023] According to the embodiments disclosed herein, the pump of the endoscope device is controlled by the output of an artificial neural network model, thereby preventing instability that may occur when using individual models that judge different categories.

[0024] Furthermore, according to the embodiments disclosed herein, the endoscopic device can be stably controlled by taking into account the similarity or consistency between categories determined frame by frame. Attached Figure Description

[0025] Figure 1 This is a block diagram of an endoscope system including an endoscope device and a computing device, which discloses an embodiment of the present invention.

[0026] Figure 2 This is a configuration diagram of an endoscope device according to one embodiment.

[0027] Figure 3 This is an illustrative diagram showing an artificial neural network model according to an embodiment of the present disclosure.

[0028] Figure 4 This is an illustrative diagram showing an operation method of a computing device according to an embodiment of the present disclosure.

[0029] Figure 5 This is a flowchart illustrating a method by which a computing device controls an endoscope device according to an embodiment of the present disclosure. Detailed Implementation

[0030] The present disclosure will now be described in detail with reference to the accompanying drawings, so that those skilled in the art (hereinafter referred to as "those skilled in the art") can easily implement the embodiments of the present disclosure. The embodiments disclosed herein are intended to enable those skilled in the art to utilize or implement the content of the disclosure. Therefore, various modifications to the embodiments of the present disclosure will be apparent to those skilled in the art. That is, the present disclosure can be implemented in various different forms and is not limited to the embodiments described below.

[0031] Throughout this specification, the same or similar reference numerals denote the same or similar elements. Furthermore, reference numerals for parts unrelated to the description of this disclosure may be omitted from the drawings for clarity.

[0032] The term "or" as used in this disclosure does not imply exclusive "or" but rather implied "or". That is, unless otherwise specified in this disclosure or its meaning is unclear in the context, "X utilizes A or B" should be understood to mean one of the natural implied substitutions. For example, unless otherwise specified in this disclosure or its meaning is unclear in the context, "X utilizes A or B" can be interpreted as one of the following: X utilizes A, X utilizes B, or X utilizes both A and B.

[0033] The term “and / or” as used in this disclosure should be understood to refer to or include all possible combinations of more than one of the related concepts listed.

[0034] The terms “comprising” and / or “including” as used in this disclosure should be understood to mean the presence of a specific feature and / or element. However, the terms “comprising” and / or “including” should be understood to not exclude the presence or addition of more than one other feature, other element, and / or combination thereof.

[0035] Unless otherwise specified in this disclosure or not clearly indicated in the context, the singular should generally be interpreted as including "one or more".

[0036] The term "Nth (where N is a natural number)" as used in this disclosure can be understood as a way of distinguishing the elements of this disclosure from each other according to a presupposed basis such as a functional viewpoint, a structural viewpoint, or for ease of explanation. For example, elements performing different functions in this disclosure can be distinguished as first elements or second elements. However, elements that are substantially the same in technical spirit as those in this disclosure but need to be distinguished for the sake of explanation can also be distinguished as first elements or second elements.

[0037] On the other hand, the terms "module" or "unit" used in this disclosure can be understood as referring to independent functional units that process computing resources, such as computer-related entities, firmware, software or a part thereof, hardware or a part thereof, or a combination of software and hardware. In this case, a "module" or "unit" can be a unit composed of a single element, or a unit represented by a combination or set of multiple elements. For example, as a protocol concept, "module" or "unit" can refer to hardware elements of a computing device or a set thereof, an application program that performs a specific function of software, a procedure implemented by executing software, or a set of instructions for executing a program. Moreover, as a broad concept, "module" or "unit" can refer to the computing device itself that makes up the system, or the application program executed on the computing device. However, the foregoing concepts are merely illustrative, and the concept of "module" or "unit" can be defined in various ways within the scope understandable to those skilled in the art, based on the content of this disclosure.

[0038] As used in this disclosure, the term "model" can be understood as a system implemented using mathematical concepts and language to solve a specific problem, a collection of software units used to solve a specific problem, or an abstract model of a process used to solve a specific problem. For example, a neural network "model" can refer to the entire system implemented by a neural network that has acquired the ability to solve problems through learning. In this case, the neural network acquires the ability to solve problems by learning to optimize the parameters of the connecting nodes or neurons. A neural network "model" can include a single neural network or a collection of neural networks composed of multiple neural networks.

[0039] As used in this disclosure, the term "image" can refer to multidimensional data composed of multiple discrete image elements. That is, the term "image" can be understood as a digital representation of an object that can be seen by the human eye. For example, "image" can refer to multidimensional data in a two-dimensional image composed of multiple elements equivalent to pixels. "Image" can refer to multidimensional data in a three-dimensional image composed of multiple elements equivalent to voxels.

[0040] As used in this disclosure, the term "video" can refer to multidimensional data consisting of multiple "images" that are sequentially transmitted over time. In other words, "video" can be understood as a term referring to a digital representation that changes over time. For example, "video" can refer to data consisting of multiple two-dimensional images (frames) captured at regular time intervals, where each frame can be multidimensional data composed of pixels. Furthermore, "video" can include data generated by sequentially listing three-dimensional images along a time axis, in which case each frame can be multidimensional data composed of voxels.

[0041] The foregoing terms are provided to aid in understanding this disclosure. Therefore, unless explicitly stated otherwise as limiting the scope of this disclosure, they are not used in the sense of limiting the technical spirit of this disclosure.

[0042] Figure 1 This is a block diagram of an endoscope system including an endoscope device and a computing device, according to one embodiment. Figure 2 This is a configuration diagram of an endoscope device according to one embodiment.

[0043] Please refer to Figure 1 The endoscope system 10 includes: an endoscope device 100, which acquires various information including medical images of the body interior, transmits the medical images to a computing device 200, and receives control signals from the computing device 200; and the computing device 200, which acquires medical images from the endoscope device 100 and generates signals for controlling the endoscope device based on the medical images.

[0044] In this specification, the endoscope device 100 can be a rigid endoscope or a flexible endoscope. Rigid endoscopes include a straight tube made of high-strength metal or plastic, while flexible endoscopes may include a tube made of a flexible material. These tubes may be referred to as endoscope tubes 150. The endoscope tubes 150 of both rigid and flexible endoscopes may contain fiber optic or lens systems for illumination and image transmission. Furthermore, they may include channels for spraying air or water, channels for inserting biopsy tools, etc. The operations and configurations described in this disclosure can be used to control either a rigid or flexible endoscope. However, the types of endoscope devices 100 applicable to this disclosure are not limited to these.

[0045] Furthermore, the endoscope device 100 described in this specification can be used on animals or humans. That is, the body referred to in this specification can refer to the body of an animal or a human. The configuration of the endoscope device 100 can be changed according to the body size and anatomical characteristics of the animal or human, and the actions and configuration described in this disclosure can also be appropriately changed according to the characteristics of the body.

[0046] The endoscope device 100 may include elements for acquiring medical images of the inside of the body, and elements for inserting tools to view the medical images while performing treatment or procedures when needed. The endoscope device 100 may include a control unit 120 for controlling the overall movement of the endoscope device 100, a scope 150 inserted into the body, and a drive unit 130 for providing the power required for the movement of the scope 150.

[0047] Please refer to Figure 2 The endoscope device 100 may include an output unit 110, a control unit 120, a drive unit 130, a pump unit 140, and a scope tube 150, and may also include a light source unit (not shown).

[0048] The output unit 110 may include a display for displaying medical images. The output unit 110 may include display modules such as liquid crystal displays (LCDs), thin film transistor liquid crystal displays (TFT LCDs), organic light-emitting diodes (OLEDs), flexible displays, and 3D displays, which are capable of outputting visual information or realizing a touch screen.

[0049] Output unit 110 may include various tools for providing medical images or information about medical images. Output unit 110 can display medical images acquired from endoscope 150 as is or can display medical images processed by control unit 120. Alternatively, output unit 110 can output information received by computing device 200 to an external device.

[0050] The output unit 110 can provide information via auditory means in addition to visual means; for example, it may include a speaker that provides warnings about medical images in an audible manner. On the other hand, although Figure 2 One output unit 110 is shown, but there may be multiple output units 110. In this case, a portion of the output unit displays medical images acquired from the endoscope 150, and a portion may display information processed by the control unit 120 or the computing device 200.

[0051] The control unit 120 can control the overall operation of the endoscope device 100. For example, the control unit 120 can perform medical image acquisition based on the endoscope tube 150, processing of the acquired medical images, control actions for performing medical actions such as cleaning water spraying or suction, and a series of calculations for controlling the movement of the endoscope tube 150.

[0052] In this specification, the control unit 120 processes or manipulates the information acquired by the endoscope device 100 to provide to the computing device 200. The control unit 120 can process or manipulate the information received from the computing device 200 to generate signals for controlling the endoscope device 100. For example, the control unit 120 can generate a control signal based on a category determination result generated by the computing device 200 to drive at least one of the gas pump, water pump, and suction pump of the endoscope device 100 to operate.

[0053] The control unit 120 may include any type of device capable of processing data. According to an exemplary embodiment, the control unit 120 may be a data processing device with physically structured circuitry built into hardware to perform functions represented by code or instructions contained within a program. Examples of data processing devices built into hardware include microprocessors, central processing units (CPUs), processor cores, multiprocessors, application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), and similar processing devices, but the spirit of this disclosure is not limited thereto.

[0054] The control unit 120 can control the movement of the endoscope tube 150 via the drive unit 130 connected to the endoscope tube 150. That is, the control unit 120 generates a control signal to be provided to the drive unit 130 in order to control the movement of the endoscope tube 150. As an example, the endoscope device 100 disclosed herein can perform a series of actions to control the endoscope tube 150 as follows: The user can input the degree or direction of bending of the endoscope tube 150 via the operation unit 151. The input information is transmitted to the control unit 120, which processes the input information and generates a signal to be provided to the drive unit 130. For example, the control unit 120 can calculate the motor position, angle, angular velocity, etc., corresponding to the degree or direction of bending set by the user and provide them to the drive unit 130. The drive unit 130 can generate power based on the signal from the control unit 120 and transmit it to the endoscope tube 150. Thus, the endoscope tube 150 can move or bend according to the value input by the user.

[0055] The drive unit 130 provides the power required for the insertion of the endoscope tube 150 into the body or for bending and moving within the body. For example, the drive unit 130 may include a motor connected to a steel wire inside the endoscope tube 150 and a tension adjustment unit for adjusting the tension of the steel wire.

[0056] The drive unit 130 controls the motor power to control the endoscope tube 150 in various directions. For example, multiple motors may be provided corresponding to the intended bending direction of the insertion portion 152 at the end of the endoscope tube 150. Alternatively, multiple motors may be provided corresponding to the internal wires of the endoscope tube 150. Specifically, the drive unit 130 may include a first motor that determines the x-axis movement of the endoscope tube 150 and a second motor that determines the y-axis movement of the endoscope tube 150. The x-axis position, y-axis position, z-axis position, roll, pitch, and yaw values ​​of the end of the endoscope tube 150 may be determined according to the control of the drive unit 130, but the configuration of the drive unit 130 is not limited to this.

[0057] The tension adjustment unit receives power from the motor and pulls the steel wires inside the lens tube 150 to generate tension, thereby enabling the lens tube 150 to bend. The tension adjustment unit 330 can adjust the tension applied to the multiple steel wires 1000 inside the lens tube 150 to bend the lens tube 150 according to a determined amount and direction of bending.

[0058] Pump unit 140 may include at least one of the following: a gas pump that injects air into the body through endoscope tube 150; a suction pump that provides negative pressure or vacuum to draw at least one of gas, liquid, or foreign matter from the body through endoscope tube 150; and a water pump that injects cleaning water into the body through endoscope tube 150. Each pump may include a valve for controlling the flow of fluid. Pump unit 140 may be turned on or off by control unit 120. At least one of the suction pump, water pump, and gas pump may be turned on or off according to a control signal from computing device 200 or control from control unit 120.

[0059] The endoscope tube 150 may include: an insertion part 152 for insertion into the digestive organs; and an operation unit 151 for receiving input from the user to control the movement of the insertion part 152 and perform various actions.

[0060] The insertion section 152 is configured to be flexibly bent, with one end connected to the drive unit 130 so that the degree or direction of bending can be determined by the drive unit 130. Medical imaging and surgical procedures are performed at the end of the insertion section 152; therefore, the endoscope tube 150 may include multiple cables and tubes extending to the end of the insertion section 152. The endoscope tube 150 may contain a light source lens 153, an objective lens 154, an operating channel 155, and a gas and water channel 156. Instruments used for treating and managing lesions during endoscopic procedures can be inserted through the operating channel 155. Air can be injected or cleaning water supplied through the gas and water channel. At least one of the gas, liquid, or foreign matter sucked in by the suction pump can be aspirated through the operating channel 155 or an additional channel. On the other hand, Figure 2 The gas and water passage 156, which serves as a water supply path for cleaning, is shown, but the invention is not limited to this. As an example, the endoscope tube 150 may have an additional water-jet passage (not shown) through which cleaning water can also be supplied.

[0061] On the other hand, the description in this specification that the lens tube 150 is bent by the control unit 120 or the drive unit 130 may refer to at least a portion of the lens tube 150 being bent, such as the insertion portion 152 being bent.

[0062] The operating unit 151 may include multiple input buttons to provide various functions (such as image capture, spraying cleaning water, etc.) for the endoscopic operator to control the direction of the insertion section 152 and to perform procedures through the operating channel 155 and the gas and water channel 156. For example, the operating unit 151 may include multiple buttons or joystick-shaped input devices that indicate the direction of the endoscope tube 150.

[0063] The light source unit may include a light source that illuminates the body through the endoscope tube 150. The light source unit may include an illumination device that generates white light, or it may include multiple illumination devices that generate light with different wavelengths. The type of light source, light intensity, white balance, etc., can be set through the light source unit. Alternatively, the aforementioned settings can also be set through the control unit 120. The light generated by the light source unit can be transmitted to the endoscope tube 150 via a path such as an optical fiber.

[0064] The computing device 200 can receive information from the endoscope device 100 and generate a signal for controlling the endoscope device 100 based on the received information, and then provide it to the endoscope device 100.

[0065] on the other hand, Figure 1 The computing device 200 is illustrated as being located inside the endoscope device 100, but it can also be located outside the endoscope device 100. In this case, the endoscope device 100 and the computing device 200 can send and receive data via a wired or wireless network connection. Furthermore, the endoscope system 10 can operate in a cloud environment. In this case, the endoscope system 10 may also include the endoscope device 100, the computing device 200, and a cloud server.

[0066] The computing device 200 disclosed in one embodiment may be a hardware device or part of a hardware device that performs comprehensive data processing and computation, or it may be a software-based computing environment connected by a communication network. For example, the computing device 200 may be a server that performs intensive data processing functions and shares resources, or it may be a client that shares resources through interaction with the server. Moreover, the computing device 200 may also be a cloud system that enables comprehensive data processing through the interaction of multiple servers and multiple clients. The foregoing description is merely an example related to the type of computing device 200, and the types of computing devices 200 can be constructed in various ways within the scope of understanding of those skilled in the art based on the content of this disclosure.

[0067] Please refer to Figure 1 The computing device 200 disclosed in one embodiment may include a processor 210, a memory 220, and a network unit 230. However, Figure 1 This is merely an example; the computing device 200 may include other elements for implementing the computing environment. Furthermore, the computing device 200 may also include only a portion of the disclosed elements.

[0068] The processor 210 of one embodiment disclosed herein can be understood as comprising hardware and / or software components for performing computations. For example, the processor 210 can read a computer program and perform data processing for machine learning. The processor 210 can perform computational processes such as processing input data for machine learning, feature extraction for machine learning, and error calculation based on backpropagation. The processor 210 for performing data processing as described above may include a central processing unit (CPU), a general purpose graphics processing unit (GPGPU), a tensor processing unit (TPU), an application-specific integrated circuit (ASIC), or a field-programmable gate array (FPGA), etc. The aforementioned types of processor 210 are merely illustrative, and various configurations of the processor 210 can be made based on the content of this disclosure within the scope understandable to those skilled in the art.

[0069] According to this disclosure, processor 210 can generate signals for controlling endoscope device 100 based on endoscopic video of the body interior acquired by endoscope device 100. These signals may be signals for activating or deactivating pumps in endoscope device 100. Processor 210 can also generate signals based on endoscopic images for activating or deactivating at least one of a gas pump, water pump, or suction pump. For these operations, processor 210 can train and utilize an artificial neural network model 300.

[0070] Hereinafter, this specification will refer to an endoscopic image in a series of endoscopic videos continuously acquired by the endoscope device 100 as a frame. Furthermore, groups of endoscopic images corresponding to different types of operation of the endoscope device 100 will be referred to as categories. For example, a first category may refer to a group of endoscopic images that drive the air pump of the endoscope device 100, a second category may refer to a group of endoscopic images that drive the water pump of the endoscope device 100, and a third category may refer to a group of endoscopic images that drive the suction pump of the endoscope device 100. Moreover, categories not belonging to the first to third categories may be referred to as a fourth category.

[0071] In this disclosure, as an example, endoscopic images can be categorized based on the type of pump in the endoscope device 100. However, this criterion is merely illustrative, and the categorization criteria can vary depending on the controllable elements of the endoscope device. That is, the categories can be categorized according to which elements in the endoscope device are intended to be activated.

[0072] The processor 210 can train an artificial neural network model 300 based on endoscopic images to determine which of the multiple pumps in the endoscope device 100 needs to operate. Furthermore, it can input endoscopic images into the trained artificial neural network model 300 to generate signals that cause the pumps to operate and stop operating. In other words, the trained artificial neural network model 300 can determine the category corresponding to the endoscopic image and generate a category determination result.

[0073] The processor 210 generates training data by labeling the endoscopic images into a first category, a second category, a third category, and a fourth category. Furthermore, the training data can be used to train an artificial neural network model 300 to infer the actions that the endoscope device 100 needs to perform based on the endoscopic images.

[0074] The processor 210 can use an artificial neural network model 300 trained in such a way as to determine the category corresponding to at least one endoscope image obtained from the endoscope device 100, the training being based on the endoscope image to infer the action that the endoscope device 100 needs to perform.

[0075] The processor 210 can determine the final action that the endoscope device 100 needs to perform on the endoscope video based on the category results for individual frames. That is, it can determine the final category based on frame-by-frame category. The endoscope device 100 can be controlled according to the final category.

[0076] The processor 210 can continuously determine the frame-by-frame category for each consecutively acquired frame. That is, it does not generate control signals for controlling the endoscope device 100 based on the category of a single frame. Instead, it considers how the frame-by-frame categories are continuously distributed across a series of endoscopic videos. This is to ensure stable operation of the endoscope device 100 by considering the similarity and even consistency between the categories determined frame by frame.

[0077] On the other hand, unlike this disclosure, using multiple models that make individual judgments for each category could result in a situation where any endoscopic image is classified as belonging to both category one and category three. If this result is used to control the endoscope device 100, post-processing is necessary for operational stability, which reduces the operating speed of the endoscope device 100, whose real-time performance is crucial. Therefore, according to this disclosure, the computing device 200 uses the output of an artificial neural network model 300 to control the pump of the endoscope device 100, thus preventing the situation where the output pump is reused when making individual category judgments.

[0078] The memory 220 disclosed in one embodiment can be understood as a hardware and / or software component that stores and manages the data processed by the computing device 200. That is, the memory 220 can store data of any form generated or determined by the processor 210 and data of any form received by the network unit 230. For example, the memory 220 may include at least one of the following storage media: flash memory, hard disk, multimedia card micro, card memory, random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic storage, magnetic disk, or optical disk. Furthermore, the memory 220 may also include a database system that controls and manages data according to a preset architecture. The aforementioned types of memory 220 are merely illustrative examples, and the types of memory 220 can be defined in various ways within the scope that can be understood by those skilled in the art based on the content of this disclosure.

[0079] The memory 220 can manage the data, combinations of data, and executable code required by the processor 210 during computation in a structured and organized manner. Furthermore, the memory 220 can store the program code that drives the processor 210 to generate training data.

[0080] The memory 220 can store endoscopic videos acquired from the endoscope device 100. Furthermore, the memory 220 can store multiple parameters generated by the processor 210 during the training of the artificial neural network model 300.

[0081] The memory 220 can store information about each frame of the endoscopic video used as training data, along with its corresponding label. For example, the memory 220 can store images labeled as the first category, such as images where the focus is blurred, the lens is contaminated with foreign objects requiring cleaning of the endoscope tube 150, or images where the observation area on the inner wall of a digestive organ is narrow, requiring the action of a gas pump. The memory 220 can store images labeled as the second category, such as images where the two corners of the endoscope tube 150 lens are contaminated with water or foreign objects, requiring cleaning of the lens, or images where foreign objects or blood are present in the lesion area, requiring cleaning of the affected area and thus requiring the action of a water pump. The memory 220 can store images labeled as the third category, such as images where the endoscope tube 150 of the endoscope device 100 is inserted into the body containing secretions, requiring the action of a suction pump. The memory 220 can store images labeled as the fourth category, such as images corresponding to the endoscope tube 150 of the endoscope device 100 contacting the inner wall of the body, or images containing noise caused by the movement of the endoscope tube 150.

[0082] The network unit 230 disclosed in this embodiment can be understood as a component that transmits and receives data through any form of known wired wireless communication system. For example, the network unit 230 can use wired wireless communication systems such as local area network (LAN), wideband code division multiple access (WCDMA), long term evolution (LTE), wireless broadband internet (WiBro), 5G, ultra-wideband, ZigBee, radio frequency (RF) communication, wireless LAN, wireless fidelity, near field communication (NFC), or Bluetooth for data transmission and reception. The aforementioned communication systems are merely examples, and various wired wireless communication systems for data transmission and reception purposes of the network unit 230 can also be applied beyond the aforementioned examples.

[0083] Network unit 230 can receive data required for computation by processor 210 via wired or wireless communication with any system or client. Furthermore, network unit 230 can transmit data generated by the computation of processor 210 via wired or wireless communication with any system or client. For example, network unit 230 can receive data containing endoscopic images via communication with medical image storage and transmission systems, cloud servers performing tasks such as medical data standardization, or endoscope devices 100. Network unit 230 can transmit various types of data generated by the computation of processor 210 via communication with the aforementioned systems, servers, or endoscope devices 100.

[0084] The network unit 230 can acquire a series of endoscopic videos from the endoscope device 100. Furthermore, if the processor 210 determines the final category of the endoscopic videos and generates control signals for controlling the endoscope device 100 based on this, the network unit 230 can transmit these control signals to the endoscope device 100.

[0085] Figure 3 This is an illustrative diagram showing an artificial neural network model according to an embodiment of the present disclosure.

[0086] Please refer to Figure 3 The artificial neural network model 300 can be trained by receiving endoscopic images as input and outputting categories corresponding to the endoscopic images. In this case, the artificial neural network model 300 can be... Figure 1 The computing unit 200 performs training and inference operations. At this time, the endoscopic images can be referred to as the frames that constitute the endoscopic video.

[0087] Labeled endoscopic images can be used as training data for training the artificial neural network model 300. The endoscopic images can be labeled into four categories. The first category can be a label corresponding to an image deemed to require the operation of the air pump of the endoscopic device 100; the second category can be a label corresponding to an image deemed to require the operation of the water pump of the endoscopic device 100; the third category can be a label corresponding to an image deemed to require the operation of the suction pump of the endoscopic device 100; and the fourth category can be a label corresponding to an image deemed not requiring the operation of any of the pumps—air pump, water pump, or suction pump.

[0088] The reason for the existence of the fourth category in this disclosure is that if only the first to third categories of pump operation existed, the pump might unexpectedly operate in an unnecessary manner during endoscopic medical procedures, potentially hindering the procedure or endangering safety. In other words, the existence of the fourth category, corresponding to the situation where the pump does not operate, ensures the stability of endoscopic medical procedures.

[0089] Category 1 may include situations where the lens of endoscope tube 150 is contaminated with foreign objects, causing the image to become blurred; situations where the lens is contaminated with water or foreign objects; or situations where the observation area formed on the inner wall of the digestive organ is narrow. Category 2 may include situations where the lens is contaminated with water or foreign objects, requiring lens cleaning; or situations where there are foreign objects or blood in the lesion area, requiring cleaning of the affected area. Category 3 may include situations where secretions (e.g., gastric juice) are present in front of the lens of endoscope tube 150, causing the area in contact with the secretions to appear as a different color in the image. Category 4 may include situations where the instrument used for endoscopic medical procedures is present in the endoscopic image; situations where the endoscope tube 150 touches the inner wall of the body; or situations where movement of the endoscope tube 150 causes motion blur or misfocus, resulting in blurred images. Category 4 may include situations that are difficult to distinguish into categories 1 through 3.

[0090] The reason for classifying the situation where the endoscope tube 150 touches the inner wall as Category 4 is that, for safety reasons, the air pump, water pump, and suction pump need to be shut down. Moreover, situations where the endoscope image is blurry or the lens is contaminated with foreign objects may fall under Category 1 or Category 2, while situations where the endoscope tube 150 moves or the lens itself is out of focus may fall under Category 3.

[0091] On the other hand, the computing device 200 can apply various enhancement techniques to the endoscopic images to ensure training data. At this time, enhancement techniques that do not cause category changes can be used, and enhancement techniques based on cropping that may cause foreign objects in the image to disappear or blurring that may generate information that is confused with foreign objects can be excluded.

[0092] The artificial neural network model 300 may include a classification model. This classification model can be a lightweight version designed to ensure the real-time performance of endoscopic medical procedures. For example, the artificial neural network model 300 may include multiple convolutional layers, batch normalization blocks, activation functions, and fully connected layers.

[0093] To train the artificial neural network model 300, specifically, the computing device 200 extracts features from the endoscopic images contained in the training data through convolution operations, batch normalization, and activation function calculation. Then, the computing device 200 outputs the category and its corresponding probability through a fully connected layer. Based on the true value label of one of the four categories contained in the training data, the difference between the predicted value of the artificial neural network model 300 and the actual true value label is calculated using the cross-entropy loss function. The gradient of the loss function with respect to each weight is calculated using the backpropagation algorithm, and the weights are updated using gradient descent based on the calculated gradient. The process of repeating the above steps to adjust the weights of the loss function is possible.

[0094] At this time, the computing device 200 can apply corresponding weights to the first to fourth categories when adjusting the weights of the loss function. The computing device 200 can set the weights for the labels of the first or second category to be higher than the weights for the labels of the third category. This is because the image patterns corresponding to the first or second category are far more complex and diverse than those corresponding to the third category; therefore, prioritizing the classification of the first or second category is necessary to improve training accuracy.

[0095] On the other hand, in addition to the aforementioned supervised learning methods, the artificial neural network model 300 can also be trained using semi-supervised learning with training data that only labels a portion of the endoscopic images, unsupervised learning that learns the structure or pattern of the endoscopic images themselves without having true value labels for the training data, or self-supervised learning that uses labels automatically generated from the endoscopic images.

[0096] The trained artificial neural network model 300 can receive endoscopic images as input and output the corresponding category of the endoscopic image. The artificial neural network model 300 infers the category for each frame of the endoscopic image and generates a judgment result for each frame.

[0097] Figure 4 This is an illustrative diagram showing an operation method of a computing device according to an embodiment of the present disclosure.

[0098] Please refer to Figure 4 (a) to Figure 4 (c) The computing device 200 can determine the final category for driving the pump action of the endoscope device 100 using the frame-by-frame category results output by the artificial neural network model 300. That is, the computing device 200 can combine multiple frame-by-frame categories to determine the final category. Therefore, the endoscope device 100 does not perform actions based on the frame-by-frame category results for every frame, but rather performs actions based on the accumulated judgment results after all the judgment results for multiple frames have been determined. That is, the computing device 200 can make a final judgment after considering the consistency or tendency among multiple frame-by-frame category results.

[0099] As an example, the processing unit 200 can store the frame-by-frame classification results into a memory 220 having a preset size. At this time, the processing unit 200 can sequentially store the frame-by-frame classification results according to the order of the acquired endoscopic images. The memory 220 can be... Figure 1 It constitutes a part of the memory 220.

[0100] As an example, memory 220 may have n spaces for storing n frame-by-frame category results, and initially all n spaces can be filled with the third category. Memory 220 is configured as a queue, where the information that is input first is output first, and the information that is input later is output later, in that order.

[0101] Furthermore, the frame-by-frame classification results output by the artificial neural network model 300 can be filled starting from the first space Q1. Moreover, if all the frame-by-frame classification results for the nth frame are stored, Qn can store the classification results for the earliest input frame, and Q1 can store the classification results for the latest frame.

[0102] Figure 4 (a) to Figure 4 (c) can show the category results stored in memory 220 at a specific point in time. In this case, the judgment result after a reference unit of time, for example, the judgment result after 1 frame, may be a frame-by-frame category result located in n spaces, each shifted one grid to the right. In this way, the category result for the most recent frame can be filled into position Q1.

[0103] The arithmetic unit 200 can determine the final category based on the continuity and quantity of the judgment results of each category in the n spaces of the memory 220.

[0104] As an example, like Figure 4 (c) If all the spaces in the n spaces of memory 220 corresponding to a base number (such as a first number) are configured in the first category, then the arithmetic device 200 can be ultimately determined to be in the first category. In this case, the first number can be greater than the third number described later. For example, the first number can be n. The second category can also be determined in the same way as the first category. That is, if all the spaces in the n spaces of memory 220 corresponding to a second number are configured in the second category, then the arithmetic device 200 can be ultimately determined to be in the second category. In this case, the second number can be greater than the third number described later.

[0105] As an example, like Figure 4 If, in (b), the third category, corresponding to a base number (such as a third number), is consecutively arranged in the n spaces of memory 220, the arithmetic device 200 can ultimately be determined as the third category. In this case, the third number can be less than n.

[0106] Figure 5 This is a flowchart illustrating a method by which a computing device controls an endoscope device according to an embodiment of the present disclosure.

[0107] Please refer to Figure 5The computing device 200 can determine, in the endoscopic video consisting of multiple frames, the frame-by-frame category corresponding to the action that the endoscope device 100 needs to perform for each frame (step S110). At this time, the category may include at least one of the following: a first category for driving the gas pump of the endoscope device 100, a second category for driving the water pump of the endoscope device 100, a third category for driving the suction pump of the endoscope device 100, and a fourth category that does not belong to the first to the third categories.

[0108] The processing unit 200 can classify images containing secretions from inside the body through which the endoscope tube is inserted as a third category. The processing unit 200 can classify images corresponding to situations where the endoscope tube is in contact with the body's inner wall as a fourth category. The processing unit 200 can classify images containing noise caused by the movement of the endoscope tube as a fourth category. In this case, the noise may include images with blurred focus or motion blur.

[0109] The processing unit 200 can determine the final category corresponding to the final action that the endoscope device needs to perform on the endoscope video based on frame-by-frame categories (step S120). The processing unit 200 can determine the final category based on the type and number of frame-by-frame categories determined for multiple frames.

[0110] The processing unit 200 is capable of inferring a category for each of a series of frames, including the first frame constituting the endoscopic video. For example, the category of the first frame is equivalent to... Figure 4 One of Q1 to Qn.

[0111] Furthermore, the computing device 200 can determine the final category based on the category determined for at least one previous frame earlier than the first frame or the category determined for at least one subsequent frame later than the first frame.

[0112] At this point, categories determined for at least one previous frame can be stored before categories determined for the first frame, and categories determined for at least one subsequent frame can be stored later than categories determined for the first frame. For example, Figure 4 After the category of the first frame is stored in Q1, if the category of the subsequent second frame is determined, the category of the first frame is stored in Q2, and the category of the second frame can be stored in Q1.

[0113] The arithmetic unit 200 can store the frame-by-frame category determination results into the memory 220. The arithmetic unit 200 can determine the final category based on the types and quantities of the stored categories. Specifically, the arithmetic unit 200 can determine the final category as the first category when a first category is stored consecutively to a first quantity. The arithmetic unit 200 can determine the final category as the second category when a second category is stored consecutively to a second quantity. Furthermore, the arithmetic unit 200 can determine the final category as the third category when a third category is stored consecutively to a third quantity. In this case, the first or second quantity may be greater than the third quantity.

[0114] The computing unit 200 can control the endoscope device 100 to perform actions corresponding to the final category. That is, the computing unit can control at least one of the gas pump, water pump, and suction pump to perform actions corresponding to the final category. The endoscope device 100 can activate the gas pump when the final category is a first category. The endoscope device 100 can activate the water pump when the final category is a second category. The endoscope device 100 can activate the suction pump when the final category is a third category.

[0115] As an example, the endoscope device 100 can also drive the water pump and then the air pump sequentially based on the frame-by-frame category and the duration of each frame. That is, the endoscope device 100 can, based on the intensity of the cleaning to be performed, allow the water pump to operate for a preset duration and then stop, and then allow the air pump to operate for a preset duration.

[0116] For example, if it is judged to be in the second category during 30 frames, the endoscope device 100 can drive the water pump to operate for the equivalent of 30 frames, and then allow the gas pump to operate for a certain duration without being affected by the frame-by-frame category result.

[0117] Even if the endoscope device 100 drives the water pump in the second category during a specific frame (e.g., 20 frames), it may be determined that the water pump still needs to operate, and therefore it may be determined that high-intensity cleaning is required. Therefore, the gas pump may be driven to operate further in order to increase the cleaning force.

[0118] Because if the endoscopic device 100 is controlled frame-by-frame based on the generation of category results, the action can change frame-by-frame. To achieve robust action, this disclosure considers controlling the endoscopic device 100 continuously across multiple frame-by-frame categories in a series of endoscopic videos. The following example illustrates the case of controlling the endoscopic device 100 solely based on frame-by-frame category. For example, for an endoscopic video consisting of three frames, where each frame is a third, fourth, and third category, the following procedure is required: drive the suction pump for one frame, then stop the suction pump for one frame, and then drive the suction pump again for one frame. In actual endoscopic medical operations, the endoscopic images vary greatly, causing the artificial neural network model 300's judgments to change rapidly frame-by-frame. Therefore, to achieve consistent and robust action, this disclosure allows control of the pump in the endoscopic device 100 when a certain number of consecutive category judgments are output.

[0119] The foregoing description of this disclosure is illustrative, and those skilled in the art to which this disclosure pertains should understand that it can be easily modified into other specific embodiments without altering the technical spirit or essential features of this disclosure. Therefore, the embodiments described above are merely illustrative and not limiting in all respects. For example, the elements described as a single integral form can also be implemented separately, and similarly, the elements described as separate can also be implemented in a combined form.

[0120] The scope of this disclosure is not defined by the foregoing detailed description but by the claims. All modifications and variations derived from the meaning and scope of the claims and their equivalents shall be interpreted as falling within the scope of this disclosure.

Claims

1. An endoscopic video analysis method, executed by a computing device including at least one processor, characterized in that, Includes the following steps: In an endoscopic video consisting of multiple frames, determine the frame-by-frame category corresponding to the action that the endoscopic device needs to perform for each frame; and, Based on the frame-by-frame categories, determine the final category corresponding to the final action that the endoscopic device needs to perform on the endoscopic video. The categories include a first category for driving the gas pump of the endoscope device, a second category for driving the water pump of the endoscope device, a third category for driving the suction pump of the endoscope device, and at least one of a fourth category that does not belong to the first to the third categories.

2. The method according to claim 1, characterized in that, The step of determining the final category includes the following steps. The final category is determined based on the category determined for at least one previous frame earlier than the first frame or the category determined for at least one subsequent frame later than the first frame.

3. The method according to claim 2, characterized in that, The step of determining the final category includes the following steps. The final category is determined based on the type and number of frame-by-frame categories identified for the multiple frames.

4. The method according to claim 3, characterized in that, The step of determining the frame-by-frame category includes the following steps. The category determined for the at least one previous frame is stored earlier than the category determined for the first frame, and the category determined for the at least one subsequent frame is stored later than the category determined for the first frame.

5. The method according to claim 4, characterized in that, The steps for determining the final category include the following steps. When the first category is stored continuously to reach a first quantity, the final category is determined as the first category; When the second category is stored continuously to reach a second quantity, the final category is determined as the second category; and, When the third category is continuously stored to reach a third quantity, the final category is determined as the third category.

6. The method according to claim 5, characterized in that, The first quantity or the second quantity is greater than the third quantity.

7. The method according to claim 1, characterized in that, It also includes the step of controlling at least one of the gas pump, the water pump and the suction pump to perform an action corresponding to the final category.

8. The method according to claim 1, characterized in that, The step of determining the frame-by-frame category includes the following steps. Images of secretions inserted into the body through the endoscope tube are identified as belonging to the third category.

9. The method according to claim 1, characterized in that, The step of determining the frame-by-frame category includes the following steps. Images representing situations where the endoscope tube of the endoscopic device contacts the inner wall of the body are identified as the fourth category.

10. The method according to claim 1, characterized in that, The step of determining the frame-by-frame category includes the following step: identifying images containing noise caused by the movement of the lens tube as the fourth category.

11. The method according to claim 10, characterized in that, The noise includes images with blurred focus or motion blur.

12. A computing device for analyzing endoscopic video, characterized in that, include: Memory, storing endoscopic video consisting of multiple frames; and The processor determines the frame-by-frame category corresponding to the action that the endoscope device needs to perform for each frame, and based on the frame-by-frame category, determines the final category corresponding to the final action that the endoscope device needs to perform for the endoscope video. The categories include a first category for driving the gas pump of the endoscope device, a second category for driving the water pump of the endoscope device, a third category for driving the suction pump of the endoscope device, and at least one of a fourth category that does not belong to the first to the third categories.

13. A computer program stored in a computer-readable storage medium, which, when executed on one or more processors, performs a plurality of actions for analyzing endoscopic video. The plurality of actions includes the following actions: In an endoscopic video consisting of multiple frames, determine the frame-by-frame category corresponding to the action that the endoscopic device needs to perform for each frame; and, Based on the frame-by-frame categories, determine the final category corresponding to the final action that the endoscopic device needs to perform on the endoscopic video. The categories include a first category for driving the gas pump of the endoscope device, a second category for driving the water pump of the endoscope device, a third category for driving the suction pump of the endoscope device, and at least one of a fourth category that does not belong to the first to the third categories.

Images