Method for generating training datasets based on endoscopic images, computer program, and computing device.

Abstract

This invention provides a method for generating training datasets based on endoscopic images taken by a computing device containing at least one processor. [Solution] The method includes the steps of: inputting raw data into multiple models trained to infer useful information about endoscopic images taken inside the body (S110); generating final useful information for the raw data based on the useful information output from each model (S120); and generating a training dataset including the raw data and the final useful information (S130). The final useful information is information necessary to train a model for analyzing endoscopic images using the training dataset, and includes labels for the raw data or additional information about the labels.

Description

Technical Field

[0001] The present disclosure relates to a technique for generating a learning dataset, and specifically, to a method, a computer program, and a computing device for generating a learning dataset based on endoscopic images.

Background Art

[0002] [[ID=V11]] An endoscope is a general term for a medical instrument that performs surgery or observes organs by inserting a scope into the body without dissection. An endoscope inserts a scope into the inside of the body of an animal or a human, irradiates light, and visualizes the light reflected on the surface of the inner wall. The types of endoscopes are classified according to the purpose and the body part, and roughly classified into a rigid endoscope in which the endoscope tube is formed of metal and a flexible endoscope typified by a gastrointestinal endoscope.

[0003] On the other hand, in order to improve the accuracy and efficiency of inspections in the field of endoscopes, various artificial neural network models are being utilized. Exemplarily, an artificial neural network model that detects polyps in real time during a colonoscopy, classifies whether it is positive or malignant, or automatically detects early lesions of cancer is being utilized. There is also an artificial neural network model for converting a low-resolution endoscopic image into a high resolution or removing various noises generated in the image to improve the image quality.

[0004] Thus, in order to utilize an artificial neural network model, first, the artificial neural network model must be learned. That is, the work of preparing learning data must be performed first. In particular, in the medical field such as the endoscope field, high-quality learning data is very important for ensuring the accuracy and reliability of the model.

[0005] However, acquiring high-quality training data in the medical field presents various challenges. Medical data contains sensitive information such as patients' health status and treatment records, making the data collection process itself complex. Furthermore, artificial neural network models must learn from data on diverse patient populations, requiring the acquisition of data with diverse characteristics such as gender, age, and medical history. Therefore, technologies are needed to acquire large amounts of training data while ensuring accuracy in the medical field, particularly in the field of endoscopy. [Prior art documents] [Patent Documents]

[0006] [Patent Document 1] Korean Published Patent Publication No. 10-2016-0035888 [Overview of the project] [Problems that the invention aims to solve]

[0007] This disclosure aims to solve the problems of the prior art described above, and relates to a method, computer program, and computing device for generating a training dataset based on endoscopic images, by generating labels or additional information about labels using raw data of endoscopic images.

[0008] However, the technical challenges that this embodiment aims to address are not limited to those mentioned above; other technical challenges may also exist. [Means for solving the problem]

[0009] One embodiment of the present disclosure for achieving the aforementioned problems is disclosed, which is a method for generating a training dataset based on endoscopic images using a computing device. The method includes the steps of: inputting raw data into a plurality of models trained to infer valid information about endoscopic images taken inside the body; generating final valid information about the raw data based on the valid information output from each model; and generating a training dataset including the raw data and the final valid information, wherein the final valid information is information necessary to train a model for analyzing the endoscopic images using the training dataset, and includes labels for the raw data or additional information about the labels.

[0010] In one embodiment, the model for analyzing the endoscopic image includes a model for detecting a target in the endoscopic image, the label includes whether the raw data contains the target, and additional information about the label may include confidence in the label or information about the location of the target.

[0011] In one embodiment, the target may include at least one of a lesion, body fluid, foreign body, endoscopic instrument, and a predefined region of interest.

[0012] In one embodiment, the model for analyzing the endoscopic image includes a model that predicts the actions that the endoscopic device must perform based on the endoscopic image, the label includes the type of action, and additional information about the label may include a confidence level for the label.

[0013] In one embodiment, the label may include a class indicating the movement of the scope of the endoscope, including at least one of up, down, left, right, forward, and backward.

[0014] In one embodiment, the label may include a class of operations that at least one of the air pump, suction pump, and water pump of the endoscope device should perform.

[0015] In one embodiment, the plurality of models may include a first model and a second model that detect targets included in the raw data and output information about the region containing the targets.

[0016] As one embodiment, the step of generating the final valid information may include the steps of calculating a score and a third region based on the first region and the confidence level for the first region output from the first model, and the second region and the confidence level for the second region output from the second model, and generating the final valid information based on the third region based on the result of comparing the score with a reference value.

[0017] In one embodiment, the plurality of models may include a first model that detects targets included in the raw data and outputs information about the region containing the targets, and a second model that outputs whether the raw data contains the targets.

[0018] In one embodiment, the step of generating the final valid information can be performed based on the first region and the confidence level for the first region output from the first model, and the class and the confidence level for the class output from the second model.

[0019] In one embodiment, the plurality of models may include a first model that detects targets included in the raw data and outputs information about the region containing the target, a second model that outputs whether the raw data contains the target, and a third model that outputs the pixel-specific class that constitutes the raw data.

[0020] In one embodiment, the step of generating the final valid information may include generating the final valid information based on the first region and confidence level for the first region output from the first model, the class and confidence level for the class output from the second model, and the pixel-specific class output from the third model.

[0021] As one embodiment, the step of generating the final valid information may include comparing the distribution values ​​of the pixel-specific classes within the first region with a first reference value, comparing the confidence level for the classes with a second reference value, and generating the final valid information based on the results of each comparison.

[0022] One embodiment of the present disclosure for achieving the aforementioned problems discloses a computer program stored on a computer-readable storage medium. When the computer program is executed on one or more processors, it performs an operation to generate a training dataset based on endoscopic images. The operation includes inputting raw data into a plurality of models trained to infer valid information about endoscopic images taken inside the body; generating final valid information for the raw data based on the valid information output from each model; and generating a training dataset containing the raw data and the final valid information, wherein the final valid information is information necessary to train a model for analyzing endoscopic images using the training dataset, and may include labels for the raw data or additional information about the labels.

[0023] According to an embodiment of the present disclosure for realizing the above-described problems, a computing device for generating a learning dataset based on an endoscopic image is disclosed. The device includes a memory for storing raw data for an endoscopic image obtained by imaging the inside of a body, and inputs the raw data into a plurality of models trained to infer valid information about the endoscopic image, generates final valid information for the raw data based on the valid information output from each model, and includes a processor for generating a learning dataset including the raw data and the final valid information, where the final valid information is information necessary for training a model for analyzing an endoscopic image using the learning dataset, and may include a label for the raw data or additional information about the label.

Advantages of the Invention

[0024] According to an embodiment of the present disclosure, by using a plurality of neural network models that output different characteristics, valid information about lesions and the like included in an endoscopic image is output, such valid information is reflected in each model, and label information for the endoscopic image can be generated using the final valid information. As a result, the accuracy of the generated label can be improved, and a large amount of high-quality learning data can be generated easily.

Brief Description of the Drawings

[0025] [Figure 1] It is a block diagram of an endoscopic system including an endoscopic device and a computing device according to an embodiment of the present disclosure. [Figure 2] It is a configuration diagram showing an artificial neural network model according to an embodiment of the present disclosure. [Figure 3] It is an exemplary diagram of an artificial neural network model according to an embodiment of the present disclosure. [Figure 4] It is an explanatory diagram showing an operation of calculating final valid information according to an embodiment of the present disclosure. [Figure 5] It is an exemplary diagram of an artificial neural network model according to an embodiment of the present disclosure. [Figure 6] This is an illustrative diagram of an artificial neural network model according to one embodiment of the present disclosure. [Figure 7] This flowchart shows the operation of a computing device according to one embodiment of the present disclosure. [Modes for carrying out the invention]

[0026] Hereinafter, embodiments of the Disclosure will be described in detail with reference to the accompanying drawings so that they can be easily implemented by a person skilled in the art (hereinafter referred to as "a person skilled in the art"). The embodiments presented in this Disclosure are provided so that a person skilled in the art can use or implement the contents of the Disclosure. Thus, various modifications of the embodiments of the Disclosure will be obvious to a person skilled in the art. That is, the Disclosure can be embodied in a variety of different forms and is not limited to the embodiments described below.

[0027] Throughout the specification of this disclosure, identical or similar reference numerals in the drawings refer to identical or similar components. Furthermore, in order to clearly explain this disclosure, reference numerals in the drawings that are not relevant to the description of this disclosure may be omitted.

[0028] As used in this disclosure, the term "or" is intended to mean an implicational "or" rather than an exclusive "or". That is, wherever not otherwise specified or where its meaning is not clear from the context, "X uses A or B" should be understood to mean one of the natural implicational substitutions. For example, wherever not otherwise specified or where its meaning is not clear from the context, "X uses A or B" can be interpreted as X using A, X using B, or X using both A and B.

[0029] The terms "and / or" as used in this disclosure should be understood to include all possible combinations of one or more of the related concepts listed.

[0030] The terms “contains” and / or “contains” as used in this disclosure should be understood to mean the presence of certain features and / or components. However, the terms “contains” and / or “contains” should be understood not to exclude the presence or addition of one or more other features, other components and / or combinations thereof.

[0031] Wherever the context does not clearly indicate otherwise or singular form in this disclosure, singular should generally be interpreted as including "one or more."

[0032] The term “nth (where n is a natural number)” as used in this disclosure can be understood as an expression used to distinguish components of this disclosure from one another based on predetermined criteria such as functional, structural, or ease of explanation. For example, components in this disclosure that perform different functional roles may be distinguished as either a first component or a second component. However, components that are substantially identical within the technical concept of this disclosure but must be distinguished for ease of explanation may also be distinguished as either a first component or a second component.

[0033] On the other hand, the terms "module" or "unit" as used in this disclosure can be understood as referring to an independent functional unit that processes computing resources, such as a computer-related entity, firmware, software or a part thereof, hardware or a part thereof, or a combination of software and hardware. Here, a "module" or "unit" may be a unit composed of a single element, or it may be a unit expressed as a combination or set of multiple elements. For example, as a concept, a "module" or "unit" may refer to a hardware element or set thereof of a computing device, an application program that performs a specific function of software, a processing procedure embodied by the execution of software, or a set of instructions for executing a program. Furthermore, as a broader concept, a "module" or "unit" may refer to the computing device itself that constitutes a system, or an application executed on a computing device. However, the above concepts are merely examples, and the concepts of "module" or "unit" can be defined in various ways to the extent that a person skilled in the art can understand them based on the content of this disclosure.

[0034] As used in this disclosure, the term "model" can be understood as a system embodied using mathematical concepts and language to solve a particular problem, a set of software units to solve a particular problem, or an abstract model of a processing step to solve a particular problem. For example, a neural network "model" can refer to any system embodied as a neural network that has problem-solving capabilities through learning. Here, a neural network can have problem-solving capabilities by optimizing the parameters connecting nodes or neurons through learning. A neural network "model" may include a single neural network or a set of neural networks that are combinations of multiple neural networks.

[0035] As used in this disclosure, the term "image" refers to multidimensional data composed of discrete image elements. In other words, "image" can be understood as a digital representation of an object that can be seen by the human eye. For example, "image" can refer to multidimensional data composed of elements corresponding to pixels in a two-dimensional image. "Image" can also refer to multidimensional data composed of elements corresponding to voxels in a three-dimensional image.

[0036] The definitions of the terms mentioned above are provided to aid in understanding this disclosure. Therefore, unless the terms mentioned above are explicitly stated to limit the content of this disclosure, it should be noted that the content of this disclosure is not intended to be used to limit the technical ideas.

[0037] Figure 1 is a block diagram of an endoscope system including an endoscope and a computing device according to one embodiment of the present disclosure.

[0038] Referring to Figure 1, the endoscopic system 10 includes an endoscope device 100 that acquires various information, including medical images of the inside of the body, and a computing device 200 that receives and analyzes information about the inside of the body from the endoscope device 100 and reconstructs the medical images of the inside of the body in three dimensions.

[0039] In this specification, the endoscopic device 100 may be a rigid endoscope or a flexible endoscope. A rigid endoscope includes a straight tube made of high-strength metal or plastic, while a flexible endoscope may include a tube made of a flexible material. Such a tube can be called a scope. Inside the scope of a rigid or flexible endoscope, optical fiber or lens systems may be provided for transmitting illumination and images. They may also include channels for spraying air or water, channels into which instruments for biopsy are inserted, and so on. The operations and configurations described herein may be for controlling a rigid endoscope or a flexible endoscope. However, the types of endoscopic devices 100 to which this disclosure applies are not limited thereto.

[0040] Furthermore, as used herein, the endoscope device 100 can be used on animals or humans. That is, the body referred to herein may be an animal body or a human body. The configuration of the endoscope device 100 can be modified to take into account the size and anatomical characteristics of the animal or human body, and the operation and configuration described herein can also be appropriately modified according to the characteristics of the body.

[0041] The endoscope device 100 may include a configuration that allows for the acquisition of medical images of the inside of the digestive tract, and a configuration that allows for the insertion of instruments and the performance of treatment or procedures while viewing the medical images, if necessary.

[0042] The endoscope device 100 may include an output unit, a control unit, a drive unit, a pump unit, and a scope, and may further include a light source unit. The endoscope device 100 may further include a network device for sending and receiving data with an external computing device 200, another endoscope device 100, or a hospital server.

[0043] The output unit may include a display for displaying medical images. The output unit may also include a display module that can output visualized information or implement a touchscreen, such as a liquid crystal display (LCD), thin-film transistor-liquid crystal display (TFTLCD), organic light-emitting diode (OLED), flexible display, or 3D display.

[0044] The output unit may include various means for providing medical images or information about medical images. The output unit can display medical images acquired by the scope or medical images processed by the control unit. In addition to visual means, the output unit may also provide information by auditory means, and may include, for example, a speaker that provides an audible alarm for medical images. On the other hand, although one output unit is shown in Figure 2, there may be multiple output units. In this case, the output unit that displays medical images acquired by the scope and the output unit that displays information processed by the control unit can be distinguished.

[0045] The control unit can control the overall operation of the endoscope device 100. For example, the control unit can perform operations such as medical image acquisition through the scope, processing of acquired medical images, control operations for performing medical operations such as spraying cleaning water and suction, and a series of calculations for controlling the movement of the scope. The control unit can include all kinds of devices capable of processing data. According to an exemplary embodiment, the control unit may be a hardware-integrated data processing device having a physically structured circuit to perform functions expressed by code or instructions contained in a program. Examples of hardware-integrated data processing devices include, but are not limited to, processing devices such as microprocessors, central processing units (CPUs), processor cores, multiprocessors, ASICs (application-specific integrated circuits), and FPGAs (field programmable gate arrays).

[0046] The control unit can control the movement of the scope via a drive unit connected to the scope. That is, the control unit can generate control signals to be provided to the drive unit in order to control the movement of the scope.

[0047] Exemplary, a series of operations by the endoscope device 100 of this disclosure to control the scope can be performed as follows: The user can input the degree or direction of curvature of the scope via the control unit. The input information is transmitted to the control unit, which can process the input information and generate a signal to be provided to the drive unit. For example, the control unit can calculate and provide to the drive unit the motor position, angle, angular velocity, etc., corresponding to the degree or direction of curvature set by the user. The drive unit can generate power based on the signal from the control unit and transmit it to the scope. Thus, the scope can move or curve in accordance with the value input by the user.

[0048] The drive unit can provide the necessary power as the scope is inserted into the body or as it bends or moves within the body. For example, the drive unit may include a motor connected to a wire inside the scope and a tension adjuster that adjusts the tension of the wire.

[0049] The drive unit can control the scope in various directions by controlling the power of the motors. For example, the motors may consist of multiple motors corresponding to the direction in which the insertion part at the end of the scope is to bend, or multiple motors corresponding to the wires inside the scope. Specifically, the drive unit may include a first motor that determines the x-axis motion of the scope and a second motor that determines the y-axis motion of the scope. The x-axis position, y-axis position, z-axis position, roll, pitch, and yaw values ​​of the end of the scope can be determined by the control of the drive unit, but the configuration of the drive unit is not limited to these.

[0050] The tension adjustment unit receives power from a motor and can generate tension by pulling the wires inside the scope. This allows the scope to bend. The tension adjustment unit can adjust the tension acting on multiple wires inside the scope to bend the scope by a determined amount and direction of curvature.

[0051] The pump unit may include at least one of the following: an air pump for injecting air into the body through a scope; a suction pump for drawing air from the body through a scope by providing negative pressure or vacuum; and a water pump for injecting cleaning water into the body through a scope. Each pump may include a valve for controlling the flow of fluid. The pump unit may be opened and closed by the control unit. At least one of the suction pump, water pump, and air pump may be opened and closed by a control signal of the computing device 200 or by the control unit.

[0052] The scope may include an insertion section that is inserted into the digestive tract and an operating section that receives input from the user to control the movement of the insertion section and perform various actions.

[0053] The insertion section is configured to be flexible, and one end is connected to a drive unit, which can determine the degree or direction of curvature. Since medical imaging and surgery are performed at the end of the insertion section, the scope may include multiple cables and tubes extending to the end of the insertion section. Inside the scope, a light source lens, an objective lens, a working channel, and air and water channels may be provided. Through the working channel, instruments for treating and manipulating lesions during endoscopic procedures may be inserted. Air can be injected and irrigation water can be supplied through the air and water channels. On the other hand, while Figure 2 shows air and water channels as passages for supplying irrigation water, it is not limited to these. Exemplarily, a separate water set channel (not shown) may be provided inside the scope, and irrigation water can also be supplied through the water set channel.

[0054] On the other hand, in this specification, expressions indicating that the scope is bent by a control unit or drive unit may mean that at least a portion of the scope, for example, the insertion portion, is bent.

[0055] The control unit may include multiple input buttons that provide various functions (such as image acquisition and irrigation water spray) to allow the endoscopist to control the direction of the insertion tube and perform the procedure through the working channel and air and water channels. For example, the control unit may include multiple buttons or joystick-type input devices that indicate the direction of the scope.

[0056] The light source unit may include a light source that illuminates the inside of the body through the endoscope scope. The light source unit may include an illumination device that generates white light, or may include multiple illumination devices that generate light with different wavelength bands. The type of light source, light intensity, white balance, etc., can be set via the light source unit. Alternatively, the aforementioned settings can also be set via the control unit. The light generated by the light source unit may be transmitted to the scope via a path such as an optical fiber.

[0057] A computing device 200 according to one embodiment of this disclosure may be a hardware device or part of a hardware device that performs comprehensive data processing and calculations, or it may be a software-based computing environment connected via a communication network. For example, the computing device 200 may be a server that performs intensive data processing functions and is the entity that shares resources, or it may be a client that shares resources through interaction with a server. Furthermore, the computing device 200 may be a cloud system that enables multiple servers and clients to interact with each other to comprehensively process data. The above description is merely an example related to the types of computing devices 200, and the types of computing devices 200 can be configured in a variety of ways within the scope that can be understood by a person skilled in the art based on the content of this disclosure.

[0058] Referring to Figure 1, a computing device 200 according to one embodiment of the present disclosure may include a processor 210, memory 220, and a network unit 230. However, since Figure 1 is merely an example, the computing device 200 may include other configurations to embody a computer environment. Furthermore, the computing device 200 may include only a portion of the disclosed configurations.

[0059] A processor 210 according to one embodiment of the present disclosure can be understood as a component unit including hardware and / or software for performing computing operations. For example, the processor 210 can read a computer program and perform data processing for machine learning. The processor 210 can handle computational processes such as processing input data for machine learning, feature extraction for machine learning, and error calculation based on backpropagation. A processor 210 for performing such data processing may include a central processing unit (CPU), a general-purpose graphics processing unit (GPGPU), a tensor processing unit (TPU), an application-specific integrated circuit (ASIC), or a field-programmable gate array (FPGA). The types of processors 210 described above are merely examples, and the types of processors 210 can be configured in a variety of ways that are understandable to those skilled in the art based on the content of the present disclosure.

[0060] The processor 210 according to this disclosure can extract useful information from endoscopic images acquired from the endoscope device 100 and generate a training dataset containing the endoscopic images and the useful information. A neural network model can then be trained using the training dataset. The neural network model may be a model for analyzing endoscopic images. The endoscopic device can be controlled based on the analysis results of the endoscopic images output by the neural network model. For example, the configuration of the endoscopic device can be physically controlled based on the analysis results, or other control information that affects the operation of the endoscopic device can be generated. Exemplarily, the neural network model can perform operations such as identifying regions containing targets, such as lesions, from the endoscopic image, or determining whether the endoscopic image is normal or contains a target.

[0061] In this specification, “target” can include at least one of the following: a lesion occurring inside the body, an area containing foreign matter such as bodily fluids or blood, and a predefined region of interest. For example, a target may include an area where blood is distributed, or an area with dense blood vessels, or an area with a blurred focus, or an area where a surgical instrument is displayed. The following description uses the example of a target being a lesion, but this is illustrative and the types of targets are not limited to this.

[0062] In this specification, effective information may include information that a neural network model trained using a training dataset utilizes as ground truth. Ground truth information may include labels for endoscopic images or additional information about labels. For example, if a neural network model is trained to identify a target, such as a region containing a lesion, the effective information may include coordinate information for the region containing the lesion. Alternatively, the effective information may include pixel-level class information for segmenting the region corresponding to the lesion in the endoscopic image. For example, if a neural network model is trained to classify input endoscopic images into normal images and images containing lesions, the effective information may include tags or labels for the endoscopic images.

[0063] The operation of the neural network model described above is illustrative; any neural network model capable of generating information for analyzing endoscopic images would be possible. For example, a neural network model could be trained to generate information for controlling the pumps of the endoscope device 100 based on the endoscopic image. In this case, the useful information could include labels indicating the class of operation that at least one of the air pump, suction pump, and water pump of the endoscope device 100 should perform.

[0064] Alternatively, a neural network model may be trained to generate information that controls the movement of the endoscope 100's scope based on the endoscopic image. In this case, the useful information may include labels indicating a class for at least one of the following movements of the scope: up, down, left, right, forward, and backward.

[0065] The processor 210 can use multiple neural network models to extract useful information from endoscopic images and generate final useful information based on the information output from each neural network model. Here, the multiple neural network models may all perform the same role, or at least one neural network model may perform a different role from the others.

[0066] The processor 210 can train multiple neural network models that output useful information. Here, each neural network model can include at least one neural network. The neural network can include, but is not limited to, network models such as DNN (Deep Neural Network), RNN (Recurrent Neural Network), BRDNN (Bidirectional Recurrent Deep Neural Network), MLP (Multilayer Perceptron), CNN (Convolutional Neural Network), and transformer.

[0067] In other words, according to this disclosure, it is possible to generate training datasets necessary for training a neural network model using only raw data without labels or additional information about labels. Furthermore, since the time and effort required for labeling correct information can be reduced, a large amount of high-quality training datasets can be obtained quickly. In addition, since the output results of multiple neural network models are combined to create a single training dataset to generate the final valid information, the accuracy of the training data can be improved.

[0068] A memory 220 according to one embodiment of the present disclosure can be understood as a component unit including hardware and / or software for storing and managing data processed by the computing device 200. That is, the memory 220 can store any form of data generated or determined by the processor 210 and any form of data received by the network unit 230. For example, the memory 220 may include at least one type of storage medium from among flash memory type, hard disk type, multimedia card micro type, card type memory, RAM (random access memory), SRAM (static random access memory), ROM (read-only memory), EEPROM (electrically erasable programmable read-only memory), PROM (programmable read-only memory), magnetic memory, magnetic disk, or optical disk. The memory 220 may also include a database system for controlling and managing data in a predetermined manner. The types of memory 220 described above are merely examples, and the types of memory 220 can be configured in a variety of ways within the scope understandable to those skilled in the art based on the content of the present disclosure.

[0069] Memory 220 can structure and organize the data, data combinations, and program code (code) that the processor 210 can execute, which are necessary for the processor 210 to perform calculations. Memory 220 can also store program code that causes the processor 210 to generate training data.

[0070] The memory 220 provided in this disclosure can store raw data of endoscopic images acquired from the endoscope device 100, i.e., endoscopic images of the inside of the body. The memory 220 can also store parameter information contained in multiple neural network models, valid information output from each neural network model, and reference values ​​used to output final valid information from the multiple valid information.

[0071] A network unit 230 according to one embodiment of the present disclosure can be understood as a component unit that transmits and receives data via any known wired wireless communication system. For example, the network unit 230 can transmit and receive data using a wired wireless communication system such as a local area network (LAN), wideband code division multiple access (WCDMA), LTE (long term evolution), WiBro (wireless broadband internet), 5th generation mobile communication (5G), ultrawide-band wireless communication (ultrawide-band), ZigBee, radio frequency (RF) communication, wireless LAN (wireless LAN), Wi-Fi (wireless fidelity), near field communication (NFC), or Bluetooth®. The above-mentioned communication systems are merely examples, and a wide variety of wired wireless communication systems for data transmission and reception of the network unit 230 are applicable beyond the examples given above.

[0072] The network unit 230 can receive data necessary for the processor 210 to perform calculations via wired or wireless communication with any system or client. The network unit 230 can also transmit data generated by the processor 210's calculations via wired or wireless communication with any system or client. For example, the network unit 230 can receive basic data, including captured images, through communication with a medical image storage and transmission system, a cloud server that performs tasks such as medical data standardization, or an endoscope device 100. The network unit 230 can also transmit various types of data generated by the processor 210's calculations through communication with the aforementioned systems, servers, or endoscope device 100.

[0073] On the other hand, although Figure 1 shows the endoscope device 100 and the computing device 200 as being separate entities, this is merely illustrative, and the computing device 200 can also be installed inside the endoscope device 100 and constitute part of the endoscope device 100.

[0074] The network unit 230 according to this disclosure can transmit valid information, final valid information, or training datasets generated by the processor 210 to an external computing device 200. For example, the training dataset can be transmitted to an external computing device 200 that trains a neural network model to control the endoscope device 100 based on the training dataset generated by the processor 210.

[0075] Figure 2 is a diagram showing an artificial neural network model according to one embodiment of the present disclosure.

[0076] Referring to Figure 2, the artificial neural network model 300 can identify the objects to be detected in the raw data and output information about the objects as final valid information. Here, the objects to be detected may mean targets. The raw data may represent images acquired from the endoscope device 100 and may contain no information about the targets. The targets may be lesions, foreign bodies, endoscopic instruments, specific regions, specific body parts, etc.

[0077] If the target is a lesion, the final valid information may include the bounding box containing the lesion, class information for the pixels corresponding to the lesion, a label indicating that the image contains the lesion, confidence level for the lesion, type of lesion, and size of the lesion.

[0078] In other words, the artificial neural network model 300 can receive raw data as input and generate label information corresponding to the raw data using multiple pieces of valid information that are the results output from multiple models. That is, it can generate information for using the raw data as training data. Herein, the artificial neural network model 300 disclosed herein does not directly use the results output from multiple models that analyze various characteristics of the raw data, but instead goes through a decision stage to ensure reliability, thereby providing a careful method for generating training data.

[0079] The artificial neural network model 300 can include multiple models, for example, a first model through an nth model. Each of the first through nth models is a trained model, and training may be performed by the computing device 200, for example.

[0080] For example, the computing device 200 can match endoscopic images and labels corresponding to those images and input them into the first to nth models, thereby training the first to nth models to output useful information from the endoscopic images. This example is an embodiment in which the first to nth models are trained using supervised learning. The first to nth models can be trained to output labels or additional information about labels from endoscopic images using a variety of learning methods, such as semi-supervised learning, unsupervised learning, reinforcement learning, or self-supervised learning.

[0081] Here, each of the first to nth models may be trained using the same method, or they may be trained using different methods. Furthermore, each of the first to nth models may be trained to output the same type of useful information, or to output different types of useful information.

[0082] Each of the first to nth models, once training is complete, can receive raw data input for an endoscopic image, analyze the images contained in the endoscopic image, and output valid information. Here, the types of the first to nth models or the parameters constituting each model may differ from one another. Therefore, although the first to nth models receive a single data input, the first to nth valid information output may differ from one another. Accordingly, the computing device 200 can generate final valid information based on the first to nth valid information in order to generate labels for the raw data or additional information about the labels.

[0083] Here, if the types of the first to the nth model are the same, the formats of the first to the nth valid information may be similar. Therefore, the computing device 200 can extract duplicate information from the first to the nth valid information and generate the final valid information by comparing the duplicate information with a reference value.

[0084] On the other hand, if the types of the first to nth models are different, the format of the first to nth valid information may differ. Therefore, the computing device 200 can generate the final valid information by using the numerical values ​​contained in the first to nth valid information as a threshold. The specific details will be described later based on Figures 3 to 6.

[0085] The computing device 200 can generate a training dataset by matching the final valid information with the raw data. For example, the final valid information may include the coordinate information of the bounding box containing the lesion and the confidence level of this bounding box. The training dataset thus generated can be used as training data for an artificial neural network model 300 that detects lesions.

[0086] In other words, according to this disclosure, the computing device 200 can easily generate large amounts of training data by generating highly reliable label information using raw data.

[0087] On the other hand, the type of model constituting the artificial neural network model 300 can be determined by the characteristics of the raw data, the characteristics of the artificial neural network model to be trained on the training dataset, and the operation of the artificial neural network model. In other words, the type and amount of training data required may vary depending on the purpose of the artificial neural network model. In order to appropriately provide the training data required by the artificial neural network model, according to this disclosure, an artificial neural network model that generates the label information required by the artificial neural network model can be selectively used. This makes it possible to generate training data efficiently.

[0088] For example, depending on the purpose or role of the artificial neural network model, i.e., the model for controlling the endoscopic device using the training dataset, a configuration can be implemented in which of the multiple artificial neural network models 300a, 300b, and 300c is selected to provide the training dataset. Depending on the role of the endoscopic device control model, an appropriate model can be recommended from among the multiple artificial neural network models 300a, 300b, and 300c.

[0089] In the following, we will explain using the case where the target is a lesion as an example. For example, if the artificial neural network model performs an action that provides a simple alarm when a lesion occurs, the artificial neural network model 300 may consist of a model optimized to determine the presence and type of lesion. For example, this may be the artificial neural network model 300b in Figure 5.

[0090] For example, if the artificial neural network model performs the function of providing the location of a lesion, the artificial neural network model 300 may consist of a model optimized for determining the location of the lesion. For example, this could be the artificial neural network model 300a shown in Figure 3. Here, the artificial neural network model 300a uses multiple lesion detection models and considers the degree of agreement of the output results of each lesion detection model, so it is suitable for having highly reliable training data.

[0091] For example, if the primary objective of the artificial neural network model is to provide the detailed location of a lesion, the artificial neural network model 300 may consist of a model optimized to provide the region identified as a lesion. For example, this could be the artificial neural network model 300c shown in Figure 6.

[0092] For example, depending on the priorities considered in the endoscopic device control model, the appropriate model can be recommended from among multiple artificial neural network models 300a, 300b, and 300c. If it is important to precisely understand areas such as lesions and foreign bodies, it may be recommended to generate training data using artificial neural network model 300c. If it is important to understand the type of lesion, it may be recommended to generate training data using artificial neural network model 300b. If it is important to confidently understand the lesion area, it may be recommended to generate training data using artificial neural network model 300a.

[0093] Figure 3 is an illustrative diagram of an artificial neural network model according to one embodiment of the present disclosure, and Figure 4 is an explanatory diagram showing the operation of calculating the final effective information according to one embodiment of the present disclosure.

[0094] Referring to Figure 3, the artificial neural network model 300a corresponds to one embodiment of the artificial neural network model 300 in Figure 2. The first to nth models can all be models trained to detect lesions and output information about regions containing lesions. As a result, each lesion detection model can detect lesions included in the raw data and output regions containing lesions. That is, each lesion detection model can output the coordinates of the region containing the lesion, class information indicating the type of lesion, and a confidence level for the inference result.

[0095] Here, the lesion detection model can output the region containing the lesion as bounding box, pixel-wise class, attention map, heat map, keypoint location information, etc.

[0096] In other words, the artificial neural network model 300a can use multiple lesion detection models that output classification results for lesions and regions containing lesions. Therefore, it is possible to generate final valid information based on the degree of agreement of the values ​​output from each lesion detection model.

[0097] Referring to Figure 4, the computing device 200 can generate final valid information based on the information output by each of the lesion detection models.

[0098] For example, the first lesion detection model can output a bounding box R1 for a lesion whose raw data includes a lesion of class 1, and output a first value (conf1) as its confidence level. The second lesion detection model can output a bounding box R2 for a lesion whose raw data includes a lesion of class 1, and output a second value (conf2) as its confidence level. Here, taking the case where the bounding box is a rectangle as an example, R1 can be determined as P1(x1, y1) and P2(x2, y2), and R2 can be determined as P3(x3, y3) and P4(x4, y4). Here, the coordinates of the bounding box R3 included in the final valid information can be calculated by the following equation 1. The coordinates of R3 can be determined as P5(x5, y5) and P6(x6, y6).

[0099]

number

[0100] In other words, the bounding box included in the final valid information can be generated by reflecting the bounding box and confidence level output by each lesion detection model. On the other hand, the shape of the bounding box does not necessarily have to be rectangular; if it is not rectangular, the bounding box reflected in the final valid information can be determined based on the coordinates and confidence levels of several points that determine the bounding box.

[0101] On the other hand, the computing device 200 can decide whether to adopt the bounding boxes thus generated as final valid information. For example, the computing device 200 can calculate a score based on each bounding box and its confidence level, and compare the score with a reference value to decide whether to adopt the bounding box. For example, the computing device 200 can calculate a score based on the IoU (Intersection over Union) and first and second values ​​indicating the confidence level of bounding boxes R1 and R2 of bounding box R3, and if the score is equal to or greater than the reference value, it can include bounding box R3 as final valid information.

[0102] Furthermore, the computing device 200 does not have to adopt the bounding box R3 as the final valid information if the class information for the lesion types output by the lesion detection models are different from each other. In this case, the raw data may be considered to not contain lesions.

[0103] Figures 5 and 6 illustrate an example of an artificial neural network model according to one embodiment of the present disclosure.

[0104] The artificial neural network model 300b in Figure 5 and the artificial neural network model 300c in Figure 6 correspond to one embodiment of the artificial neural network model 300 in Figure 2. The following sections will omit details similar to the artificial neural network model 300 in Figure 3 mentioned above.

[0105] Referring to Figure 5, the artificial neural network model 300b corresponds to one embodiment of the artificial neural network model 300 in Figure 2. The artificial neural network model 300b includes a model trained to output information about regions containing lesions by detecting lesions from endoscopic images, and may include a model trained to determine whether the endoscopic image contains a lesion or what type of lesion it contains. Here, the classification model can output a class indicating whether a lesion is present, class information indicating the type of lesion if a lesion is present, and a confidence level for the inference result.

[0106] The computing device 200 can generate final valid information based on the bounding box and confidence level output from the lesion detection model, and the class and confidence level output from the classification model.

[0107] For example, the bounding box included in the final valid information may be output from the lesion detection model, and the class information included in the final valid information may be a class output from the classification model. The computing device 200 then compares the confidence level output from the classification model with a reference value, and if it is equal to or greater than the reference value, it can be adopted as the final valid information.

[0108] Here, the classification model may have a higher ability to infer whether a lesion is present and what type of lesion it is compared to the lesion detection model. Therefore, if the class information for a lesion output from the classification model and the class information for a lesion output from the lesion detection model are different, the results output from the classification model can be used. In other words, when the classification model and the lesion detection model are used together, the classification model plays a role in compensating for the shortcomings of the lesion detection model.

[0109] Furthermore, when training a classification model using supervised learning, image and class information for lesions can be used as training data. When training a lesion detection model using supervised learning, image, class information for lesions, and bounding box information can be used as training data. Therefore, by using a classification model, the difficulty of learning can be reduced and the learning speed can be increased.

[0110] In other words, in situations where the goal is to generate training data to input into a model that accurately determines the type of lesion rather than its location, the artificial neural network model 300b in Figure 5 is more suitable than other artificial neural network models. For example, a model trained using the final valid information generated in Figure 5 might be a model that performs endoscopic procedures, detects lesions in real time, and provides alarms. In this case, it is considered most important that a model trained on a wide variety of lesion cases accurately provides the type of lesion. Therefore, to train a model that performs such operations, it is most appropriate to use the artificial neural network model 300b in Figure 5 to provide training data.

[0111] On the other hand, while Figure 5 shows a case where there is only one lesion detection model, there may be multiple lesion detection models. In this case, similar to the method described based on Figures 3 and 4, the bounding box can be determined, and the final valid information can be generated based on the coordinates and information of the determined bounding box and the information output from the classification model.

[0112] Referring to Figure 6, the artificial neural network model 300c corresponds to one embodiment of the artificial neural network model 300 in Figure 2. The artificial neural network model 300c can include a lesion detection model, a lesion classification model, and a region segmentation model. The region segmentation model can output pixel-specific classes that constitute the endoscopic image. This allows the region segmentation model to display regions corresponding to target areas, such as lesions, foreign bodies, or bodily fluids, on a pixel-by-pixel basis on the endoscopic image.

[0113] The computing device 200 can generate final valid information based on bounding boxes and confidence levels output from the lesion detection model, classes and confidence levels output from the classification model, and pixel-specific classes output from the region segmentation model.

[0114] For example, the computing device 200 can determine regions within the bounding box where the class output from the lesion detection model is the same as the class output from the region segmentation model. Then, the computing device 200 can decide whether to include the bounding box in the final valid information based on the proportion that the region occupies within the bounding box. Secondarily, it can decide whether to include the bounding box based on the confidence level output from the classification model.

[0115] Artificial neural network model 300c, by using a region segmentation model, can improve the accuracy of bounding boxes compared to artificial neural network model 300a in Figure 3, which uses only a lesion detection model. In other words, it can reduce the likelihood that non-lesion regions are included in the bounding box.

[0116] To supervise training a segmentation model, image and pixel-level class information can be used as training data. This can make training more difficult compared to classification or lesion detection models, as it is harder to obtain sufficient training data. However, if location information is deemed more important than class information, and accuracy of inference is required despite the increased training difficulty, it may be necessary to use the artificial neural network model 300c to generate the final valid information.

[0117] For example, a model trained using the final valid information generated in Figure 6 could be a model that predicts the actions that the endoscopic device should perform based on the endoscopic image. For instance, the model could supply irrigation water or perform suction based on the endoscopic image. In this case, the location information of the lesion to which the irrigation water is sprayed and the location information of bodily fluids, foreign objects, etc., to which suction is to be performed are considered to be of utmost importance. Therefore, to train a model that performs such actions, it is most appropriate to use the artificial neural network model 300c in Figure 6 to provide training data.

[0118] According to this disclosure, multiple neural network models that output different characteristics from each other can be used to output useful information about lesions and other features for a single endoscopic image. Each model then reflects this useful information, and the final useful information is used to generate label information for the endoscopic image. This improves the accuracy of the generated labels and allows for the easy and efficient generation of large amounts of high-quality training data.

[0119] Figure 7 is a flowchart showing the operation of a computing device according to one embodiment of the present disclosure.

[0120] Referring to Figure 7, the computing device 200 can input raw data into multiple models that have been trained to infer useful information about endoscopic images taken inside the body (S110).

[0121] The computing device 200 can generate final valid information for the raw data based on the valid information output from each model (S120). Here, the final valid information may include information necessary to train a model for analyzing endoscopic images using a training dataset. That is, it may include labels for the raw data or additional information about said labels. The results of analyzing the endoscopic images may be used to physically control the configuration of the endoscopic device or to generate other control information that affects the operation of the endoscopic device.

[0122] As one embodiment, a model for analyzing endoscopic images may include a model for detecting targets in the endoscopic images. Here, a label may indicate whether the raw data contains a target. Additional information about the label may include confidence in the label or information about the location of the target. Here, the target may include at least one of lesions, foreign bodies, bodily fluids, and a predefined region of interest.

[0123] As one embodiment, a model for analyzing endoscopic images may include a model that predicts the actions that an endoscopic device should perform based on the endoscopic images. Here, the labels may include the type of action. Additional information about the labels may include a confidence level for the labels. For example, the labels may include a class indicating the action of the scope of the endoscopic device, which includes at least one of up, down, left, right, forward, and backward. Alternatively, the labels may include a class indicating the action that at least one of the air pump, suction pump, and water pump of the endoscopic device should perform.

[0124] The computing device 200 can generate a training dataset containing raw data and final valid information (S130).

[0125] As one embodiment, the multiple models may include a first model and a second model that detect targets included in the raw data and output information about the region containing the target. Here, in step S120, the computing device 200 can calculate a score and a third region based on the first region and the confidence level for the first region output from the first model, and the second region and the confidence level for the second region output from the second model. Then, the computing device 200 can generate final valid information based on the third region based on the comparison result of the score and the reference value.

[0126] In one embodiment, the multiple models may include a first model that detects targets included in the raw data and outputs information about the region containing the target, and a second model that outputs whether the raw data contains the target. Here, in step S120, the computing device 200 can generate final valid information based on the first region and confidence level for the first region output from the first model, and the class and confidence level for the class output from the second model.

[0127] In one embodiment, the multiple models may include a first model that detects targets included in the raw data and outputs information about the region containing the target, a second model that outputs whether the raw data contains the target, and a third model that outputs the pixel-specific class that constitutes the raw data. Here, in step S120, the computing device 200 can generate final valid information based on the first region and confidence level for the first region output from the first model, the class and confidence level for the class output from the second model, and the pixel-specific class output from the third model.

[0128] The computing device 200 can then compare the distribution values ​​of individual pixel classes within the first region with a first reference value, compare the confidence level for each class with a second reference value, and generate final valid information based on the results of each comparison.

[0129] Here, the aforementioned target could be, for example, a lesion. In addition to lesions, it could include at least one of the following: body fluids, foreign bodies, endoscopic instruments, a specific area, a specific site, or an area with a blurred focus.

[0130] The descriptions of this disclosure set forth herein are illustrative, and a person with ordinary skill in the art to which this disclosure pertains will understand that it can be easily adapted to other specific forms without altering the technical idea or essential features of this disclosure. Therefore, the embodiments of this disclosure described above should be understood to be illustrative and not limiting in all respects. For example, each component described as a single type can be implemented in a distributed manner, and similarly, components described as distributed can be implemented in a combined form.

[0131] The scope of this disclosure is determined by the claims set forth below rather than by the detailed description above, and all forms of modification or alteration derived from the meaning, scope, and equivalents of the claims should be interpreted as being included within the scope of this disclosure. [Explanation of Symbols]

[0132] 10 Endoscopy Systems 100 Endoscopes 200 computing devices 210 processors 220 memory 230 Network Department 300, 300a, 300b, 300c Artificial Neural Network Models

Claims

1. A method for generating a training dataset based on endoscopic images, performed by a computing device including at least one processor, The process involves inputting raw data into multiple models trained to infer useful information about endoscopic images taken from inside the body, A step of generating final valid information for the raw data based on the valid information output from each model, A step of generating a training dataset including the raw data and the final valid information, Includes, The aforementioned final valid information is information generated based on the valid information output from the multiple models, and is necessary for training a model to analyze the endoscopic images using the training dataset, and includes labels for the raw data or additional information about the labels. method.

2. The model for analyzing the endoscopic image includes a model for detecting a target in the endoscopic image. The label includes whether the raw data includes the target, The additional information regarding the label includes information about the confidence level of the label or the location of the target. The method according to claim 1.

3. The target includes at least one of the following: lesions, bodily fluids, foreign bodies, endoscopic instruments, and predefined regions of interest. The method according to claim 2.

4. The model for analyzing the endoscopic image includes a model that predicts the actions that the endoscopic device should perform based on the endoscopic image. The label includes the type of operation, The additional information regarding the label includes the confidence level for the label, The method according to claim 1.

5. The label includes a class indicating the movement of the scope of the endoscope, which includes at least one of the following: up, down, left, right, forward, and backward. The method according to claim 4.

6. The label includes a class of operation that at least one of the air pump, suction pump, and water pump of the endoscope device should perform. The method according to claim 4.

7. The aforementioned plurality of models include a first model and a second model that detect targets included in the raw data and output information about the region containing the target. The method according to claim 1.

8. The step of generating the aforementioned final valid information is: A step of calculating a confidence score and a third region based on the first region and the confidence level for the first region output from the first model, and the second region and the confidence level for the second region output from the second model, The process includes a step of generating final valid information based on a third domain, based on a comparison result between a score based on the aforementioned reliability and a predetermined reference value. The method according to claim 7.

9. The aforementioned plurality of models include a first model that detects targets included in the raw data and outputs information about the region containing the target, and a second model that outputs whether the raw data contains the target. The method according to claim 1.

10. The step of generating the final valid information includes generating the final valid information based on the first region and the confidence level for the first region output from the first model, and the class and the confidence level for the class output from the second model. The method according to claim 9.

11. The aforementioned plurality of models include a first model that detects targets included in the raw data and outputs information about the region containing the target, a second model that outputs whether the raw data contains the target, and a third model that outputs the pixel-specific class that constitutes the raw data. The method according to claim 1.

12. The step of generating the final valid information includes generating the final valid information based on the first region and confidence level for the first region output from the first model, the class and confidence level for the class output from the second model, and the pixel-specific class output from the third model. The method according to claim 11.

13. The step of generating the final valid information includes comparing the distribution values ​​of the pixel-specific classes within the first region with a predetermined first reference value, comparing the confidence level for the classes with a predetermined second reference value, and generating the final valid information based on the results of each comparison. The method according to claim 12.

14. A computer program stored on a computer-readable storage medium, When the computer program is run on one or more processors, it performs an operation to generate a training dataset based on endoscopic images. The process involves inputting raw data into multiple models trained to infer useful information about endoscopic images taken from inside the body, The operation of generating final valid information for the raw data based on the valid information output from each model, The operation of generating a training dataset including the raw data and the final valid information, Includes, The aforementioned final valid information is information generated based on the valid information output from the multiple models, and is necessary for training a model to analyze endoscopic images using the training dataset, and includes labels for the raw data or additional information about the labels. Computer program.

15. A computing device for generating training datasets based on endoscopic images, Memory for storing raw data for endoscopic images taken from inside the body, A processor that inputs the raw data into multiple models trained to infer valid information about endoscopic images, generates final valid information for the raw data based on the valid information output from each model, and generates a training dataset including the raw data and the final valid information. Includes, The aforementioned final valid information is information generated based on the valid information output from the multiple models, and is necessary for training a model to analyze endoscopic images using the training dataset, and includes labels for the raw data or additional information about the labels. Device.

Images