Endoscope device and method of acquiring lesion size information

Abstract

To provide an endoscope device and a method of acquiring lesion size information.SOLUTION: The invention relates to an endoscope device that identifies a lesion based on a pre-trained model and calculates size information of the identified lesion, and to a method of acquiring lesion size information. The method includes the steps of: acquiring an image of the inside of the body from an image sensor; identifying at least one lesion from the image based on the pre-trained model; and calculating size information of the lesion.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to an endoscopic device and a method for acquiring size information of a lesion, and more particularly to an endoscopic device that identifies a lesion based on a pre-trained model and calculates size information of the identified lesion, and a method for acquiring size information of the lesion. [Background technology]

[0002] An endoscopic device is a medical instrument that inserts a scope into the body to observe organs and, if necessary, perform treatment or therapy. When using an endoscopic device, the user typically simultaneously operates the scope and performs medical procedures to check for abnormalities inside the body. In such a situation where the user's concentration is divided, there is a significant risk of overlooking lesions that require observation or therapy, depending on the user's level of skill or fatigue.

[0003] When a lesion is discovered during medical treatment using an endoscopic device, the treatment method differs depending on the size of the lesion, so it is necessary to measure the size of the lesion. Conventionally, the size of the lesion has been measured with the naked eye, but this has the problem of low accuracy.

[0004] The aforementioned background art is technical information that the inventor possessed in order to derive the present invention or that was acquired in the process of deriving the present invention, and is not necessarily publicly known art that was made public to the general public prior to the filing of the invention application. Summary of the Invention [Problem to be solved by the invention]

[0005] The present invention is intended to solve the above-mentioned problems, and aims to provide an endoscopic device that identifies lesions based on a pre-trained model and calculates size information of the identified lesions, as well as a method for acquiring size information of the lesions.

[0006] However, such problems are merely examples, and the problems to be solved by the present invention are not limited thereto. Problems not mentioned above will be clearly understood by a person having ordinary skill in the art to which the present invention pertains from the present specification and the attached drawings. [Means for solving the problem]

[0007] One embodiment of the present invention discloses a method for acquiring size information of a lesion, comprising the steps of acquiring an image of an interior of a body from an image sensor, identifying at least one lesion from the image based on a pre-trained model, and calculating size information of the lesion.

[0008] In this embodiment, the method may further include calculating a first distance, and the step of calculating size information of the lesion may calculate the size information of the lesion based on the first distance.

[0009] In this embodiment, calculating the first distance also includes calculating the first distance based on brightness information of the lesion represented in the image.

[0010] In this embodiment, the step of calculating the first distance also includes the steps of acquiring a first image when the tip is at a first angle and a second image when the tip is at a second angle, and measuring the difference between the first angle and the second angle and the visual deviation between the first image and the second image.

[0011] In this embodiment, identifying the lesion also includes calculating lesion size information on the image, which indicates size information of the lesion on the image.

[0012] In this embodiment, the lesion size information on the image also includes a bounding box displayed around the lesion.

[0013] This embodiment also includes controlling the tip so that it tracks the lesion.

[0014] In this embodiment, the angle of view of the image sensor includes a first region, and controlling the tip portion also includes controlling the tip portion so that the lesion is located within the first region.

[0015] In this embodiment, controlling the tip also includes calculating a first value.

[0016] Another embodiment of the present invention discloses an endoscopic device including a distal end portion having an image sensor, and a control unit that acquires an image of the inside of a body from the image sensor, identifies at least one lesion from the image based on a pre-trained model, and calculates size information of the lesion.

[0017] In the present embodiment, the control unit can calculate a first distance and calculate size information of the lesion based on the first distance.

[0018] In this embodiment, the control unit may calculate the first distance based on brightness information of the lesion represented in the image.

[0019] In this embodiment, the control unit may acquire a first image when the tip is at a first angle and a second image when the tip is at a second angle, and measure the difference between the first angle and the second angle and the visual deviation between the first image and the second image.

[0020] In this embodiment, the control unit can calculate lesion size information on the image that indicates size information of the lesion on the image.

[0021] In this embodiment, the lesion size information on the image also includes a bounding box displayed around the lesion.

[0022] In this embodiment, the control unit may control the tip unit so that the tip unit tracks the lesion.

[0023] In this embodiment, the angle of view of the image sensor includes a first region, and the control unit can control the tip unit so that the lesion is located within the first region.

[0024] In this embodiment, the control unit may calculate a first value.

[0025] Other aspects, features, and advantages in addition to those described above will become apparent from the following detailed description of the invention, the claims, and the drawings. [Effects of the Invention]

[0026] An endoscopic device and method for acquiring size information of a lesion according to one embodiment of the present invention can detect lesions inside the body without being affected by the user's level of skill or fatigue by using a pre-trained model to identify the lesion.

[0027] The endoscopic device and method for acquiring lesion size information according to one embodiment of the present invention allows the endoscopic device to automatically calculate size information of identified lesions, thereby obtaining accurate and consistent size information of the lesion without deviation due to the user's experience or skill level, and based on this, selecting the correct way to deal with the lesion, thereby improving the quality of medical care.

[0028] The effects of the present invention are not limited to those mentioned above, and other effects not mentioned will be clearly understood by those having ordinary skill in the art to which the present invention pertains from this specification and the accompanying drawings. [Brief explanation of the drawings]

[0029] [Figure 1] 1 is a diagram illustrating an endoscope apparatus according to an embodiment of the present invention. [Figure 2]FIG. 1 illustrates an illustration of a pre-trained model identifying lesions from internal body images, according to an embodiment of the present invention. [Figure 3] FIG. 10 is a diagram showing an image output on the display unit when a lesion is identified. [Figure 4] FIG. 1 is a diagram showing a first distance inside the body. [Figure 5] FIG. 10 is a diagram showing the concept of acquiring lesion size information based on lesion size information on an image. [Figure 6] FIG. 10 is a diagram showing an image output on the display unit when a lesion is located within a first region. [Figure 7] 10A and 10B are diagrams showing a method for calculating a first value based on a position difference on an image caused by an angle difference of the tip portion. [Figure 8] 1 is a flowchart illustrating a method for obtaining size information of a lesion, according to one embodiment of the present invention. [Figure 9] 4 is a flowchart illustrating the steps of calculating a first distance in accordance with an embodiment of the present invention; [Figure 10] 10 is a flowchart illustrating a method for obtaining size information of a lesion according to another embodiment of the present invention. [Figure 11] 1 is a flowchart illustrating the steps broken down for controlling a tip according to one embodiment of the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0030] The terms used in the present invention are merely used to describe specific embodiments and are not intended to limit the scope of other embodiments. A singular expression includes a plural expression unless the context clearly dictates otherwise. Terms used herein, including technical or scientific terms, may have the same meaning as commonly understood by a person of ordinary skill in the technical field described in the present invention. Among the terms used in the present invention, commonly defined terms should be interpreted as meanings that are identical to or similar to the meanings they have in the context of the related art, and should not be interpreted as ideal or overly formal unless explicitly defined in the present invention. In some cases, even if a term is defined in the present invention, it should not be interpreted as excluding embodiments of the present invention.

[0031] Hereinafter, various embodiments will be described in detail with reference to the accompanying drawings so that those skilled in the art can easily implement the present invention. However, the technical concept of the present invention may be embodied in various forms and is not limited to the embodiments described herein. In describing the embodiments disclosed herein, if a detailed description of related known technology is deemed to obscure the gist of the technical concept of the present invention, the detailed description of the known technology will be omitted. Identical or similar elements will be designated by the same reference numerals, and redundant description thereof will be omitted.

[0032] Here, the term "module" used in this embodiment refers to a component that performs a specific function, which is performed by software or hardware such as a field programmable gate array (FPGA) or an application-specific integrated circuit (ASIC). However, the "module" is not limited to being performed by software or hardware. The "module" may exist in the form of data stored on an addressable recording medium, or may be embodied by an instruction word, and configured to cause one or more processors to execute a specific function.

[0033] Software may include computer programs, code, instructions, or a combination of one or more of these, which may configure a processing device to operate as desired or may instruct the processing device, either independently or collectively. The software and / or data may be permanently or temporarily embodied in some type of machine, component, physical device, virtual device, computer storage medium or device, or transmitted signal wave to be interpreted by the processing device or to provide instructions or data to the processing device. The software may also be distributed across network-coupled computer systems, stored or executed in a distributed manner. The software and data may be stored in one or more computer-readable storage media. The software may be read into main memory from other computer-readable media, such as a data storage device, or from another device via a communication interface. The software instructions stored in main memory may cause a processor to perform the processes or steps described in detail below. Alternatively, hard-wired circuitry may be used in place of or in combination with software instructions to implement processes consistent with the principles of the invention. Thus, embodiments consistent with the principles of the invention are not limited to any specific combination of hardware circuitry and software.

[0034] The terms used in this application are merely used to describe specific embodiments and are not intended to limit the present invention. The singular terms include the plural terms unless the context clearly dictates otherwise. In this application, terms such as "comprise" or "have" specify the presence of a specified feature, number, step, operation, component, part, or combination thereof, but should be understood not to preclude the presence or addition of one or more other features, numbers, steps, operations, components, parts, or combinations thereof. Terms such as "first" and "second" may be used to describe various components, but the components are not limited by these terms. These terms are used solely to distinguish one component from another.

[0035] The term "learning model" as used herein includes all types of algorithms or methodologies used to learn or understand specific patterns or structures from data. That is, the term "learning model" includes not only machine learning models such as regression models, decision trees, random forests, support vector machines, k-nearest neighbors, naive phase, and clustering algorithms, but also deep learning models such as neural networks, convolutional neural networks, recurrent neural networks, Transformer-based neural networks, generative adversarial networks (GANs), and autoencoders. A "learning model" refers to a set of learned parameters or weights used to predict or classify an output for a specific input. The model can be trained through methods such as supervised learning, unsupervised learning, semi-supervised learning, and reinforcement learning. It also includes not only single models but also various learning methods and structures such as ensemble models, multimodal models, and models via transfer learning. Such learning models can be pre-trained on a computer device separate from the computer device that predicts the output for the input and used on another computer device.

[0036] The learning model according to an embodiment of the present invention also includes a model related to object detection and position estimation.

[0037] An endoscopic device and a method for obtaining size information of a lesion according to an embodiment of the present invention will now be described with reference to FIGS.

[0038] FIG. 1 is a diagram showing an endoscope apparatus 100 according to an embodiment of the present invention.

[0039] The endoscopic device 100 is a medical instrument that can insert a scope 150 into the body to observe organs and, if necessary, perform treatment or therapy. Referring to FIG. 1, the endoscopic device 100 also includes a display unit 110, a control unit 120, a drive unit 130, an operation unit 140, and the scope 150.

[0040] An image may be output to the display unit 110. In other words, an internal body image acquired from an image sensor 153a (described later) may be output to the display unit 110. The output internal body image may include x-axis coordinates and y-axis coordinates (see FIG. 3).

[0041] The display unit 110 may include a display module that outputs visualized information, such as a liquid crystal display (LCD), a thin film transistor-liquid crystal display (TFT LCD), an organic light-emitting diode (OLED), a flexible display, or a 3D display, or that can implement a touch screen.

[0042] The display unit 110 can output lesion position information (P) on the image, which will be described later.

[0043] The control unit 120 can control the overall operation of the endoscopic device 100. The control unit 120 can also include all types of components that can process data. In one embodiment, the control unit 120 can also include a data processing device built into hardware having physically structured circuits to perform functions expressed by codes or instructions contained in a program. The data processing device built into hardware can also include processing devices such as a microprocessor, a central processing unit (CPU), a processor core, a multiprocessor, an application-specific integrated circuit (ASIC), and a field programmable gate array (FPGA).

[0044] The control unit 120 can calculate size information (R) of the identified lesion. A detailed description of the control unit 120 will be given later.

[0045] The drive unit 130 may provide the power necessary for the scope 150, which will be described later, to be inserted into the body or to move within the body. For example, the drive unit 130 may include a plurality of motors connected to wires inside the scope 150 and a tension adjustment unit that adjusts the tension of the wires.

[0046] The operation unit 140 can input user commands and includes a plurality of buttons that provide various functions, such as controlling the angle of the distal end 153 (described later) or performing various surgical procedures inside the body.

[0047] The scope 150 can be directly inserted into the body. Specifically, the scope 150 includes an insertion section 151, a bending section 152, and a tip section 153.

[0048] The insertion unit 151 can serve to insert a distal end unit 153 (described later) to any position inside the body that is the target of observation and treatment. The insertion unit 151 can be connected to one end of the operation unit 140.

[0049] The bending portion 152 may be connected to one end of the insertion portion 151. The bending portion 152 may change the angle of the tip portion 153, which will be described later. The bending portion 152 may be flexibly bent. The bending of the bending portion 152 may change the angle of the tip portion 153. The bending portion 152 may be connected to the driving portion 130, and may be supplied with a force required to change the angle of the tip portion 153. The degree or direction of bending of the bending portion 152 may be determined by the driving portion 130.

[0050] The tip 153 may be connected to one end of the curved portion 152. The tip 153 may be used to take images of the inside of the body and, if necessary, to perform treatment or therapy. The tip 153 may also include an image sensor 153a, a lens 153b, a light 153c, a working channel 153d, and an air / water channel 153e.

[0051] The image sensor 153a may serve to acquire an image of the inside of the body. The image of the inside of the body may include a video image formed by a series of multiple frames. The image of the inside of the body may be output via the display unit 110.

[0052] The angle of view of image sensor 153a also includes a first region. The first region also includes the center of the angle of view. In image 300 (FIG. 3) of the inside of the body, portion 310 (FIG. 3) corresponding to the first region also includes the center of image 300 (see FIG. 3). Therefore, if an object is located within the first region, the object may be located near the center of the image output via display unit 110.

[0053] In one embodiment, the image sensor 153a includes a plurality of image sensors.

[0054] The lens 153b can serve as a passage through which light reflected inside the body can enter the image sensor 153a. The illuminator 153c can irradiate light into the inside of the body so that the image sensor 153a can capture an image of the inside of the body. The number of illuminators 153c is not particularly limited. The working channel 153d can be used to insert a tool for treating and processing the lesion LE. The air / water channel 153e can be used to supply air or irrigation water.

[0055] The control unit 120 will be specifically described below.

[0056] 2 is a diagram illustrating an example in which a pre-trained model 200 identifies a lesion LE (FIG. 3) from an internal body image A according to an embodiment of the present invention. FIG. 3 illustrates an image 300 output to the display unit 110 when a lesion LE is identified.

[0057] The control unit 120 also includes a processor for performing various calculations or operations, which will be described later. The processor may interpret a computer program and perform data processing for machine learning. The processor may process input data for machine learning, feature extraction for machine learning, and error calculations based on backpropagation. Processors for performing such data processing may include a central processing unit (CPU), a general-purpose graphics processing unit (GPGPU), a tensor processing unit (TPU), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA), etc. However, these are merely examples, and the type of processor may be configured in a variety of ways within the scope of what a person of ordinary skill in the art can understand based on the present disclosure.

[0058] The control unit 120 can acquire an image of the inside of the body from the image sensor 153a, identify at least one lesion LE (Figure 3) from the image based on the pre-trained model 200, and calculate size information (R) of the lesion.

[0059] The control unit 120 may train the model 200 in advance. The model 200 may receive an image of the inside of the body as input, and may be trained to identify a lesion LE from the image of the inside of the body and calculate lesion size information (S) on the image. The model 200 may receive the image of the inside of the body as input training data. The model 200 may also receive label data corresponding to each image of the inside of the body, in which the size of the lesion is displayed in a bounding box.

[0060] The model 200 may also include a network structure such as a fully convolutional network (FCN), a conditional adversarial network (CAN), a recurrent neural network (RNN), or a matching cost-CNN (MC-CNN).

[0061] Referring to FIG. 2, the control unit 120 may identify a lesion LE from an image A based on a pre-trained model 200. The control unit 120 may input an image A of the inside of the body acquired from the image sensor 153a to the pre-trained model 200. The pre-trained model 200 to which the image A of the inside of the body is input may identify the lesion LE and calculate and output lesion size information (S) on the image. The lesion size information (S) on the image is also information indicating the size of the lesion LE on the image output to the display unit 110. In one embodiment, as shown in FIG. 2, the lesion size information (S) on the image also includes a bounding box B (FIG. 3) displayed to surround the lesion LE.

[0062] For example, as shown in FIG. 2, if an internal body image A1 is input to pre-trained model 200, pre-trained model 200 may identify a lesion LEa contained in internal body image A1 and output a bounding box Ba surrounding the lesion LEa. If an internal body image A2 is input to pre-trained model 200, pre-trained model 200 does not output a bounding box because there is no lesion in internal body image A2. If an internal body image A3 is input to pre-trained model 200, pre-trained model 200 may identify a lesion LEb contained in internal body image A3 and output a bounding box Bb surrounding the lesion LEb.

[0063] In one embodiment, the lesion size information (S) on the image also includes coordinate information on the image related to the four line segments of the bounding box B. That is, the pre-trained model 200 can calculate coordinate information on the image related to the four line segments of the bounding box B.

[0064] For example, referring to FIG. 3, the pre-trained model 200 may calculate, as lesion size information (S) on an image, the x-coordinate (x1) of the left line segment of bounding box B, the x-coordinate (x2) of the right line segment of bounding box B, the y-coordinate (y1) of the line segment of bounding box B, and the y-coordinate (y2) of the top line segment of bounding box B. The horizontal size (xw) of bounding box B is the value obtained by subtracting the x-coordinate (x1) of the left line segment of bounding box B from the x-coordinate (x2) of the right line segment of bounding box B (xw = x2 - x1). The vertical size (yh) of bounding box B is the value obtained by subtracting the y-coordinate (y1) of the bottom line segment of bounding box B from the y-coordinate (y2) of the top line segment of bounding box B (yh = y2 - y1). The lesion size information (S) on the image also includes the horizontal size (xw) of the bounding box B and the vertical size (yh) of the bounding box B.

[0065] FIG. 4 is a diagram illustrating the first distance D inside the body.

[0066] The control unit 120 may calculate the first distance D. The first distance D is also a value required to calculate size information (R) of the lesion. Specifically, the first distance D can serve as a medium connecting numerical information on the image with numerical information on the actual interior of the body. The first distance D also includes the distance between the lesion LE and the tip 153. More specifically, the first distance D also includes the distance between the lesion LE and the lens 153b.

[0067] In one embodiment, the control unit 120 may calculate the first distance D based on brightness information of the lesion LE represented in the image. The control unit 120 also stores in advance the distance from the tip 153 to an object inside the body and corresponding brightness data on the image of the object inside the body reflected by the light 153c. Therefore, by substituting the brightness of the lesion LE on the image into the data, the distance from the tip 153 to the lesion LE, i.e., the first distance D, may be calculated.

[0068] In one embodiment, the first distance D may be calculated by averaging the distances obtained by substituting the brightness at multiple points within the lesion LE into the data.

[0069] In another embodiment, the control unit 120 may acquire a first image (i1) when the tip 153 is at a first angle (a) and a second image (i2) when the tip 153 is at a second angle (b), and measure the difference (Δθ) between the first angle (a) and the second angle (b) and the visual deviation (ΔL) between the first image (i1) and the second image (i2).

[0070] The first angle (a) and the second angle (b) are two arbitrary angles different from each other for calculating the first distance D, and may refer to the angle between the distal end 153 formed by bending the bending portion and the insertion portion 151. Note that a change in the angle of the distal end 153 causes a change in the position of the image sensor 153a, and the change in the position of the image sensor 153a may cause the position of a given point (W) on the image to appear to have changed. The visual disparity may refer to the degree to which the position of a given point (W) on the image appears to have changed due to the change in the position of the image sensor 153a. Specifically, the control unit 120 may calculate the first distance D using the following Equation 1: [Number 1] D=(RΔθ×f) / ΔL

[0071] In Equation 1, Δθ is the difference between the first angle (a) and the second angle (b), and ΔL is the visual deviation between the first image (i1) and the second image (i2). R is the radius of rotation of the curved portion 152, so RΔθ represents the displacement of the image sensor 153a. f is the focal length of the lens 153b.

[0072] In yet another embodiment, the control unit 120 may calculate the first distance D using a stereo method based on a plurality of image sensors 153a.

[0073] In yet another embodiment, the control unit 120 may calculate the first distance D by irradiating the lesion LE with structured light, such as a stripe or grid pattern, and measuring the distortion of the light pattern returned from the lesion LE to the image sensor 153a.

[0074] In yet another embodiment, the control unit 120 may calculate the first distance D by irradiating the lesion LE with light and then measuring the travel time of the light reflected from the lesion LE.

[0075] FIG. 5 is a diagram showing the concept of acquiring lesion size information (R) based on lesion size information (S) on an image.

[0076] The control unit 120 can calculate size information (R) of the lesion based on the first distance D. The control unit 120 can calculate size information based on the lesion size information (S) on the image and the first distance D.

[0077] In one embodiment, the control unit 120 may obtain the size information (R) of the lesion by using a regression model for a proportional value relating to how the size of one pixel on the image corresponds to the actual size at a point a first distance D away from the distal end 153. Referring to FIGS. 3 and 5, the size information (R) of the lesion may be obtained by multiplying the proportional value by the number of pixels corresponding to the size information on the image of a bounding box B surrounding the lesion LE. As a specific example, the actual width (RW) of the lesion may be obtained by multiplying the width (xw) of the bounding box B by the proportional value. The actual height (RH) of the lesion may be obtained by multiplying the height (yh) of the bounding box B by the proportional value.

[0078] In another embodiment, the control unit 120 may obtain size information (R) of the lesion using the first distance D and the focal length (f) of the lens 153b. The focal length (f) of the lens 153b, the size (xw, yh) of the bounding box B on the image, the first distance D, and the actual sizes RW and RH of the lesion may be expressed by a proportional formula such as the following Equation 2. [Number 2] xw f=RW:D yh f=RH:D

[0079] Therefore, the actual sizes RW and RH of the lesion included in the size information (R) of the lesion can be calculated using the following Equation 3, which is a modification of Equation 2. [Number 3] RW={(xw)×D} / f RH={(yh)×D} / f

[0080] 6 is a diagram showing an image 300′ output on the display unit 110 when a lesion LE is located within the first region. FIG. 7 is a diagram specifically showing a method for calculating the first value (K) based on the position difference on the image caused by the angle difference of the tip 153.

[0081] The control unit 120 may control the tip unit 153 so that the tip unit 153 tracks the lesion LE. Specifically, the control unit 120 may control the tip unit 153 so that the lesion LE is located within a first region, as shown in FIG. 6 . That is, by controlling the tip unit 153 so that the identified lesion LE is located within the first region, the identified lesion LE may be located close to the center of the image 300′ output via the display unit 110. As a result, the identified lesion LE may be continuously output to the display unit 110.

[0082] In one embodiment, the center coordinates of the bounding box B can be calculated from coordinate information on the image related to the four line segments of the bounding box B. Referring to FIG. 3, in the image 300 of the inside of the body output to the display unit 110, the center coordinates (X O ,Y O The center coordinates (X O ,Y O ) can be said to be the center coordinate of the identified lesion LE. [Number 4] (X O =(X1+X2) / 2,Y O =(Y1+Y2) / 2)

[0083] The control unit 120 calculates the first value (K) and calculates the center coordinates (X O ,Y O), and the first value (K), the total movement angle (TMA) of the tip is calculated, and the movement angular velocity (as) of the tip 153 and the control period (cp) are taken into consideration to calculate the target angle (ag) for the current time, and the tip 153 can be moved by the target angle (ag) for the current time.

[0084] The control unit 120 may calculate a first value (K). The first value (K) is a ratio of a position difference on an image to an angle change of the tip portion 153. The first value (K) is also a value required to calculate a total movement angle (TMA) of the tip portion 153.

[0085] As an embodiment, referring to FIG. 7, the control unit 120 may acquire a third image (i3) when the tip portion 153 is at a third angle (c) and a fourth image (i4) when the tip portion 153 is at a fourth angle (d), and measure the difference (Δδ) between the third angle (c) and the fourth angle (d) and the position difference (ΔX) between the third image (i3) and the fourth image (i4).

[0086] The third angle (c) and the fourth angle (d) are any two different angles for calculating the first value (K), and may refer to the angle between the distal end 153 formed by bending the bending portion 152 and the insertion portion 151. The position difference (ΔX) between the third image (i3) and the fourth image (i4) also includes the difference in the coordinates on the images relating to an arbitrary point (X) inside the body. In other words, it also includes the difference between the coordinate of the arbitrary point (X) on the third image (i3) and the coordinate of the arbitrary point (X) on the fourth image (i4).

[0087] 7, when the control unit 120 operates the bending portion 152 to set the distal end portion 153 at a third angle (c), the image sensor 153a may acquire a third image (i3). At this time, the coordinates of an arbitrary point (X) inside the body on the image are also (xc, yc). The method of specifying the arbitrary point (X) is not particularly limited. In one embodiment, the arbitrary point (X) may be the position of a specific type of blood vessel inside the body found on the image.

[0088] Furthermore, when the control unit 120 operates the bending portion 152 and sets the distal end portion 153 at a fourth angle (d), the image sensor 153a can acquire a fourth image (i4). At this time, the coordinates on the image of an arbitrary point (X) inside the body are also (xd, yd).

[0089] The coordinate difference between the third image (i3) and the fourth image (i4) can be expressed as (Δx, Δy), which is the difference in the coordinates of an arbitrary point (X) on the image. In this case, Δx = xd-xc, and Δy = yd-yc. Using this, the position difference (ΔX) between the third image (i3) and the fourth image (i4) can be expressed as the following Equation 5.

number

[0090] The first value (K) can be calculated by substituting the difference (Δδ) between the third angle (c) and the fourth angle (d) and the position difference (ΔX) between the third image (i3) and the fourth image (i4) into the following equation 6. [Number 6] K=ΔX / Δδ

[0091] In one embodiment, calculating a first value (S241) (FIG. 11) may use the first image (i1) and the second image (i2) acquired in calculating a first distance (S130) (FIG. 8). Specifically, the difference (Δδ) between the third angle (c) and the fourth angle (d) may be calculated using the difference (Δθ) between the first angle (a) and the second angle (b). In addition, the position difference (ΔX) between the third image (i3) and the fourth image (i4) may be calculated using the difference between the coordinates (xa, ya) of the arbitrary point (W) on the first image (i1) and the coordinates (xb, yb) of the arbitrary point (W) on the second image (i2).

[0092] The control unit 120 calculates the center coordinates (X O ,Y O ), and the first value (K), the total movement angle (TMA) of the tip can be calculated. The total movement angle (TMA) is calculated based on the center coordinates (X O ,Y O) to the center coordinates (0, 0) on the image. The length (Q) from the center of the lesion LE to the center on the image is given by the following Equation 7.

number

[0093] The total movement angle (TMA) can be calculated by substituting the length (Q) from the center of the lesion LE to the center on the image and the calculated first value (K) into the following equation 8. [Number 8] (TMA)=Q / K

[0094] The control unit 120 can calculate the target angle (ag) at the current time in consideration of the movement angular velocity (as) and control period (cp) of the tip unit 153. The movement angular velocity (as) and control period (cp) of the tip unit 153 can be set in advance.

[0095] In one embodiment, the target angle (ag) at the current time may be calculated via a polynomial trajectory. For example, the target angle (ag) may be set by moving the tip unit 153 at a constant speed through a total movement angle (TMA). The preset angular velocity (as) of the tip unit 153 is 30° / s, and the preset control period (cp) is 2 ms. In this case, the target angle (ag) at the current time may be 30° / s×2 ms=0.2°. However, this is merely an example for illustrative purposes and is not limiting.

[0096] In another embodiment, the target angle (ag) at the current time may be calculated using a Bezier curve locus. That is, the method for calculating the target angle (ag) at the current time is not limited to a polynomial locus. In other words, the target angle (ag) at the current time may be calculated using various loci that can be thought of by ordinary skilled artisans.

[0097] The control unit 120 may move the tip unit 153 by a target angle (ag) at the current time. The control unit 120 may calculate a force required to move the tip unit 153 by the target angle (ag) at the current time, taking into account the dynamic characteristics of the bending unit 152. The control unit 120 may transmit information related to the calculated force to the driving unit 130, and move the tip unit 153 by the target angle (ag) at the current time.

[0098] 6, the control unit 120 may repeatedly perform the process of identifying the lesion, calculating the first distance D, and controlling the tip unit 153 until the lesion LE is located in the first region. As a result, the tip unit 153 may track the lesion LE, and the lesion LE may be continuously output to the display unit 110.

[0099] By controlling the distal end 153 so that the identified lesion LE is located within the first region, the lesion LE is positioned close to the center (C) of the angle of view of the image sensor 153a, which can increase the convenience and accuracy of calculating the first distance D. This can improve the reliability of the lesion size information (R). Furthermore, even a less skilled user can easily track and observe the lesion LE, allowing the user to focus on observing, treating, and managing the identified lesion LE without having to worry about operating the scope, thereby improving the quality of medical care.

[0100] FIG. 8 is a flowchart illustrating a method (M1) for obtaining size information of a lesion, according to one embodiment of the present invention.

[0101] The method (M1) for acquiring size information of a lesion is a method for identifying a lesion LE in an image of the inside of a body acquired from the endoscope device 100 and acquiring size information of the identified lesion LE.

[0102] Referring to FIG. 8, the method (M1) for acquiring size information of a lesion also includes a step (S110) of acquiring an image of the inside of the body from an image sensor 153a, a step (S120) of identifying at least one lesion LE from the image based on a pre-trained model 200, a step (S130) of calculating a first distance D, and a step (S140) of calculating size information (R) of the lesion.

[0103] The endoscope device 100 may acquire an image A of the inside of the body from the image sensor 153a (S110). The scope 150 may be inserted into the inside of the body, and the image of the inside of the body may be acquired via the image sensor 153a provided at the tip 153. The image of the inside of the body may include a video image formed by a series of multiple frames. The image of the inside of the body may be output via the display unit 110.

[0104] Referring to FIG. 2, the endoscope device 100 may identify a lesion LE from an image A based on a pre-trained model 200 (S120). The endoscope device 100 may input an image A of the inside of the body acquired from the image sensor 153a into the pre-trained model 200. The pre-trained model 200 to which the image A of the inside of the body is input may identify the lesion LE and calculate and output lesion size information (S) on the image. The lesion size information (S) on the image is also information indicating the size of the lesion LE on the image output to the display unit 110. In one embodiment, as shown in FIG. 2, the lesion size information (S) on the image also includes a bounding box B displayed to surround the lesion LE.

[0105] For example, as shown in FIG. 2, if an internal body image A1 is input to pre-trained model 200, pre-trained model 200 may identify a lesion LEa contained in internal body image A1 and output a bounding box Ba surrounding the lesion LEa. If an internal body image A2 is input to pre-trained model 200, pre-trained model 200 does not output a bounding box because there is no lesion in internal body image A2. If an internal body image A3 is input to pre-trained model 200, pre-trained model 200 may identify a lesion LEb contained in internal body image A3 and output a bounding box Bb surrounding the lesion LEb.

[0106] Model 200 (FIG. 2) can be trained to receive images of the inside of the body, identify lesions LE from the images, and calculate lesion size information (S) on the images. Model 200 can receive images of the inside of the body as training data. Model 200 can also receive label data corresponding to each image of the inside of the body, in which the location of the lesion is displayed in a bounding box.

[0107] The model 200 may also include a network structure such as a fully convolutional network (FCN), a conditional adversarial network (CAN), a recurrent neural network (RNN), or a matching cost-CNN (MC-CNN).

[0108] In one embodiment, the lesion size information (S) on the image also includes coordinate information on the image related to the four line segments of the bounding box B. That is, the pre-trained model 200 can calculate coordinate information on the image related to the four line segments of the bounding box B.

[0109] For example, referring to FIG. 3, the pre-trained model 200 may calculate, as lesion size information (S) on the image, the x-coordinate (x1) of the left line segment of bounding box B, the x-coordinate (x2) of the right line segment of bounding box B, the y-coordinate (y1) of the bottom line segment of bounding box B, and the y-coordinate (y2) of the top line segment of bounding box B. The horizontal size (xw) of bounding box B is the value obtained by subtracting the x-coordinate (x1) of the left line segment of bounding box B from the x-coordinate (x2) of the right line segment of bounding box B (xw = x2 - x1). The vertical size (yh) of bounding box B is the value obtained by subtracting the y-coordinate (y1) of the bottom line segment of bounding box B from the y-coordinate (y2) of the top line segment of bounding box B (yh = y2 - y1). The lesion size information (S) on the image also includes the horizontal size (xw) of the bounding box B and the vertical size (yh) of the bounding box B.

[0110] FIG. 9 is a flowchart illustrating the step of calculating the first distance (S130) in detail according to an embodiment of the present invention.

[0111] The endoscope device 100 can calculate a first distance D (S130). The first distance D is also a value required to calculate size information (R) of the lesion. Specifically, the first distance D can serve as a medium connecting numerical information on the image with numerical information about the actual interior of the body. The first distance D also includes the distance between the lesion LE and the distal end 153. More specifically, the first distance D also includes the distance between the lesion LE and the lens 153b.

[0112] 9, in one embodiment, the step of calculating the first distance (S130) also includes a step of calculating the first distance (S131) ​​based on brightness information of the lesion LE displayed in an image. The distance from the tip 153 to an object inside the body and corresponding brightness data on an image of the object inside the body reflected by the light 153c are also stored in advance. Therefore, by substituting the brightness of the lesion LE on the image into the data, the distance from the tip 153 to the lesion LE, i.e., the first distance D, can be calculated.

[0113] In one embodiment, an average may be used as the first distance D. Specifically, the first distance D may be calculated by averaging the distances obtained by substituting the brightness at multiple points within the lesion LE into the data.

[0114] In another embodiment, referring to FIG. 9, the step of calculating the first distance (S130) may also include a step (S132a) of acquiring a first image (i1) when the tip 153 is at a first angle (a) and a second image (i2) when the tip 153 is at a second angle (b), as well as a step (S132b) of measuring the difference (Δθ) between the first angle (a) and the second angle (b) and the visual deviation (ΔL) between the first image (i1) and the second image (i2).

[0115] The first angle (a) and the second angle (b) are two arbitrary angles different from each other for calculating the first distance D, and may refer to the angle between the distal end 153 formed by bending the bending portion 152 and the insertion portion 151. As the angle of the distal end 153 changes, the position of the image sensor 153a also changes, and as the position of the image sensor 153a changes, the position of an arbitrary point (W) viewed through an image appears to change. Visual disparity may refer to the degree to which the position of an arbitrary point (W) viewed through an image appears to change as the position of the image sensor 153a changes. Specifically, the first distance D may be calculated using Equation 1.

[0116] In yet another embodiment, the first distance D may be calculated using a stereo method based on a plurality of image sensors 153a.

[0117] In yet another embodiment, the first distance D can be calculated by irradiating the lesion LE with structured light of a specific pattern, such as a stripe pattern or grid, and measuring the distortion of the light pattern returning from the lesion LE to the image sensor 153a.

[0118] In yet another embodiment, the first distance D can be calculated by irradiating the lesion LE with light and then measuring the travel time of the light reflected from the lesion LE.

[0119] The endoscope device 100 can calculate size information (R) of the lesion based on the first distance D (S140). The size information can be calculated based on the lesion size information (S) on the image and the first distance D.

[0120] In one embodiment, the endoscope device 100 may obtain size information (R) of the lesion by using a regression model to obtain a proportional value relating to how the size of one pixel on the image corresponds to the actual size at a point a first distance D away from the distal end 153. Referring to FIGS. 3 and 5, the size information (R) of the lesion may be obtained by multiplying the proportional value by the number of pixels corresponding to the size information on the image of a bounding box B surrounding the lesion LE. As a specific example, the actual width RW of the lesion may be obtained by multiplying the width (xw) of the bounding box B by the proportional value. The actual height RH of the lesion may be obtained by multiplying the height (yh) of the bounding box B by the proportional value.

[0121] In another embodiment, the endoscope device 100 may obtain size information (R) of the lesion using the first distance D and the focal length (f) of the lens 153b. The focal length (f) of the lens 153b, the size (xw, yh) of the bounding box B on the image, the first distance D, and the actual sizes RW and RH of the lesion may be expressed by a proportional formula such as Equation 2.

[0122] Therefore, the actual sizes RW and RH of the lesion included in the size information (R) of the lesion can be obtained by using Equation 3 obtained by modifying Equation 2.

[0123] Figure 10 is a flowchart illustrating a method (M2) for acquiring size information of a lesion according to another embodiment of the present invention. Figure 11 is a flowchart illustrating a detailed step (S240) of controlling the tip portion 153 according to an embodiment of the present invention.

[0124] 10, the method M2 for acquiring size information of a lesion includes steps of acquiring an image A of the inside of the body from the image sensor 153a (S210), identifying at least one lesion LE from the image A based on the pre-trained model 200 (S220), calculating a first distance D (S230), controlling the tip unit 153 so that the tip unit 153 tracks the lesion LE (S240), and calculating size information of the lesion LE (S250). Here, the steps of acquiring an image (S210), identifying the lesion (S220), calculating the first distance (S230), and calculating size information of the lesion (S250) are the same as or similar to those described in the method M1 for acquiring size information of a lesion, and therefore, detailed description thereof will be omitted, and the following description will focus on the differences.

[0125] The endoscope device 100 can control the tip portion 153 so that the tip portion 153 tracks the lesion LE (S240). The step of controlling the tip portion (S240) also includes a step of controlling the tip portion 153 so that the lesion LE is located within the first region.

[0126] In one embodiment, the center coordinates of the bounding box B can be calculated from coordinate information on the image related to the four line segments of the bounding box B. Referring to FIG. 3, in the image 300 of the inside of the body output to the display unit 110, the center coordinates (XO ,Y O ) can be calculated from the coordinate information on the image related to the four line segments of the bounding box B as shown in Equation 4. O ,Y O ) can be said to be the center coordinate of the identified lesion LE.

[0127] The angle of view of image sensor 153a also includes the first region. The first region also includes the center of the angle of view. Referring to FIG. 6, in image 300′ of the inside of the body, portion 310 corresponding to the first region also includes the center of image 300′. Therefore, if an object is located within the first region, the object may be located close to the center of the image output via display unit 110.

[0128] That is, by controlling the tip 153 so that the identified lesion LE is located within the first region, the identified lesion LE can be positioned close to the center of the image output via the display 110. As a result, the identified lesion LE can be continuously output to the display 110.

[0129] Referring to FIG. 11, the step of controlling the tip (S240) includes a step of calculating a first value (K) (S241), a step of calculating the center coordinate (X) of the lesion on the image, and a step of calculating a second value (X). O ,Y O ), and the first value (K), a step (S242) of calculating a total movement angle (TMA) of the tip portion based on the first value (K), a step (S243) of calculating a target angle (ag) for the current time taking into account the movement angular velocity (as) of the tip portion 153 and the control period (cp), and a step (S244) of moving the tip portion 153 by the target angle (ag) for the current time.

[0130] The endoscope device 100 can calculate a first value (K) (S241). The first value (K) is a ratio of a position difference on an image to an angle change of the tip portion 153. The first value (K) is also a value required to calculate a total movement angle (TMA) of the tip portion 153.

[0131] In one embodiment, the step of calculating the first value (S241) also includes the steps of acquiring a third image (i3) when the tip 153 is at a third angle (c) and a fourth image (i4) when the tip 153 is at a fourth angle (d), and measuring the difference (Δδ) between the third angle (c) and the fourth angle (d) and the position difference (ΔX) between the third image (i3) and the fourth image (i4).

[0132] The third angle (c) and the fourth angle (d) are any two different angles for calculating the first value (K), and may refer to the angle between the distal end 153 formed by bending the bending portion 152 and the insertion portion 151. The position difference (ΔX) between the third image (i3) and the fourth image (i4) also includes the difference in the coordinates on the images relating to an arbitrary point (X) inside the body. In other words, it also includes the difference between the coordinate of the arbitrary point (X) on the third image (i3) and the coordinate of the arbitrary point (X) on the fourth image (i4).

[0133] 7, when the bending portion 152 is manipulated to set the distal end 153 at a third angle (c), a third image (i3) can be acquired by the image sensor 153a. At this time, the coordinates of an arbitrary point (X) inside the body on the image are also (xc, yc). The method of specifying the arbitrary point (X) is not particularly limited. In one embodiment, the arbitrary point (X) is the position of a specific type of blood vessel inside the body found on the image.

[0134] Furthermore, when the bending portion 152 is manipulated to set the distal end portion 153 at a fourth angle (d), a fourth image (i4) can be acquired by the image sensor 153a. At this time, the coordinates of an arbitrary point (X) inside the body on the image are also (xd, yd).

[0135] The coordinate difference between the third image (i3) and the fourth image (i4) can be expressed as (Δx, Δy), which is the difference in the coordinates of an arbitrary point (X) on the image. In this case, Δx = xd-xc, and Δy = yd-yc. Using this, the position difference (ΔX) between the third image (i3) and the fourth image (i4) can be expressed as Equation 5 above.

[0136] By substituting the difference (Δδ) between the third angle (c) and the fourth angle (d) and the position difference (ΔX) between the third image (i3) and the fourth image (i4) into Equation 6, the first value (K) can be calculated.

[0137] In one embodiment, calculating the first value (S241) may use the first image (i1) and the second image (i2) acquired in calculating the first distance (S130). Specifically, the difference (Δδ) between the third angle (c) and the fourth angle (d) may be calculated using the difference (Δθ) between the first angle (a) and the second angle (b). In addition, the position difference (ΔX) between the third image (i3) and the fourth image (i4) may be calculated using the difference between the coordinates (xa, ya) of the arbitrary point (W) on the first image (i1) and the coordinates (xb, yb) of the arbitrary point (W) on the second image (i2).

[0138] The endoscope device 100 calculates the center coordinates (X O ,Y O ), and the first value (K), the total movement angle (TMA) of the tip can be calculated (S242). The total movement angle (TMA) is calculated based on the center coordinates (X O ,Y O ) to the center coordinates (0, 0) on the image. The length (Q) from the center of the lesion LE to the center on the image is given by Equation 7 above.

[0139] The total movement angle (TMA) can be calculated by substituting the length (Q) from the center of the lesion LE to the center on the image and the calculated first value (K) into Equation 8.

[0140] The endoscope device 100 can calculate a target angle (ag) at the current time (S243) in consideration of the movement angular velocity (as) and control cycle (cp) of the tip portion 153. The movement angular velocity (as) and control cycle (cp) of the tip portion 153 can be set in advance.

[0141] In one embodiment, the target angle (ag) at the current time may be calculated via a polynomial trajectory. For example, the target angle (ag) may be set by moving the tip unit 153 at a constant speed through a total movement angle (TMA). The preset angular velocity (as) of the tip unit 153 is 30° / s, and the preset control period (cp) is 2 ms. In this case, the target angle (ag) at the current time is 30° / s×2 ms=0.2°. However, this is merely an example for illustrative purposes and is not limiting.

[0142] In another embodiment, the target angle (ag) at the current time may be calculated using a Bezier curve locus. That is, the method for calculating the target angle (ag) at the current time is not limited to the polynomial locus. In other words, the target angle (ag) at the current time may be calculated using various loci that ordinary skilled artisans can conceive.

[0143] The endoscopic device 100 may move the distal end portion 153 by the target angle (ag) at the current time (S244). The endoscopic device 100 may calculate a force required to move the distal end portion 153 by the target angle (ag) at the current time, taking into account the dynamic characteristics of the bending portion 152. The endoscopic device 100 may transmit information related to the calculated force to the driving unit 130, and move the distal end portion 153 by the target angle (ag) at the current time.

[0144] The step of identifying the lesion (S220), the step of calculating the first distance (S230), and the step of controlling the tip unit 153 (S240) may be repeatedly performed until the lesion LE is located in the first region, whereby the tip unit 153 may track the lesion LE and the lesion LE may be continuously output to the display unit 110.

[0145] By further including the step of controlling the tip portion 153 (S240), the lesion LE is positioned close to the center of the field of view (C) of the image sensor 153a, which may increase the convenience and accuracy of calculating the first distance D. This may improve the reliability of the lesion size information (R). Furthermore, even a less skilled user may be able to track and observe the lesion LE without difficulty, allowing the user to focus on observing, treating, and administering treatment to the identified lesion LE without having to worry about operating the scope, thereby improving the quality of medical care.

[0146] Although the present invention has been described above with reference to the embodiments shown in the drawings, these are merely examples. Those skilled in the art will appreciate that various modifications and equivalent embodiments are possible from the present embodiments. Therefore, the true technical scope of protection of the present invention should be determined based on the claims.

[0147] The specific description content described in this embodiment is one embodiment and does not limit the technical scope of this embodiment. For the sake of concise and clear description of the invention, descriptions of conventional general technologies and configurations may be omitted. Furthermore, line connections or connecting members between components shown in the drawings are illustrative of functional connections and / or physical or circuit connections, and may be expressed by various functional connections, physical connections, or circuit connections that can be substituted or added in an actual device. Furthermore, unless specifically referred to as "essential" or "important," a component is not necessarily required for application of the present invention.

[0148] Unless otherwise specified, the word "the," "the," or similar indicators in the description of the invention and the claims may refer to either the singular or the plural. Furthermore, when a range is described in the present embodiment, it encompasses the invention to which each individual value within that range is applied (unless otherwise specified), and each individual value constituting the range is described in the description of the invention. Furthermore, unless a clear order is stated for steps constituting a method according to the present embodiment, or unless there is a contrary statement, the steps may be performed in any suitable order. The order in which the steps are described does not necessarily limit the present embodiment. The use of all examples or exemplary terms (e.g., "etc.") in the present embodiment is merely intended to describe the present embodiment in detail. Since the scope of the present embodiment is not limited by the claims, the scope of the present embodiment is not limited by the examples or exemplary terms. Furthermore, a person of ordinary skill in the art will recognize that various modifications, combinations, and variations may be made depending on design conditions and factors within the scope of the claims or their equivalents. [Explanation of symbols]

[0149] 1 Endoscopic device 110 Display section 120 control section 130 Drive unit 140 Operation section 150 Scope 151 Insertion part 152 curved section 153 Tip 153a Image sensor 153b Lens 153c lighting 200 models How to obtain size information for M1 and M2 lesions LE lesions B Bounding Box D First distance Lesion size information on S images R Lesion size information K first value

Claims

1. acquiring an image of the interior of the body from an image sensor; identifying at least one lesion from the image based on a pre-trained model; and calculating size information of the lesion.

2. further comprising calculating a first distance; The step of calculating size information of the lesion includes: The method for acquiring size information of a lesion according to claim 1 , further comprising calculating size information of the lesion based on the first distance.

3. The step of calculating the first distance includes: The method for acquiring size information of a lesion according to claim 2 , further comprising the step of calculating the first distance based on brightness information of the lesion represented in the image.

4. The step of calculating the first distance includes: acquiring a first image when the tip is at a first angle and a second image when the tip is at a second angle; and measuring a difference between the first angle and the second angle and a visual deviation between the first image and the second image.

5. identifying the lesion includes: The method of claim 1 , further comprising the step of calculating lesion size information on the image, the size information of the lesion being indicated on the image.

6. The method of claim 5 , wherein the lesion size information on the image includes a bounding box displayed around the lesion.

7. The method of claim 1 , further comprising controlling the tip so that the tip tracks the lesion.

8. the image sensor has an angle of view that includes a first region; The step of controlling the tip includes:

8. The method of claim 7, further comprising controlling the tip so that the lesion is located within the first region.

9. The step of controlling the tip includes:

8. The method of claim 7, further comprising calculating a first value.

10. a tip portion having an image sensor; and a control unit that acquires an image of the inside of the body from the image sensor, identifies at least one lesion from the image based on a pre-trained model, and calculates size information of the lesion.

11. The control unit Calculating a first distance; The endoscope apparatus according to claim 10 , further comprising: a first distance calculating unit configured to calculate size information of the lesion based on the first distance.

12. The control unit The endoscope apparatus according to claim 11 , wherein the first distance is calculated based on brightness information of the lesion represented in the image.

13. The control unit acquiring a first image when the tip is at a first angle and a second image when the tip is at a second angle; The endoscope apparatus of claim 11 , further comprising: measuring a difference between the first angle and the second angle and a visual deviation between the first image and the second image.

14. The control unit The endoscope apparatus according to claim 10 , further comprising: calculating lesion size information on the image that indicates size information of the lesion on the image.

15. The endoscope device according to claim 14 , wherein the lesion size information on the image includes a bounding box displayed to surround the lesion.

16. The control unit The endoscope device according to claim 10 , wherein the tip portion is controlled so that the tip portion tracks the lesion.

17. the image sensor has an angle of view that includes a first region; The control unit The endoscope device according to claim 16, wherein the distal end portion is controlled so that the lesion is located within the first region.

18. The endoscope apparatus according to claim 16 , wherein the control unit calculates a first value.

Images