Endoscope device and method of controlling that endoscope device

Abstract

To provide an endoscope device and a method of controlling the endoscope device.SOLUTION: The invention relates to an endoscope device that identifies a lesion based on a pre-trained model and tracks the identified lesion, and to a method of controlling the endoscope device. The method includes the steps of: acquiring an image of the inside of the body from an image sensor; identifying a lesion from the image based on the pre-trained model; and controlling a distal end portion so that the distal end portion tracks the lesion.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to an endoscope apparatus and a control method for an endoscope apparatus, and more particularly to an endoscope apparatus that identifies a lesion based on a pre-trained model and tracks the identified lesion, and a control method for the endoscope apparatus. [Background technology]

[0002] An endoscopic device is a medical instrument that inserts a scope into the body to observe organs and, if necessary, perform treatment or therapy. When using an endoscopic device, the user typically simultaneously operates the scope and performs medical procedures to check for abnormalities inside the body. In such a situation where the user's concentration is divided, there is a significant risk of overlooking lesions that require observation or therapy, depending on the user's level of skill or fatigue.

[0003] Furthermore, when a lesion is found during medical treatment using an endoscopic device, in order to observe the lesion or perform a tissue examination on the lesion, the scope must be continuously operated so that the image sensor continues to point toward the lesion. However, while concentrating on the medical treatment, it is often the case that the scope is overlooked. In such cases, the scope shakes, causing the lesion to move off the screen, and the problem of having to search for the lesion again arises.

[0004] The aforementioned background art is technical information that the inventor possessed in order to derive the present invention or that was acquired in the process of deriving the present invention, and is not necessarily publicly known art that was made public to the general public prior to the filing of the invention application. Summary of the Invention [Problem to be solved by the invention]

[0005] The present invention is intended to solve the above-mentioned problems, and aims to provide an endoscopic device that identifies lesions based on a pre-trained model and tracks the identified lesions, as well as a control method for the endoscopic device.

[0006] However, such problems are merely examples, and the problems to be solved by the present invention are not limited thereto. Problems not mentioned above will be clearly understood by a person having ordinary skill in the art to which the present invention pertains from the present specification and the attached drawings. [Means for solving the problem]

[0007] One embodiment of the present invention discloses a method for controlling an endoscopic device, including steps of acquiring an image of the inside of a body from an image sensor, identifying a lesion from the image based on a pre-trained model, and controlling the tip so that the tip tracks the lesion.

[0008] In this embodiment, the angle of view of the image sensor includes a first region, and controlling the tip portion also includes controlling the tip portion so that the lesion is located within the first region.

[0009] In this embodiment, the first region also includes the center of the angle of view.

[0010] In this embodiment, identifying the lesion also includes calculating lesion position information on the image, which indicates the location of the lesion on the image.

[0011] In this embodiment, controlling the tip also includes calculating a first value that is a ratio of a position difference on the image to an angle change of the tip.

[0012] In this embodiment, controlling the tip also includes acquiring a first image when the tip is at a first angle and a second image when the tip is at a second angle, and measuring a difference between the first angle and the second angle and a position difference between the first image and the second image.

[0013] In this embodiment, controlling the tip also includes calculating a total angle of movement of the tip based on the first value.

[0014] Another embodiment of the present invention discloses a method for controlling an endoscopic device, including steps of acquiring an image of the inside of a body from an image sensor, identifying multiple lesions from the image based on a pre-trained model, and controlling the tip portion so that the tip portion tracks one target lesion among the multiple lesions.

[0015] In this embodiment, the method further includes a step of outputting lesion position information on an image relating to the identified lesions, and a step of selecting the target lesion from the lesions in response to a user input on an operation unit.

[0016] In this embodiment, the user input may also include at least one of a single input, an input for more than a critical time, and a double input within a critical time.

[0017] Yet another embodiment of the present invention discloses an endoscopic device including a tip portion having an image sensor, and a control unit that controls the tip portion to acquire images of the inside of a body from the image sensor, identify lesions from the images based on a pre-trained model, and track the lesions.

[0018] In this embodiment, the angle of view of the image sensor includes a first region, and the control unit can control the tip unit so that the lesion is located within the first region.

[0019] In this embodiment, the first region also includes the center of the angle of view.

[0020] In this embodiment, the control unit can calculate lesion position information on the image that indicates the position of the lesion on the image.

[0021] In this embodiment, the control unit may calculate a first value that is a ratio of a position difference on the image to an angle change of the tip.

[0022] In this embodiment, the control unit may acquire a first image when the tip is at a first angle and a second image when the tip is at a second angle, and measure the difference between the first angle and the second angle and the position difference between the first image and the second image.

[0023] In the present embodiment, the control unit can calculate a total movement angle of the tip portion based on the first value.

[0024] Yet another embodiment of the present invention discloses an endoscopic device including a tip portion having an image sensor, and a control portion that controls the tip portion to acquire an image of the inside of a body from the image sensor, identify multiple lesions from the image based on a pre-trained model, and track one target lesion among the multiple lesions.

[0025] In this embodiment, the system further includes a display unit that outputs the image and an operation unit that inputs user commands, and the control unit outputs lesion position information on the image related to the identified multiple lesions to the display unit, and the target lesion can be selected from the multiple lesions in response to user input on the operation unit.

[0026] In this embodiment, the user input may also include at least one of a single input, an input for more than a critical time, and a double input within a critical time.

[0027] Other aspects, features, and advantages in addition to those described above will become apparent from the following detailed description of the invention, the claims, and the drawings. [Effects of the Invention]

[0028] An endoscopic device and a control method for an endoscopic device according to one embodiment of the present invention can detect lesions inside the body without being affected by the user's level of skill or fatigue, by using a pre-trained model to identify lesions.

[0029] In an endoscopic device and a control method for an endoscopic device according to one embodiment of the present invention, the tip automatically tracks identified lesions, allowing even less skilled users to track and observe lesions without difficulty. This allows the user to focus on observing the identified lesions or performing treatment and procedures without having to worry about operating the scope, thereby improving the quality of medical care.

[0030] The effects of the present invention are not limited to those mentioned above, and other effects not mentioned will be clearly understood by those having ordinary skill in the art to which the present invention pertains from this specification and the accompanying drawings. [Brief explanation of the drawings]

[0031] [Figure 1] 1 is a diagram illustrating an endoscope apparatus according to an embodiment of the present invention. [Figure 2] 1 is a diagram illustrating an example of a pre-trained model identifying lesions from internal body images, according to an embodiment of the present invention. [Figure 3] FIG. 10 is a diagram showing an image output on the display unit when a lesion is identified. [Figure 4] 10 is a diagram specifically showing a method for calculating a first value based on a position difference on an image caused by an angle difference of the tip portion. FIG. [Figure 5] FIG. 10 is a diagram showing an image output on the display unit when a lesion is located within a first region. [Figure 6]FIG. 10 is a diagram showing an example in which a target lesion is selected from among a plurality of lesions in response to a user input on an operation unit. [Figure 7] 1 is a flowchart illustrating a control method for an endoscope apparatus according to an embodiment of the present invention. [Figure 8] 1 is a flowchart illustrating the steps broken down for controlling a tip according to one embodiment of the present invention. [Figure 9] 10 is a flowchart illustrating a control method for an endoscope apparatus according to another embodiment of the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0032] The terms used in the present invention are merely used to describe specific embodiments and are not intended to limit the scope of other embodiments. A singular expression includes a plural expression unless the context clearly dictates otherwise. Terms used herein, including technical or scientific terms, may have the same meaning as commonly understood by a person of ordinary skill in the technical field described in the present invention. Among the terms used in the present invention, commonly defined terms should be interpreted as meanings that are identical to or similar to the meanings they have in the context of the related art, and should not be interpreted as ideal or overly formal unless explicitly defined in the present invention. In some cases, even if a term is defined in the present invention, it should not be interpreted as excluding embodiments of the present invention.

[0033] Hereinafter, various embodiments will be described in detail with reference to the accompanying drawings so that those skilled in the art can easily implement the present invention. However, the technical concept of the present invention may be embodied in various forms and is not limited to the embodiments described herein. In describing the embodiments disclosed herein, if a detailed description of related known technology is deemed to obscure the gist of the technical concept of the present invention, the detailed description of the known technology will be omitted. Identical or similar elements will be designated by the same reference numerals, and redundant description thereof will be omitted.

[0034] Here, the term "module" used in this embodiment refers to a component that performs a specific function, which is performed by software or hardware such as a field programmable gate array (FPGA) or an application-specific integrated circuit (ASIC). However, the "module" is not limited to being performed by software or hardware. The "module" may exist in the form of data stored on an addressable recording medium, or may be embodied by an instruction word, and configured to cause one or more processors to execute a specific function.

[0035] Software may include computer programs, code, instructions, or a combination of one or more of these, which may configure a processing device to operate as desired or may instruct the processing device, either independently or collectively. The software and / or data may be permanently or temporarily embodied in some type of machine, component, physical device, virtual device, computer storage medium or device, or transmitted signal wave to be interpreted by the processing device or to provide instructions or data to the processing device. The software may also be distributed across network-coupled computer systems, stored or executed in a distributed manner. The software and data may be stored in one or more computer-readable storage media. The software may be read into main memory from other computer-readable media, such as a data storage device, or from another device via a communication interface. The software instructions stored in main memory may cause a processor to perform the processes or steps described in detail below. Alternatively, hard-wired circuitry may be used in place of or in combination with software instructions to implement processes consistent with the principles of the invention. Thus, embodiments consistent with the principles of the invention are not limited to any specific combination of hardware circuitry and software.

[0036] The terms used in this application are merely used to describe specific embodiments and are not intended to limit the present invention. The singular terms include the plural terms unless the context clearly dictates otherwise. In this application, terms such as "comprise" or "have" specify the presence of a specified feature, number, step, operation, component, part, or combination thereof, but should be understood not to preclude the presence or addition of one or more other features, numbers, steps, operations, components, parts, or combinations thereof. Terms such as "first" and "second" may be used to describe various components, but the components are not limited by these terms. These terms are used solely to distinguish one component from another.

[0037] The term "learning model" as used herein includes all types of algorithms or methodologies used to learn or understand specific patterns or structures from data. That is, the term "learning model" includes not only machine learning models such as regression models, decision trees, random forests, support vector machines, k-nearest neighbors, naive phase, and clustering algorithms, but also deep learning models such as neural networks, convolutional neural networks, recurrent neural networks, Transformer-based neural networks, generative adversarial networks (GANs), and autoencoders. A "learning model" refers to a set of learned parameters or weights used to predict or classify an output for a specific input. The model can be trained through methods such as supervised learning, unsupervised learning, semi-supervised learning, and reinforcement learning. It also includes not only single models but also various learning methods and structures such as ensemble models, multimodal models, and models via transfer learning. Such learning models can be pre-trained on a computer device separate from the computer device that predicts the output for the input and used on another computer device.

[0038] The learning model according to an embodiment of the present invention also includes a model related to object detection and position estimation.

[0039] An endoscope apparatus and a control method for an endoscope apparatus according to an embodiment of the present invention will be described below with reference to FIGS.

[0040] FIG. 1 is a diagram showing an endoscope apparatus 100 according to an embodiment of the present invention.

[0041] The endoscopic device 100 is a medical instrument that can insert a scope 150 into the body to observe organs and, if necessary, perform treatment or therapy. Referring to FIG. 1, the endoscopic device 100 also includes a display unit 110, a control unit 120, a drive unit 130, an operation unit 140, and the scope 150.

[0042] An image may be output to the display unit 110. In other words, an internal body image acquired from an image sensor 153a (described later) may be output to the display unit 110. The output internal body image may include x-axis coordinates and y-axis coordinates (see FIG. 3).

[0043] The display unit 110 may include a display module that outputs visualized information, such as a liquid crystal display (LCD), a thin film transistor-liquid crystal display (TFT LCD), an organic light-emitting diode (OLED), a flexible display, or a 3D display, or that can implement a touch screen.

[0044] The display unit 110 can output lesion position information (P) on the image, which will be described later.

[0045] The control unit 120 can control the overall operation of the endoscopic device 100. The control unit 120 can also include all types of components that can process data. In one embodiment, the control unit 120 can also include a data processing device built into hardware having physically structured circuits to perform functions expressed by codes or instructions contained in a program. The data processing device built into hardware can also include processing devices such as a microprocessor, a central processing unit (CPU), a processor core, a multiprocessor, an application-specific integrated circuit (ASIC), and a field programmable gate array (FPGA).

[0046] The control unit 120 can control the angle of the tip unit 153 via the driving unit 130, which will be described later. A detailed description of the control unit 120 will be given later.

[0047] The drive unit 130 can provide the power necessary for the scope 150 (described later) to be inserted into or move within the body. For example, the drive unit 130 can include a plurality of motors connected to wires inside the scope 150 and a tension adjustment unit that adjusts the tension of the wires.

[0048] The operation unit 140 can input user commands and includes a plurality of buttons that provide various functions to control the angle of the distal end portion 153 (described later) and to perform various surgeries inside the body.

[0049] In one embodiment, the user's command may include a command to select a target lesion T (FIG. 6) from among the multiple lesions LE (FIG. 3) identified in the image. The target lesion T refers to the lesion LE that is the target to be tracked by the tip 153. In one embodiment, the operation unit 140 may include a selection button (not shown). Specific examples of this will be described later.

[0050] The scope 150 can be directly inserted into the body. Specifically, the scope 150 includes an insertion section 151, a bending section 152, and a tip section 153.

[0051] The insertion unit 151 can serve to insert a distal end unit 153 (described later) to any position inside the body that is the target of observation and treatment. The insertion unit 151 can be connected to one end of the operation unit 140.

[0052] The bending portion 152 may be connected to one end of the insertion portion 151. The bending portion 152 may change the angle of the tip portion 153, which will be described later. The bending portion 152 may be flexibly bent. The bending of the bending portion 152 may change the angle of the tip portion 153. The bending portion 152 may be connected to the driving portion 130, and may be supplied with a force required to change the angle of the tip portion 153. The degree or direction of bending of the bending portion 152 may be determined by the driving portion 130.

[0053] The tip 153 may be connected to one end of the curved portion 152. The tip 153 may be used to take images of the inside of the body and, if necessary, to perform treatment or therapy. The tip 153 may also include an image sensor 153a, a lens 153b, a light 153c, a working channel 153d, and an air / water channel 153e.

[0054] The image sensor 153a may serve to acquire an image of the inside of the body. The image of the inside of the body may include a video image formed by a series of multiple frames. The image of the inside of the body may be output via the display unit 110.

[0055] The angle of view of image sensor 153a also includes a first region. The first region also includes the center of the angle of view. In image 300 (FIG. 3) of the inside of the body, portion 310 (FIG. 3) corresponding to the first region also includes the center of image 300 (see FIG. 3). Therefore, if an object is located within the first region, the object may be located near the center of the image output via display unit 110.

[0056] The lens 153b can serve as a passage through which light reflected inside the body can enter the image sensor 153a. The illuminator 153c can irradiate light into the inside of the body so that the image sensor 153a can capture an image of the inside of the body. The number of illuminators 153c is not particularly limited. The working channel 153d can be used to insert a tool for treating and processing the lesion LE. The air / water channel 153e can be used to supply air or irrigation water.

[0057] The control unit 120 will be specifically described below.

[0058] 2 is a diagram illustrating an example in which a pre-trained model 200 identifies a lesion LE (FIG. 3) from an internal body image A according to an embodiment of the present invention. FIG. 3 illustrates an image 300 output to the display unit 110 when a lesion LE is identified.

[0059] The control unit 120 also includes a processor for performing various calculations or operations, which will be described later. The processor may interpret a computer program and perform data processing for machine learning. The processor may process input data for machine learning, feature extraction for machine learning, and error calculations based on backpropagation. Processors for performing such data processing may include a central processing unit (CPU), a general-purpose graphics processing unit (GPGPU), a tensor processing unit (TPU), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA), etc. However, these are merely examples, and the type of processor may be configured in various ways within the scope of what a person of ordinary skill in the art can understand based on the present disclosure.

[0060] The control unit 120 can acquire an image of the inside of the body from the image sensor 153a, identify a lesion LE from the image based on the pre-trained model 200, and control the tip unit 153 so that the tip unit 153 tracks the lesion LE.

[0061] The control unit 120 may train the model 200 in advance. The model 200 may receive an image of the inside of the body as input, and may be trained to identify a lesion LE from the image of the inside of the body and calculate lesion position information (P) on the image. The model 200 may receive the image of the inside of the body as input training data. The model 200 may also receive label data corresponding to each image of the inside of the body, in which the position of the lesion is displayed in a bounding box.

[0062] The model 200 may also include a network structure such as a fully convolutional network (FCN), a conditional adversarial network (CAN), a recurrent neural network (RNN), or a matching cost-CNN (MC-CNN).

[0063] Referring to FIG. 2, the control unit 120 may identify a lesion LE from an image A based on a pre-trained model 200. The control unit 120 may input an image A of the inside of the body acquired from the image sensor 153a to the pre-trained model 200. The pre-trained model 200 to which the image A of the inside of the body is input may identify the lesion LE and calculate and output lesion position information (P) on the image. The lesion position information (P) on the image is also information indicating the position of the lesion LE on the image output to the display unit 110. In one embodiment, as shown in FIG. 2, the lesion position information (P) on the image also includes a bounding box B (FIG. 3) surrounding the periphery of the lesion LE.

[0064] For example, as shown in FIG. 2, when an internal body image A1 is input in time 200, the pre-trained model 200 may identify a lesion LEa contained in the internal body image A1 and output a bounding box Ba surrounding the lesion LEa. When an internal body image A2 is input in time 200, the pre-trained model 200 does not output a bounding box because there is no lesion in the internal body image A2. When an internal body image A3 is input in time 200, the pre-trained model 200 may identify a lesion LEb contained in the internal body image A3 and output a bounding box Bb surrounding the lesion LEb.

[0065] In one embodiment, the lesion position information (P) on the image also includes coordinate information on the image related to the four line segments of the bounding box B. That is, the pre-trained model 200 can calculate coordinate information on the image related to the four line segments of the bounding box B.

[0066] In one embodiment, the center coordinates of bounding box B can be calculated from image coordinate information related to the four line segments of bounding box B. Referring to FIG. 3, in an internal body image 300 output on display unit 110, the center coordinates (x1, y1) of bounding box B of lesion LE identified by pre-trained model 200 can be calculated from image coordinate information related to the four line segments of bounding box B. The center coordinates (x1, y1) of bounding box B can be said to be the center coordinates of the identified lesion LE.

[0067] Figure 4 is a diagram specifically illustrating a method for calculating the first value (K) based on the position difference on the image caused by the angle difference of the tip 153. Figure 5 is a diagram illustrating an image 300' output on the display unit 110 when the lesion LE is located within the first region.

[0068] The control unit 120 may control the tip unit 153 so that the tip unit 153 tracks the lesion LE. Specifically, the control unit 120 may control the tip unit 153 so that the lesion LE is located within the first region. This may position the identified lesion LE close to the center of the image 300′ output via the display unit 110. As a result, the identified lesion LE may be continuously output to the display unit 110.

[0069] The control unit 120 calculates a first value (K) which is the ratio of the position difference on the image to the angle change of the tip unit 153, calculates the total movement angle (TMA) of the tip unit based on the lesion position information (P) on the image and the first value (K), calculates the target angle (ag) at the current time taking into account the movement angular velocity (as) and control period (cp) of the tip unit 153, and moves the tip unit 153 by the target angle (ag) at the current time.

[0070] The control unit 120 may calculate a first value (K) that is a ratio of a position difference on an image to an angle change of the tip unit 153. The first value (K) is a value required to calculate a total movement angle (TMA) of the tip unit 153, which will be described later.

[0071] In one embodiment, the control unit 120 may acquire a first image (i1) when the tip portion 153 is at a first angle a and a second image (i2) when the tip portion 153 is at a second angle b, and measure a difference (Δθ) between the first angle a and the second angle b and a position difference (ΔX) between the first image (i1) and the second image (i2). The first angle a and the second angle b are two arbitrary angles different from each other for calculating the first value (K), and may refer to the angle between the tip portion 153 formed by bending the bending portion 152 and the insertion portion 151.

[0072] 4, when the control unit 120 operates the bending portion 152 and sets the tip portion 153 at a first angle a, the control unit 120 may acquire a first image (i1) using the image sensor 153a. At this time, the coordinates of an arbitrary point X inside the body on the image are also (xa, ya). The method of specifying the arbitrary point X is not particularly limited. In one embodiment, the arbitrary point X may be the position of a blood vessel of a particular shape inside the body found on the image.

[0073] Furthermore, when the control unit 120 operates the bending portion 152 to set the tip portion 153 at a second angle b, the image sensor 153a can acquire a second image (i2). At that time, the coordinates on the image of an arbitrary point X inside the body are also (xb, yb).

[0074] The coordinate difference between the first image (i1) and the second image (i2) can be expressed as (Δx, Δy), which is the difference in the coordinates on the image of an arbitrary point X. In this case, Δx = xb - xa, and Δy = yb - ya. Using this, the position difference (ΔX) between the first image (i1) and the second image (i2) can be expressed as the following Equation 1.

number

[0075] The first value (K) can be calculated by substituting the difference (Δθ) between the first angle a and the second angle b, and the position difference (ΔX) between the first image (i1) and the second image (i2) into the following equation 2.

number

[0076] The control unit 120 may calculate a total movement angle (TMA) of the tip 153 based on the lesion position information (P) on the image and the first value (K). The total movement angle (TMA) refers to the total angle that the tip 153 must move in order to move the center coordinates (x1, y1) of the lesion LE to the center coordinates (0, 0) on the image. Referring to FIG. 3, the length (Q) from the center of the lesion LE to the center on the image is given by the following Equation 3:

number

[0077] The total movement angle (TMA) can be calculated by substituting the length (Q) from the center of the lesion LE to the center on the image and the calculated first value (K) into the following equation 4.

number

[0078] The control unit 120 may calculate the target angle (ag) at the current time in consideration of the angular velocity (as) and control period (cp) of the tip unit 153. The angular velocity (as) and control period (cp) of the tip unit 153 may be set in advance.

[0079] In one embodiment, the target angle (ag) at the current time may be calculated via a polynomial trajectory. For example, the target angle (ag) may be set by moving the tip unit 153 at a constant speed through a total movement angle (TMA). The preset angular velocity (as) of the tip unit 153 is 30° / s, and the preset control period (cp) is 2 ms. In this case, the target angle (ag) at the current time is 30° / s×2 ms=0.2°. However, this is merely an example for illustrative purposes and is not limiting.

[0080] In another embodiment, the target angle (ag) at the current time may be calculated using a Bezier curve locus. That is, the method for calculating the target angle (ag) at the current time is not limited to a polynomial locus. In other words, the target angle (ag) at the current time may be calculated using various loci that can be thought of by ordinary skilled artisans.

[0081] The control unit 120 may move the tip unit 153 by a target angle (ag) at the current time. The control unit 120 may calculate a force required to move the tip unit 153 by the target angle (ag) at the current time, taking into account the dynamic characteristics of the bending unit 152. The control unit 120 may transmit information related to the calculated force to the driving unit 130, and move the tip unit 153 by the target angle (ag) at the current time.

[0082] 5, the control unit 120 may repeatedly perform the process of identifying the lesion, calculating the first value (K), and controlling the tip unit 153 until the lesion LE is located in the first region, so that the tip unit 153 can track the lesion LE and the lesion LE can be continuously output to the display unit 110.

[0083] FIG. 6 is a diagram showing an example in which a target lesion T is selected from among a plurality of lesions LE in response to a user input on the operation unit 140. In FIG.

[0084] The control unit 120 may identify multiple lesions LE from an image based on the pre-trained model 200. For example, referring to FIG. 6, a first lesion LE1, a second lesion LE2, and a third lesion LE3 may be identified in an image acquired from the image sensor 153a. The pre-trained model 200 may output bounding boxes surrounding the first lesion LE1, the second lesion LE2, and the third lesion LE3, respectively.

[0085] A target lesion T can be selected from among the multiple lesions LE in response to a user input on the operation unit 140. Specifically, in response to a user input, the lesion LE regarded as the target lesion can be changed from among the multiple lesions LE, and the lesion LE regarded as the target lesion can be selected as the target lesion T. As described above, the user can select the target lesion T from among the multiple lesions LE using a selection button (not shown) provided on the operation unit 140.

[0086] In one embodiment, the user input may include at least one of a single input, an input for a critical time or more, and a double input within the critical time. For example, the single input may be a command to change the lesion LE regarded as the lesion LE among the multiple lesions LE. Meanwhile, the double input within the critical time may be a command to select the lesion LE regarded as the lesion LE as the target lesion T. However, these are merely examples and are not limited thereto.

[0087] In one embodiment, as shown in FIG. 6, the lesion LE that has been identified may be identified by the difference in thickness of the bounding box. Using the above example, this can be explained in more detail as follows: If the user presses the selection button once, the lesion LE that has been identified may change from the first lesion LE1 to the second lesion LE2. If the user presses the selection button again, the lesion LE that has been identified may change from the second lesion LE2 to the third lesion LE3. If the user then presses the selection button a second time within the critical time, the third lesion LE3 that has been identified may be selected as the target lesion T.

[0088] When multiple lesions LE are identified in one image, the usefulness of the endoscope device 100 can be increased by tracking a target lesion T selected from the multiple lesions LE.

[0089] FIG. 7 is a flowchart showing a control method (M1) for an endoscope apparatus according to one embodiment of the present invention.

[0090] The control method (M1) of the endoscopic device is a method of identifying a lesion LE in an image of the inside of the body acquired from the endoscopic device 100, and controlling the endoscopic device 100 so that the tip 153 tracks the identified lesion LE and continuously outputs the lesion LE to the display unit 110.

[0091] Referring to FIG. 7, the control method (M1) of the endoscopic device also includes a step (S110) of acquiring an image A of the inside of the body from the image sensor 153a, a step (S120) of identifying a lesion LE from the image A based on a pre-trained model 200, and a step (S130) of controlling the tip portion 153 so that the tip portion 153 tracks the lesion LE.

[0092] The endoscope device 100 may acquire an image A of the inside of the body from the image sensor 153a (S110). The scope 150 may be inserted into the inside of the body, and the image of the inside of the body may be acquired via the image sensor 153a provided at the tip 153. The image of the inside of the body may include a video image formed by a series of multiple frames. The image of the inside of the body may be output via the display unit 110.

[0093] Referring to FIG. 2, the endoscope device 100 may identify a lesion LE from an image A based on a pre-trained model 200 (S120). The endoscope device 100 may input an image A of the inside of the body acquired from the image sensor 153a into the pre-trained model 200. The pre-trained model 200 to which the image A of the inside of the body is input may identify the lesion LE and calculate and output lesion position information (P) on the image. The lesion position information (P) on the image is also information indicating the position of the lesion LE on the image output to the display unit 110. In one embodiment, as shown in FIG. 2, the lesion position information (P) on the image also includes a bounding box B surrounding the periphery of the lesion LE.

[0094] For example, as shown in FIG. 2, if an internal body image A1 is input to pre-trained model 200, pre-trained model 200 may identify a lesion LEa contained in internal body image A1 and output a bounding box Ba surrounding the lesion LEa. If an internal body image A2 is input to pre-trained model 200, pre-trained model 200 does not output a bounding box because there is no lesion in internal body image A2. If an internal body image A3 is input to pre-trained model 200, pre-trained model 200 may identify a lesion LEb contained in internal body image A3 and output a bounding box Bb surrounding the lesion LEb.

[0095] Model 200 can be trained to receive input images of the inside of the body, identify lesions LE from the images of the inside of the body, and calculate lesion position information (P) on the images. Model 200 can receive input images of the inside of the body as training data. Model 200 can also receive input label data corresponding to each image of the inside of the body, in which the position of the lesion is displayed in a bounding box.

[0096] The model 200 may also include a network structure such as a fully convolutional network (FCN), a conditional adversarial network (CAN), a recurrent neural network (RNN), or a matching cost-CNN (MC-CNN).

[0097] In one embodiment, the lesion position information (P) on the image also includes coordinate information on the image related to the four line segments of the bounding box B. That is, the pre-trained model 200 can calculate coordinate information on the image related to the four line segments of the bounding box B.

[0098] In one embodiment, the center coordinates of bounding box B can be calculated from image coordinate information related to the four line segments of bounding box B. Referring to FIG. 3, in an internal body image 300 output on display unit 110, the center coordinates (x1, y1) of bounding box B of lesion LE identified by pre-trained model 200 can be calculated from image coordinate information related to the four line segments of bounding box B. The center coordinates (x1, y1) of bounding box B can be said to be the center coordinates of the identified lesion LE.

[0099] The angle of view of image sensor 153a also includes the first region. The first region also includes the center of the angle of view. Referring to FIG. 3, in image 300 of the inside of the body, portion 310 corresponding to the first region also includes the center of image 300. Therefore, if an object is located within the first region, the object may be located close to the center of the image output via display unit 110.

[0100] FIG. 8 is a flowchart showing a detailed step of controlling the tip unit (S130) according to an embodiment of the present invention.

[0101] The endoscope device 100 may control the tip unit 153 so that the tip unit 153 tracks the lesion LE (S130). The step of controlling the tip unit (S130) also includes a step of controlling the tip unit 153 so that the lesion LE is located within the first region. As a result, the identified lesion LE may be located close to the center of the image output via the display unit 110. As a result, the identified lesion LE may be continuously output to the display unit 110.

[0102] Referring to FIG. 8, the step of controlling the tip (S130) also includes a step of calculating a first value (K) which is a ratio of a position difference on an image to an angle change of the tip 153 (S131), a step of calculating a total movement angle (TMA) of the tip based on lesion position information on the image (P) and the first value (K) (S132), a step of calculating a target angle (ag) at the current time in consideration of the movement angular velocity (as) and control period (cp) of the tip 153, and a step of moving the tip 153 by the target angle (ag) at the current time (S134).

[0103] The endoscope device 100 can calculate (S131) ​​a first value (K) that is a ratio of a position difference on an image to an angle change of the tip portion 153. The first value (K) is a value required to calculate a total movement angle (TMA) of the tip portion 153.

[0104] In one embodiment, the step of calculating the first value (S131) ​​includes the steps of acquiring a first image (i1) when the tip portion 153 is at a first angle a, and a second image (i2) when the tip portion 153 is at a second angle b, and measuring a difference (Δθ) between the first angle a and the second angle b, and a position difference (ΔX) between the first image (i1) and the second image (i2). The first angle a and the second angle b are two arbitrary angles different from each other for calculating the first value (K), and may refer to the angle between the tip portion 153 and the insertion portion 151, formed by bending the bending portion 152.

[0105] 4, when the bending portion 152 is operated and the tip portion 153 is set at a first angle a, a first image (i1) can be acquired by the image sensor 153a. At this time, the coordinates of an arbitrary point X inside the body on the image are also (xa, ya). The method of specifying the arbitrary point X is not particularly limited. In one embodiment, the arbitrary point X is the position of a specific type of blood vessel inside the body found on the image.

[0106] Furthermore, when the bending portion 152 is manipulated to set the distal end portion 153 at a second angle b, a second image (i2) can be acquired by the image sensor 153a. At that time, the coordinates of an arbitrary point X inside the body on the image are also (xb, yb).

[0107] The coordinate difference between the first image (i1) and the second image (i2) can be expressed as (Δx, Δy), which is the difference in the coordinates on the image of an arbitrary point X. In this case, Δx = xb - xa, and Δy = yb - ya. Using this, the position difference (ΔX) between the first image (i1) and the second image (i2) can be expressed as Equation 1 above.

[0108] By substituting the difference (Δθ) between the first angle a and the second angle b and the position difference (ΔX) between the first image (i1) and the second image (i2) into Equation 2, the first value (K) can be calculated.

[0109] The endoscope device 100 may calculate a total movement angle (TMA) of the distal end portion based on the lesion position information (P) on the image and the first value (K) (S132). The total movement angle (TMA) refers to the total angle that the distal end portion 153 must move in order to move the center coordinates (x1, y1) of the lesion LE to the center coordinates (0, 0) on the image. Referring to FIG. 3, the length (Q) from the center of the lesion LE to the center on the image is given by Equation 3.

[0110] The total movement angle (TMA) can be calculated by substituting the length (Q) from the center of the lesion LE to the center on the image and the calculated first value (K) into Equation 4.

[0111] The endoscope device 100 can calculate a target angle (ag) at the current time (S133) in consideration of the movement angular velocity (as) and control period (cp) of the tip portion 153. The movement angular velocity (as) and control period (cp) of the tip portion 153 can be set in advance.

[0112] In one embodiment, the target angle (ag) at the current time may be calculated via a polynomial trajectory. For example, the target angle (ag) may be set by moving the tip unit 153 at a constant speed through a total movement angle (TMA). The preset angular velocity (as) of the tip unit 153 is 30° / s, and the preset control period (cp) is 2 ms. In this case, the target angle (ag) at the current time is 30° / s×2 ms=0.2°. However, this is merely an example for illustrative purposes and is not limiting.

[0113] In another embodiment, the target angle (ag) at the current time may be calculated using a Bezier curve locus. That is, the method for calculating the target angle (ag) at the current time is not limited to a polynomial locus. In other words, the target angle (ag) at the current time may be calculated using various loci that can be thought of by ordinary skilled artisans.

[0114] The endoscope device 100 may move the distal end portion 153 by the target angle (ag) at the current time (S134). The endoscope device 100 may calculate a force required to move the distal end portion 153 by the target angle (ag) at the current time, taking into account the dynamic characteristics of the bending portion 152. The endoscope device 100 may transmit information related to the calculated force to the driving unit 130, and move the distal end portion 153 by the target angle (ag) at the current time.

[0115] The step of identifying the lesion (S120) and the step of controlling the tip (S130) may be repeatedly performed until the lesion LE is located in the first region, whereby the tip 153 may track the lesion LE and the lesion LE may be continuously output to the display 110.

[0116] FIG. 9 is a flowchart showing a control method (M2) for an endoscope apparatus according to another embodiment of the present invention.

[0117] 9, the control method (M2) of an endoscope device includes a step (S210) of acquiring an image of the inside of a body from the image sensor 153a, a step (S220) of identifying a plurality of lesions LE from the image based on the pre-trained model 200, a step (S230) of outputting lesion position information (P) on the image related to the identified plurality of lesions LE, a step (S240) of selecting a target lesion T from the plurality of lesions LE in response to a user input on the operation unit 140, and a step (S250) of controlling the tip unit 153 so that the tip unit 153 tracks one target lesion T among the plurality of lesions LE. Among these, the step (S210) of acquiring the image, the step (S230) of outputting the position information on the image, and the step (S250) of controlling the tip unit are the same as or similar to those described in the control method (M1) of an endoscope device, and therefore detailed description thereof will be omitted, and the following description will focus on the differences.

[0118] The endoscope device 100 may identify multiple lesions LE from the image (S220) based on the pre-trained model 200. For example, referring to Fig. 6, a first lesion LE1, a second lesion LE2, and a third lesion LE3 may be identified in the image acquired from the image sensor 153a.

[0119] In the step (S230) of outputting lesion position information (P) on the image, the pre-trained model 200 may output bounding boxes surrounding the first lesion LE1, the second lesion LE2, and the third lesion LE3, respectively.

[0120] A target lesion T can be selected from the plurality of lesions LE in response to a user input on the operation unit 140 (S240). Specifically, the lesion LE regarded as the target lesion can be changed from the plurality of lesions LE in response to the user input, and the lesion LE regarded as the target lesion can be selected as the target lesion T. The user can select the target lesion T from the plurality of lesions LE using a selection button (not shown) provided on the operation unit 140.

[0121] In one embodiment, the user input may include at least one of a single input, an input for a critical time or more, and a double input within the critical time. For example, the single input may be a command to change the lesion LE regarded as the lesion LE among the multiple lesions LE. Note that the double input within the critical time may be a command to select the lesion LE regarded as the lesion LE as the target lesion T. However, these are merely examples and are not limited thereto.

[0122] In one embodiment, as shown in FIG. 6 , the lesion LE that has been deemed may be identified by the difference in thickness of the bounding box B. To explain this in more detail using the above example, if the user presses the selection button once, the deemed lesion LE may change from the first lesion LE1 to the second lesion LE2. If the user presses the selection button again, the deemed lesion LE may change from the second lesion LE2 to the third lesion LE3. If the user then presses the selection button a second time within the critical time, the deemed third lesion LE3 may be selected as the target lesion T.

[0123] When multiple lesions LE are identified in one image, the usefulness of the endoscope device 100 can be increased by tracking a target lesion T selected from the multiple lesions LE.

[0124] Although the present invention has been described above with reference to the embodiments shown in the drawings, these are merely examples. Those skilled in the art will appreciate that various modifications and equivalent embodiments are possible from the present embodiments. Therefore, the true technical scope of protection of the present invention should be determined based on the claims.

[0125] The specific description content described in this embodiment is one embodiment and does not limit the technical scope of this embodiment. For the sake of concise and clear description of the invention, descriptions of conventional general technologies and configurations may be omitted. Furthermore, line connections or connecting members between components shown in the drawings are illustrative of functional connections and / or physical or circuit connections, and may be expressed by various functional connections, physical connections, or circuit connections that can be substituted or added in an actual device. Furthermore, unless specifically referred to as "essential" or "important," a component is not necessarily required for application of the present invention.

[0126] Unless otherwise specified, the word "the," "the," or similar indicators in the description of the invention and the claims may refer to either the singular or the plural. Furthermore, when a range is described in the present embodiment, it encompasses the invention to which each individual value within that range is applied (unless otherwise specified), and each individual value constituting the range is described in the description of the invention. Furthermore, unless a clear order is stated for steps constituting a method according to the present embodiment, or unless there is a contrary statement, the steps may be performed in any suitable order. The order in which the steps are described does not necessarily limit the present embodiment. The use of all examples or exemplary terms (e.g., "etc.") in the present embodiment is merely intended to describe the present embodiment in detail. Since the scope of the present embodiment is not limited by the claims, the scope of the present embodiment is not limited by the examples or exemplary terms. Furthermore, a person of ordinary skill in the art will recognize that various modifications, combinations, and variations may be made depending on design conditions and factors within the scope of the claims or their equivalents. [Explanation of symbols]

[0127] 1 Endoscopic device 110 Display section 120 control section 130 Drive unit 140 Operation section 150 Scope 151 Insertion part 152 curved section 153 Tip 153a Image sensor 153b Lens 153c lighting 200 pre-trained models B Bounding Box K first value LE lesions M1, M2 Control method of endoscope device Lesion location information on P images

Claims

1. acquiring an image of the interior of the body from an image sensor; identifying lesions from the image based on a pre-trained model; and controlling the tip portion so that the tip portion tracks the lesion.

2. the image sensor has an angle of view that includes a first region; The step of controlling the tip includes: The method for controlling an endoscope apparatus according to claim 1 , further comprising the step of controlling the distal end portion so that the lesion is located within the first region.

3. The method for controlling an endoscope apparatus according to claim 2 , wherein the first region includes a center of the angle of view.

4. identifying the lesion includes: The control method for an endoscope apparatus according to claim 1 , further comprising the step of calculating lesion position information on the image that indicates the position of the lesion on the image.

5. The step of controlling the tip includes: The method for controlling an endoscope apparatus according to claim 1 , further comprising the step of calculating a first value that is a ratio of a position difference on the image to an angle change of the tip portion.

6. The step of calculating the first value includes: acquiring a first image when the tip is at a first angle and a second image when the tip is at a second angle; The method for controlling an endoscope apparatus according to claim 5, further comprising measuring a difference between the first angle and the second angle and a position difference between the first image and the second image.

7. The step of controlling the tip includes: The method for controlling an endoscope apparatus according to claim 5 , further comprising the step of calculating a total movement angle of the tip portion based on the first value.

8. acquiring an image of the interior of the body from an image sensor; identifying a plurality of lesions from the image based on a pre-trained model; and controlling the tip portion so that the tip portion tracks one target lesion among the plurality of lesions.

9. outputting lesion location information on the image relating to the identified lesions; The control method for an endoscope apparatus according to claim 8 , further comprising: selecting the target lesion from among the plurality of lesions in response to a user input on an operation unit.

10. The method of claim 9 , wherein the user input includes at least one of a single input, an input for more than a critical time, and a double input within a critical time.

11. a tip portion having an image sensor; and a control unit that acquires an image of the inside of the body from the image sensor, identifies a lesion from the image based on a pre-trained model, and controls the tip unit so that the tip unit tracks the lesion.

12. the image sensor has an angle of view that includes a first region; The control unit The endoscope device according to claim 11 , wherein the distal end portion is controlled so that the lesion is located within the first region.

13. The endoscope apparatus according to claim 12 , wherein the first region includes a center of the angle of view.

14. The control unit The endoscope apparatus according to claim 11 , further comprising: calculating lesion position information on the image that indicates the position of the lesion on the image.

15. The control unit The endoscope apparatus according to claim 11 , further comprising: a first value that is a ratio of a position difference on the image to an angle change of the tip portion.

16. The control unit acquiring a first image when the tip is at a first angle and a second image when the tip is at a second angle; The endoscope apparatus according to claim 15, wherein the difference between the first angle and the second angle and the position difference between the first image and the second image are measured.

17. The control unit The endoscope apparatus according to claim 15 , wherein a total movement angle of the tip portion is calculated based on the first value.

18. a tip portion having an image sensor; an endoscope device including: a control unit that acquires an image of the inside of the body from the image sensor, identifies a plurality of lesions from the image based on a pre-trained model, and controls the tip unit so that the tip unit tracks one target lesion among the plurality of lesions.

19. a display unit on which the image is output; an operation unit for inputting a user's command, The control unit outputting lesion position information on the image relating to the identified plurality of lesions to the display unit; The endoscope device according to claim 18 , wherein the target lesion is selected from the plurality of lesions in response to a user input on the operation unit.

20. The endoscopic device of claim 19 , wherein the user input includes at least one of a single input, an input for more than a critical time, and a double input within a critical time.

Images