Endoscope device and control method for said endoscope device

Abstract

To provide an endoscope device and a method of controlling the endoscope device.SOLUTION: The invention relates to an endoscope device that identifies a lesion based on a pre-trained model and tracks the identified lesion, and to a method of controlling the endoscope device. The method includes the steps of: acquiring an image of the inside of the body from an image sensor; identifying a lesion from the image based on the pre-trained model; and controlling a distal end portion so that the distal end portion tracks the lesion.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] The present invention relates to an endoscope device and a method for controlling an endoscope device. More specifically, the present invention relates to an endoscope device that identifies a lesion based on a pre-learned model and tracks the identified lesion, and a method for controlling the endoscope device.

Background Art

[0002] An endoscope device refers to a medical instrument that inserts a scope into the body, observes organs, and performs treatment or procedures if necessary. When using the endoscope device, a user generally performs the act of operating the scope and the medical act of checking for any abnormalities inside the body simultaneously. In such a situation where the user's concentration is dispersed, there are often problems of missing lesions that require observation or treatment due to the user's proficiency or fatigue level.

[0003] In addition, when performing medical treatment using an endoscope device and a lesion is found, in order to observe the discovered lesion or perform a tissue examination on the lesion, etc., it is necessary to continuously operate the scope so that the image sensor continuously faces the lesion. At that time, when concentrating on the medical act, it often happens that the scope is missed. In that case, the scope shakes, the lesion moves out of the screen, and the problem of having to search for the lesion again often occurs.

[0004] The above background art is technical information that the inventor possessed for deriving the present invention or acquired in the process of deriving the present invention, and it cannot necessarily be regarded as publicly known technology publicly disclosed to the general public before the filing of the invention.

Summary of the Invention

Problems to be Solved by the Invention

[0005] The present invention aims to solve the aforementioned problems and to provide an endoscopic device that identifies lesions based on a pre-trained model and tracks the identified lesions, as well as a method for controlling the endoscopic device.

[0006] However, such problems are illustrative, and the problems that the present invention seeks to solve are not limited thereto. Problems not mentioned can be clearly understood by a person with ordinary skill in the art to which the present invention pertains, from this specification and the accompanying drawings. [Means for solving the problem]

[0007] One embodiment of the present invention discloses a method for controlling an endoscope, which includes the steps of acquiring an image of the inside of a body from an image sensor, identifying a lesion from the image based on a pre-trained model, and controlling the tip so that the tip tracks the lesion.

[0008] In this embodiment, the field of view of the image sensor includes a first region, and the step of controlling the tip portion also includes the step of controlling the tip portion so that the lesion is located within the first region.

[0009] In this embodiment, the first region also includes the center of the field of view.

[0010] In this embodiment, the step of identifying the lesion also includes the step of calculating image lesion position information that indicates the location of the lesion on the image.

[0011] In this embodiment, the step of controlling the tip also includes the step of calculating a first value which is the ratio of the positional difference on the image to the angle change of the tip.

[0012] In this embodiment, the step of controlling the tip portion also includes the step of acquiring a first image when the tip portion is at a first angle and a second image when the tip portion is at a second angle, and the step of measuring the difference between the first angle and the second angle, and the positional difference between the first image and the second image.

[0013] In this embodiment, the step of controlling the tip portion also includes the step of calculating the total movement angle of the tip portion based on the first value.

[0014] Another embodiment of the present invention discloses a method for controlling an endoscope, which includes the steps of: acquiring an image of the inside of a body from an image sensor; identifying a plurality of lesions from the image based on a pre-trained model; and controlling the tip so that the tip tracks one of the plurality of lesions.

[0015] In this embodiment, the process further includes the step of outputting image-based lesion position information relating to the identified plurality of lesions, and the step of selecting the target lesion from among the plurality of lesions in response to user input in the operation unit.

[0016] In this embodiment, the user input also includes at least one of the following: a single input, an input exceeding a critical time, and a double input within a critical time.

[0017] A further embodiment of the present invention discloses an endoscope device comprising a tip equipped with an image sensor, and a control unit that acquires an image of the inside of a body from the image sensor, identifies a lesion from the image based on a pre-trained model, and controls the tip so that the tip tracks the lesion.

[0018] In this embodiment, the field of view of the image sensor includes a first region, and the control unit can control the tip portion so that the lesion is located within the first region.

[0019] In the present embodiment, the first region also includes the center of the viewing angle.

[0020] In the present embodiment, the control unit can calculate lesion position information on the image indicating the position of the lesion on the image.

[0021] In the present embodiment, the control unit can calculate a first value that is the ratio of the positional difference on the image with respect to the angular change of the tip.

[0022] In the present embodiment, the control unit acquires a first image when the tip is at a first angle and a second image when the tip is at a second angle, and can measure the difference between the first angle and the second angle and the positional difference between the first image and the second image.

[0023] In the present embodiment, the control unit can calculate the total movement angle of the tip based on the first value.

[0024] Still another embodiment of the present invention discloses an endoscope apparatus including a tip provided with an image sensor, and a control unit that acquires an image inside the body from the image sensor, discriminates a plurality of lesions from the image based on a pre-learned model, and controls the tip so as to track one target lesion among the plurality of lesions.

[0025] In the present embodiment, it further includes a display unit for outputting the image and an operation unit for inputting a user's command, and the control unit outputs the lesion position information on the image related to the identified plurality of lesions to the display unit, and in response to a user input at the operation unit, the target lesion can be selected from among the plurality of lesions.

[0026] In the present embodiment, the user input also includes at least one of a single input, an input for a critical time or more, and a double input within a critical time.

[0027] Other aspects, features, and advantages other than those described above will become clear from the specific content, claims, and drawings for implementing the following invention.

Effect of the Invention

[0028] According to an embodiment of the present invention, an endoscope device and a control method for an endoscope device can detect a lesion inside the body without being affected by the proficiency and fatigue of the user by identifying the lesion with a pre-trained model.

[0029] According to an embodiment of the present invention, an endoscope device and a control method for an endoscope device can track and observe a lesion without difficulty even by a user with low proficiency by automatically tracking the lesion identified at the tip, enabling the user to observe the identified lesion or concentrate on performing treatment and procedures without using nerves for the scope operation behavior, and improving the quality of medical treatment.

[0030] The effects of the present invention are not limited to the effects mentioned above, and other effects not mentioned can be clearly understood by those with ordinary knowledge in the technical field to which the present invention belongs from the present specification and the accompanying drawings.

Brief Explanation of Drawings

[0031] [Figure 1] It is a diagram showing an endoscope device according to an embodiment of the present invention. [Figure 2] It is a drawing showing an example of a pre-trained model identifying a lesion from an image inside the body according to an embodiment of the present invention. [Figure 3] It is a diagram showing an image output to the display unit when a lesion is identified. [Figure 4] It is a diagram specifically showing a method of calculating a first value based on a positional difference on an image due to an angular difference at the tip. [Figure 5] It is a diagram showing an image output to the display unit when a lesion is located within a first region. [Figure 6]This diagram illustrates an example of how a target lesion is selected from among multiple lesions in response to user input on the control panel. [Figure 7] This is a flowchart illustrating a control method for an endoscope device according to one embodiment of the present invention. [Figure 8] This is a flowchart illustrating the steps for controlling the tip of an embodiment of the present invention. [Figure 9] This is a flowchart illustrating a control method for an endoscope device according to another embodiment of the present invention. [Modes for carrying out the invention]

[0032] The terms used in this invention are used solely to describe specific embodiments and are not intended to limit the scope of other embodiments. Singular expressions also include plural expressions unless explicitly stated in the context. Terms used herein, including technical and scientific terms, may have the same meaning as those generally understood by a person of ordinary skill in the art described herein. General, predefined terms used herein are to be interpreted as having the same or similar meaning as they do in the context of the relevant art, and not as to be idealistic or overly formal unless explicitly defined herein. Where applicable, a term defined herein is not to be construed as excluding embodiments of the invention.

[0033] In the following, various embodiments will be described in detail with reference to the accompanying drawings, so that they can be easily implemented by a person with ordinary skill in the art to which the present invention pertains. However, the technical idea of ​​the present invention can be embodied in various forms and is not limited to the embodiments described herein. In the description of embodiments disclosed herein, if it is determined that specifically describing related prior art would obscure the gist of the technical idea of ​​the present invention, then specific descriptions relating to such prior art will be omitted. Identical or similar components will be given the same reference numeral, and redundant descriptions relating thereto will be omitted.

[0034] Here, the term "~part" as used in this embodiment means a component that performs a specific function, whether performed by software or hardware such as an FPGA (field programmable gate array) or an application-specific integrated circuit (ASIC). However, "~part" is not limited to those performed by software or hardware. "~part" may also exist as data stored on an addressable recording medium, be embodied by instructions, and be configured so that one or more processors perform a specific function.

[0035] Software may include computer programs, code, instructions, or a combination of one or more of these, which can configure a processing unit to operate as desired, or independently or collectively, instruct the processing unit. Software and / or data may be permanently or temporarily embodied in a type of machine, component, physical device, virtual device, computer recording medium or device, or transmitted signal wave, in order to be interpreted by a processing unit or to provide instructions or data to a processing unit. The software may also be distributed on a network of computer systems, stored in a distributed manner, or executed. The software and data may be stored on a recording medium readable by one or more computers. The software may be read into main memory from other computer-readable media, such as data storage devices, or from other devices via a communication interface. Software instructions stored in main memory may cause the processor to perform processes or steps described in detail below. Alternatively, fixed wiring circuits can be used in place of, or in combination with, software instructions to perform processes consistent with the principles of the present invention. Therefore, embodiments consistent with the principles of the present invention are not limited to any particular combination of hardware circuits and software.

[0036] The terms used in this application are used solely to describe specific embodiments and are not intended to limit the invention. A singular expression includes plural expressions unless the context explicitly limits it to that singular expression. In this application, terms such as “includes” or “having” should be understood to indicate the existence of features, numbers, stages, operations, components, parts, or combinations thereof described in the specification, and not to preemptively exclude the possibility of the existence or addition of one or more other features, numbers, stages, operations, components, parts, or combinations thereof. Terms such as the first and second may be used to describe a variety of components, but such components are not limited by the terms. The terms are used solely for the purpose of distinguishing one component from others.

[0037] The term "learning model" as used in this invention also includes all forms of algorithms or methodologies used to learn or understand specific patterns or structures from data. That is, the learning model includes not only machine learning models such as regression models, decision trees, random forests, support vector machines, K-nearest neighbors, naive phases, and clustering algorithms, but also deep learning models such as neural networks, convolutional neural networks, circulatory neural networks, Transformer-based neural networks, GANs (Generative Adversarial Networks), and autoencoders. The "learning model" refers to a set of learned parameters or weights used to predict or classify an output for a given input, and the model may be learned through methods such as directed learning, undirected learning, semi-directed learning, and reinforcement learning. Furthermore, it includes not only single models, but also diverse learning methods and structures such as ensemble models, multimodal models, and models via transfer learning. Such learning models may be pre-trained on a separate computer device from the computer device used to predict the output for a given input, and then used on the other computer device.

[0038] The learning model according to one embodiment of the present invention also includes models related to object detection and position estimation.

[0039] Hereinafter, with reference to Figures 1 to 9, an endoscope device and a control method for the endoscope device according to embodiments of the present invention will be described.

[0040] Figure 1 is a drawing showing an endoscope device 100 according to one embodiment of the present invention.

[0041] The endoscopic device 100 refers to a medical instrument that allows for the insertion of a scope 150 into the body to observe organs and, if necessary, to perform treatment or procedures. Referring to Figure 1, the endoscopic device 100 also includes a display unit 110, a control unit 120, a drive unit 130, an operating unit 140, and a scope 150.

[0042] The display unit 110 can output an image. In other words, the display unit 110 can output an internal body image acquired from the image sensor 153a, which will be described later. The output internal body image also includes x-axis and y-axis coordinates (see Figure 3).

[0043] The display unit 110 may also include a display module that can output visualized information or implement a touch screen, such as a liquid crystal display (LCD), thin-film transistor-liquid crystal display (TFT LCD), organic light-emitting diode (OLED), flexible display, or 3D display.

[0044] The display unit 110 can output lesion location information (P) on the image, which will be described later.

[0045] The control unit 120 can control the overall operation of the endoscope device 100. The control unit 120 also includes all kinds of components that can process data. In one embodiment, the control unit 120 also includes a hardware-integrated data processing device that has a physically structured circuit to perform a function expressed by code or instructions contained in the program. The hardware-integrated data processing device may include processing devices such as a microprocessor, a central processing unit (CPU), a processor core, a multiprocessor, an ASIC (application-specific integrated circuit), or an FPGA (field programmable gate array).

[0046] The control unit 120 can control the angle of the tip portion 153 via the drive unit 130, which will be described later. A detailed explanation of the control unit 120 will be given later.

[0047] The drive unit 130 can provide the necessary power for the scope 150, which will be described later, as it is inserted into or moves inside the body. For example, the drive unit 130 may also include a plurality of motors connected to wires inside the scope 150, and a tension adjustment unit for adjusting the tension of the wires.

[0048] The control unit 140 can receive user commands. The control unit 140 also includes several buttons that provide various functions to control the angle of the tip 153 (described later) and to perform various surgical procedures inside the body.

[0049] In one embodiment, the user's command may include a command to select a target lesion T (Figure 6) from among multiple lesion LEs (Figure 3) identified from the image. The target lesion T refers to the lesion LE that the tip 153 will track. In one embodiment, the operation unit 140 may also include a selection button (not shown). Specific examples related to this will be described later.

[0050] The scope 150 can be inserted directly into the body. Specifically, the scope 150 also includes an insertion section 151, a bending section 152, and a tip section 153.

[0051] The insertion portion 151 can be used to insert the tip portion 153, which will be described later, to any desired position inside the body that is the target of observation and treatment. The insertion portion 151 can be connected to one end of the operating portion 140.

[0052] The curved portion 152 can be connected to one end of the insertion portion 151. The curved portion 152 can change the angle of the tip portion 153, which will be described later. The curved portion 152 can bend flexibly. By bending the curved portion 152, the angle of the tip portion 153 can be changed. The curved portion 152 can be connected to the drive unit 130 and supplied with the force necessary for changing the angle of the tip portion 153. The degree or direction of bending of the curved portion 152 can be determined by the drive unit 130.

[0053] The tip 153 may be connected to one end of the curved section 152. The tip 153 can be used to image the inside of the body, and treatment or procedures may be performed if necessary. The tip 153 also includes an image sensor 153a, a lens 153b, illumination 153c, a working channel 153d, and an air / water channel 153e.

[0054] The image sensor 153a can acquire images of the inside of the body. These images of the inside of the body may include video images consisting of multiple consecutive frames. These images of the inside of the body can be output via the display unit 110.

[0055] The field of view of the image sensor 153a also includes a first region. This first region also includes the center of the field of view. In the image 300 of the inside of the body (Figure 3), the portion 310 (Figure 3) corresponding to the first region also includes the center of the image 300 (see Figure 3). Therefore, if an object is located within the first region, that object may be located close to the center of the image output via the display unit 110.

[0056] Lens 153b can act as a passage for light reflected from within the body to enter the image sensor 153a. Illumination 153c can irradiate light into the body so that the image sensor 153a can acquire an image of the inside of the body. The number of illuminations 153c is not particularly limited. Working channel 153d may be used to insert tools for the treatment and processing of lesion LE. Air / water channel 153e may be supplied with air or irrigation water.

[0057] The control unit 120 will be described in detail below.

[0058] Figure 2 is a diagram illustrating an example of a pre-trained model 200 according to one embodiment of the present invention, in which a lesion LE (Figure 3) is identified from an image A of the inside of the body. Figure 3 is a diagram showing the image 300 output to the display unit 110 when a lesion LE is identified.

[0059] The control unit 120 also includes a processor for performing various calculations or operations, as described later. This processor can interpret computer programs and perform data processing for machine learning. This processor can handle computational processes such as processing input data for machine learning, feature extraction for machine learning, and error calculation based on backpropagation. Processors for such data processing may include a central processing unit (CPU), a general-purpose graphics processing unit (GPGPU), a tensor processing unit (TPU), an application-specific integrated circuit (ASIC), or a field-programmable gate array (FPGA). However, these are merely examples, and the type of processor can be configured in a variety of ways based on this information, within the scope understandable to an average engineer.

[0060] The control unit 120 can acquire an image of the inside of the body from the image sensor 153a, identify a lesion LE from the image based on a pre-trained model 200, and control the tip 153 so that it tracks the lesion LE.

[0061] The control unit 120 can pre-train the model 200. The model 200 can be trained to receive an image of the inside of a body, identify lesions (LEs) from the image, and calculate lesion location information (P) on the image. The model 200 can receive images of the inside of a body as training data. In addition, the model 200 can receive label data corresponding to each image of the inside of a body, which is label data in which the location of the lesion is displayed in a bounding box.

[0062] Model 200 also includes network structures such as FCN (fully convolutional network), CAN (conditional adversarial network), RNN (recurrent neural network), or MC-CNN (matching cost-CNN).

[0063] As shown in Figure 2, the control unit 120 can identify lesions LE from image A based on a pre-trained model 200. The control unit 120 can input the internal body image A acquired from the image sensor 153a to the pre-trained model 200. The pre-trained model 200, upon receiving the internal body image A, can identify lesions LE and calculate and output lesion position information (P) on the image. The lesion position information (P) on the image is also information indicating the position of lesions LE on the image output to the display unit 110. In one embodiment, as shown in Figure 2, the lesion position information (P) on the image also includes a bounding box B (Figure 3) surrounding the lesion LE.

[0064] For example, as shown in Figure 2, if an internal body image A1 is input within 200 seconds, the pre-trained model 200 may identify the lesion LEa contained in internal body image A1 and output a bounding box Ba surrounding lesion LEa. If an internal body image A2 is input within 200 seconds, since there is no lesion in internal body image A2, the pre-trained model 200 will not output a bounding box. If an internal body image A3 is input within 200 seconds, the pre-trained model 200 may identify the lesion LEb contained in internal body image A3 and output a bounding box Bb surrounding lesion LEb.

[0065] In one embodiment, the lesion location information (P) in the image also includes image coordinate information relating to the four line segments of the bounding box B. That is, the pre-trained model 200 can calculate image coordinate information relating to the four line segments of the bounding box B.

[0066] In one embodiment, the center coordinates of bounding box B can be calculated from the image coordinate information relating to the four line segments of bounding box B. Referring to Figure 3, in the image 300 of the inside of the body output to the display unit 110, the center coordinates (x1, y1) of the bounding box B of the lesion LE identified by the pre-trained model 200 can be calculated from the image coordinate information relating to the four line segments of bounding box B. The center coordinates (x1, y1) of bounding box B can be said to be the center coordinates of the identified lesion LE.

[0067] Figure 4 is a diagram illustrating a method for calculating the first value (K) based on the image position difference due to the angle difference of the tip portion 153. Figure 5 is a diagram showing the image 300' output to the display unit 110 when the lesion LE is located within the first region.

[0068] The control unit 120 can control the tip 153 so that it tracks the lesion LE. Specifically, the control unit 120 can control the tip 153 so that the lesion LE is located within a first region. This allows the identified lesion LE to be located close to the center of the image 300' output via the display unit 110. As a result, the identified lesion LE can be output to the display unit 110 in succession.

[0069] The control unit 120 calculates a first value (K), which is the ratio of the positional difference on the image to the angular change of the tip portion 153. Based on the lesion position information on the image (P) and the first value (K), it calculates the total movement angle (TMA) of the tip portion. Considering the angular velocity (as) and control period (cp) of the tip portion 153, it calculates the target angle (ag) at the current time and moves the tip portion 153 by the target angle (ag) at the current time.

[0070] The control unit 120 can calculate a first value (K), which is the ratio of the positional difference in the image to the angle change of the tip portion 153. The first value (K) is a value necessary to calculate the total movement angle (TMA) of the tip portion 153, which will be described later.

[0071] In one embodiment, the control unit 120 can acquire a first image (i1) when the tip portion 153 is at a first angle a, and a second image (i2) when the tip portion 153 is at a second angle b, and can measure the difference (Δθ) between the first angle a and the second angle b, and the positional difference (ΔX) between the first image (i1) and the second image (i2). The first angle a and the second angle b are any two different angles for calculating a first value (K), and may refer to the angle between the tip portion 153, which is formed by the curvature of the curved portion 152, and the insertion portion 151.

[0072] Referring to Figure 4, when the control unit 120 operates the curved portion 152 and the tip portion 153 is set to a first angle a, the image sensor 153a can acquire a first image (i1). At that time, the image coordinates of any point X inside the body are also (xa, ya). The method of specifying any point X is not particularly limited. In one embodiment, any point X is also the location of a specific type of blood vessel inside the body that is found in the image.

[0073] Furthermore, when the control unit 120 manipulates the curved portion 152 to set the tip portion 153 to a second angle b, the image sensor 153a can acquire a second image (i2). At that time, the image coordinates of any point X inside the body are (xb, yb).

[0074] The coordinate difference between the first image (i1) and the second image (i2) can be expressed as the difference in coordinates on the image of an arbitrary point X, (Δx, Δy). In this case, Δx = xb - xa and Δy = yb - ya. Using this, the positional difference (ΔX) between the first image (i1) and the second image (i2) can be shown as given by equation 1 below.

number

[0075] The first value (K) can be calculated by substituting the difference (Δθ) between the first angle a and the second angle b, and the positional difference (ΔX) between the first image (i1) and the second image (i2) into the following equation 2.

number

[0076] The control unit 120 can calculate the total movement angle (TMA) of the tip based on the lesion position information (P) and the first value (K) on the image. The total movement angle (TMA) is the total angle that the tip 153 must move in order to move the center coordinates (x1, y1) of the lesion LE to the center coordinates (0, 0) on the image. Referring to Figure 3, the length (Q) from the center of the lesion LE to the center on the image is given by the following formula 3.

number

[0077] The total displacement angle (TMA) can be calculated by substituting the distance (Q) from the center of the lesion LE to the center of the image, and the calculated first value (K), into the following formula 4.

number

[0078] The control unit 120 can calculate the target angle (ag) at the current time, taking into account the angular velocity (as) and control period (cp) of the tip 153. The angular velocity (as) and control period (cp) of the tip 153 can be set in advance.

[0079] In one embodiment, the target angle (ag) at the current time can be calculated via a polynomial trajectory. For example, the tip 153 can be set by moving at a constant velocity to achieve a total travel angle (TMA). The pre-set angular velocity (as) of the tip 153 is 30° / s, and the pre-set control period (cp) is 2ms. In such a case, the target angle (ag) at the current time is 30° / s × 2ms = 0.2°. However, this is merely one embodiment for illustrative purposes and is not limited thereto.

[0080] In another embodiment, the target angle (ag) at the current time can be calculated via a Bézier curve trajectory. That is, the method for calculating the target angle (ag) at the current time is not limited to a polynomial trajectory. In other words, the target angle (ag) at the current time can be calculated via a variety of trajectories that an ordinary engineer might conceive.

[0081] The control unit 120 can move the tip portion 153 by a target angle (ag) at the current time. The control unit 120 can calculate the force required to move the tip portion 153 by a target angle (ag) at the current time, taking into account the dynamic characteristics of the curved portion 152. The control unit 120 transmits the calculated force information to the drive unit 130, which can then move the tip portion 153 by a target angle (ag) at the current time.

[0082] As shown in Figure 5, the control unit 120 can repeatedly perform the process of identifying the lesion LE until it is located in the first region, calculating a first value (K), and controlling the tip unit 153. As a result, the tip unit 153 tracks the lesion LE, and the lesion LE can be continuously output to the display unit 110.

[0083] Figure 6 is a diagram illustrating an example in which the target lesion T is selected from among multiple lesions LE in response to user input in the operation unit 140.

[0084] The control unit 120 can identify multiple lesions LE from an image based on a pre-trained model 200. For example, referring to Figure 6, the image acquired from the image sensor 153a may identify a first lesion LE1, a second lesion LE2, and a third lesion LE3. The pre-trained model 200 can output bounding boxes surrounding each of the first lesion LE1, the second lesion LE2, and the third lesion LE3.

[0085] In response to user input on the operation unit 140, the target lesion T can be selected from among multiple lesion LEs. Specifically, in response to user input, the lesion LE that is considered to be the target lesion can be changed from among multiple lesion LEs, and the lesion LE that is considered to be the target lesion T can be selected. As described above, the user can select the target lesion T from among multiple lesion LEs using the selection buttons (not shown) provided on the operation unit 140.

[0086] In one embodiment, the user input may include at least one of a single input, an input exceeding a critical time, and a double input within a critical time. For example, the single input may be a command to change a lesion LE that is identified as such among multiple lesion LEs. On the other hand, the double input within a critical time may be a command to select the identified lesion LE as the target lesion T. However, these are only embodiments and are not limited to them.

[0087] In one embodiment, as shown in Figure 6, the identified lesion LE can be identified by the difference in the thickness of the bounding box. Using the example described above, a concrete explanation is as follows: If the user presses the selection button once, the identified lesion LE may change from the first lesion LE1 to the second lesion LE2. If the user presses the selection button again, the identified lesion LE may change from the second lesion LE2 to the third lesion LE3. At that time, if the user presses the selection button twice within the critical time, the identified third lesion LE3 may be selected as the target lesion T.

[0088] When multiple lesions (LEs) are identified in a single image, the practicality of the endoscopic device 100 can be increased by tracking a selected target lesion T from among the multiple lesions (LEs).

[0089] Figure 7 is a flowchart showing a control method (M1) for an endoscope device according to one embodiment of the present invention.

[0090] The control method (M1) for the endoscope device is a method of controlling the endoscope device 100 so that the lesion LE is continuously output to the display unit 110 by having the tip 153 track the identified lesion LE in the internal body image acquired from the endoscope device 100.

[0091] The control method (M1) of the endoscope device, as shown in Figure 7, also includes the steps of acquiring an image A of the inside of the body from the image sensor 153a (S110), identifying a lesion LE from image A based on a pre-trained model 200 (S120), and controlling the tip 153 so that the tip 153 tracks the lesion LE (S130).

[0092] The endoscope device 100 can acquire an image A of the inside of the body from the image sensor 153a (S110). The scope 150 is inserted into the body, and an image of the inside of the body can be acquired via the image sensor 153a provided at the tip 153. The image of the inside of the body may also include a video image consisting of multiple consecutive frames. The image of the inside of the body can be output via the display unit 110.

[0093] As shown in Figure 2, the endoscope device 100 can identify lesions LE from image A based on a pre-trained model 200 (S120). The endoscope device 100 can input an image A of the inside of the body acquired from the image sensor 153a to the pre-trained model 200. The pre-trained model 200, having received the image A of the inside of the body as input, can identify lesions LE and calculate and output lesion position information (P) on the image. The lesion position information (P) on the image is also information indicating the position of lesions LE on the image output to the display unit 110. In one embodiment, as shown in Figure 2, the lesion position information (P) on the image also includes a bounding box B surrounding the lesion LE.

[0094] For example, as shown in Figure 2, if an internal body image A1 is input to a pre-trained model 200, the pre-trained model 200 may identify the lesion LEa contained in internal body image A1 and output a bounding box Ba surrounding lesion LEa. If an internal body image A2 is input to the pre-trained model 200, since there is no lesion in internal body image A2, the pre-trained model 200 will not output a bounding box. If an internal body image A3 is input to the pre-trained model 200, the pre-trained model 200 may identify the lesion LEb contained in internal body image A3 and output a bounding box Bb surrounding lesion LEb.

[0095] Model 200 can be trained to receive internal body images as input, identify lesions (LEs) from these images, and calculate lesion location information (P) on the images. Model 200 can receive internal body images as training data. Additionally, Model 200 can receive label data corresponding to each internal body image, which is label data where the location of the lesion is displayed in a bounding box.

[0096] Model 200 also includes network structures such as FCN (fully convolutional network), CAN (conditional adversarial network), RNN (recurrent neural network), or MC-CNN (matching cost-CNN).

[0097] In one embodiment, the lesion location information (P) in the image also includes image coordinate information relating to the four line segments of the bounding box B. That is, the pre-trained model 200 can calculate image coordinate information relating to the four line segments of the bounding box B.

[0098] In one embodiment, the center coordinates of bounding box B can be calculated from the image coordinate information relating to the four line segments of bounding box B. Referring to Figure 3, in the image 300 of the inside of the body output to the display unit 110, the center coordinates (x1, y1) of the bounding box B of the lesion LE identified by the pre-trained model 200 can be calculated from the image coordinate information relating to the four line segments of bounding box B. The center coordinates (x1, y1) of bounding box B can be said to be the center coordinates of the identified lesion LE.

[0099] Furthermore, the field of view of the image sensor 153a also includes the first region. This first region also includes the center of the field of view. Referring to Figure 3, in the image 300 of the inside of the body, the portion 310 corresponding to the first region also includes the center of the image 300. Therefore, if an object is located within the first region, that object may be located close to the center of the image output via the display unit 110.

[0100] Figure 8 is a flowchart showing a breakdown of the step (S130) for controlling the tip portion according to one embodiment of the present invention.

[0101] The endoscope device 100 can control the tip 153 so that it tracks the lesion LE (S130). The step of controlling the tip (S130) also includes a step of controlling the tip 153 so that the lesion LE is located within the first region. This allows the identified lesion LE to be located close to the center of the image output via the display unit 110. As a result, the identified lesion LE can be output continuously to the display unit 110.

[0102] The step of controlling the tip (S130), as shown in Figure 8, also includes the steps of: calculating a first value (K), which is the ratio of the positional difference on the image to the angular change of the tip 153 (S131); calculating the total movement angle (TMA) of the tip based on the lesion position information on the image (P) and the first value (K) (S132); calculating the target angle (ag) at the current time, taking into account the angular velocity (as) and control period (cp) of the tip 153 (S133); and moving the tip 153 to the target angle (ag) at the current time (S134).

[0103] The endoscope device 100 can calculate a first value (K), which is the ratio of the positional difference in the image to the angular change of the tip 153 (S131). The first value (K) is a value necessary to calculate the total movement angle (TMA) of the tip 153.

[0104] In one embodiment, the step of calculating the first value (S131) ​​also includes the steps of obtaining a first image (i1) when the tip portion 153 is at a first angle a, and a second image (i2) when the tip portion 153 is at a second angle b, and measuring the difference (Δθ) between the first angle a and the second angle b, and the positional difference (ΔX) between the first image (i1) and the second image (i2). The first angle a and the second angle b are any two different angles for calculating the first value (K), and may refer to the angle between the tip portion 153, which is formed by the curvature of the curved portion 152, and the insertion portion 151.

[0105] Referring to Figure 4, when the curved portion 152 is manipulated and the tip portion 153 is set to a first angle a, the first image (i1) can be acquired by the image sensor 153a. At that time, the image coordinates of any point X inside the body are also (xa, ya). The method of specifying any point X is not particularly limited. In one embodiment, any point X is also the location of a specific type of blood vessel inside the body that is found in the image.

[0106] Furthermore, when the curved portion 152 is manipulated to set the tip portion 153 to a second angle b, a second image (i2) can be acquired by the image sensor 153a. At that time, the image coordinates of any point X inside the body are also (xb, yb).

[0107] The coordinate difference between the first image (i1) and the second image (i2) can be expressed as the difference in coordinates on the image of an arbitrary point X, (Δx, Δy). In this case, Δx = xb - xa and Δy = yb - ya. Using this, the positional difference (ΔX) between the first image (i1) and the second image (i2) can be shown as given by equation 1 above.

[0108] By substituting the difference between the first angle a and the second angle b (Δθ), and the positional difference between the first image (i1) and the second image (i2) (ΔX) into the above formula 2, the first value (K) can be calculated.

[0109] The endoscope device 100 can calculate the total movement angle (TMA) of the tip based on the lesion position information (P) and a first value (K) on the image (S132). The total movement angle (TMA) is the total angle that the tip 153 must move in order to move the central coordinates (x1,y1) of the lesion LE to the central coordinates (0,0) on the image. Referring to Figure 3, the length (Q) from the center of the lesion LE to the center on the image is given by the above formula 3.

[0110] By substituting the length from the center of the lesion LE to the center of the image (Q) and the calculated first value (K) into the above formula 4, the total displacement angle (TMA) can be calculated.

[0111] The endoscope device 100 can calculate the target angle (ag) at the current time, taking into account the angular velocity (as) and control period (cp) of the tip 153 (S133). The angular velocity (as) and control period (cp) of the tip 153 can be set in advance.

[0112] In one embodiment, the target angle (ag) at the current time can be calculated via a polynomial trajectory. Specifically, it can be set by the tip 153 moving at a constant velocity over a total travel angle (TMA). The pre-set angular velocity (as) of the tip 153 is 30° / s, and the pre-set control period (cp) is 2ms. In such a case, the target angle (ag) at the current time is 30° / s × 2ms = 0.2°. However, this is merely one embodiment for illustrative purposes and is not limited thereto.

[0113] In another embodiment, the target angle (ag) at the current time can be calculated via a Bézier curve trajectory. That is, the method for calculating the target angle (ag) at the current time is not limited to a polynomial trajectory. In other words, the target angle (ag) at the current time can be calculated via a variety of trajectories that an ordinary engineer might conceive.

[0114] The endoscope device 100 can move its tip 153 by a target angle (ag) at the current time (S134). The endoscope device 100 can calculate the force required to move the tip 153 by a target angle (ag) at the current time, taking into account the dynamic characteristics of the bending section 152. The endoscope device 100 transmits the calculated force information to the drive unit 130, and can move the tip 153 by a target angle (ag) at the current time.

[0115] The steps of identifying the lesion (S120) and controlling the tip (S130) can be repeated until the lesion LE is located in the first region. As a result, the tip 153 tracks the lesion LE, and the lesion LE can be continuously output to the display unit 110.

[0116] Figure 9 is a flowchart showing a control method (M2) for an endoscope device according to another embodiment of the present invention.

[0117] The control method for the endoscope device (M2), as shown in Figure 9, includes the steps of: acquiring an image of the inside of the body from the image sensor 153a (S210); identifying multiple lesions LE from the image based on a pre-trained model 200 (S220); outputting lesion position information (P) on the image related to the identified lesions LE (S230); selecting a target lesion T from among the multiple lesions LE in response to user input in the operation unit 140 (S240); and controlling the tip 153 so that the tip 153 tracks one target lesion T from among the multiple lesions LE (S250). Of these, the steps of acquiring an image (S210), outputting position information on the image (S230), and controlling the tip (S250) are the same as or similar to those described in the control method for the endoscope device (M1) above, so a detailed explanation will be omitted, and the focus will be on the differences.

[0118] The endoscope device 100 can identify multiple lesions LE from an image based on a pre-trained model 200 (S220). For example, referring to Figure 6, the first lesion LE1, the second lesion LE2, and the third lesion LE3 can be identified in the image acquired from the image sensor 153a.

[0119] In the stage (S230) where lesion location information (P) on the image is output, the pre-trained model 200 can output bounding boxes surrounding the first lesion LE1, the second lesion LE2, and the third lesion LE3, respectively.

[0120] In response to user input in the operation unit 140, the target lesion T can be selected from among multiple lesion LEs (S240). Specifically, in response to user input, the lesion LE that is considered to be the target lesion can be changed from among multiple lesion LEs, and the lesion LE that is considered to be the target lesion T can be selected. The user can select the target lesion T from among multiple lesion LEs using the selection buttons (not shown) provided on the operation unit 140.

[0121] In one embodiment, the user input may include at least one of a single input, an input exceeding a critical time, and a double input within a critical time. For example, the single input may be a command to change a lesion LE that is identified as such among multiple lesion LEs. The double input within a critical time may also be a command to select the identified lesion LE as the target lesion T. However, these are only embodiments and are not limited to them.

[0122] In one embodiment, as shown in Figure 6, the identified lesion LE can be identified by the difference in the thickness of the bounding box B. Using the example described above, a concrete explanation is as follows: If the user presses the selection button once, the identified lesion LE may change from the first lesion LE1 to the second lesion LE2. If the user presses the selection button again, the identified lesion LE may change from the second lesion LE2 to the third lesion LE3. At that time, if the user presses the selection button twice within the critical time, the identified third lesion LE3 may be selected as the target lesion T.

[0123] When multiple lesions (LEs) are identified in a single image, the practicality of the endoscopic device 100 can be increased by tracking a selected target lesion T from among the multiple lesions (LEs).

[0124] As described above, the present invention has been explained with reference to the embodiments illustrated in the drawings, but these are merely illustrative. A person with ordinary skill in the art will be able to fully understand that a variety of modifications and equivalent other embodiments are possible from these embodiments. Therefore, the true scope of technical protection of the present invention must be determined by the claims.

[0125] The specific descriptions provided in this embodiment represent only one embodiment and do not limit the technical scope of this embodiment. In order to describe the invention concisely and clearly, descriptions relating to prior art and configurations may be omitted. Furthermore, the linear connections or connecting members between components shown in the drawings are illustrative examples of functional and / or physical or circuit connections, and in actual devices, they may be represented by a variety of functional, physical, or circuit connections that are interchangeable or added. Also, unless specifically mentioned as "essential" or "important," components are not necessarily required for the application of the present invention.

[0126] The “the foregoing” or similar demonstrative pronouns in the description of the invention and the claims may refer to either the singular or plural, unless otherwise specified. Furthermore, where a range is described in this embodiment, it includes inventions applying the individual values ​​belonging to that range (unless otherwise stated), as the individual values ​​constituting that range are described in the description of the invention. Also, with respect to the steps constituting the method according to this embodiment, unless otherwise stated, the steps may be performed in any order. The order in which the steps are described does not necessarily limit this embodiment. In this embodiment, the use of all examples or exemplary terms (e.g., “etc.”) is merely for the purpose of detailing this embodiment and, unless otherwise stated in the claims, does not limit the scope of this embodiment. Furthermore, a person of ordinary skill will know that the claims, or their equivalents, can be comprised of various modifications, combinations, and changes, and may be comprised of design conditions and factors. [Explanation of Symbols]

[0127] 1. Endoscope equipment 110 Display section 120 Control Unit 130 Drive unit 140 Operation section 150 Scope 151 Insertion part 152 Curved section 153 Tip 153a Image Sensor 153b lens 153c lighting 200 pre-trained models B Bounding Box K 1st value LE lesions M1, M2 Endoscope Control Method P Image of lesion location information

Claims

1. The control unit acquires image data from the image sensor. The control unit processes the image data to automatically identify a lesion, wherein the identification is performed by a pre-trained model. The step includes the step of the control unit generating a control signal based on the lesion to control the tip of the endoscope so that the tip of the endoscope tracks the lesion, A method for controlling an endoscope, wherein the control unit calculates a first value which is the ratio of the positional difference in the image data to the angle change of the tip.

2. The field of view of the image sensor includes a first region, The step of controlling the tip portion is: A method for controlling an endoscope according to claim 1, comprising the step of controlling the tip portion so that the lesion is located within the first region.

3. The control method for an endoscope device according to claim 2, wherein the first region includes the center of the field of view.

4. The step of identifying the lesion is, A method for controlling an endoscope device according to claim 1, comprising the step of calculating lesion position information on the image data, which indicates the position of the lesion on the image data.

5. The step of calculating the first value is: A step of obtaining a first image when the tip is at a first angle, and a second image when the tip is at a second angle, A method for controlling an endoscope device according to claim 1, comprising the steps of measuring the difference between the first angle and the second angle, and the positional difference between the first image and the second image.

6. The step of controlling the tip portion is: A control method for an endoscope device according to claim 1, comprising the step of calculating the total movement angle of the tip based on the first value.

7. The control unit acquires image data from the image sensor. The control unit processes the image data to automatically identify multiple lesions, wherein the identification is performed by a pre-trained model. The process includes the step of the control unit generating a control signal based on the plurality of lesions to control the tip of the endoscope so that the tip of the endoscope tracks one of the plurality of lesions, A method for controlling an endoscope, wherein the control unit calculates a first value which is the ratio of the positional difference in the image data to the angle change of the tip.

8. The control unit outputs image lesion position information relating to the plurality of identified lesions, A method for controlling an endoscope device according to claim 7, further comprising the step of the control unit selecting the target lesion from among the plurality of lesions in response to user input in the operation unit.

9. The control method for an endoscope device according to claim 8, wherein the user input includes at least one of a single input, an input exceeding a critical time, and a dual input within a critical time.

10. The tip portion is equipped with an image sensor, An endoscope device comprising: an image sensor that acquires an image of the inside of the body; a pre-trained model that identifies a lesion from the image; a first value that is the ratio of the positional difference on the image to the angle change of the tip of the endoscope device; and a control unit that controls the tip so that the tip tracks the lesion.

11. The field of view of the image sensor includes a first region, The control unit, The endoscope device according to claim 10, wherein the tip portion is controlled so that the lesion is located within the first region.

12. The endoscope apparatus according to claim 11, wherein the first region includes the center of the field of view.

13. The control unit, The image is used to calculate lesion position information on the image, which indicates the location of the lesion on the image. The endoscopic device according to claim 10.

14. The control unit, A first image is obtained when the tip is at a first angle, and a second image is obtained when the tip is at a second angle. The endoscope apparatus according to claim 10, which measures the difference between the first angle and the second angle, and the positional difference between the first image and the second image.

15. The control unit, The endoscope device according to claim 10, wherein the total movement angle of the tip is calculated based on the first value.

16. The tip portion is equipped with an image sensor, An endoscope device comprising: an image sensor that acquires an image of the inside of the body; a pre-trained model that identifies multiple lesions from the image; a first value that is the ratio of the positional difference on the image to the angle change of the tip of the endoscope device; and a control unit that controls the tip so that the tip tracks one target lesion among the multiple lesions.

17. A display unit that outputs the aforementioned image, It further includes an operating unit for inputting user commands, The control unit, The lesion position information on the image relating to the identified plurality of lesions is output to the display unit. The endoscope device according to claim 16, wherein, in response to user input in the control unit, the target lesion is selected from among the plurality of lesions.

18. The endoscope apparatus according to claim 17, wherein the user input includes at least one of a single input, an input exceeding a critical time, and a double input within a critical time.

Images