Endoscope device for acquiring upper gastrointestinal tract images and method of controlling the same

Abstract

To provide an endoscope device for acquiring upper gastrointestinal tract images and a method of controlling the same.SOLUTION: The invention belongs to the field of medical imaging equipment and automation control technology, and particularly relates to an endoscope device associated with imaging of the upper gastrointestinal tract using an endoscope, and to a method of controlling the endoscope device. The method includes the steps of: acquiring an image of the upper gastrointestinal tract from an image sensor; sensing at least one first body part from the image based on a pre-trained model; calculating relative position information between the sensed first body part and a distal end portion; generating a first control signal related to the position of the first body part based on the relative position information; and transmitting the first control signal to a drive unit.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] The present invention relates to the field of medical imaging equipment and automated control technology, and more particularly to an endoscopic device for imaging the upper gastrointestinal tract using an endoscope. [Background technology]

[0002] An endoscope is a medical instrument that is inserted into the body to observe organs without performing surgery or autopsies (pathological anatomy). An endoscope is inserted into the human body, irradiates it with light, and visualizes the light reflected from the surface of the internal wall. Endoscopes are classified according to their purpose and the body part they are used in. Broadly speaking, they can be divided into rigid endoscopes, in which the endoscopic tube is made of metal, and flexible endoscopes, such as those used in gastrointestinal endoscopes.

[0003] Today, when endoscopists perform endoscopic examinations and find a lesion, they must perform additional actions, such as inserting instruments to perform tissue examinations and pressing buttons on the scope. In these actions, they may let go of the scope, causing the scope to shake and the lesion to move out of the field of view.

[0004] Such endoscopic techniques primarily rely on manual operation by medical professionals to adjust the distal end of the endoscope and acquire images of the patient's internal body parts. This process is highly dependent on the skill and experience of the operator, which makes it difficult to acquire accurate and clear images of the desired body part. In particular, unnecessary movements and difficulty in precise adjustment during the operation of the endoscope can make it difficult to acquire sufficient images of the specific part required for accurate diagnosis.

[0005] Furthermore, endoscopic techniques have limitations in precisely adjusting the position and angle of the distal end of the endoscope, making it difficult to obtain images of areas such as the upper gastrointestinal tract, which has many curves. Such limitations reduce the efficiency and accuracy of endoscopic diagnosis in terms of early detection and accurate localization of diseases. Summary of the Invention [Problem to be solved by the invention]

[0006] The present invention has been proposed to solve the above-mentioned problems, and the problem that the present invention aims to solve is to provide a technology for controlling the position and direction of the tip of an endoscope based on an artificial neural network. [Means for solving the problem]

[0007] To achieve the above-mentioned object, according to one embodiment of the present specification, a control method for an endoscopic device also includes the steps of acquiring an image related to the upper gastrointestinal tract from an image sensor, detecting at least one first body part from the image based on a pre-trained model, calculating relative position information between the detected first body part and the tip portion, generating a first control signal for steering in correspondence with the first body part based on the relative position information, and transmitting the first control signal to a drive unit.

[0008] The step of acquiring images of the upper gastrointestinal tract includes a step of acquiring a first image at a first point and a step of acquiring a second image at a second point, and the step of calculating the relative position information also includes a step of calculating a target rotation angle of the tip based on i) an angle change of the tip between the first point and the second point, and ii) a change between the first image and the second image.

[0009] The method further includes identifying at least one second body part from the image based on the pre-trained model; calculating relative pose information between the identified second body part and the tip; generating a second control signal related to photographing the second body part based on the relative pose information; and transmitting the second control signal to the driving unit.

[0010] The step of calculating the relative pose information also includes the steps of identifying a brightness difference in the image based on the acquired image and brightness information related to lighting, estimating a distance between the tip and the second body part based on the identified brightness difference, and calculating the relative pose information based on the estimated distance.

[0011] The step of acquiring an image of the upper gastrointestinal tract includes a step of acquiring a first image at a first location and a step of acquiring a second image at a second location, and the step of calculating the relative pose information also includes a step of estimating a distance between the tip and the second body part based on i) an angle change of the tip between the first location and the second location, and ii) a change between the first image and the second image, and a step of calculating the relative pose information.

[0012] The pre-trained models may include a classification model and a detection model trained using a dataset of labeled images of a first body part and a second body part related to the upper gastrointestinal tract.

[0013] The step of generating the second control signal also includes the steps of identifying at least one imaging location corresponding to the identified second body part, generating a second control signal based on the at least one imaging location and the relative position information, and capturing an image when the position of the tip corresponds to the imaging location.

[0014] The method also includes adjusting tension of at least one wire and controlling a rotation angle of the tip based on the first control signal.

[0015] The driving unit may further include a step of generating torque feedback and transmitting the torque feedback to an operating unit.

[0016] The torque feedback may have a positive correlation with a target rotation angle to which the tip must move in response to a first control signal related to the image capture.

[0017] In order to achieve the above-mentioned object, an endoscopic device according to one embodiment of the present specification also includes a tip portion having an image sensor, a drive unit that controls the rotation angle of the tip portion, and a control unit that detects at least one first body part from the image based on a pre-trained model, calculates relative position information between the detected first body part and the tip portion, generates a first control signal related to the position of the first body part based on the relative position information, and transmits the first control signal to the drive unit.

[0018] The control unit may acquire a first image at a first point and a second image at a second point, and may calculate a target rotation angle of the tip based on i) an angle change of the tip between the first point and the second point, and ii) the change between the first image and the second image.

[0019] The control unit may identify at least one second body part from the image based on the pre-trained model, calculate relative pose information between the identified second body part and the tip, generate a second control signal related to photographing the second body part based on the relative pose information, and transmit the second control signal to the driving unit.

[0020] The control unit may identify brightness differences in the image based on the acquired image and brightness information related to lighting, estimate a distance between the tip and the second body part based on the identified brightness difference, and calculate the relative pose information based on the estimated distance.

[0021] The control unit may acquire a first image at a first location and a second image at a second location, and estimate a distance between the tip and the second body part based on i) an angle change of the tip between the first location and the second location, and ii) a change between the first image and the second image, and calculate the relative pose information.

[0022] The pre-trained models may include a classification model and a detection model trained using a dataset of labeled images of a first body part and a second body part related to the upper gastrointestinal tract.

[0023] The control unit may identify at least one imaging location corresponding to the identified body part, generate a second control signal based on the at least one imaging location and the relative position information, and capture an image when the position of the tip corresponds to the imaging location.

[0024] The endoscopic device further includes a bending portion connected to the tip portion, and the control unit controls the drive unit based on the control signal, adjusts the tension of at least one wire connected to the drive unit, and adjusts the rotation angle of the tip portion based on the bending of the bending portion.

[0025] The endoscope device further includes an operation unit having a bending steering unit, and the control unit can generate torque feedback based on the drive unit and transmit the torque feedback to the bending steering unit.

[0026] The torque feedback may have a positive correlation with a target rotation angle to which the tip must move in response to a first control signal related to the image capture. [Effects of the Invention]

[0027] According to an embodiment of the present invention, a control unit of an endoscope device controls the position of a distal end portion to accurately and efficiently acquire images of a specific body part, thereby reducing the difficulty of endoscopic operation and improving the accuracy of medical diagnosis.

[0028] The effects of this embodiment are not limited to the effects mentioned above, and other effects not mentioned will be clearly understood by those of ordinary skill in the art from the description in the claims. [Brief explanation of the drawings]

[0029] [Figure 1] 1 is a diagram schematically illustrating an endoscopic device according to an embodiment of the present invention. [Figure 2] 10A to 10C are diagrams illustrating a process in which a bending section is controlled by a driving section and a wire according to an embodiment of the present invention. [Figure 3] 1 is a flowchart illustrating an operation for generating a learning model according to one embodiment of the present invention. [Figure 4A] 1 is a flowchart illustrating the operation of an endoscopic device, in accordance with one embodiment of the present invention. [Figure 4B] 10 is a flowchart illustrating the operation of an endoscopic device in accordance with another embodiment of the present invention. [Figure 4C] 10 is a flowchart illustrating the operation of an endoscopic device according to yet another embodiment of the present invention. [Figure 5] 10 is a flowchart illustrating an operation of calculating relative pose information of an endoscopic device in accordance with an embodiment of the present invention. [Figure 6] 10 is a flowchart illustrating an operation of calculating relative pose information of an endoscopic device according to another embodiment of the present invention. [Figure 7] 10 is a flowchart illustrating an operation of calculating relative pose information of an endoscopic device according to yet another embodiment of the present invention. [Figure 8]1 is a flowchart illustrating operations involved in capturing images in an endoscopic device, in accordance with one embodiment of the present invention. [Figure 9] 10 is a flowchart illustrating operations relating to torque feedback in an endoscopic device, in accordance with one embodiment of the present invention. [Figure 10] FIG. 1 is a block diagram illustrating a block configuration of a computer device according to an embodiment of the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0030] The terms used in the present invention are merely used to describe specific embodiments and are not intended to limit the scope of other embodiments. A singular expression includes a plural expression unless the context clearly dictates otherwise. Terms used herein, including technical or scientific terms, may have the same meaning as commonly understood by a person of ordinary skill in the technical field described in the present invention. Among the terms used in the present invention, commonly defined terms should be interpreted as meanings that are identical to or similar to the meanings they have in the context of the related art, and should not be interpreted as ideal or overly formal unless explicitly defined in the present invention. In some cases, even if a term is defined in the present invention, it should not be interpreted as excluding embodiments of the present invention.

[0031] Hereinafter, various embodiments will be described in detail with reference to the accompanying drawings so that those skilled in the art can easily implement the present invention. However, the technical concept of the present invention may be embodied in various forms and is not limited to the embodiments described herein. In describing the embodiments disclosed herein, if a detailed description of related known technology is deemed to obscure the gist of the technical concept of the present invention, the detailed description of the known technology will be omitted. Identical or similar elements will be designated by the same reference numerals, and redundant description thereof will be omitted.

[0032] Here, the term "module" used in this embodiment refers to a component that performs a specific function, which is performed by software or hardware such as a field programmable gate array (FPGA) or an application-specific integrated circuit (ASIC). However, the "module" is not limited to being performed by software or hardware. The "module" may exist in the form of data stored on an addressable recording medium, or may be embodied by an instruction word, and configured to cause one or more processors to execute a specific function.

[0033] Software may include computer programs, code, instructions, or a combination of one or more of these, which may configure a processing device to operate as desired or may instruct the processing device, either independently or collectively. The software and / or data may be permanently or temporarily embodied in some type of machine, component, physical device, virtual device, computer storage medium or device, or transmitted signal wave to be interpreted by the processing device or to provide instructions or data to the processing device. The software may also be distributed across network-coupled computer systems, stored or executed in a distributed manner. The software and data may be stored in one or more computer-readable storage media. The software may be read into main memory from other computer-readable media, such as a data storage device, or from another device via a communication interface. The software instructions stored in main memory may cause a processor to perform the processes or steps described in detail below. Alternatively, hard-wired circuitry may be used in place of or in combination with software instructions to implement processes consistent with the principles of the invention. Thus, embodiments consistent with the principles of the invention are not limited to any specific combination of hardware circuitry and software.

[0034] The terms used in this application are merely used to describe specific embodiments and are not intended to limit the present invention. The singular terms include the plural terms unless the context clearly dictates otherwise. In this application, terms such as "comprise" or "have" specify the presence of a specified feature, number, step, operation, component, part, or combination thereof, but should be understood not to preclude the presence or addition of one or more other features, numbers, steps, operations, components, parts, or combinations thereof. Terms such as "first" and "second" may be used to describe various components, but the components are not limited by these terms. These terms are used solely to distinguish one component from another.

[0035] The term "learning model" as used herein includes all types of algorithms or methodologies used to learn or understand specific patterns or structures from data. That is, the term "learning model" includes not only machine learning models such as regression models, decision trees, random forests, support vector machines, k-nearest neighbors, naive phase, and clustering algorithms, but also deep learning models such as neural networks, convolutional neural networks, recurrent neural networks, Transformer-based neural networks, generative adversarial networks (GANs), and autoencoders. A "learning model" refers to a set of learned parameters or weights used to predict or classify an output for a specific input. The model can be trained through methods such as supervised learning, unsupervised learning, semi-supervised learning, and reinforcement learning. It also includes not only single models but also various learning methods and structures such as ensemble models, multimodal models, and models via transfer learning. Such learning models can be pre-trained on a computer device separate from the computer device that predicts the output for the input and used on another computer device.

[0036] The learning model according to an embodiment of the present invention may include at least one model related to object classification, object detection, and position estimation.

[0037] FIG. 1 schematically illustrates an endoscopic device according to one embodiment of the present invention.

[0038] 1, an endoscopic device 100 according to an embodiment of the present invention is a flexible endoscope, specifically a digestive endoscope. The endoscopic device 100 includes a component for acquiring medical images of the inside of the digestive system, and a component for inserting a tool and performing treatment or therapy while viewing the medical images, if necessary.

[0039] The endoscope device 100 also includes an output unit 110 , a control unit 120 , a driving unit 130 , a scope 140 , and an operation unit 160 .

[0040] The output unit 110 may also include a display that displays medical images. The output unit 110 may output visualized information using a display module, such as a liquid crystal display (LCD), a thin film transistor-liquid crystal display (TFT LCD), an organic light-emitting diode (OLED), a flexible display, or a 3D display, or may include a display module that implements a touch screen and supports functions such as image display, image enlargement, and image reduction. Furthermore, the touch screen function allows the user to manipulate the image and obtain necessary information. The output unit 110 may display medical images acquired by the scope 140 or medical images processed by the control unit 120.

[0041] The drive unit 130 may provide the necessary power when the scope 140 is inserted into or moves within the body. For example, the drive unit 130 may include a plurality of motors connected to wires inside the scope 140 and a tension adjustment unit that adjusts the tension of the wires. The drive unit 130 may control the power of each of the plurality of motors to control the scope 140 in various directions. Specifically, the drive unit 130 may control the power of each of the plurality of motors to adjust the tension of the wires, bend the bending portion 142, and adjust the rotation angle of the tip portion 143.

[0042] The scope 140 also includes an insertion section 141, a bending section 142, and a tip section 143. The insertion section 141 is a section that is inserted into the inside of the body, and can be steered by the bending section 142 and moved to an internal organ.

[0043] The bending section 142 is connected to the insertion section 141 and can adjust the direction of entry of the scope 140 into the body. The bending section 142 can adjust its rotation angle in response to a command from the user or a control signal from the control section.

[0044] The tip 143 is located at the tip of the scope 140 and can perform various operations in response to commands from a user or control signals from the control unit. The tip 143 also includes an image sensor 151, a nozzle 152, a light 153, a lens 154, and a working channel 155.

[0045] The image sensor 151 may capture an image of the endoscope device. For example, the image sensor 151 may be a complementary metal-oxide-semiconductor (CMOS) sensor or a charge-coupled device (CCD) sensor.

[0046] The nozzle 152 can spray a solution, medication, etc. to clean the lens 154. The nozzle 152 can also spray medication required for tissue examination or treatment into the body.

[0047] The lighting 153 may emit a light source with a certain illuminance so that the image sensor 151 can capture an image. Information regarding the brightness of the lighting 153 may be stored in the control unit 120 in advance.

[0048] The lens 154 may focus light so that a suitable image can be captured by the image sensor 151. Such a lens 154 may also include a wide-angle or zoom function.

[0049] Working channel 155 may refer to a channel for transmitting a separate instrument or sampling tool into the human body.

[0050] The operation unit 160 may refer to a user interface that actually operates the endoscope. The operation unit 160 also includes a bending / steering unit 161 and various buttons, dials, and levers for controlling various functions of the endoscope. A user can input user commands based on the configuration provided in the operation unit 160.

[0051] The bending steering unit 161 may be used to adjust the bending portion 142. The bending steering unit 161 may be embodied in the form of a rotary knob, joystick, or lever, and a user may rotate or move it to steer the direction of the bending portion 142. The bending steering unit 161 may receive torque feedback from the driving unit 130. For example, the torque feedback may be a physical signal based on a control signal from the driving unit 130 that replicates the force generated when the tip portion 143 contacts internal human tissue or is based on a pre-trained model.

[0052] The controller 120 may control the overall operation of the endoscopic device 100 and perform the operation of the endoscopic device according to an embodiment. The controller 120 may control the movement of the scope 140 via a driver 130 connected to the scope 140. The controller 120 may perform various control operations for capturing images of the inside of the digestive tract through the scope 140. The controller 120 may perform various processes on the medical image acquired through the scope 140.

[0053] According to an embodiment, the controller 120 may acquire an image of the upper gastrointestinal tract from an image sensor, detect at least one body part from the image based on a pre-trained model, calculate relative position information between the body part and the distal end, generate a control signal for image capture based on the identified body part and relative pose information, and transmit the control signal to a driver. Such relative position information may refer to steering information for moving the distal end from a first position, which is the current position of the distal end, to a second position. For example, the controller 120 may use the relative position information to calculate a target rotation angle by which the distal end can be steered toward a body part that is located approximately x and y coordinates away on the image. The controller 120 may generate a control signal based on the target rotation angle to control the driver, and adjust the pitch and yaw of the distal end to steer the distal end toward a body part that is located approximately x and y coordinates away on the image.

[0054] According to an embodiment, the controller 120 may acquire an image of the upper gastrointestinal tract from an image sensor, identify at least one body part from the image based on a pre-trained model, calculate relative pose information between the body part and the distal end, generate a control signal for image capture based on the identified body part and the relative pose information, and transmit the control signal to a driver. Such relative pose information may refer to information for moving the distal end to a first pose, which is the current pose of the distal end, and a second pose for effectively capturing the identified body part. A pose may also include the position and orientation of an object in space. For example, the position may be expressed as x, y, and z coordinates in a coordinate system, and the orientation may be expressed as pitch (roll around x-axis), yaw (roll around y-axis), and roll (roll around z-axis).

[0055] The control unit 120 may include a central processing unit (CPU), random access memory (RAM), read-only memory (ROM), a system bus, etc. The control unit 120 may be implemented by a single CPU or multiple CPUs (or digital signal processors (DSPs), system-on-chip (SoC)). In one embodiment, the control unit 120 may be implemented by a digital signal processor (DSP) that processes digital signals, a microprocessor, or a time controller (TCON). However, the control unit 120 may include or be defined by one or more of a central processing unit (CPU), a microcontroller unit (MCU), a micro processing unit (MPU), a controller, an application processor (AP), a communication processor (CP), or an ARM processor, without being limited thereto. The control unit 120 may also be implemented as a system-on-chip (SoC) or large scale integration (LSI) with a built-in processing algorithm, or as a field programmable gate array (FPGA). The control unit 120 may also include a neural processing unit (NPU), a graphics processing unit (GPU), and a tensor processing unit (TPU).

[0056] 2 is a diagram illustrating a process in which a bending section is controlled by a driving section and wires according to an embodiment of the present invention. For convenience of explanation, the operation of controlling the direction of the bending section 142 using two motors 200 is illustrated. However, it will be clear to those skilled in the art that multiple motors can be used to adjust the tension of each wire and adjust the rotation angle of the distal end.

[0057] 2, the endoscope device 100 also includes a motor 200, a first wire 210, and a second wire 220 included in a driving unit 130. In order to bend the bending portion 142 to one side, the endoscope device 100 controls the motor 200 to increase the tension of the first wire 210 and decrease the tension of the second wire 220. In addition, in order to bend the bending portion 142 to the other side, the endoscope device 100 controls the motor 200 to decrease the tension of the first wire 210 and increase the tension of the second wire 220.

[0058] In this manner, the endoscope device 100 can adjust the rotation angle of the tip portion 143.

[0059] 3 is a flowchart illustrating operations for generating a learning model for object detection according to one embodiment of the present invention. The learning model in FIG. 3 may correspond to a pre-trained model. While such operations are disclosed as being learned by a separate computer device for convenience of explanation, it will be apparent to one skilled in the art that they may be performed by the endoscope system 100 or a separate computer device.

[0060] Referring to FIG. 3, a computer device may review an endoscopic image and tag or label specific body parts in step S310. Based on user input, the computer device may label parts of the upper gastrointestinal tract in the image with bounding boxes. For example, the upper gastrointestinal tract may include at least one of the oral cavity, pharynx, esophagus, stomach, and duodenum. The endoscopic image may also include images of the lumen of the oral cavity, pharynx, esophagus, stomach, and duodenum.

[0061] According to an embodiment, the computer device may generate a dataset including labeled images in step S320. Such a dataset may include various lighting conditions, viewing angles, body part states, etc.

[0062] According to an embodiment, the computer device may train a neural network model using the generated dataset in step S330. The computer device may select the neural network model and train the model based on the dataset. For example, such a model may be an object detection model, including a convolutional neural network.

[0063] According to another embodiment, a computer device may generate a learning model for object classification, which may be a model for identifying objects for image capture.

[0064] In another embodiment, the computer device may review the endoscopic image and tag or label a specific body part in step S310. Based on user input, the computer device may label the image with a body part of the upper gastrointestinal tract. Such a body part of the upper gastrointestinal tract may indicate the body part for which the image was taken.

[0065] According to an embodiment, the computer device may generate a dataset including labeled images in step S320. Such a dataset may include various lighting conditions, viewing angles, body part states, etc.

[0066] According to an embodiment, the computer device may train a neural network model using the generated dataset in step S330. The computer device may select the neural network model and train the model based on the dataset. For example, such a model may be an object classification model, including a convolutional neural network.

[0067] According to still another embodiment, the computer device may generate a learning model using endoscopic images and control history information corresponding to a user's clinical skills. Such control history information may refer to control history information used to observe the endoscopic image and adjust the direction, angle, depth, etc. of the endoscope. According to another embodiment, the computer device may acquire endoscopic images and control history information related to the upper gastrointestinal tract in step S310. For example, the upper gastrointestinal tract may include at least one of the oral cavity, pharynx, esophagus, stomach, and duodenum. The endoscopic images may also include images of the lumen of the oral cavity, pharynx, esophagus, stomach, and duodenum.

[0068] According to yet another embodiment, the computer device may generate a data set including the endoscopic image and the control history information in step S320.

[0069] According to still another embodiment, the computer device may train a neural network model using the generated dataset in step S330. The computer device may select a neural network model and train the model based on the dataset. For example, such a model may include a classification model such as a convolutional neural network (CNN) for image processing, and at least one of a recurrent neural network (RNN) and a long short-term memory (LSTM) network for sequence processing of control history information.

[0070] FIG. 4A is a flowchart illustrating the operation of an endoscopic device, according to one embodiment of the present invention.

[0071] 4A, the endoscopic device may acquire an image of the upper gastrointestinal tract from an image sensor in step S410a. For example, the distal end of the endoscopic device may enter the upper gastrointestinal tract and acquire an image in which an optical signal is converted into an electrical signal from the image sensor.

[0072] According to an embodiment, the endoscope device may detect at least one body part from the image based on a pre-trained model in step S420a. The endoscope device may detect the body part based on a pre-trained model labeled with a bounding box for the upper gastrointestinal tract and confirm the image location of the at least one body part within the image.

[0073] According to an embodiment, the endoscopic device may calculate relative position information between the body part and the distal end of the endoscopic device in step S430a. The endoscopic device may calculate the relative position information based on an image change caused by an angle change of the distal end. The endoscopic device may calculate the relative position information based on disparity and the angle change of the distal end.

[0074] In step S440a, the endoscope apparatus according to an embodiment may generate a control signal for steering based on the relative position information and the position of the body part.

[0075] In step S450a, the endoscopic device according to an embodiment may transmit a control signal to the driving unit, and may adjust the tension of at least one wire to control the rotation angle of the distal end portion so that the distal end portion can be steered in response to the detected body part.

[0076] As a specific example, an endoscope device according to an embodiment may acquire a first image at a first point, and then acquire a second image at a second point rotated by a certain amount Δθ. If the time difference between the first image and the second image is ΔL, then the distance L from the center coordinate of the detected body part bounding box is target θ for steering the remote tip to the sensed body part target can be expressed as in Equation 1. [Number 1] θ target =(Δθ / ΔL)L target

[0077] Since Δθ can be calculated from encoder information of the drive unit and ΔL can be calculated from the time difference between the images, a relational expression can be derived between the target distance on the image and the target angle of tip rotation by the drive unit.

[0078] Equation 1 can be expressed as a coordinate system. That is, since the rotation of the endoscope device based on the direction in which the distal end of the endoscope device advances into the body is called roll, the endoscope device can be steered toward a body part that is distant by the x and y coordinates on the image through control of the pitch and yaw by the driving unit.

[0079] FIG. 4B is a flowchart illustrating the operation of an endoscopic device according to another embodiment of the present invention.

[0080] 4B, the endoscope device may acquire an image of the upper gastrointestinal tract from an image sensor in step S410b. For example, the distal end of the endoscope device may enter the upper gastrointestinal tract and acquire an image in which an optical signal is converted into an electrical signal from the image sensor.

[0081] In another embodiment, the endoscopic device may identify at least one body part from the image based on a pre-trained model in step S420b. The endoscopic device may classify the body part based on a pre-trained model labeled for the upper gastrointestinal tract and identify at least one body part within the image.

[0082] In another embodiment, the endoscopic device may calculate relative pose information including the distance between the body part and the distal end of the endoscopic device in step S430b. For example, the endoscopic device may calculate the relative pose information based on the brightness of the lighting, the image change due to the angle change of the distal end, and the movement trajectory of the distal end. The operation of the endoscopic device to calculate such relative pose information may correspond to FIGS. 5, 6, and 7.

[0083] In another embodiment, the endoscope device may generate control signals for image capture based on the identified body part and relative pose information in step S440b, including signals for adjusting the distal end of the endoscope.

[0084] In step S450b, the endoscopic device according to another embodiment may transmit a control signal to the driving unit. In step S450b, the endoscopic device according to one embodiment may adjust the tension of at least one wire to control the rotation angle of the distal end portion so as to correspond to at least one preset imaging point related to the identified body part.

[0085] According to another embodiment, an endoscopic device may capture an image based on the identified body part and relative pose information in step S460b. The endoscopic device may capture an image when the distal end of the endoscopic device is located at at least one preset imaging position related to the identified body part.

[0086] FIG. 4C is a flowchart illustrating the operation of an endoscopic device according to yet another embodiment of the present invention.

[0087] 4C, the endoscope device may acquire an image of the upper gastrointestinal tract from an image sensor in step S410c. For example, the distal end of the endoscope device may enter the upper gastrointestinal tract and acquire an image in which an optical signal is converted into an electrical signal from the image sensor.

[0088] In yet another embodiment, an endoscopic device may generate a control signal for image capture based on a pre-trained model using control history information corresponding to the endoscopic image and the user's clinical skills in step S420c. The endoscopic device may input the endoscopic image to a classification model and a training model, which is a sequence processing model, to generate a control signal. Such a control signal may also include a signal for adjusting the distal end of the endoscope.

[0089] According to still another embodiment, the endoscope device may transmit a control signal to the driver in step S430c, and may adjust the tension of at least one wire to control the rotation angle of the distal end portion so as to correspond to at least one preset imaging point related to the identified body part in step S430c.

[0090] According to another embodiment, an endoscope device may capture an image based on the identified body part and relative pose information in step S440c. The endoscope device may capture an image when the distal end is located at at least one predetermined capture position related to the identified body part.

[0091] 5 is a flowchart illustrating an operation of calculating relative pose information of an endoscopic device according to an embodiment of the present invention. The operation of the endoscopic device in FIG. 5 may correspond to step S430b in FIG. 4B. Such relative pose information may refer to information for moving from a first pose, which is the current pose of the tip, to a second pose for effectively capturing an image of an identified body part.

[0092] 5, the endoscope system may identify brightness differences in an image based on brightness information related to the acquired image and illumination in operation S510. The endoscope system may analyze the difference between the stored brightness information related to illumination and the brightness information of the image captured by the image sensor and the illumination. The endoscope system may also identify brightness patterns indicated by characteristics of internal tissues, and the intensity and direction of illumination.

[0093] According to an embodiment, the endoscope device may calculate relative pose information based on the identified brightness difference in step S520. The endoscope device may estimate a distance between the distal end and a specific body part in the current pose of the distal end based on the identified brightness difference, and may calculate position information for the distal end to move and direction information for the image sensor of the distal end to capture an image based on the estimated distance, the center coordinates in the image, and the coordinates of the identified body part. The relative pose information may also include the path, direction, and required angle change of the distal end passing through the inside of the body.

[0094] According to an embodiment, the endoscope device may project a specific pattern of light based on structured light in addition to brightness differences, and calculate relative pose information based on changes in the pattern. For example, the endoscope device may estimate a distance based on the change in the pattern, and calculate position information for the distal end to move and direction information for the image sensor provided at the distal end to capture an image based on the estimated distance, the center coordinates in the image, and the coordinates of the identified body part.

[0095] An endoscopic device according to one embodiment may use multiple image sensors to analyze the pixel difference between two images and calculate the distance from the tip to the lesion.

[0096] 6 is a flowchart illustrating the operation of calculating relative pose information of an endoscope apparatus according to another embodiment of the present invention. The operation of the endoscope apparatus in FIG. 6 may correspond to step S430b in FIG. 4B.

[0097] Referring to FIG. 6, the endoscope device may acquire a first image at a first location in step S610.

[0098] According to another embodiment, the endoscope apparatus may acquire a second image at a second location in step S620.

[0099] In another embodiment, the endoscope apparatus may calculate relative pose information based on the change in angle of the distal end portion and the change in pixels on the screen while moving from the first point to the second point in step S630.

[0100] For example, an endoscope may analyze an input image using a convolution operation. The endoscope may calculate the location of an identified body part based on a pre-trained model that is trained by extracting features from the input image through various convolution layers, using bounding box coordinates. The coordinates may consist of left, top, right, and bottom, and the output bounding box coordinates may be converted to the center coordinates of the bounding box through a post-processing process.

[0101] The center coordinates of such a bounding box can be expressed as Equation 2.

number

[0102] The endoscopic device can calculate the angle change (Δθ) of the tip while moving from the first point to the second point based on the encoder information, and can calculate the distance between the identified body part and the tip using the pixel change (ΔL) of the image.

[0103] This can be shown as in Equation 3.

number

[0104] The above-described operation of the endoscope device can be realized not only by a single image sensor but also by stereo vision based on multiple image sensors, where ΔB is the distance between the image sensors.

[0105] The endoscopic device can estimate the distance and, based on the estimated distance, the center coordinates in the image, and the coordinates of the identified body part, calculate position information for the movement of the tip and directional information for the image sensor provided at the tip to take images.

[0106] 7 is a flowchart illustrating the operation of calculating relative pose information of an endoscope apparatus according to yet another embodiment of the present invention. The operation of the endoscope apparatus in FIG. 7 may correspond to step S430b in FIG. 4B.

[0107] 7, the endoscope device may acquire movement trajectory data of the distal end in step S710. Such movement trajectory data is data on the movement of the distal end inside the body, and may include at least one of sensor-based tracking data, such as a magnetic field sensor, a gyroscope, and an accelerometer, and image-based tracking data based on an image sensor.

[0108] According to still another embodiment, the endoscope device may calculate relative pose information between the body part and the distal end based on the movement trajectory of the distal end and the acquired image in step S720. For example, the endoscope device may calculate relative pose information for moving the distal end from a first pose, which is the current position, to a second pose, which is the target position, based on position information based on the encoder value of the motor and a depth estimation algorithm.

[0109] 8 is a flowchart illustrating operations related to image capture by an endoscopic device according to an embodiment of the present invention. The operations of the endoscopic device in FIG. 8 may correspond to steps S440b through S460b in FIG. 4B and steps S420c through S440c in FIG. 4C.

[0110] 8, the endoscope device may identify at least one preset imaging position for the identified body part in step S810. The at least one preset imaging position is a position previously determined for imaging or treating the body part, and may refer to a position previously set by a user based on the user's clinical skills, or a position where structural information of the identified body part can be obtained.

[0111] In one embodiment, the endoscope device may generate a control signal for controlling the rotation of the distal end portion based on at least one imaging position and relative pose information in step S820. Specifically, the endoscope device may calculate relative pose information between the body part and the endoscope device, and generate a control signal for positioning the distal end portion at at least one imaging position based on the calculated relative pose information and the distance and direction from at least one predetermined imaging position.

[0112] In one embodiment, an endoscopic device transmits a control signal to a driving unit in step S830, and the driving unit can adjust the tension of at least one wire based on the control signal, bend the bending portion, and control the rotation angle of the tip portion.

[0113] In step S840, an endoscope apparatus according to an embodiment may capture an image based on an image sensor at each of the at least one imaging point when the distal end portion is disposed at each of the at least one imaging point.

[0114] 9 is a flowchart illustrating operations related to torque feedback in an endoscopic device, according to one embodiment of the present invention. Such torque feedback can be a physical signal based on a drive control signal that replicates the forces generated when the tip contacts internal human tissue or can be based on a pre-trained model.

[0115] 9, the endoscope device may generate torque feedback related to the movement of the distal end portion in step S910. Such torque feedback may be generated based on a control signal related to image capture. The torque feedback is generated based on information related to the direction and amount of rotation of the distal end portion of the endoscope device, and the movement distance and the magnitude of the torque feedback may have a positive correlation.

[0116] The endoscope device may transmit the torque feedback to the operation unit in step S920. For example, the endoscope device may control the distal end according to a rotation angle that must be controlled for image capture, and the bending steering unit may transmit torque feedback in a direction in which the distal end is steered, thereby providing the feedback to the user.

[0117] FIG. 10 is a block diagram illustrating the block configuration of a computer device according to one embodiment of the present invention.

[0118] The computing device 1000 also includes a memory 1010 and a processor 1020. The computing device 1000 may be a separate device from the endoscope device or may be included in the control unit of the endoscope device, and may execute one or more sets of instructions that cause the computing device to perform any one or more of the methodologies described herein.

[0119] The memory 1010 may store a set of instructions, including system-related instructions that cause any one or more of the methodology functions described herein and user interface-related instructions. The memory 1010 temporarily or permanently stores data such as basic programs, applications, and configuration information for device operation. The memory 1010 may include, but is not limited to, RAM, ROM, and a permanent mass storage device such as a disk drive. Such software components may be loaded from a computer-readable recording medium separate from the memory 1010 using a drive mechanism. Such separate computer-readable recording media may include computer-readable recording media such as a floppy drive, disk, tape, DVD (digital versatile disc) / CD-ROM (compact disc read only memory) drive, and memory card. According to one embodiment, the software components may be loaded into the memory 1010 via a communication unit rather than a computer-readable recording medium. Additionally, the memory 1010 may provide stored data at the request of the processor 1020. According to an embodiment of the present invention, the memory 1010 may store configuration information.

[0120] The processor 1020 controls the overall operation of the computing device. The processor 1020 may be configured to process instructions by performing basic arithmetic, logic, and input / output operations. The instructions may be provided to the processor 1020 by the memory 1010. For example, the processor 1020 may be configured to execute instructions received from program code stored in a storage device such as the memory 1010. For example, the processor 1020 may control the device to perform operations according to various embodiments described above.

[0121] According to an embodiment of the present invention, the processor 1020 may review the endoscopic image and tag or label specific body parts. Based on user input or the like, the computing device may label or classify parts of the upper gastrointestinal tract in the image with bounding boxes. For example, such upper gastrointestinal tract may include at least one of the oral cavity, pharynx, esophagus, stomach, and duodenum.

[0122] According to an embodiment of the present invention, the processor 1020 may generate a dataset including labeled images, including datasets that include various lighting conditions, viewing angles, body part conditions, etc.

[0123] According to an embodiment of the present invention, the processor 1020 may use the generated dataset to train a neural network model. The computer device may select the neural network model and train the model based on the dataset. For example, such a model may be at least one of an object detection model and an object classification model, and may include a convolutional neural network.

[0124] According to another embodiment of the present invention, the processor 1020 may generate a learning model using endoscopic images and control history information corresponding to the user's clinical skills. Such control history information may refer to control history information used to observe the endoscopic images and adjust the direction, angle, depth, etc. of the endoscope. The processor 1020 may acquire endoscopic images and control history information related to the upper gastrointestinal tract. For example, the upper gastrointestinal tract may include at least one of the oral cavity, pharynx, esophagus, stomach, and duodenum.

[0125] In other embodiments, the processor 1020 may generate a data set including endoscopic images and control history information.

[0126] In another embodiment, the processor 1020 may use the generated dataset to train a neural network model. The computer device may select the neural network model and train the model based on the dataset. For example, such a model may include a classification model, such as a convolutional neural network (CNN), for image processing, and / or a sequence processing model, such as a recurrent neural network (RNN) and a long short-term memory (LSTM) network, for sequence processing of control history information.

[0127] Although the present embodiment has been described above by way of limited embodiments and drawings, those skilled in the art will appreciate that various modifications and variations may be made from the foregoing description. For example, the described techniques may be performed in a different order than described, and / or the described components, such as systems, structures, devices, and circuits, may be combined or combined in a different manner than described, or may be substituted or replaced by other components or equivalents, and still achieve suitable results.

[0128] Accordingly, other implementations, other embodiments, and equivalents of the claims are intended to be within the scope of the claims. [Explanation of symbols]

[0129] 100: Endoscopic device 110: Output section 120: Control unit 130: Drive unit 140:Scope 141: Insertion section 142: Curved section 143:Tip 151: Image sensor 152: Nozzle 153: Lighting 154: Lens 155: Working Channel 160:Operation unit 161: Curved steering section 200: Motor 210: First wire 220: Second wire 1000: Computer equipment 1010:Memory 1020: Processor

Claims

1. 1. A control method for an endoscope apparatus, comprising: acquiring an image relating to the upper gastrointestinal tract from an image sensor; detecting at least one first body part from the image based on a pre-trained model; calculating relative position information between the sensed first body part and the tip; generating a first control signal for steering the first body part relative to the first body part based on the relative position information; transmitting the first control signal to a driver.

2. The step of acquiring an image relating to the upper gastrointestinal tract comprises: acquiring a first image at a first location; acquiring a second image at a second location; The step of calculating the relative position information includes:

2. The method of claim 1, further comprising calculating a target rotation angle of the tip based on: i) an angular change of the tip between the first point and the second point; and ii) a change between the first image and the second image.

3. identifying at least one second body part from the image based on the pre-trained model; calculating relative pose information between the identified second body part and a tip part; generating a second control signal for imaging the second body part based on the relative pose information; The method of claim 1 , further comprising: transmitting the second control signal to the driver.

4. The step of calculating the relative pose information includes: identifying brightness differences in the image based on the acquired image and brightness information related to lighting; estimating a distance between the tip and the second body part based on the identified brightness difference; and calculating the relative pose information based on the estimated distance.

5. The step of acquiring an image relating to the upper gastrointestinal tract comprises: acquiring a first image at a first location; acquiring a second image at a second location; The step of calculating the relative pose information includes: i) estimating a distance between the tip and the second body part based on the change in angle of the tip between the first point and the second point, and ii) the change in angle between the first image and the second image; and calculating the relative pose information.

6. 4. The method of claim 3, wherein the pre-trained models include a classification model and a detection model trained on a dataset of labeled images of a first body part and a second body part associated with the upper gastrointestinal tract.

7. The step of generating the second control signal comprises: identifying at least one image capture location corresponding to the identified second body part; generating the second control signal based on the at least one photographing location and the relative pose information; and capturing an image when the tip position corresponds to the at least one image capture location.

8. The method of claim 1 , further comprising adjusting tension of at least one wire and controlling a rotation angle of the tip based on the first control signal.

9. The method of claim 1 , further comprising the step of: the drive unit generating torque feedback; and transmitting the torque feedback to an operating unit.

10. 10. The method of claim 9, wherein the torque feedback is positively correlated with a target rotation angle that the tip must move due to the first control signal.

11. In an endoscope device, a tip portion having an image sensor; a drive unit for controlling the rotation angle of the tip portion; a control unit that detects at least one first body part from the image based on a pre-trained model, calculates relative position information between the detected first body part and the tip end, generates a first control signal for steering in correspondence with the first body part based on the relative position information, and transmits the first control signal to a drive unit.

12. The control unit capturing a first image at a first location and a second image at a second location; The endoscope device according to claim 11, wherein a target rotation angle of the tip portion is calculated based on i) an angle change of the tip portion between the first point and the second point, and ii) a change between the first image and the second image.

13. The control unit 12. The endoscope device according to claim 11, further comprising: identifying at least one second body part from the image based on the pre-trained model; calculating relative pose information between the identified second body part and the tip portion; generating a second control signal related to imaging of the second body part based on the relative pose information; and transmitting the second control signal to the drive unit.

14. The control unit The endoscopic device according to claim 13, further comprising: identifying a brightness difference in the image based on the acquired image and brightness information related to lighting; estimating a distance between the tip and the second body part based on the identified brightness difference; and calculating the relative pose information based on the estimated distance.

15. The control unit capturing a first image at a first location and a second image at a second location; The endoscopic device according to claim 13, wherein the distance between the tip and the second body part is estimated based on i) an angle change of the tip between the first point and the second point, and ii) a change between the first image and the second image, and the relative pose information is calculated.

16. The endoscope device of claim 13, wherein the pre-trained models include a classification model and a detection model trained using a dataset of labeled images of a first body part and a second body part related to the upper gastrointestinal tract.

17. The control unit The endoscopic device according to claim 13, further comprising: identifying at least one imaging point corresponding to the identified body part; generating the second control signal based on the at least one imaging point and the relative pose information; and capturing an image when the position of the tip corresponds to the at least one imaging point.

18. further comprising a curved portion connected to the tip portion; The control unit The endoscope device according to claim 11, wherein the drive unit is controlled based on the control signal, the tension of at least one wire connected to the drive unit is adjusted, and the rotation angle of the tip portion is adjusted based on the bending of the bending portion.

19. Further including an operating unit having a bending steering unit, The control unit The endoscope device according to claim 11 , wherein torque feedback is generated based on the drive unit, and the torque feedback is transmitted to the bending steering unit.

20. The endoscope apparatus according to claim 19, wherein the torque feedback has a positive correlation with a target rotation angle to which the distal end portion must move in response to the first control signal.

Images