Endoscope device for acquiring lower gastrointestinal tract images and method of controlling the same

Abstract

To provide an endoscope device for acquiring lower gastrointestinal tract images and a method of controlling the same.SOLUTION: The invention belongs to the field of medical imaging equipment and automation control technology, and particularly relates to an endoscope device associated with imaging of the lower gastrointestinal tract using an endoscope, and to a method of controlling the endoscope device. The method includes the steps of: acquiring an image of the lower gastrointestinal tract from an image sensor; acquiring environmental information related to a distal end portion; sensing at least one first body part from the image based on a pre-trained model; calculating relative position information between the first body part and the distal end portion of the endoscope device; generating a first control signal for steering in correspondence with the first body part based on the environmental information and the relative position information; and transmitting the first control signal to a drive unit.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to the field of medical imaging equipment and automated control technology, and more particularly to an endoscopic device for imaging the lower gastrointestinal tract using an endoscope. [Background technology]

[0002] An endoscope is a medical instrument that is inserted into the body to observe organs without performing surgery or autopsies (pathological anatomy). An endoscope is inserted into the human body, irradiates it with light, and visualizes the light reflected from the surface of the internal wall. Endoscopes are classified according to their purpose and the body part they are used in. Broadly speaking, they can be divided into rigid endoscopes, in which the endoscopic tube is made of metal, and flexible endoscopes, such as those used in gastrointestinal endoscopes.

[0003] Today, when endoscopists perform endoscopic examinations and find a lesion, they must perform additional actions, such as inserting instruments to perform tissue examinations and pressing buttons on the scope. In these actions, they may let go of the scope, causing the scope to shake and the lesion to move out of the field of view.

[0004] Such endoscopic techniques primarily rely on manual operation by medical professionals to adjust the distal end of the endoscope and acquire images of the patient's internal body parts. This process is highly dependent on the skill and experience of the operator, which makes it difficult to acquire accurate and clear images of the desired body part. In particular, unnecessary movements and difficulty in precise adjustment during the operation of the endoscope can make it difficult to acquire sufficient images of the specific part required for accurate diagnosis.

[0005] Furthermore, endoscopic techniques have limitations in precisely adjusting the position and angle of the distal end of the endoscope, making it difficult to obtain images of areas such as the lower gastrointestinal tract, which has many curves. Such limitations reduce the efficiency and accuracy of endoscopic diagnosis in terms of early detection and accurate localization of diseases. Summary of the Invention [Problem to be solved by the invention]

[0006] The present invention has been proposed to solve the above-mentioned problems, and the problem that the present invention aims to solve is to provide a technology for controlling the position and direction of the tip of an endoscope based on an artificial neural network. [Means for solving the problem]

[0007] To achieve the above-mentioned object, a control method for an endoscopic device according to one embodiment of the present specification also includes the steps of acquiring an image related to the lower gastrointestinal tract from an image sensor, acquiring environmental information related to the tip, detecting at least one first body part from the image based on a pre-trained model, calculating relative position information between the first body part and the tip of the endoscopic device, generating a first control signal for steering in correspondence with the first body part based on the environmental information and the relative position information, and transmitting the first control signal to a drive unit.

[0008] The step of acquiring the environmental information also includes the steps of calculating rotation information based on an encoder value of the driving unit, and calculating position information based on the acquired image and the rotation information.

[0009] The step of calculating the position information also includes the steps of generating a stitched image based on feature points of the acquired images, and generating a body shape structure based on the stitched image.

[0010] The step of acquiring an image of the lower gastrointestinal tract includes a step of acquiring a first image at a first point and a step of acquiring a second image at a second point, and the step of calculating the relative position information also includes a step of calculating a target rotation angle of the tip based on i) an angle change of the tip between the first point and the second point, and ii) a change between the first image and the second image.

[0011] The method further includes identifying at least one second body part from the image based on the pre-trained model; calculating relative pose information between the identified second body part and the tip; generating a second control signal related to photographing the second body part based on the relative pose information; and transmitting the second control signal to the driving unit.

[0012] The pre-trained models may include a classification model and a detection model trained using a dataset of labeled images of a first body part and a second body part related to the lower gastrointestinal tract.

[0013] The step of generating the second control signal also includes the steps of identifying at least one imaging location corresponding to the identified second body part, and generating a second control signal for controlling rotation of the tip based on the at least one imaging location and the relative pose information.

[0014] The method may also further include capturing an image when the tip position corresponds to the capture point.

[0015] The method further includes displaying the environmental information based on a display unit.

[0016] The method also includes generating torque feedback based on the drive unit and transmitting the torque feedback to an operating unit.

[0017] To achieve the above-mentioned object, an endoscopic device according to one embodiment of the present specification also includes a distal end portion having an image sensor capable of acquiring images related to the lower gastrointestinal tract, a driving unit that controls the rotation angle of the distal end portion, and a control unit that acquires images related to the lower gastrointestinal tract and environmental information related to the distal end portion, detects at least one first body part from the images based on a pre-trained model, calculates relative position information between the first body part and the distal end portion of the endoscopic device, generates a first control signal for steering in correspondence with the first body part based on the environmental information and the relative position information, and transmits the first control signal to the driving unit.

[0018] The control unit may calculate rotation information based on an encoder value of the driving unit, and calculate position information based on the image and the rotation information obtained by calculating rotation information.

[0019] The control unit may generate a stitched image based on feature points of the acquired images, and generate a body shape structure based on the stitched image.

[0020] In the endoscopic device, the tip portion further includes a light, and the control unit acquires a first image at a first point and a second image at a second point, and can calculate a target rotation angle of the tip portion based on i) an angle change of the tip portion between the first point and the second point, and ii) the change between the first image and the second image.

[0021] The control unit may identify at least one second body part from the image based on the pre-trained model, calculate relative pose information between the identified second body part and the tip, generate a second control signal related to photographing the second body part based on the relative pose information, and transmit the second control signal to the driving unit.

[0022] The pre-trained models may include a classification model and a detection model trained using a dataset of labeled images of a first body part and a second body part related to the lower gastrointestinal tract.

[0023] The control unit may identify at least one imaging point corresponding to the identified second body part, and generate a second control signal for controlling rotation of the tip unit based on the at least one imaging point and the relative pose information.

[0024] The control unit may capture an image when the position of the tip corresponds to the image capturing point.

[0025] The device may further include a display unit, and the control unit may display the pause information based on the display unit.

[0026] The robot may further include an operation unit having a bending steering unit, and the control unit may generate torque feedback based on the drive unit and transmit the torque feedback to the bending steering unit. [Effects of the Invention]

[0027] According to an embodiment of the present invention, a control unit of an endoscope device controls the position of a distal end portion to accurately and efficiently acquire images of a specific body part, thereby reducing the difficulty of endoscopic operation and improving the accuracy of medical diagnosis.

[0028] The effects of this embodiment are not limited to the effects mentioned above, and other effects not mentioned will be clearly understood by those of ordinary skill in the art from the description in the claims. [Brief explanation of the drawings]

[0029] [Figure 1] 1 is a diagram schematically illustrating an endoscopic device according to an embodiment of the present invention. [Figure 2]10A to 10C are diagrams illustrating a process in which a bending section is controlled by a driving section and a wire according to an embodiment of the present invention. [Figure 3] 1 is a flowchart illustrating an operation for generating a learning model according to one embodiment of the present invention. [Figure 4A] 1 is a flowchart illustrating the operation of an endoscopic device, in accordance with one embodiment of the present invention. [Figure 4B] 10 is a flowchart illustrating the operation of an endoscopic device according to another embodiment of the present invention. [Figure 5] 10 is a flowchart illustrating an operation of acquiring environmental information of an endoscopic device in accordance with one embodiment of the present invention. [Figure 6] 10 is a flowchart illustrating an operation of calculating relative pose information of an endoscopic device in accordance with an embodiment of the present invention. [Figure 7] 10 is a flowchart illustrating an operation of calculating relative pose information of an endoscope apparatus according to another embodiment of the present invention. [Figure 8] 1 is a flowchart illustrating operations involved in capturing images in an endoscopic device, in accordance with one embodiment of the present invention. [Figure 9] 10 is a flowchart illustrating operations relating to torque feedback in an endoscopic device, in accordance with one embodiment of the present invention. [Figure 10] FIG. 1 is a block diagram illustrating a block configuration of a computer device according to an embodiment of the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0030] The terms used in the present invention are merely used to describe specific embodiments and are not intended to limit the scope of other embodiments. A singular expression includes a plural expression unless the context clearly dictates otherwise. Terms used herein, including technical or scientific terms, may have the same meaning as commonly understood by a person of ordinary skill in the technical field described in the present invention. Among the terms used in the present invention, commonly defined terms should be interpreted as meanings that are identical to or similar to the meanings they have in the context of the related art, and should not be interpreted as ideal or overly formal unless explicitly defined in the present invention. In some cases, even if a term is defined in the present invention, it should not be interpreted as excluding embodiments of the present invention.

[0031] Hereinafter, various embodiments will be described in detail with reference to the accompanying drawings so that those skilled in the art can easily implement the present invention. However, the technical concept of the present invention may be embodied in various forms and is not limited to the embodiments described herein. In describing the embodiments disclosed herein, if a detailed description of related known technology is deemed to obscure the gist of the technical concept of the present invention, the detailed description of the known technology will be omitted. Identical or similar elements will be designated by the same reference numerals, and redundant description thereof will be omitted.

[0032] Here, the term "module" used in this embodiment refers to a component that performs a specific function, which is performed by software or hardware such as a field programmable gate array (FPGA) or an application-specific integrated circuit (ASIC). However, the "module" is not limited to being performed by software or hardware. The "module" may exist in the form of data stored on an addressable recording medium, or may be embodied by an instruction word, and configured to cause one or more processors to execute a specific function.

[0033] Software may include computer programs, code, instructions, or a combination of one or more of these, which may configure a processing device to operate as desired or may instruct the processing device, either independently or collectively. The software and / or data may be permanently or temporarily embodied in some type of machine, component, physical device, virtual device, computer storage medium or device, or transmitted signal wave to be interpreted by the processing device or to provide instructions or data to the processing device. The software may also be distributed across network-coupled computer systems, stored or executed in a distributed manner. The software and data may be stored in one or more computer-readable storage media. The software may be read into main memory from other computer-readable media, such as a data storage device, or from another device via a communication interface. The software instructions stored in main memory may cause a processor to perform the processes or steps described in detail below. Alternatively, hard-wired circuitry may be used in place of or in combination with software instructions to implement processes consistent with the principles of the invention. Thus, embodiments consistent with the principles of the invention are not limited to any specific combination of hardware circuitry and software.

[0034] The terms used in this application are merely used to describe specific embodiments and are not intended to limit the present invention. The singular terms include the plural terms unless the context clearly dictates otherwise. In this application, terms such as "comprise" or "have" specify the presence of a specified feature, number, step, operation, component, part, or combination thereof, but should be understood not to preclude the presence or addition of one or more other features, numbers, steps, operations, components, parts, or combinations thereof. Terms such as "first" and "second" may be used to describe various components, but the components are not limited by these terms. These terms are used solely to distinguish one component from another.

[0035] The term "learning model" as used herein includes all types of algorithms or methodologies used to learn or understand specific patterns or structures from data. That is, the term "learning model" includes not only machine learning models such as regression models, decision trees, random forests, support vector machines, k-nearest neighbors, naive phase, and clustering algorithms, but also deep learning models such as neural networks, convolutional neural networks, recurrent neural networks, Transformer-based neural networks, generative adversarial networks (GANs), and autoencoders. A "learning model" refers to a set of learned parameters or weights used to predict or classify an output for a specific input. The model can be trained through methods such as supervised learning, unsupervised learning, semi-supervised learning, and reinforcement learning. It also includes not only single models but also various learning methods and structures such as ensemble models, multimodal models, and models via transfer learning. Such learning models can be pre-trained on a computer device separate from the computer device that predicts the output for the input and used on another computer device.

[0036] The learning model according to an embodiment of the present invention may include at least one model related to object classification, object detection, and position estimation.

[0037] FIG. 1 schematically illustrates an endoscopic device according to one embodiment of the present invention.

[0038] 1, an endoscopic device 100 according to an embodiment of the present invention is a flexible endoscope, specifically a digestive endoscope. The endoscopic device 100 includes a component for acquiring medical images of the inside of the digestive system, and a component for inserting a tool and performing treatment or therapy while viewing the medical images, if necessary.

[0039] The endoscope device 100 also includes an output unit 110 , a control unit 120 , a driving unit 130 , a scope 140 , and an operation unit 160 .

[0040] The output unit 110 may also include a display that displays medical images. The output unit 110 may output visualized information using a display module, such as a liquid crystal display (LCD), a thin film transistor-liquid crystal display (TFT LCD), an organic light-emitting diode (OLED), a flexible display, or a 3D display, or may include a display module that implements a touch screen and supports functions such as image display, image enlargement, and image reduction. Furthermore, the touch screen function allows the user to manipulate the image and obtain necessary information. The output unit 110 may display medical images acquired by the scope 140 or medical images processed by the control unit 120.

[0041] The drive unit 130 may provide the necessary power when the scope 140 is inserted into or moves within the body. For example, the drive unit 130 may include a plurality of motors connected to wires inside the scope 140 and a tension adjustment unit that adjusts the tension of the wires. The drive unit 130 may control the power of each of the plurality of motors to control the scope 140 in various directions. Specifically, the drive unit 130 may control the power of each of the plurality of motors to adjust the tension of the wires, bend the bending portion 142, and adjust the rotation angle of the tip portion 143.

[0042] The scope 140 also includes an insertion section 141, a bending section 142, and a tip section 143. The insertion section 141 is a section that is inserted into the inside of the body, and can be steered by the bending section 142 and moved to an internal organ.

[0043] The bending section 142 is connected to the insertion section 141 and can adjust the direction of entry of the scope 140 into the body. The bending section 142 can adjust its rotation angle in response to a command from the user or a control signal from the control section.

[0044] The tip 143 is located at the tip of the scope 140 and can perform various operations in response to commands from a user or control signals from the control unit. The tip 143 also includes an image sensor 151, a nozzle 152, a light 153, a lens 154, and a working channel 155.

[0045] The image sensor 151 may capture an image of the endoscope device. For example, the image sensor 151 may be a complementary metal-oxide-semiconductor (CMOS) sensor or a charge-coupled device (CCD) sensor.

[0046] The nozzle 152 can spray a solution, medication, etc. to clean the lens 154. The nozzle 152 can also spray medication required for tissue examination or treatment into the body.

[0047] The lighting 153 may emit a light source with a certain illuminance so that the image sensor 151 can capture an image. Information about the brightness of the lighting 153 may be stored in the control unit 120 in advance.

[0048] The lens 154 may focus light so that a suitable image can be captured by the image sensor 151. Such a lens 154 may also include a wide-angle function or a zoom function.

[0049] Working channel 155 may refer to a channel for transmitting a separate instrument or sampling tool into the human body.

[0050] The operation unit 160 may refer to a user interface that actually operates the endoscope. The operation unit 160 also includes a bending / steering unit 161 and various buttons, dials, and levers for controlling various functions of the endoscope. A user can input user commands based on the configuration provided in the operation unit 160.

[0051] The bending steering unit 161 may be used to adjust the bending portion 142. The bending steering unit 161 may be embodied in the form of a rotary knob, joystick, or lever, and a user may rotate or move it to steer the direction of the bending portion 142. The bending steering unit 161 may receive torque feedback from the driving unit 130. For example, the torque feedback may be a physical signal based on a control signal from the driving unit 130 that replicates the force generated when the tip portion 143 contacts internal human tissue or is based on a pre-trained model.

[0052] The controller 120 may control the overall operation of the endoscopic device 100 and perform the operation of the endoscopic device according to an embodiment. The controller 120 may control the movement of the scope 140 via a driver 130 connected to the scope 140. The controller 120 may perform various control operations for capturing images of the inside of the digestive tract through the scope 140. The controller 120 may perform various processes on the medical image acquired through the scope 140.

[0053] According to an embodiment, the controller 120 may acquire an image of the lower gastrointestinal tract from an image sensor, detect at least one body part from the image based on a pre-trained model, calculate relative position information between the body part and the distal end, generate a control signal for image capture, and transmit the control signal to a driver. Such relative position information may refer to steering information for moving the distal end from a first position, which is the current position of the distal end, to a second position. For example, the controller 120 may use the relative position information to calculate a target rotation angle by which the distal end can be steered toward a body part that is located approximately x and y coordinates away on the image. The controller 120 may also generate a control signal to control the driver based on the target rotation angle, and adjust the pitch and yaw of the distal end to steer the distal end toward a body part that is located approximately x and y coordinates away on the image.

[0054] According to an embodiment, the controller 120 may acquire an image of the lower gastrointestinal tract from an image sensor, identify at least one body part from the image based on a pre-trained model, calculate relative pose information between the body part and the tip, generate a control signal for image capture based on the identified body part and the relative pose information, and transmit the control signal to a driver. Such relative pose information may refer to information for moving the tip from a first pose, which is the current pose of the tip, to a second pose for effectively capturing the identified body part. A pose may also include the position and orientation of an object in space. For example, the position may be expressed as x, y, and z coordinates in a coordinate system, and the orientation may be expressed as pitch (roll around x-axis), yaw (roll around y-axis), and roll (roll around z-axis).

[0055] The control unit 120 may include a central processing unit (CPU), random access memory (RAM), read-only memory (ROM), a system bus, etc. The control unit 120 may be implemented by a single CPU or multiple CPUs (or digital signal processors (DSPs), system-on-chip (SoC)). In one embodiment, the control unit 120 may be implemented by a digital signal processor (DSP) that processes digital signals, a microprocessor, or a time controller (TCON). However, the control unit 120 may include or be defined by one or more of a central processing unit (CPU), a microcontroller unit (MCU), a micro processing unit (MPU), a controller, an application processor (AP), a communication processor (CP), or an ARM processor, without being limited thereto. The control unit 120 may also be implemented as a system-on-chip (SoC) or large scale integration (LSI) with a built-in processing algorithm, or as a field programmable gate array (FPGA). The control unit 120 may also include a neural processing unit (NPU), a graphics processing unit (GPU), and a tensor processing unit (TPU).

[0056] 2 is a diagram illustrating a process in which a bending section is controlled by a driving section and wires according to an embodiment of the present invention. For convenience of explanation, the operation of controlling the direction of the bending section 142 using two motors 200 is illustrated. However, it will be clear to those skilled in the art that multiple motors can be used to adjust the tension of each wire.

[0057] 2, the endoscope device 100 also includes a motor 200, a first wire 210, and a second wire 220 included in the driving unit 130. In order to bend the bending portion 142 to one side, the endoscope device 100 may control the motor 200 to increase the tension of the first wire 210 and decrease the tension of the second wire 220. In addition, in order to bend the bending portion 142 to the other side, the endoscope device 100 may control the motor 200 to decrease the tension of the first wire 210 and increase the tension of the second wire 220.

[0058] In this manner, the endoscope device 100 can adjust the rotation angle of the tip portion 143.

[0059] 3 is a flowchart illustrating operations for generating a learning model for object detection according to one embodiment of the present invention. The learning model in FIG. 3 may correspond to a pre-trained model. While such operations are disclosed as being learned by a separate computer device for convenience of explanation, it will be apparent to one skilled in the art that they may be performed by the endoscope system 100 or a separate computer device.

[0060] Referring to FIG. 3 , in step S310, a computer device may review an endoscopic image and tag or label specific body parts. Based on user input, the computer device may label parts of the lower gastrointestinal tract with bounding boxes in the image. Based on user input, the computer device may label parts of the lower gastrointestinal tract in the image. For example, the lower gastrointestinal tract may include at least one of the duodenum, jejunum, ileum, cecum, appendix, colon, rectum, and anus. The endoscopic image may also include images of the lumens of the ileum, cecum, appendix, colon, rectum, and anus.

[0061] According to an embodiment, the computer device may generate a dataset including labeled images in step S320. Such a dataset may include various lighting conditions, viewing angles, body part states, etc.

[0062] According to an embodiment, the computer device may train a neural network model using the generated dataset in step S330. The computer device may select the neural network model and train the model based on the dataset. For example, such a model may be an object detection model, including a convolutional neural network.

[0063] According to another embodiment, a computer device may generate a learning model for object classification, which may be a model for identifying objects for image capture.

[0064] In another embodiment, the computer device may review the endoscopic image and tag or label a specific body part in step S310. Based on user input, the computer device may label the image with a lower gastrointestinal tract part. Such a lower gastrointestinal tract part may indicate the body part for which the image was taken.

[0065] According to an embodiment, the computer device may generate a dataset including labeled images in step S320. Such a dataset may include various lighting conditions, viewing angles, body part states, etc.

[0066] According to an embodiment, the computer device may train a neural network model using the generated dataset in step S330. The computer device may select the neural network model and train the model based on the dataset. For example, such a model may be an object classification model, including a convolutional neural network.

[0067] FIG. 4A is a flowchart illustrating the operation of an endoscopic device, according to one embodiment of the present invention.

[0068] 4A, the endoscope device may acquire an image of the lower gastrointestinal tract from an image sensor in step S410a. For example, the distal end of the endoscope device may enter the lower gastrointestinal tract, and acquire an image in which an optical signal is converted into an electrical signal from the image sensor.

[0069] According to an embodiment, the endoscope apparatus may acquire environmental information related to the distal end portion in step S420a. According to an embodiment, the environmental information may include at least one of i) the spatial position (e.g., x, y, z coordinates) of the distal end portion, ii) information including the directionality (e.g., rotation angle, tilt angle), and iii) structural information related to the body shape. For example, the environmental information may be determined by sensing a magnetic field, inertia, or mechanical deformation using a sensor built into the distal end portion, or may be determined through a separate calculation method. The operation of acquiring such environmental information may correspond to FIG. 5.

[0070] According to an embodiment, the endoscope device may detect at least one body part from the image based on a pre-trained model in step S430a. The endoscope device may detect the body part based on a pre-trained model labeled with a bounding box for the lower gastrointestinal tract and confirm the position of the at least one body part within the image.

[0071] According to an embodiment, in step S440a, an endoscopic device may calculate relative position information including a distance between a body part and a distal end of the endoscopic device. The endoscopic device may calculate the relative position information based on an image change caused by an angle change of the distal end. The endoscopic device may calculate the relative position information based on disparity and the angle change of the distal end.

[0072] In one embodiment, the endoscope device may generate a steering control signal for a position relative to a body part based on the relative position information and the environmental information in step S450a. The endoscope device may associate the distal end with the body part based on the relative position information, and generate a steering signal based on structural information related to the body part and body shape based on the environmental information.

[0073] As a specific example, an endoscope device according to an embodiment may acquire a first image at a first point, and then acquire a second image at a second point rotated by a certain amount Δθ. If the time difference between the first image and the second image is ΔL, then the center coordinates of the detected body part bounding box can be calculated by L. target θ for steering the remote tip to the sensed body part target can be expressed as in Equation 1. [Number 1] θ target =(Δθ / ΔL)L target

[0074] Since Δθ can be calculated from encoder information of the drive unit and ΔL can be calculated from the time difference between the images, a relational expression can be derived between the target distance on the image and the target angle of tip rotation by the drive unit.

[0075] Equation 1 can be expressed as a coordinate system. That is, since the rotation of the endoscope device based on the direction in which the distal end of the endoscope device advances into the body is called roll, the endoscope device can be steered toward a body part that is distant by the x and y coordinates on the image through control of the pitch and yaw by the driving unit.

[0076] In step S460a, the endoscopic device according to an embodiment may transmit a control signal to the driving unit, and may adjust the tension of at least one wire to control the rotation angle of the distal end portion so that the distal end portion can be steered in response to the detected body part.

[0077] According to an embodiment, the endoscope device may display environmental information using a display unit in step S470a. The endoscope device may display environmental information including at least one of i) a spatial position (e.g., x, y, z coordinates) of the distal end, ii) information including directionality (e.g., rotation angle, tilt angle), and iii) structural information related to the body shape on the display unit, thereby displaying environmental information of the current distal end.

[0078] FIG. 4B is a flowchart illustrating the operation of an endoscopic device according to another embodiment of the present invention.

[0079] 4B, the endoscope device may acquire an image of the lower gastrointestinal tract from an image sensor in step S410b. For example, the distal end of the endoscope device may enter the lower gastrointestinal tract and acquire an image in which an optical signal is converted into an electrical signal from the image sensor.

[0080] In one embodiment, the endoscopy device may identify at least one body part from the image based on the environmental information and a pre-trained model in step S420b. The endoscopy device may classify the body part based on a labeled pre-trained model for the lower gastrointestinal tract and identify the at least one body part within the image.

[0081] In step S430b, an endoscope according to an embodiment may calculate relative pose information including a distance between a body part and a distal end of the endoscope. For example, the endoscope may calculate the relative pose information based on the brightness of lighting or based on an image change caused by a change in the angle of the distal end. The operation of the endoscope to calculate such relative pose information may correspond to FIGS. 6 and 7.

[0082] In accordance with an embodiment, the endoscope device may generate control signals for image capture based on the identified body part and relative pose information in step S440b, including signals for adjusting the distal end of the endoscope.

[0083] In step S450b, the endoscopic device according to an embodiment may transmit a control signal to the driving unit, and may adjust the tension of at least one wire to control the rotation angle of the distal end portion so as to correspond to at least one preset imaging point related to the identified body part.

[0084] According to an embodiment, the endoscope device may capture an image based on the identified body part and relative pose information in step S460b. The endoscope device may capture an image when the distal end is located at at least one predetermined capture position related to the identified body part.

[0085] 5 is a flowchart illustrating an operation of acquiring environmental information of an endoscope apparatus according to an embodiment of the present invention. The operation of the endoscope apparatus in FIG. 5 may correspond to step S420a in FIG. 4A.

[0086] According to one embodiment, the environmental information may include at least one of i) the spatial position of the tip (e.g., x, y, z coordinates), ii) information including directionality (e.g., rotation angle, tilt angle), and iii) structural information related to the body shape. For example, an endoscopic device may acquire environmental information through simultaneous localization and mapping (SLAM). The endoscopic device may extract feature points from an image and acquire the environmental information based on data association between the feature points and previously acquired images.

[0087] 5, the endoscope apparatus may calculate rotation information based on encoder values ​​of the driving units in step S510. For example, the rotation information may be determined based on an encoder value of a first motor that determines the x-axis rotational motion and an encoder value of a second motor that determines the y-axis rotational motion of the scope for each frame.

[0088] According to an embodiment, the endoscopic device may generate a stitched image based on feature points of the acquired images in step S520. For example, the endoscopic device may capture successive images while the distal end enters the body and stitch them together based on feature points of the images.

[0089] According to an embodiment, an endoscopic device may generate a body shape structure based on the stitched images in step S530. The endoscopic device may grasp the morphology of specific body parts or structures and convert them into structural information. Such an endoscopic device may capture various images from various angles of the target environment or object and generate structural information through 3D reconstruction. For example, a point cloud may be generated in 3D space through feature detection and feature matching, and the morphology of specific body parts or structures may be grasped and converted into structural information through processes such as mesh generation and texture mapping.

[0090] According to an embodiment, the endoscope device may calculate the position information of the distal end based on stitched images or body shape structure information acquired from continuously captured images and rotation information in step S540. The endoscope device may calculate the position information of the distal end based on the continuous images captured while passing through the interior of the body and a visual map of the internal structure of the body obtained by analyzing the continuous images. In this case, since the interior of the human body is a dynamic environment where tissue movement and deformation may occur, it is possible to store environment information for each patient and track the movement of internal structures due to breathing and heartbeat to update the SLAM information.

[0091] Through a series of processes, the endoscopic device can acquire environmental information including at least one of i) the spatial position of the tip (e.g., x, y, z coordinates), ii) information including directionality (e.g., rotation angle, tilt angle), and iii) structural information related to the body shape.

[0092] 6 is a flowchart illustrating an operation of calculating relative pose information of an endoscopic device according to an embodiment of the present invention. The operation of the endoscopic device in FIG. 6 may correspond to step S430b in FIG. 4B. Such relative pose information may refer to position information for moving from a first pose, which is the current pose of the tip, to a second pose for effectively capturing an image of an identified body part.

[0093] 6, the endoscopy device may identify brightness differences in an image based on brightness information related to the acquired image and illumination in operation S610. The endoscopy device may analyze the difference between the stored brightness information related to illumination and the brightness information of the image captured by the image sensor and the illumination. The endoscopy device may also identify brightness patterns indicated by characteristics of internal tissues, and the intensity and direction of illumination.

[0094] According to an embodiment, the endoscope device may calculate relative pose information based on the identified brightness difference in step S620. The endoscope device may estimate a distance between the distal end and a specific body part at the current pose of the distal end based on the identified brightness difference, and may calculate position information for the distal end to move and direction information for the image sensor included in the distal end to capture an image based on the estimated distance, the center coordinates in the image, and the coordinates of the identified body part. The relative pose information may also include the path, direction, and required angle change of the distal end passing through the inside of the body.

[0095] According to an embodiment, the endoscope device may project a specific pattern of light based on structured light in addition to brightness differences, and calculate relative pose information based on changes in the pattern. For example, the endoscope device may estimate a distance based on the change in the pattern, and calculate position information for the distal end to move and direction information for the image sensor provided at the distal end to capture an image based on the estimated distance, the center coordinates in the image, and the coordinates of the identified body part.

[0096] An endoscopic device according to one embodiment may use multiple image sensors to analyze the pixel difference between two images and calculate the distance from the tip to the lesion.

[0097] 7 is a flowchart illustrating the operation of calculating relative pose information of an endoscope apparatus according to another embodiment of the present invention. The operation of the endoscope apparatus in FIG. 7 may correspond to step S430b in FIG. 4B.

[0098] Referring to FIG. 7, the endoscope device may acquire a first image at a first location in step S710.

[0099] According to another embodiment, the endoscope apparatus may acquire a second image at a second location in step S720.

[0100] In another embodiment, the endoscope apparatus may calculate relative pose information based on the change in angle of the tip and the change in pixels on the screen while moving from the first point to the second point in step S730.

[0101] For example, an endoscope may analyze an input image using a convolution operation. The endoscope may calculate the location of an identified body part based on a pre-trained model that is trained by extracting features from the input image through various convolution layers, using bounding box coordinates. The coordinates may consist of left, top, right, and bottom, and the output bounding box coordinates may be converted to the center coordinates of the bounding box through a post-processing process.

[0102] The center coordinates of such a bounding box can be expressed as Equation 2.

number

[0103] JPEG2025129141000003.jpg6163 indicates the center coordinate of the x-axis, which is the average value of the boundary between the left (l) and right (r) sides. JPEG2025129141000004.jpg8163 indicates the center coordinate on the y-axis, which is the average value of the boundary between the upper (t) and lower (b) sides.

[0104] The endoscopic device can calculate the angle change (Δθ) of the tip while moving from the first point to the second point based on the encoder information, and can calculate the distance between the identified body part and the tip using the pixel change (ΔL) of the image.

[0105] This can be shown as in Equation 3.

number

[0106] JPEG2025129141000006.jpg9163 is the distance between the lesion and the tip, R is the radius of rotation of the tip, Δθ is the angular change of the endoscope tip, f is the focal length, ΔB is the displacement of the image sensor, and ΔL is the pixel change on the screen, which refers to the disparity of the same object on the two images. Based on the calculated rotation angle, the endoscopic device can calculate a target angle to which the tip of the endoscope must move using a polynomial trajectory. The endoscopic device can analyze the difference between the calculated target movement angle and the current angle of the tip and generate a control signal.

[0107] The above-described operation of the endoscope device can be realized not only by a single image sensor but also by stereo vision based on multiple image sensors, where ΔB is the distance between the image sensors.

[0108] The endoscopic device can estimate the distance and, based on the estimated distance, the center coordinates in the image, and the coordinates of the identified body part, calculate position information for the movement of the tip and directional information for the image sensor provided at the tip to take images.

[0109] 8 is a flowchart illustrating operations related to image capture by an endoscope according to an embodiment of the present invention. The operations of the endoscope in FIG. 8 may correspond to steps S440b through S460b in FIG. 4B.

[0110] 8, the endoscope device may identify at least one preset imaging position for the identified body part in step S810. The at least one preset imaging position is a position previously determined for imaging or treating the body part, and may refer to a position previously set by a user based on the user's clinical skills, or a position where structural information of the identified body part can be obtained.

[0111] In one embodiment, the endoscope device may generate a control signal for controlling the rotation of the distal end portion based on at least one imaging position and relative pose information in step S820. Specifically, the endoscope device may calculate relative pose information between the body part and the endoscope device, and generate a control signal for positioning the distal end portion at at least one imaging position based on the calculated relative pose information and the distance and direction from at least one predetermined imaging position.

[0112] In one embodiment, an endoscopic device transmits a control signal to a driving unit in step S830, and the driving unit can adjust the tension of at least one wire based on the control signal, bend the bending portion, and control the rotation angle of the tip portion.

[0113] In step S840, an endoscope apparatus according to an embodiment may capture an image based on an image sensor at each of the at least one imaging point when the distal end portion is disposed at each of the at least one imaging point.

[0114] 9 is a flowchart illustrating operations related to torque feedback in an endoscopic device, according to one embodiment of the present invention. Such torque feedback can be a physical signal based on a drive control signal that replicates the forces generated when the tip contacts internal human tissue or can be based on a pre-trained model.

[0115] 9, the endoscope device may generate torque feedback related to the movement of the distal end portion in step S910. Such torque feedback may be generated based on a control signal related to image capture. The torque feedback is generated based on information related to the direction and amount of rotation of the distal end portion of the endoscope device, and the movement distance and the magnitude of the torque feedback may have a positive correlation.

[0116] The endoscope device may transmit the torque feedback to the operation unit in step S920. For example, the endoscope device may control the distal end according to a rotation angle that must be controlled for image capture, and the bending steering unit may transmit torque feedback in a direction in which the distal end is steered, thereby providing the feedback to the user.

[0117] FIG. 10 is a block diagram illustrating the block configuration of a computer device according to one embodiment of the present invention.

[0118] The computing device 1000 also includes a memory 1010 and a processor 1020. The computing device 1000 may be a separate device from the endoscope device or may be included in the control unit of the endoscope device, and may execute one or more sets of instructions that cause the computing device to perform any one or more of the methodologies described herein.

[0119] The memory 1010 may store a set of instructions, including system-related instructions that cause any one or more of the methodology functions described herein and user interface-related instructions. The memory 1010 temporarily or permanently stores data such as basic programs, applications, and configuration information for device operation. The memory 1010 may include, but is not limited to, RAM, ROM, and a permanent mass storage device such as a disk drive. Such software components may be loaded from a computer-readable recording medium separate from the memory 1010 using a drive mechanism. Such separate computer-readable recording media may include computer-readable recording media such as a floppy drive, disk, tape, DVD (digital versatile disc) / CD-ROM (compact disc read only memory) drive, and memory card. According to one embodiment, the software components may be loaded into the memory 1010 via a communication unit rather than a computer-readable recording medium. Additionally, the memory 1010 may provide stored data at the request of the processor 1020. According to an embodiment of the present invention, the memory 1010 may store configuration information.

[0120] The processor 1020 controls the overall operation of the computing device. The processor 1020 may be configured to process instructions by performing basic arithmetic, logic, and input / output operations. The instructions may be provided to the processor 1020 by the memory 1010. For example, the processor 1020 may be configured to execute instructions received from program code stored in a storage device such as the memory 1010. For example, the processor 1020 may control the device to perform operations according to various embodiments described above.

[0121] According to an embodiment of the present invention, the processor 1020 may review the endoscopic image and tag or label specific body parts. Based on user input or the like, the computing device may label or classify parts of the lower gastrointestinal tract in the image with bounding boxes. For example, such lower gastrointestinal tract may include at least one of the duodenum, jejunum, ileum, cecum, appendix, colon, rectum, and anus.

[0122] According to an embodiment of the present invention, the processor 1020 may generate a dataset including labeled images, including datasets that include various lighting conditions, viewing angles, body part conditions, etc.

[0123] According to an embodiment of the present invention, the processor 1020 may use the generated dataset to train a neural network model. The computer device may select the neural network model and train the model based on the dataset. For example, such a model may be at least one of an object detection model and an object classification model, and may include a convolutional neural network.

[0124] Although the present embodiment has been described above by way of limited embodiments and drawings, those skilled in the art will appreciate that various modifications and variations may be made from the foregoing description. For example, the described techniques may be performed in a different order than described, and / or the described components, such as systems, structures, devices, and circuits, may be combined or combined in a different manner than described, or may be substituted or replaced by other components or equivalents, and still achieve suitable results.

[0125] Accordingly, other implementations, other embodiments, and equivalents of the claims are intended to be within the scope of the claims. [Explanation of symbols]

[0126] 100: Endoscopic device 110: Output section 120: Control unit 130: Drive unit 140:Scope 141: Insertion section 142: Curved section 143:Tip 151: Image sensor 152: Nozzle 153: Lighting 154: Lens 155: Working Channel 160:Operation unit 161: Curved steering section 200: Motor 210: First wire 220: Second wire 1000: Computer equipment 1010:Memory 1020: Processor

Claims

1. 1. A control method for an endoscope apparatus, comprising: acquiring an image relating to the lower gastrointestinal tract from an image sensor; acquiring environmental information relating to the tip; detecting at least one first body part from the image based on a pre-trained model; calculating relative position information between the first body part and a distal end portion of the endoscope device; generating a first control signal for steering the first body part corresponding to the environmental information and the relative position information; transmitting the first control signal to a driver.

2. The step of acquiring environmental information includes: calculating rotation information based on the encoder value of the driving unit; and calculating position information based on the acquired image and the rotation information.

3. The step of calculating the position information includes: generating a stitched image based on feature points of the acquired images; and generating a body shape structure based on the stitched images.

4. The step of acquiring an image relating to the lower gastrointestinal tract includes: acquiring a first image at a first location; acquiring a second image at a second location; The step of calculating the relative position information includes:

2. The method of claim 1, further comprising calculating a target rotation angle of the tip based on: i) an angular change of the tip between the first point and the second point; and ii) a change between the first image and the second image.

5. identifying at least one second body part from the image based on the pre-trained model; calculating relative pose information between the identified second body part and a tip part; generating a second control signal for imaging the second body part based on the relative pose information; The method of claim 1 , further comprising: transmitting the second control signal to the driver.

6. 6. The method of claim 5, wherein the pre-trained models include a classification model and a detection model trained on a dataset of labeled images of a first body part and a second body part related to the lower gastrointestinal tract.

7. The step of generating the second control signal comprises: identifying at least one image capture location corresponding to the identified second body part; and generating a second control signal for controlling rotation of the tip based on the at least one image capture location and the relative pose information.

8. The method of claim 7 , further comprising capturing an image when the tip position corresponds to the capture location.

9. The method of claim 1 , further comprising displaying the environmental information based on a display unit.

10. The method of claim 1 , further comprising generating torque feedback based on the drive unit and transmitting the torque feedback to an operating unit.

11. In an endoscope device, a distal end portion having an image sensor capable of acquiring images relating to the lower gastrointestinal tract; a drive unit for controlling the rotation angle of the tip portion; an endoscope device including a control unit that acquires an image related to the lower gastrointestinal tract, acquires environmental information related to the tip, detects at least one first body part from the image based on a pre-trained model, calculates relative position information between the first body part and the tip of the endoscope device, generates a first control signal for steering in correspondence with the first body part based on the environmental information and the relative position information, and transmits the first control signal to a drive unit.

12. The control unit The endoscope apparatus according to claim 11 , wherein rotation information is calculated based on an encoder value of the driving unit, and position information is calculated based on the image and the rotation information acquired by calculating the rotation information.

13. The control unit The endoscope apparatus according to claim 12, further comprising: generating a stitched image based on feature points of the acquired images; and generating a body shape structure based on the stitched image.

14. the tip further comprises a light; The control unit capturing a first image at a first location and a second image at a second location; The endoscope apparatus according to claim 11, wherein a target rotation angle of the tip portion is calculated based on i) an angle change of the tip portion between the first point and the second point, and ii) a change between the first image and the second image.

15. The control unit 12. The endoscope apparatus according to claim 11, further comprising: identifying at least one second body part from the image based on the pre-trained model; calculating relative pose information between the identified second body part and the distal end portion; generating a second control signal related to imaging of the second body part based on the relative pose information; and transmitting the second control signal to the driving unit.

16. The endoscope device of claim 15, wherein the pre-trained models include a classification model and a detection model trained using a dataset of labeled images of a first body part and a second body part related to the lower gastrointestinal tract.

17. The control unit The endoscopic device according to claim 15, further comprising: identifying at least one imaging point corresponding to the identified second body part; and generating a second control signal for controlling rotation of the tip portion based on the at least one imaging point and the relative pose information.

18. The control unit The endoscope device according to claim 17 , wherein an image is captured when the position of the tip portion corresponds to the imaging point.

19. The endoscope apparatus according to claim 11 , further comprising a display unit, wherein the control unit displays the pose information based on the display unit.

20. The endoscope device according to claim 11 , further comprising an operation unit having a bending steering unit, wherein the control unit generates torque feedback based on the drive unit and transmits the torque feedback to the bending steering unit.

Images