Endoscopic device for acquiring images of the lower gastrointestinal tract, and method for controlling the same.

Abstract

To provide an endoscope device for acquiring lower gastrointestinal tract images and a method of controlling the same.SOLUTION: The invention belongs to the field of medical imaging equipment and automation control technology, and particularly relates to an endoscope device associated with imaging of the lower gastrointestinal tract using an endoscope, and to a method of controlling the endoscope device. The method includes the steps of: acquiring an image of the lower gastrointestinal tract from an image sensor; acquiring environmental information related to a distal end portion; sensing at least one first body part from the image based on a pre-trained model; calculating relative position information between the first body part and the distal end portion of the endoscope device; generating a first control signal for steering in correspondence with the first body part based on the environmental information and the relative position information; and transmitting the first control signal to a drive unit.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] The present invention belongs to the technical fields of medical imaging equipment and automation control technology, and particularly relates to an endoscope device related to image capture of the lower gastrointestinal tract using an endoscope.

Background Art

[0002] An endoscope is a general term for medical instruments that can observe organs by inserting a scope into the body without performing surgery or autopsy (pathological autopsy). The endoscope inserts a scope into the human body, irradiates light, and visualizes the light reflected from the surface of the inner wall. Depending on the purpose and body part, the types of endoscopes are classified, and generally, they can be classified into rigid endoscopes in which the endoscope tube is formed of metal and flexible endoscopes represented by gastrointestinal endoscopes.

[0003] Today, when an endoscopy specialist performs an endoscopic examination and discovers a lesion, additional operations such as inserting an instrument for tissue examination or pressing a button on the scope must be performed. At this time, while releasing the hand holding the scope, the scope shakes and the lesion moves out of the video field.

[0004] Such endoscope technology mainly depends on the manual operation of medical experts to adjust the tip of the endoscope and obtain an image of the internal body part of the patient. Such a process highly depends on the proficiency and experience of the operator, and thus has the problem that it is difficult to obtain an accurate and clear image of the desired body part. In particular, during the process of operating the endoscope, due to unnecessary movements or difficulties in precise adjustment, an image of a specific part necessary for an accurate diagnosis cannot be sufficiently obtained.

[0005] Furthermore, endoscopic techniques have limitations in precisely adjusting the position and angle of the endoscope tip, which presents particular difficulties when acquiring images of areas with many curves, such as the lower gastrointestinal tract. Such limitations reduce the efficiency and accuracy of endoscopic diagnosis in terms of early detection of disease and accurate localization. [Overview of the project] [Problems that the invention aims to solve]

[0006] This invention was proposed to solve the aforementioned problems, and the problem that this invention aims to solve is to provide a technology for controlling the position and direction of the tip of an endoscope based on an artificial neural network. [Means for solving the problem]

[0007] A control method for an endoscope device according to one embodiment of this specification for achieving the aforementioned objectives includes the steps of: acquiring an image relating to the lower gastrointestinal tract from an image sensor; acquiring environmental information relating to the tip; sensing at least one first body part from the image based on a pre-learned model; calculating relative position information between the first body part and the tip of the endoscope device; generating a first control signal for steering in correspondence with the first body part based on the environmental information and the relative position information; and transmitting the first control signal to a drive unit.

[0008] The step of acquiring the environmental information also includes the step of calculating rotation information based on the encoder value of the drive unit, and the step of calculating position information based on the acquired image and the rotation information.

[0009] The step of calculating the positional information also includes the step of generating a stitched image based on the feature points of the acquired image, and the step of generating a body shape structure based on the stitched image.

[0010] The step of acquiring an image relating to the lower gastrointestinal tract includes a step of acquiring a first image at a first location and a step of acquiring a second image at a second location, and the step of calculating the relative position information also includes a step of calculating a target rotation angle of the tip based on i) the change in angle of the tip between the first location and the second location, and ii) the change between the first image and the second image.

[0011] The method further includes the steps of: identifying at least one second body part from the image based on the pre-trained model; calculating relative pose information between the identified second body part and its tip; generating a second control signal related to photographing the second body part based on the relative pose information; and transmitting the second control signal to the drive unit.

[0012] The aforementioned pre-trained models also include classification and sensing models trained on a dataset of labeled images of the first and second body parts related to the lower gastrointestinal tract.

[0013] The step of generating the second control signal also includes the step of identifying at least one shooting location corresponding to the identified second body part, and the step of generating a second control signal for controlling the rotation of the tip based on the at least one shooting location and the relative pose information.

[0014] The method also further includes the step of capturing an image if the position of the tip corresponds to the shooting location.

[0015] The above method also further includes the step of displaying the environmental information based on the display unit.

[0016] The method further includes the step of generating torque feedback based on the drive unit and transmitting the torque feedback to the operating unit.

[0017] An endoscope according to one embodiment of this specification for achieving the aforementioned objectives also includes: a tip equipped with an image sensor capable of acquiring an image relating to the lower gastrointestinal tract; a drive unit for controlling the rotation angle of the tip; and a control unit that acquires an image relating to the lower gastrointestinal tract, acquires environmental information relating to the tip, senses at least one first body part from the image based on a pre-learned model, calculates relative position information between the first body part and the tip of the endoscope, generates a first control signal for steering in correspondence with the first body part based on the environmental information and the relative position information, and transmits the first control signal to the drive unit.

[0018] The control unit can calculate rotation information based on the encoder value of the drive unit and calculate position information based on the acquired image and rotation information.

[0019] The control unit can generate a stitched image based on the feature points of the acquired image, and generate a body shape structure based on the stitched image.

[0020] The endoscope device further includes illumination at its tip, and the control unit can acquire a first image at a first location, acquire a second image at a second location, and calculate a target rotation angle of the tip based on i) the change in angle of the tip between the first and second locations, and ii) the change between the first and second images.

[0021] The control unit may identify at least one second body part from the image based on the pre-learned model, calculate relative pose information between the identified second body part and the tip, generate a second control signal related to photographing the second body part based on the relative pose information, and transmit the second control signal to the drive unit.

[0022] The pre-trained model also includes a classification model and a perception model that are trained using, as a dataset, images labeled for a first body part and a second body part related to the lower gastrointestinal tract.

[0023] The control unit can identify at least one imaging location corresponding to the identified second body part and generate a second control signal for controlling the rotation of the distal end based on the at least one imaging location and the relative pose information.

[0024] When the position of the distal end corresponds to the imaging location, the control unit can capture an image.

[0025] The endoscopic apparatus further includes a display unit, and the control unit can display the pose information based on the display unit.

[0026] The endoscopic apparatus further includes an operation unit having a bending steering unit, and the control unit can generate torque feedback based on the drive unit and transmit the torque feedback to the bending steering unit.

Advantages of the Invention

[0027] According to an embodiment of the present invention, the control unit of the endoscope apparatus can control the position of the distal end and accurately and efficiently acquire an image of a specific body part. Thereby, the difficulty of endoscope operation can be reduced, and the accuracy of medical diagnosis can be improved.

[0028] The effects of this embodiment are not limited to the effects mentioned above, and other effects not mentioned can be clearly understood by those of ordinary skill in the art from the description of the claims.

Brief Description of the Drawings

[0029] [Figure 1] This is a diagram schematically illustrating an endoscope apparatus according to an embodiment of the present invention. [Figure 2]This figure illustrates the process by which a curved section is controlled by a drive unit and a wire according to one embodiment of the present invention. [Figure 3] This is a flowchart illustrating the operation of generating a learning model according to one embodiment of the present invention. [Figure 4A] This is a flowchart illustrating the operation of an endoscope device according to one embodiment of the present invention. [Figure 4B] This is a flowchart illustrating the operation of an endoscope device according to another embodiment of the present invention. [Figure 5] This is a flowchart illustrating the operation of acquiring environmental information from an endoscope device according to one embodiment of the present invention. [Figure 6] This is a flowchart illustrating the operation of calculating relative pause information of an endoscope device according to one embodiment of the present invention. [Figure 7] This is a flowchart illustrating the operation for calculating relative pause information of an endoscope device according to another embodiment of the present invention. [Figure 8] This is a flowchart illustrating the operations related to image acquisition by an endoscope device according to one embodiment of the present invention. [Figure 9] This is a flowchart illustrating the operation related to torque feedback in an endoscope device according to one embodiment of the present invention. [Figure 10] This is a block diagram illustrating the block configuration of a computer device according to one embodiment of the present invention. [Modes for carrying out the invention]

[0030] The terms used in this invention are used solely to describe specific embodiments and are not intended to limit the scope of other embodiments. Singular expressions also include plural expressions unless explicitly stated in the context. Terms used herein, including technical and scientific terms, may have the same meaning as those generally understood by a person of ordinary skill in the art described herein. General, predefined terms used herein are to be interpreted as having the same or similar meaning as they do in the context of the relevant art, and not as to be idealistic or overly formal unless explicitly defined herein. Where applicable, a term defined herein is not to be construed as excluding embodiments of the invention.

[0031] In the following, various embodiments will be described in detail with reference to the accompanying drawings, so as to be easily implemented by a person with ordinary skill in the art to which the present invention pertains. However, the technical idea of ​​the present invention can be embodied in various forms and is not limited to the embodiments described herein. In the description of embodiments disclosed herein, if it is determined that specifically describing related prior art would obscure the gist of the technical idea of ​​the present invention, then specific descriptions relating to such prior art will be omitted. Identical or similar components will be given the same reference numeral, and redundant descriptions relating thereto will be omitted.

[0032] Here, the term "~part" as used in this embodiment refers to a component that performs a specific function, whether by software or hardware such as an FPGA (field programmable gate array) or an application-specific integrated circuit (ASIC). However, "~part" is not limited to those performed by software or hardware. "~part" may also exist as data stored on an addressable recording medium, be embodied by instructions, and be configured so that one or more processors perform a specific function.

[0033] Software may include computer programs, code, instructions, or a combination of one or more of these, which can configure a processing unit to operate as desired, or independently or collectively, instruct the processing unit. Software and / or data may be permanently or temporarily embodied in a type of machine, component, physical device, virtual device, computer recording medium or device, or transmitted signal wave, in order to be interpreted by a processing unit or to provide instructions or data to a processing unit. The software may also be distributed on a network of computer systems, stored in a distributed manner, or executed. The software and data may be stored on a recording medium readable by one or more computers. The software may be read into main memory from other computer-readable media, such as data storage devices, or from other devices via a communication interface. Software instructions stored in main memory may cause the processor to perform processes or steps described in detail below. Alternatively, fixed wiring circuits may be used in place of, or in combination with, software instructions to perform processes consistent with the principles of the present invention. Therefore, embodiments consistent with the principles of the present invention are not limited to any particular combination of hardware circuits and software.

[0034] The terms used in this application are used solely to describe specific embodiments and are not intended to limit the invention. A singular expression includes plural expressions unless the context explicitly limits it to that singular expression. In this application, terms such as “includes” or “having” should be understood to indicate the existence of features, numbers, stages, operations, components, parts, or combinations thereof described in the specification, and not to preemptively exclude the possibility of the existence or addition of one or more other features, numbers, stages, operations, components, parts, or combinations thereof. Terms such as the first and second may be used to describe a variety of components, but such components are not limited by the terms. The terms are used solely for the purpose of distinguishing one component from others.

[0035] The term "learning model" as used in this invention also includes all forms of algorithms or methodologies used to learn or understand specific patterns or structures from data. That is, the learning model includes not only machine learning models such as regression models, decision trees, random forests, support vector machines, K-nearest neighbors, naive phases, and clustering algorithms, but also deep learning models such as neural networks, convolutional neural networks, circulatory neural networks, Transformer-based neural networks, GANs (Generative Adversarial Networks), and autoencoders. The "learning model" refers to a set of learned parameters or weights used to predict or classify an output for a given input, and the model may be learned through methods such as directed learning, undirected learning, semi-directed learning, and reinforcement learning. Furthermore, it includes not only single models, but also diverse learning methods and structures such as ensemble models, multimodal models, and models via transfer learning. Such learning models may be pre-trained on a separate computer device from the computer device used to predict the output for a given input, and then used on the other computer device.

[0036] A learning model according to one embodiment of the present invention also includes at least one model related to object classification, object detection, and position estimation.

[0037] Figure 1 schematically illustrates an endoscope device according to one embodiment of the present invention.

[0038] Referring to Figure 1, the endoscopic device 100 according to one embodiment of the present invention is also a flexible endoscope, specifically a gastrointestinal endoscope. The endoscopic device 100 also includes a configuration that can acquire medical images of the inside of the digestive tract, and a configuration that, if necessary, allows for the insertion of tools and the performance of treatment or procedures while viewing the medical images.

[0039] The endoscope device 100 also includes an output unit 110, a control unit 120, a drive unit 130, a scope 140, and an operating unit 160.

[0040] The output unit 110 also includes a display for displaying medical images. The output unit 110 includes a display module that can output visualized information or implement a touch screen, such as a liquid crystal display (LCD), thin-film transistor-liquid crystal display (TFT LCD), organic light-emitting diode (OLED), flexible display, or 3D display, and supports image representation functions, image enlargement functions, image reduction functions, etc. Furthermore, the user can manipulate the image via the touch screen function and obtain the necessary information. Such an output unit 110 can display medical images acquired by the scope 140 or medical images processed by the control unit 120.

[0041] The drive unit 130 can provide the necessary power as the scope 140 is inserted into or moves inside the body. For example, the drive unit 130 may also include a plurality of motors connected to wires inside the scope 140, and a tension adjustment unit for adjusting the tension of the wires. The drive unit 130 can control the power of each of the plurality of motors and control the scope 140 in various directions. Specifically, the drive unit 130 can control the power of each of the plurality of motors, adjust the tension of the wires, bend the curved section 142, and adjust the rotation angle of the tip section 143.

[0042] The scope 140 also includes an insertion section 141, a bending section 142, and a tip section 143. The insertion section 141 is the part inserted into the body and can be manipulated by the bending section 142 to move to internal organs.

[0043] The curved section 142 is connected to the insertion section 141 and can adjust the direction in which the scope 140 enters the body. The rotation angle of the curved section 142 can be adjusted by user commands or control signals from the control unit.

[0044] The tip section 143 is located at the tip of the scope 140 and can perform a variety of operations in response to user commands or control signals from the control unit. The tip section 143 also includes an image sensor 151, a nozzle 152, illumination 153, a lens 154, and a working channel 155.

[0045] The image sensor 151 can capture images of the endoscopic device. For example, the image sensor 151 is also a CMOS (complementary metal-oxide-semiconductor) sensor and a CCD (charge-coupled device) sensor.

[0046] The nozzle 152 can spray solutions, drugs, etc., to clean the lens 154. The nozzle 152 can also be used to inject drugs necessary for tissue examination or treatment into the body.

[0047] The illumination 153 can emit light at a constant illuminance so that the image sensor 151 can capture an image. Information related to the brightness of the illumination 153 is also stored in the control unit 120 in advance.

[0048] The lens 154 can focus light so that the image sensor 151 can capture a suitable image. Such a lens 154 may also include a wide-angle function or a zoom function.

[0049] Working channel 155 may refer to a channel for transmitting information about other devices or sampling tools into the human body.

[0050] The control unit 160 may refer to the user interface for actually operating the endoscope. The control unit 160 may also include the bending and steering unit 161, and various buttons, dials, and levers for controlling the various functions of the endoscope. The user can input user commands based on the configuration provided in such a control unit 160.

[0051] The curved steering unit 161 can be used to adjust the curved section 142. The curved steering unit 161 can be embodied in the form of a rotary knob, joystick, and lever, which the user can rotate and move to steer the direction of the curved section 142. The curved steering unit 161 can receive torque feedback from the drive unit 130. For example, this torque feedback is also a physical signal based on a control signal from the drive unit 130, which may mimic the force generated when the tip 143 contacts internal human tissue or is based on a pre-learned model.

[0052] The control unit 120 controls the overall operation of the endoscope device 100 and can perform the operation of the endoscope device according to one embodiment. The control unit 120 can control the movement of the scope 140 via the drive unit 130 which is connected to the scope 140. The control unit 120 can perform various control operations for imaging the inside of the digestive tract through the scope 140. The control unit 120 can perform various processing of medical images acquired through the scope 140.

[0053] In one embodiment, the control unit 120 acquires an image relating to the lower gastrointestinal tract from an image sensor, senses at least one body part from the image based on a pre-learned model, calculates relative position information between the body part and the tip, generates a control signal related to image acquisition, and transmits the control signal to the drive unit. Such relative position information may refer to steering information for moving the tip from a first position, which is the current position of the tip, to a second position. For example, the control unit 120 can use the relative position information to calculate a target rotation angle that allows the tip to be directed toward a body part that is as far away as the x,y coordinates on the image. Furthermore, the control unit 120 generates a control signal based on the target rotation angle to control the drive unit, and through this control, adjusts the pitch and yaw of the tip to direct it toward a body part that is as far away as the x,y coordinates on the image.

[0054] In one embodiment, the control unit 120 acquires an image relating to the lower gastrointestinal tract from the image sensor, identifies at least one body part from the image based on a pre-learned model, calculates relative pose information between the body part and the tip, generates a control signal related to image capture based on the identified body part and relative pose information, and transmits the control signal to the drive unit. Such relative pose information may refer to a first pose, which is the current pose of the tip, and information for moving to a second pose for effectively capturing the identified body part. A pose also includes the position and orientation of an object in space. For example, the position may be expressed as x, y, z coordinates in a coordinate system, and the orientation may be expressed as pitch (roll around x-axis), yaw (roll around y-axis), and roll (roll around z-axis).

[0055] The control unit 120 also includes a CPU (central processing unit), RAM (random access memory), ROM (read-only memory), system bus, etc. The control unit 120 can be implemented by a single CPU or multiple CPUs (or a DSP (digital signal processor), SoC (system-on-chip)). In one embodiment, the control unit 120 can be implemented by a digital signal processor (DSP), a microprocessor, or a TCON (time controller) that processes digital signals. However, it is not limited to these, and may include or be defined by one or more of the following: a central processing unit (CPU), an MCU (microcontroller unit), an MPU (microprocessing unit), a controller, an application processor (AP), a communication processor (CP), or an ARM processor. Furthermore, the control unit 120 can also be implemented as a SoC (system-on-chip) or LSI (large-scale integration) with a built-in processing algorithm, and as an FPGA (field-programmable gate array). Moreover, the control unit 120 may also include a neural processing unit (NPU), a graphics processing unit (GPU), and a tensor processing unit (TPU).

[0056] Figure 2 is a diagram illustrating the process by which a curved section is controlled by a drive unit and wires according to one embodiment of the present invention. For ease of explanation, the diagram illustrates the operation of controlling the direction of the curved section 142 using two motors 200, and it will be clear to those skilled in the art that this can be extended to use multiple motors and adjust the tension of each wire.

[0057] Referring to Figure 2, the endoscope device 100 also includes a motor 200, a first wire 210, and a second wire 220, which are included in the drive unit 130. The endoscope device 100 can control the motor 200 to increase the tension of the first wire 210 and decrease the tension of the second wire 220 in order to bend the bending section 142 to one side. The endoscope device 100 can also control the motor 200 to decrease the tension of the first wire 210 and increase the tension of the second wire 220 in order to bend the bending section 142 to the other side.

[0058] The endoscope device 100 can adjust the rotation angle of the tip portion 143 in such a manner.

[0059] Figure 3 is a flowchart illustrating the operation of generating a learning model related to object sensing according to one embodiment of the present invention. The learning model in Figure 3 may correspond to a pre-trained model. Although such operation is disclosed to be learned by a separate computer device for the sake of explanation, it will be obvious to those skilled in the art that it can be operated by the endoscope device 100 or a separate computer device.

[0060] Referring to Figure 3, the computer device may, in step S310, examine the endoscopic image and tag or label specific body parts. Based on user input, the computer device may label parts of the lower gastrointestinal tract in the image with bounding boxes. For example, such lower gastrointestinal tract may include at least one of the following: duodenum, jejunum, ileum, cecum, appendix, colon, rectum, and anus. The endoscopic image may also include images of the lumens of the ileum, cecum, appendix, colon, rectum, and anus.

[0061] In one embodiment, the computer device may generate a dataset in step S320 that includes images with specified labels. Such a dataset may also include diverse lighting conditions, viewing angles, and the state of body parts.

[0062] In one embodiment, the computer device may, in step S330, use the generated dataset to train a neural network model. The computer device may select a neural network model and train the model based on the dataset. For example, such a model may be an object sensing model and may include a convolutional neural network.

[0063] Computer devices according to other embodiments can generate learning models related to object classification. Such learning models related to object classification are also models for identifying objects for image capture.

[0064] In other embodiments, the computer device may, in step S310, examine the endoscopic image and tag or label specific body parts. Based on user input or the like, the computer device may label parts of the lower gastrointestinal tract in the image. Such parts of the lower gastrointestinal tract may refer to body parts used for image acquisition.

[0065] In one embodiment, the computer device may generate a dataset in step S320 that includes images with specified labels. Such a dataset may also include diverse lighting conditions, viewing angles, and the state of body parts.

[0066] In one embodiment, the computer device may, in step S330, train a neural network model using the generated dataset. The computer device may select a neural network model and train the model based on the dataset. For example, such a model may be an object classification model and may include a convolutional neural network.

[0067] Figure 4A is a flowchart illustrating the operation of an endoscope device according to one embodiment of the present invention.

[0068] Referring to Figure 4A, the endoscopic device can acquire an image of the lower gastrointestinal tract from the image sensor at stage S410a. For example, the tip of the endoscopic device enters the lower gastrointestinal tract, and an image can be acquired from the image sensor by converting the optical signal into an electrical signal.

[0069] An endoscope according to one embodiment can acquire environmental information relating to the tip at step S420a. The environmental information according to one embodiment includes at least one of the following: i) spatial position of the tip (e.g., x, y, z coordinates), ii) directionality (e.g., rotation angle, tilt angle), and iii) structural information relating to the body shape. For example, this environmental information may be determined by sensing a magnetic field, inertia, or mechanical deformation based on a sensor built into the tip, or by a separate calculation method. The operation for acquiring such environmental information may correspond to Figure 5.

[0070] An endoscope according to one embodiment can sense at least one body part from the image based on a pre-learned model at step S430a. The endoscope can sense body parts in the lower gastrointestinal tract based on a pre-learned model labeled with bounding boxes and can confirm the image position of at least one body part within the image.

[0071] An endoscope according to one embodiment can calculate relative position information, including the distance between a body part and the tip of the endoscope, at step S440a. The endoscope can calculate relative position information based on image changes due to changes in the angle of the tip. The endoscope can calculate such relative position information based on the time difference (disparity) and the change in the angle of the tip.

[0072] An endoscope according to one embodiment can generate a control signal for steering based on relative position information and environmental information in step S450a, with respect to the position relative to a body part. Based on the relative position information, the endoscope can associate its tip with a body part, and based on the environmental information, it can generate a steering signal based on structural information relating to the body part and body shape.

[0073] As a specific example, an endoscope according to one embodiment can acquire a first image at a first location and a second image at a second location rotated by a certain amount Δθ. If the time difference between the first image and the second image is ΔL, then from the center coordinates of the bounding box of the sensed body part, L target θ for maneuvering a distant tip towards a sensed body part target This can be shown as in Equation 1. [Mathematics 1] θ target =(Δθ / ΔL)L target

[0074] Since Δθ can be calculated from the encoder information of the drive unit and ΔL can be calculated from the time difference between images, a relationship between the target distance on the image and the target angle of tip rotation by the drive unit can be derived.

[0075] Such equation 1 can be expressed as a coordinate system. That is, since the rotation of the endoscope device with respect to the direction of travel in which the tip moves into the body is called roll, the endoscope device can be manipulated to face body parts that are as far away as the x,y coordinates in the image, through control of pitch and yaw by the drive unit.

[0076] In one embodiment, the endoscope device can transmit a control signal to the drive unit in step S460a. In one embodiment, the endoscope device can adjust the tension of at least one wire and control the rotation angle of the tip so that it can be steered in accordance with the sensed body part in step S460a.

[0077] An endoscope device according to one embodiment can display environmental information using a display unit at stage S470a. The endoscope device can display environmental information on the display unit that includes at least one of the following: i) spatial position of the tip (e.g., x, y, z coordinates), ii) direction (e.g., rotation angle, tilt angle), and iii) structural information related to body shape, and can display the current environmental information of the tip.

[0078] Figure 4B is a flowchart illustrating the operation of an endoscope device according to another embodiment of the present invention.

[0079] Referring to Figure 4B, the endoscopic device can acquire an image of the lower gastrointestinal tract from the image sensor at stage S410b. For example, the tip of the endoscopic device enters the lower gastrointestinal tract, and an image is acquired from the image sensor by converting the optical signal into an electrical signal.

[0080] An endoscope according to one embodiment can identify at least one body part from the image in step S420b based on environmental information and a pre-trained model. The endoscope can classify body parts in the lower gastrointestinal tract based on a labeled pre-trained model and identify at least one body part within the image.

[0081] In one embodiment, the endoscope can calculate relative pose information, including the distance between a body part and the tip of the endoscope, at step S430b. For example, the endoscope can calculate relative pose information based on the brightness of the illumination, or based on the image change due to a change in the angle of the tip. The operation by which the endoscope calculates such relative pose information can be seen in Figures 6 and 7.

[0082] In one embodiment, the endoscopic device can generate control signals for image acquisition based on identified body parts and relative pose information at step S440b. Such control signals also include signals for adjusting the tip of the endoscope.

[0083] In one embodiment, the endoscope device can transmit a control signal to the drive unit in step S450b. In one embodiment, the endoscope device can adjust the tension of at least one wire to correspond to at least one pre-set imaging point related to an identified body part, thereby controlling the rotation angle of the tip.

[0084] In one embodiment, the endoscope can capture an image in step S460b based on the identified body part and relative pose information. The endoscope can capture an image when its tip is positioned at at least one pre-set imaging point related to the identified body part.

[0085] Figure 5 is a flowchart illustrating the operation of an endoscope device to acquire environmental information according to one embodiment of the present invention. The operation of the endoscope device in Figure 5 can correspond to step S420a in Figure 4A.

[0086] The environmental information according to one embodiment also includes at least one of the following: i) spatial position of the tip (e.g., x, y, z coordinates), ii) directionality (e.g., rotation angle, tilt angle), and iii) structural information related to body shape. For example, an endoscope can acquire environmental information via SLAM (simultaneous localization and mapping). The endoscope can extract feature points from an image and acquire the environmental information based on the data association between the feature points and previously acquired images.

[0087] Referring to Figure 5, the endoscope device can calculate rotational information in step S510 based on the encoder value of the drive unit. For example, this rotational information may be determined based on the encoder value of the first motor that determines the x-axis rotational motion and the encoder value of the second motor that determines the y-axis rotational motion of the scope for each frame.

[0088] An endoscope according to one embodiment can generate a stitched image based on the feature points of the acquired image in step S520. For example, the endoscope can continuously capture images and stitch them together based on the feature points of the images as its tip penetrates into the body.

[0089] An endoscope according to one embodiment can generate body shape and structure based on stitched images in step S530. The endoscope can grasp the morphology of specific parts and structures of the body and convert them into structural information. Such an endoscope can also capture various images of the target environment and the object from various angles and generate structural information through 3D reconstruction. For example, it can generate a point cloud in 3D space through feature detection and feature matching, grasp the morphology of specific parts and structures of the body through processes such as mesh generation and texture mapping, and convert them into structural information.

[0090] In one embodiment, the endoscope can calculate the position information of the tip based on stitched images or body shape and structure information acquired from continuously captured images and rotation information in step S540. The endoscope can calculate the position information of the tip by utilizing continuous images captured as it passes through the inside of the body and a visual map of the internal body structure obtained by analyzing those images. At this time, the inside of the human body is a dynamic environment in which tissue movement and deformation can occur, but it is also possible to save environmental information for each patient, track the movement of internal structures due to respiration and heartbeat, and update the SLAM information.

[0091] The endoscope device can acquire environmental information through a series of processes, including at least one of the following: i) spatial position of the tip (e.g., x, y, z coordinates), ii) directionality (e.g., rotation angle, tilt angle), and iii) structural information related to body shape.

[0092] Figure 6 is a flowchart illustrating the operation of calculating relative pose information of an endoscope device according to one embodiment of the present invention. The operation of the endoscope device in Figure 6 may correspond to step S430b in Figure 4B. Such relative pose information may refer to positional information for moving from a first pose, which is the current pose of the tip, to a second pose for effectively photographing the identified body part.

[0093] Referring to Figure 6, the endoscope can identify brightness differences in an image at step S610 based on the acquired image and brightness information related to illumination. The endoscope can analyze the difference between the stored brightness information related to illumination and the brightness information from the image captured by the image sensor and the illumination. The endoscope can also identify brightness patterns indicated by the characteristics of internal tissue, illumination intensity, and direction.

[0094] An endoscope according to one embodiment can calculate relative pose information in step S620 based on the identified brightness difference. Based on the identified brightness difference, the endoscope can estimate the distance of the tip to a specific body part in the current pose, and calculate position information for the movement of the tip and direction information for the image sensor equipped on the tip to take an image, based on the estimated distance, the center coordinates in the image, and the coordinates of the identified body part. The relative pose information also includes the path, direction, and necessary angular changes of the tip as it passes through the inside of the body.

[0095] An endoscope according to one embodiment can project a specific pattern of light based not only on brightness differences but also on structured light, and calculate relative pose information based on changes in the pattern. For example, the endoscope can estimate distance based on pattern changes, and calculate positional information for the tip to move and directional information for the image sensor equipped at the tip to take images, based on the estimated distance, the center coordinates in the image, and the coordinates of the identified body part.

[0096] An endoscope according to one embodiment can use multiple image sensors to analyze the pixel difference between two images and calculate the distance from the tip to the lesion.

[0097] Figure 7 is a flowchart illustrating the operation of calculating relative pose information of an endoscope device according to another embodiment of the present invention. The operation of the endoscope device in Figure 7 may correspond to step S430b in Figure 4B.

[0098] Referring to Figure 7, the endoscopic device can acquire the first image at the first location in step S710.

[0099] In other embodiments of the endoscopic device, a second image at a second location can be acquired in step S720.

[0100] In other embodiments of the endoscope device, relative pose information can be calculated in step S730 based on the change in the angle of the tip and the change in pixels on the screen while moving from the first point to the second point.

[0101] For example, an endoscope may use convolutional operations to analyze an input image. The endoscope may extract features from the input image through various convolutional layers and calculate the position of identified body parts based on a pre-trained model, using bounding box coordinates. These coordinates consist of left, top, right, and bottom, and the output bounding box coordinates may be converted to the center coordinates of the bounding box through a post-processing process.

[0102] The coordinates of the center of such a bounding box can be expressed as shown in Equation 2.

number

[0103] JPEG0007860293000002.jpg6163 points to the central x-axis coordinate obtained by taking the average of the boundaries between the left (l) and right (r) sides. JPEG0007860293000003.jpg8163 refers to the central coordinate of the y-axis, which is the average of the boundary between the upper (t) and lower (b) sides.

[0104] The endoscope device can calculate the change in the angle of its tip (Δθ) while moving from a first point to a second point, based on encoder information, and use the change in the image pixels (ΔL) to calculate the distance between the identified body part and the tip.

[0105] This can be shown as in equation 3.

number

[0106] In JPEG0007860293000005.jpg9163, the distance between the lesion and the tip is given by R, the radius of rotation of the tip is given by Δθ, the angle change of the endoscope tip is given by f, the focal length is given by ΔB, the displacement of the image sensor is given by ΔL, and the pixel change on the screen is given by ΔL, which represents the time difference (disparity) of the same object in the two images. Based on the calculated rotation angle, the endoscope device can calculate the target angle that the tip of the endoscope must move to, using a multinomial trajectory. The endoscope device can analyze the difference between the calculated target movement angle and the current angle of the tip and generate a control signal.

[0107] The operation of the endoscopic device described above can be realized not only through a single image sensor, but also through stereo vision based on multiple image sensors. ΔB is also the distance between the image sensors.

[0108] The endoscope device can estimate distance and, based on the estimated distance, the center coordinates in the image, and the coordinates of the identified body part, calculate positional information for the tip to move and directional information for the image sensor equipped on the tip to take an image.

[0109] Figure 8 is a flowchart illustrating the operation of an endoscope device for image acquisition according to one embodiment of the present invention. The operation of the endoscope device in Figure 8 may correspond to steps S440b to S460b in Figure 4B.

[0110] Referring to Figure 8, the endoscopic device can identify at least one pre-set imaging location for the identified body part in step S810. Such at least one pre-set imaging location is a predetermined position for imaging or treating the body part, and may refer to a position set in advance by the user based on the user's clinical skills, or a position from which structural information of the identified body part can be obtained.

[0111] An endoscope according to one embodiment may generate a control signal for controlling the rotation of the tip based on at least one shooting location and relative pose information in step S820. Specifically, the endoscope may calculate relative pose information between a body part and the endoscope, and generate a control signal to position the tip at at least one shooting location based on the calculated relative pose information and the distance and direction to at least one pre-set shooting location.

[0112] In one embodiment, the endoscope device transmits a control signal to a drive unit in step S830, and the drive unit, based on the control signal, can adjust the tension of at least one wire to bend the curved section and control the rotation angle of the tip.

[0113] In one embodiment, when the tip of the endoscope is positioned at at least one shooting location in step S840, an image can be captured at the position of at least one shooting location based on the image sensor.

[0114] Figure 9 is a flowchart illustrating the operation related to torque feedback in an endoscope according to one embodiment of the present invention. Such torque feedback is also a physical signal based on a control signal from a drive unit that simulates the force generated when the tip comes into contact with internal human tissue or is based on a pre-learned model.

[0115] Referring to Figure 9, the endoscope device may generate torque feedback related to the movement of its tip in step S910. Such torque feedback may be generated based on control signals related to image acquisition. This torque feedback is generated based on information regarding the direction and amount of rotation required for the tip of the endoscope device, and there may be a positive correlation between the distance traveled and the magnitude of the torque feedback.

[0116] The endoscope device can transmit the torque feedback to the operating unit in step S920. For example, the endoscope device can control the tip by a rotation angle that must be controlled for image acquisition, and the bending steering unit can transmit torque feedback in the direction in which the tip is being steered, thereby providing this feedback to the user.

[0117] Figure 10 is a block diagram illustrating the block configuration of a computer device according to one embodiment of the present invention.

[0118] The computer device 1000 also includes a memory 1010 and a processor 1020. The computer device 1000 may be a separate device from the endoscope device or may be included in the control unit. It can execute one or more sets of instructions that perform any one or more of the methodologies described herein.

[0119] Memory 1010 may store a set of instruction words, including instruction words related to the system and instruction words related to the user interface, which perform any one or more of the methodological functions described herein. Memory 1010 temporarily or permanently stores data such as basic programs, applications, and configuration information for device operation. Memory 1010 may also include, but is not limited to, permanent mass storage devices such as RAM, ROM, and disk drives. Such software components may be loaded from memory 1010 and a separate computer-readable recording medium using a drive mechanism. Such separate computer-readable recording media may include computer-readable recording media such as floppy drives, disks, tapes, DVD (digital versatile disc) / CD-ROM (compact disc read only memory) drives, and memory cards. In one embodiment, software components may also be loaded into memory 1010 via a communication unit, rather than from a computer-readable recording medium. Furthermore, the memory 1010 may provide stored data at the request of the processor 1020. The memory 1010 according to one embodiment of the present invention may store configuration information.

[0120] The processor 1020 controls the overall operation of the computer device. The processor 1020 may also be configured to process instructions by performing basic arithmetic, logic, and input / output operations. These instructions may be provided to the processor 1020 by the memory 1010. For example, the processor 1020 may be configured to execute instructions received by program code stored in a recording device such as the memory 1010. For example, the processor 1020 may control the device to perform the operations described in the various embodiments above.

[0121] A processor 1020 according to one embodiment of the present invention can examine endoscopic images and tag or label specific body parts. Based on user input or the like, the computer device can label or classify parts of the lower gastrointestinal tract in the image with bounding boxes. For example, such lower gastrointestinal tract may also include at least one of the duodenum, jejunum, ileum, cecum, appendix, colon, rectum, and anus.

[0122] A processor 1020 according to one embodiment of the present invention may generate a dataset containing labeled images. Such a dataset may also include diverse lighting conditions, viewing angles, and the state of body parts.

[0123] A processor 1020 according to one embodiment of the present invention can use a generated dataset to train a neural network model. A computer device can select a neural network model and train the model based on the dataset. For example, such a model may be at least one of an object sensing model and an object classification model, and may also include a convolutional neural network.

[0124] As described above, even though this embodiment has been described by limited embodiments and drawings, a person with ordinary skill in the art will be able to make various modifications and variations from the above description. For example, the described technique may be performed in a different order than described, and / or components such as the described system, structure, apparatus, and circuit may be combined or combined in a different manner than described, or replaced or substituted by other components or equivalents, and the appropriate results may still be achieved.

[0125] Accordingly, other manifestations, other embodiments, and those equivalent to the claims also fall within the scope of the claims. [Explanation of Symbols]

[0126] 100: Endoscope equipment 110: Output section 120: Control Unit 130: Drive unit 140: Scope 141: Insertion part 142: Curved section 143:Tip 151: Image sensor 152: Nozzle 153: Lighting 154: Lens 155: Working Channel 160:Operation unit 161: Curved steering section 200: Motor 210: First wire 220: Second wire 1000: Computer device 1010: Memory 1020: Processor

Claims

1. In a control method for an endoscope device, The control unit acquires an image related to the lower gastrointestinal tract from the image sensor. The control unit acquires environmental information related to the tip of the endoscope device, The control unit processes image data and detects at least one first body part from the image, wherein the detection is performed by a pre-trained machine learning model. The control unit processes the image data to calculate relative position information between the at least one first body part and the tip of the endoscope device, The control unit processes the image data and generates a first control signal for maneuvering the tip portion to correspond to at least one first body part, based on the environmental information and the relative position information. The control unit transmits the first control signal to the drive unit, The control unit processes the image data to identify at least one second body part from the image, wherein the identification is performed by a pre-trained machine learning model. The control unit processes the image data to calculate relative pose information between the identified at least one second body part and the tip, wherein the relative pose information represents the change in pose of the tip from a first pose to a second pose. The control unit processes the image data and generates a second control signal related to the imaging of at least one pre-set imaging location corresponding to the at least one second body part, based on the relative pose information. The control unit transmits the second control signal to the drive unit, Methods that include...

2. The step of acquiring the aforementioned environmental information is: The control unit calculates rotation information based on the encoder value of the drive unit, The method according to claim 1, further comprising the step of a control unit calculating position information based on the acquired image and rotation information.

3. The step of calculating the aforementioned location information is: The control unit generates a stitched image based on the feature points of the acquired image, The method according to claim 2, comprising the step of a control unit generating a body shape structure based on a stitched image.

4. The step of acquiring an image related to the lower gastrointestinal tract is, The stage of acquiring the first image at the first location, This includes the step of acquiring a second image at a second location, The step of calculating the relative position information is: The method according to claim 1, comprising the steps of: i) calculating a target rotation angle of the tip based on the change in angle of the tip between the first point and the second point, and ii) calculating a target rotation angle of the tip based on the change between the first image and the second image.

5. The method according to claim 1, wherein the pre-trained machine learning model includes a classification model and a sensing model trained on a dataset of images labeled for a first body part and a second body part related to the lower gastrointestinal tract.

6. The step of generating the second control signal is: The control unit identifies at least one imaging location corresponding to the identified second body part, The method according to claim 1, comprising the step of a control unit generating a second control signal based on the at least one shooting location and the relative pose information.

7. The method according to claim 6, further comprising the step of the control unit capturing an image when the position of the tip corresponds to the shooting location.

8. The method according to claim 1, further comprising the step of the control unit displaying the environmental information based on the display unit.

9. The method according to claim 1, further comprising the step of the control unit generating torque feedback based on the drive unit and transmitting the torque feedback to the operation unit.

10. In endoscopes, The tip is equipped with an image sensor capable of acquiring images related to the lower gastrointestinal tract, The rotation angle control drive unit of the aforementioned tip portion, Includes a control unit, The control unit, To acquire an image related to the lower gastrointestinal tract, Environmental information related to the aforementioned tip is acquired, Based on a pre-trained model, at least one first body part is detected from the image. The relative position information between the at least one first body part and the tip of the endoscope device is calculated. Based on the environmental information and the relative position information, a first control signal is generated for maneuvering the tip portion to correspond to the first body part. The first control signal is transmitted to the drive unit, Based on a pre-trained model, at least one second body part is identified from the image. The relative pose information between the identified at least one second body part and the tip is calculated, where the relative pose information represents the change in pose of the tip from a first pose to a second pose. Based on the relative pose information, a second control signal is generated for capturing images of at least one pre-set shooting location corresponding to the at least one second body part. An endoscope device configured to transmit the second control signal to the drive unit.

11. The control unit, The endoscope apparatus according to claim 10, wherein rotation information is calculated based on the encoder value of the drive unit, and position information is calculated based on the image obtained and the rotation information.

12. The control unit, The endoscope device according to claim 11, which generates a stitched image based on the feature points of the acquired image, and generates a body shape structure based on the stitched image.

13. The aforementioned tip further includes lighting, The control unit, First image obtained at location 1, second image obtained at location 2, i) the change in angle of the tip between the first point and the second point, and ii) the change between the first image and the second image, which is used to calculate the target rotation angle of the tip, according to claim 10.

14. The endoscope device according to claim 10, wherein the pre-trained machine learning model includes a classification model and a sensing model trained on a dataset of labeled images of a first body part and a second body part related to the lower gastrointestinal tract.

15. The control unit, The endoscope apparatus according to claim 10, which identifies at least one imaging location corresponding to the identified second body part, and generates the second control signal for controlling the rotation of the tip based on the at least one imaging location and the relative pose information.

16. The control unit, The endoscope apparatus according to claim 15, wherein an image is taken when the position of the tip corresponds to the shooting location.

17. The endoscope apparatus according to claim 10, further comprising a display unit, wherein the control unit displays the environmental information based on the display unit.

18. The endoscope apparatus according to claim 10, further comprising an operating unit having a bending steering unit, wherein the control unit generates torque feedback based on the drive unit and transmits the torque feedback to the bending steering unit.

Images