Endoscopic device for acquiring images of the upper gastrointestinal tract, and method for controlling the same.

Abstract

To provide an endoscope device for acquiring upper gastrointestinal tract images and a method of controlling the same.SOLUTION: The invention belongs to the field of medical imaging equipment and automation control technology, and particularly relates to an endoscope device associated with imaging of the upper gastrointestinal tract using an endoscope, and to a method of controlling the endoscope device. The method includes the steps of: acquiring an image of the upper gastrointestinal tract from an image sensor; sensing at least one first body part from the image based on a pre-trained model; calculating relative position information between the sensed first body part and a distal end portion; generating a first control signal related to the position of the first body part based on the relative position information; and transmitting the first control signal to a drive unit.SELECTED DRAWING: Figure 2

Description

Technical Field

[0001] The present invention belongs to the technical fields of medical imaging devices and automation control technologies, and particularly relates to an endoscope device related to imaging of the upper gastrointestinal tract using an endoscope.

Background Art

[0002] An endoscope is a general term for medical instruments that observe organs by inserting a scope into the body without performing surgery or autopsy (pathological autopsy). The endoscope inserts a scope into the human body, irradiates light, and visualizes the light reflected from the surface of the inner wall. Depending on the purpose and body part, the types of endoscopes are classified, and generally speaking, they can be classified into rigid endoscopes formed of metal with an endoscope tube and flexible endoscopes represented by gastrointestinal endoscopes.

[0003] Today, when an endoscopy specialist performs endoscopy and discovers a lesion, additional operations such as inserting an instrument for tissue examination or pressing a button on the scope must be performed. At this time, while releasing the hand holding the scope, the scope shakes and the lesion moves out of the video field.

[0004] Such endoscopy technology mainly depends on the manual operation of medical experts to adjust the tip of the endoscope and obtain an image of the internal body part of the patient. Such a process highly depends on the proficiency and experience of the operator, and thus has the problem that it is difficult to obtain an accurate and clear image of the desired body part. In particular, in the process of operating the endoscope, due to unnecessary movements or difficulties in precise adjustment, an image of a specific site necessary for accurate diagnosis cannot be sufficiently obtained.

[0005] Furthermore, endoscopic techniques have limitations in precisely adjusting the position and angle of the endoscope tip, which presents particular difficulties when acquiring images of areas with many curves, such as the upper gastrointestinal tract. Such limitations reduce the efficiency and accuracy of endoscopic diagnosis, hindering the early detection of diseases and precise localization. [Overview of the project] [Problems that the invention aims to solve]

[0006] This invention was proposed to solve the aforementioned problems, and the problem that this invention aims to solve is to provide a technology for controlling the position and direction of the tip of an endoscope based on an artificial neural network. [Means for solving the problem]

[0007] A control method for an endoscope according to one embodiment of this specification for achieving the aforementioned objectives includes the steps of: acquiring an image relating to the upper gastrointestinal tract from an image sensor; sensing at least one first body part from the image based on a pre-learned model; calculating relative position information between the sensed first body part and the tip; generating a first control signal for steering in correspondence with the first body part based on the relative position information; and transmitting the first control signal to a drive unit.

[0008] The step of acquiring an image relating to the upper gastrointestinal tract includes a step of acquiring a first image at a first location and a step of acquiring a second image at a second location, and the step of calculating the relative position information also includes a step of calculating a target rotation angle of the tip based on i) the change in angle of the tip between the first location and the second location, and ii) the change between the first image and the second image.

[0009] The method further includes the steps of: identifying at least one second body part from the image based on the pre-trained model; calculating relative pose information between the identified second body part and its tip; generating a second control signal related to photographing the second body part based on the relative pose information; and transmitting the second control signal to the drive unit.

[0010] The step of calculating the relative pose information also includes the steps of: identifying brightness differences in the image based on the acquired image and brightness information related to the lighting; estimating the distance between the tip and the second body part based on the identified brightness difference; and calculating the relative pose information based on the estimated distance.

[0011] The step of acquiring an image relating to the upper gastrointestinal tract includes the step of acquiring a first image at a first location and the step of acquiring a second image at a second location, and the step of calculating relative pose information also includes the step of estimating the distance between the tip and the second body part based on i) the change in angle of the tip between the first location and the second location, and ii) the change between the first image and the second image, and the step of calculating the relative pose information.

[0012] The aforementioned pre-trained models also include classification and sensing models trained on a dataset of labeled images of the first and second body parts related to the upper gastrointestinal tract.

[0013] The step of generating the second control signal also includes the steps of: identifying at least one shooting location corresponding to the identified second body part; generating the second control signal based on the at least one shooting location and the relative position information; and capturing an image if the position of the tip corresponds to the shooting location.

[0014] The method also further includes the step of adjusting the tension of at least one wire and controlling the rotation angle of the tip based on the first control signal.

[0015] The drive unit further includes the step of generating torque feedback and transmitting the torque feedback to the operating unit.

[0016] The torque feedback may have a positive correlation with the target rotation angle to which the tip must move in response to the first control signal related to image acquisition.

[0017] An endoscope apparatus according to one embodiment of this specification for achieving the aforementioned objectives also includes a tip equipped with an image sensor, a drive unit for controlling the rotation angle of the tip, and a control unit that, based on a pre-learned model, senses at least one first body part from the image, calculates relative position information between the sensed first body part and the tip, generates a first control signal relating to the position of the first body part based on the relative position information, and transmits the first control signal to the drive unit.

[0018] The control unit can acquire a first image at a first location, acquire a second image at a second location, and calculate a target rotation angle of the tip based on i) the change in angle of the tip between the first and second locations, and ii) the change between the first and second images.

[0019] The control unit may identify at least one second body part from the image based on the pre-learned model, calculate relative pose information between the identified second body part and the tip, generate a second control signal related to photographing the second body part based on the relative pose information, and transmit the second control signal to the drive unit.

[0020] Based on the acquired image and the brightness information related to the illumination, the control unit identifies the brightness difference on the image, estimates the distance between the tip and the second body part based on the identified brightness difference, and can calculate the relative pose information based on the estimated distance.

[0021] The control unit acquires a first image at a first location and a second image at a second location, and can estimate the distance between the tip and the second body part based on i) the angular change of the tip between the first location and the second location, and ii) the change between the first image and the second image, and calculate the relative pose information.

[0022] The pre-trained model also includes a classification model and a perception model learned using images labeled for the first body part and the second body part related to the upper gastrointestinal tract as a dataset.

[0023] The control unit identifies at least one imaging location corresponding to the identified body part, generates a second control signal based on the at least one imaging location and the relative position information, and can capture an image when the position of the tip corresponds to the imaging location.

[0024] The endoscope device further includes a bending part connected to the tip, and the control unit controls the driving part based on the control signal, adjusts the tension of at least one wire connected to the driving part, and can adjust the rotation angle of the tip based on the bending of the bending part.

[0025] The endoscope device further includes an operation part having a bending steering part, and the control unit generates torque feedback based on the driving part and can transmit the torque feedback to the bending steering part.

[0026] The torque feedback may have a positive correlation with the target rotation angle by which the tip must move by the first control signal related to the image capture.

Advantages of the Invention

[0027] According to an embodiment of the present invention, the control unit of the endoscope device can control the position of the distal end and accurately and efficiently acquire an image of a specific body part. Through this, the difficulty of endoscope operation can be reduced, and the accuracy of medical diagnosis can be improved.

[0028] The effects of this embodiment are not limited to the effects mentioned above, and other effects not mentioned can be clearly understood by those of ordinary skill in the art from the description of the claims.

Brief Description of the Drawings

[0029] [Figure 1] It is a diagram schematically showing an endoscope device according to an embodiment of the present invention. [Figure 2] It is a diagram for explaining the process in which a bending part is controlled by a driving part and a wire according to an embodiment of the present invention. [Figure 3] It is a flowchart illustrating an operation of generating a learning model according to an embodiment of the present invention. [Figure 4A] It is a flowchart illustrating an operation of an endoscope device according to an embodiment of the present invention. [Figure 4B] It is a flowchart illustrating an operation of an endoscope device according to another embodiment of the present invention. [Figure 4C] It is a flowchart illustrating an operation of an endoscope device according to still another embodiment of the present invention. [Figure 5] It is a flowchart illustrating an operation of calculating relative pose information of an endoscope device according to an embodiment of the present invention. [Figure 6] It is a flowchart illustrating an operation of calculating relative pose information of an endoscope device according to another embodiment of the present invention. [Figure 7] It is a flowchart illustrating an operation of calculating relative pose information of an endoscope device according to still another embodiment of the present invention. [Figure 8]This is a flowchart illustrating the operations related to image acquisition by an endoscope device according to one embodiment of the present invention. [Figure 9] This is a flowchart illustrating the operation related to torque feedback in an endoscope device according to one embodiment of the present invention. [Figure 10] This is a block diagram illustrating the block configuration of a computer device according to one embodiment of the present invention. [Modes for carrying out the invention]

[0030] The terms used in this invention are used solely to describe specific embodiments and are not intended to limit the scope of other embodiments. Singular expressions also include plural expressions unless explicitly stated in the context. Terms used herein, including technical and scientific terms, may have the same meaning as those generally understood by a person of ordinary skill in the art described herein. General, predefined terms used herein are to be interpreted as having the same or similar meaning as they do in the context of the relevant art, and not as to be idealistic or overly formal unless explicitly defined herein. Where applicable, a term defined herein is not to be construed as excluding embodiments of the invention.

[0031] In the following, various embodiments will be described in detail with reference to the accompanying drawings, so as to be easily implemented by a person with ordinary skill in the art to which the present invention pertains. However, the technical idea of ​​the present invention can be embodied in various forms and is not limited to the embodiments described herein. In the description of embodiments disclosed herein, if it is determined that specifically describing related prior art would obscure the gist of the technical idea of ​​the present invention, then specific descriptions relating to such prior art will be omitted. Identical or similar components will be given the same reference numeral, and redundant descriptions relating thereto will be omitted.

[0032] Here, the term "~part" as used in this embodiment refers to a component that performs a specific function, whether by software or hardware such as an FPGA (field programmable gate array) or an application-specific integrated circuit (ASIC). However, "~part" is not limited to those performed by software or hardware. "~part" may also exist as data stored on an addressable recording medium, be embodied by instructions, and be configured so that one or more processors perform a specific function.

[0033] Software may include computer programs, code, instructions, or a combination of one or more of these, which can configure a processing unit to operate as desired, or independently or collectively, instruct the processing unit. Software and / or data may be permanently or temporarily embodied in a type of machine, component, physical device, virtual device, computer recording medium or device, or transmitted signal wave, in order to be interpreted by a processing unit or to provide instructions or data to a processing unit. The software may also be distributed on a network of computer systems, stored in a distributed manner, or executed. The software and data may be stored on a recording medium readable by one or more computers. The software may be read into main memory from other computer-readable media, such as data storage devices, or from other devices via a communication interface. Software instructions stored in main memory may cause the processor to perform processes or steps described in detail below. Alternatively, fixed wiring circuits may be used in place of, or in combination with, software instructions to perform processes consistent with the principles of the present invention. Therefore, embodiments consistent with the principles of the present invention are not limited to any particular combination of hardware circuits and software.

[0034] The terms used in this application are used solely to describe specific embodiments and are not intended to limit the invention. A singular expression includes plural expressions unless the context explicitly limits it to that singular expression. In this application, terms such as “includes” or “having” should be understood to indicate the existence of features, numbers, stages, operations, components, parts, or combinations thereof described in the specification, and not to preemptively exclude the possibility of the existence or addition of one or more other features, numbers, stages, operations, components, parts, or combinations thereof. Terms such as the first and second may be used to describe a variety of components, but such components are not limited by the terms. The terms are used solely for the purpose of distinguishing one component from others.

[0035] The term "learning model" as used in this invention also includes all forms of algorithms or methodologies used to learn or understand specific patterns or structures from data. That is, the learning model includes not only machine learning models such as regression models, decision trees, random forests, support vector machines, K-nearest neighbors, naive phases, and clustering algorithms, but also deep learning models such as neural networks, convolutional neural networks, circulatory neural networks, Transformer-based neural networks, GANs (Generative Adversarial Networks), and autoencoders. The "learning model" refers to a set of learned parameters or weights used to predict or classify an output for a given input, and the model may be learned through methods such as directed learning, undirected learning, semi-directed learning, and reinforcement learning. Furthermore, it includes not only single models, but also diverse learning methods and structures such as ensemble models, multimodal models, and models via transfer learning. Such learning models may be pre-trained on a separate computer device from the computer device used to predict the output for a given input, and then used on the other computer device.

[0036] A learning model according to one embodiment of the present invention also includes at least one model related to object classification, object detection, and position estimation.

[0037] Figure 1 schematically illustrates an endoscope device according to one embodiment of the present invention.

[0038] Referring to Figure 1, the endoscopic device 100 according to one embodiment of the present invention is also a flexible endoscope, specifically a gastrointestinal endoscope. The endoscopic device 100 also includes a configuration that can acquire medical images of the inside of the digestive tract, and a configuration that, if necessary, allows for the insertion of tools and the performance of treatment or procedures while viewing the medical images.

[0039] The endoscope device 100 also includes an output unit 110, a control unit 120, a drive unit 130, a scope 140, and an operating unit 160.

[0040] The output unit 110 also includes a display for displaying medical images. The output unit 110 includes a display module that can output visualized information or implement a touch screen, such as a liquid crystal display (LCD), thin-film transistor-liquid crystal display (TFT LCD), organic light-emitting diode (OLED), flexible display, or 3D display, and supports image representation functions, image enlargement functions, image reduction functions, etc. Furthermore, the user can manipulate the image via the touch screen function and obtain the necessary information. Such an output unit 110 can display medical images acquired by the scope 140 or medical images processed by the control unit 120.

[0041] The drive unit 130 can provide the necessary power as the scope 140 is inserted into or moves inside the body. For example, the drive unit 130 may also include a plurality of motors connected to wires inside the scope 140, and a tension adjustment unit for adjusting the tension of the wires. The drive unit 130 can control the power of each of the plurality of motors and control the scope 140 in various directions. Specifically, the drive unit 130 can control the power of each of the plurality of motors, adjust the tension of the wires, bend the curved section 142, and adjust the rotation angle of the tip section 143.

[0042] The scope 140 also includes an insertion section 141, a bending section 142, and a tip section 143. The insertion section 141 is the part inserted into the body and can be manipulated by the bending section 142 to move to internal organs.

[0043] The curved section 142 is connected to the insertion section 141 and can adjust the direction in which the scope 140 enters the body. The rotation angle of the curved section 142 can be adjusted by user commands or control signals from the control unit.

[0044] The tip section 143 is located at the tip of the scope 140 and can perform a variety of operations in response to user commands or control signals from the control unit. The tip section 143 also includes an image sensor 151, a nozzle 152, illumination 153, a lens 154, and a working channel 155.

[0045] The image sensor 151 can capture images of the endoscopic device. For example, the image sensor 151 is also a CMOS (complementary metal-oxide-semiconductor) sensor and a CCD (charge-coupled device) sensor.

[0046] The nozzle 152 can spray solutions, drugs, etc., to clean the lens 154. The nozzle 152 can also be used to inject drugs necessary for tissue examination or treatment into the body.

[0047] The illumination 153 can emit a light source at a constant illuminance so that the image sensor 151 can capture an image. Information related to the brightness of the illumination 153 is also stored in advance in the control unit 120.

[0048] The lens 154 can focus light so that the image sensor 151 can capture a suitable image. Such a lens 154 may also include a wide-angle function or a zoom function.

[0049] Working channel 155 may refer to a channel for transmitting information about other devices or sampling tools into the human body.

[0050] The control unit 160 may refer to the user interface for actually operating the endoscope. The control unit 160 may also include the bending and steering unit 161, and various buttons, dials, and levers for controlling the various functions of the endoscope. The user can input user commands based on the configuration provided in such a control unit 160.

[0051] The curved steering unit 161 can be used to adjust the curved section 142. The curved steering unit 161 can be embodied in the form of a rotary knob, joystick, and lever, which the user can rotate and move to steer the direction of the curved section 142. The curved steering unit 161 can receive torque feedback from the drive unit 130. For example, this torque feedback is also a physical signal from the control signal of the drive unit 130, which may mimic the force generated when the tip 143 contacts internal human tissue or be based on a pre-learned model.

[0052] The control unit 120 controls the overall operation of the endoscope device 100 and can perform the operation of the endoscope device according to one embodiment. The control unit 120 can control the movement of the scope 140 via the drive unit 130 which is connected to the scope 140. The control unit 120 can perform various control operations for imaging the inside of the digestive tract through the scope 140. The control unit 120 can perform various processing of medical images acquired through the scope 140.

[0053] In one embodiment, the control unit 120 acquires an image relating to the upper gastrointestinal tract from an image sensor, senses at least one body part from the image based on a pre-learned model, calculates relative position information between the body part and the tip, generates a control signal related to image capture based on the identified body part and relative pose information, and transmits the control signal to the drive unit. Such relative position information may refer to steering information for moving the tip from a first position, which is the current position of the tip, to a second position. For example, the control unit 120 can use the relative position information to calculate a target rotation angle that allows the tip to be directed toward a body part that is as far away as the x,y coordinates on the image. Furthermore, the control unit 120 generates a control signal based on the target rotation angle to control the drive unit, and through this control, the pitch and yaw of the tip can be adjusted to direct the tip toward a body part that is as far away as the x,y coordinates on the image.

[0054] In one embodiment, the control unit 120 acquires an image relating to the upper gastrointestinal tract from the image sensor, identifies at least one body part from the image based on a pre-learned model, calculates relative pose information between the body part and the tip, generates a control signal related to image capture based on the identified body part and relative pose information, and transmits the control signal to the drive unit. Such relative pose information may refer to a first pose, which is the current pose of the tip, and information for moving to a second pose for effectively capturing the identified body part. A pose also includes the position and orientation of an object in space. For example, the position may be expressed as x, y, z coordinates in a coordinate system, and the orientation may be expressed as pitch (roll around x-axis), yaw (roll around y-axis), and roll (roll around z-axis).

[0055] The control unit 120 also includes a CPU (central processing unit), RAM (random access memory), ROM (read-only memory), system bus, etc. The control unit 120 can be implemented by a single CPU or multiple CPUs (or a DSP (digital signal processor), SoC (system-on-chip)). In one embodiment, the control unit 120 can be implemented by a digital signal processor (DSP), a microprocessor, or a TCON (time controller) that processes digital signals. However, it is not limited to these, and may include or be defined by one or more of the following: a central processing unit (CPU), an MCU (microcontroller unit), an MPU (microprocessing unit), a controller, an application processor (AP), a communication processor (CP), or an ARM processor. Furthermore, the control unit 120 can also be implemented as a SoC (system-on-chip) or LSI (large-scale integration) with a built-in processing algorithm, and as an FPGA (field-programmable gate array). Moreover, the control unit 120 may also include a neural processing unit (NPU), a graphics processing unit (GPU), and a tensor processing unit (TPU).

[0056] Figure 2 is a diagram illustrating the process by which a curved section is controlled by a drive unit and wires according to one embodiment of the present invention. For ease of explanation, the diagram illustrates the operation of controlling the direction of the curved section 142 using two motors 200. It will be clear to those skilled in the art that this can be extended to use multiple motors, adjust the tension of each wire, and adjust the rotation angle of the tip.

[0057] Referring to Figure 2, the endoscope device 100 also includes a motor 200, a first wire 210, and a second wire 220, which are included in the drive unit 130. The endoscope device 100 can control the motor 200 to increase the tension of the first wire 210 and decrease the tension of the second wire 220 in order to bend the bending section 142 to one side. The endoscope device 100 can also control the motor 200 to decrease the tension of the first wire 210 and increase the tension of the second wire 220 in order to bend the bending section 142 to the other side.

[0058] The endoscope device 100 can adjust the rotation angle of the tip portion 143 in such a manner.

[0059] Figure 3 is a flowchart illustrating the operation of generating a learning model related to object sensing according to one embodiment of the present invention. The learning model in Figure 3 may correspond to a pre-trained model. Although such operation is disclosed to be learned by a separate computer device for the sake of explanation, it will be obvious to those skilled in the art that it can be operated by the endoscope device 100 or a separate computer device.

[0060] Referring to Figure 3, the computer device may, in step S310, examine the endoscopic image and tag or label specific body parts. Based on user input, the computer device may label parts of the upper gastrointestinal tract in the image with bounding boxes. For example, such an upper gastrointestinal tract may include at least one of the following: oral cavity, pharynx, esophagus, stomach, and duodenum. The endoscopic image may also include images of the lumens of the oral cavity, pharynx, esophagus, stomach, and duodenum.

[0061] In one embodiment, the computer device may generate a dataset in step S320 that includes images with specified labels. Such a dataset may also include diverse lighting conditions, viewing angles, and the state of body parts.

[0062] In one embodiment, the computer device may, in step S330, use the generated dataset to train a neural network model. The computer device may select a neural network model and train the model based on the dataset. For example, such a model may be an object sensing model and may include a convolutional neural network.

[0063] Computer devices according to other embodiments can generate learning models related to object classification. Such learning models related to object classification are also models for identifying objects for image capture.

[0064] In other embodiments, the computer device may, in step S310, examine the endoscopic image and tag or label specific body parts. Based on user input or the like, the computer device may label parts of the upper gastrointestinal tract in the image. Such parts of the upper gastrointestinal tract may refer to body parts used for image acquisition.

[0065] In one embodiment, the computer device may generate a dataset in step S320 that includes images with specified labels. Such a dataset may also include diverse lighting conditions, viewing angles, and the state of body parts.

[0066] In one embodiment, the computer device may, in step S330, train a neural network model using the generated dataset. The computer device may select a neural network model and train the model based on the dataset. For example, such a model may be an object classification model and may also include a convolutional neural network.

[0067] Furthermore, a computer device according to another embodiment may generate a learning model using endoscopic images and control history information corresponding to the user's clinical skills. Such control history information may refer to control history information used to observe the endoscopic image and adjust the direction, angle, depth, etc., of the endoscope. In step S310, a computer device according to another embodiment may acquire endoscopic images and control history information relating to the upper gastrointestinal tract. For example, such an upper gastrointestinal tract may include at least one of the oral cavity, pharynx, esophagus, stomach, and duodenum. The endoscopic image may also include images of the lumen of the oral cavity, pharynx, esophagus, stomach, and duodenum.

[0068] Furthermore, a computer device according to another embodiment may generate a dataset including endoscopic images and control history information in step S320.

[0069] Furthermore, in other embodiments, the computer device may, in step S330, use the generated dataset to train a neural network model. The computer device may select a neural network model and train the model based on the dataset. For example, such a model may include a classification model such as a CNN (convolutional neural network) and perform image processing, and sequence processing of control history information may include at least one of sequence processing models such as an RNN (recurrent neural network) and an LSTM (long short-term memory) network.

[0070] Figure 4A is a flowchart illustrating the operation of an endoscope device according to one embodiment of the present invention.

[0071] Referring to Figure 4A, the endoscopic device can acquire an image of the upper gastrointestinal tract from the image sensor at stage S410a. For example, the tip of the endoscopic device enters the upper gastrointestinal tract, and an image can be acquired from the image sensor by converting the optical signal into an electrical signal.

[0072] An endoscope according to one embodiment can sense at least one body part from the image based on a pre-learned model at step S420a. The endoscope can sense body parts in the upper gastrointestinal tract based on a pre-learned model labeled with bounding boxes and can confirm the image position of at least one body part within the image.

[0073] An endoscope according to one embodiment can calculate relative position information between a body part and the tip of the endoscope in step S430a. The endoscope can calculate relative position information based on image changes due to changes in the angle of the tip. The endoscope can calculate such relative position information based on the time difference (disparity) and the change in the angle of the tip.

[0074] In one embodiment, the endoscope device can generate control signals for steering based on relative position information in step S440a, with respect to the position relative to the body part.

[0075] In one embodiment, the endoscope device can transmit a control signal to the drive unit in step S450a. In one embodiment, the endoscope device can adjust the tension of at least one wire and control the rotation angle of the tip so that it can be steered in accordance with the sensed body part in step S450a.

[0076] As a specific example, an endoscope according to one embodiment can acquire a first image at a first location and a second image at a second location rotated by a certain amount Δθ. If the time difference between the first image and the second image is ΔL, then L can be obtained from the center coordinates of the bounding box of the sensed body part. target θ for maneuvering a distant tip towards a sensed body part target This can be shown as in Equation 1. [Mathematics 1] θ target =(Δθ / ΔL)L target

[0077] Since Δθ can be calculated from the encoder information of the drive unit and ΔL can be calculated from the time difference between images, a relationship between the target distance on the image and the target angle of tip rotation by the drive unit can be derived.

[0078] Such equation 1 can be expressed as a coordinate system. That is, since the rotation of the endoscope device with respect to the direction of travel in which the tip moves into the body is called roll, the endoscope device can be manipulated to face body parts that are as far away as the x,y coordinates in the image, through control of pitch and yaw by the drive unit.

[0079] Figure 4B is a flowchart illustrating the operation of an endoscope device according to another embodiment of the present invention.

[0080] Referring to Figure 4B, the endoscopic device can acquire an image of the upper gastrointestinal tract from the image sensor at stage S410b. For example, the tip of the endoscopic device enters the upper gastrointestinal tract, and an image is acquired from the image sensor by converting the optical signal into an electrical signal.

[0081] In other embodiments, the endoscopic device may, in step S420b, identify at least one body part from the image based on a pre-trained model. The endoscopic device may classify body parts based on a pre-trained model labeled for the upper gastrointestinal tract and identify at least one body part within the image.

[0082] In other embodiments of the endoscopic device, relative pose information, including the distance between a body part and the tip of the endoscopic device, can be calculated at step S430b. For example, the endoscopic device can calculate relative pose information based on the brightness of the illumination, the image change due to the change in the angle of the tip, and the movement trajectory of the tip. The operation by which the endoscopic device calculates such relative pose information can be seen in Figures 5, 6, and 7.

[0083] In other embodiments of the endoscopic device, at step S440b, control signals for image acquisition may be generated based on identified body parts and relative pose information. Such control signals may also include signals for adjusting the tip of the endoscope.

[0084] In other embodiments of the endoscope device, a control signal may be transmitted to the drive unit in step S450b. In one embodiment of the endoscope device, in step S450b, the tension of at least one wire may be adjusted to correspond to at least one pre-set imaging point relating to an identified body part, thereby controlling the rotation angle of the tip.

[0085] In other embodiments, the endoscopic device can capture an image in step S460b based on the identified body part and relative pose information. The endoscopic device can capture an image when its tip is positioned at at least one pre-set imaging point related to the identified body part.

[0086] Figure 4C is a flowchart illustrating the operation of an endoscope device according to yet another embodiment of the present invention.

[0087] Referring to Figure 4C, the endoscopic device can acquire an image of the upper gastrointestinal tract from the image sensor at stage S410c. For example, the tip of the endoscopic device enters the upper gastrointestinal tract, and an image can be acquired from the image sensor by converting the optical signal into an electrical signal.

[0088] Furthermore, in other embodiments of the endoscopic device, at step S420c, control history information corresponding to the endoscopic image and the user's clinical skills may be used to generate control signals for image acquisition based on a pre-learned model. The endoscopic device may input the endoscopic image into a classification model and a learning model, which is a sequence processing model, to generate control signals. Such control signals may also include signals for adjusting the tip of the endoscope.

[0089] Furthermore, in other embodiments of the endoscopic device, a control signal may be transmitted to the drive unit in step S430c. In step S430c, the endoscopic device may adjust the tension of at least one wire to correspond to at least one pre-set imaging point relating to an identified body part, thereby controlling the rotation angle of the tip.

[0090] Furthermore, in other embodiments, the endoscopic device can perform image acquisition in step S440c based on the identified body part and relative pose information. The endoscopic device can acquire an image when its tip is positioned at at least one pre-set imaging point related to the identified body part.

[0091] Figure 5 is a flowchart illustrating the operation of calculating relative pose information for an endoscope device according to one embodiment of the present invention. The operation of the endoscope device in Figure 5 may correspond to step S430b in Figure 4B. Such relative pose information may refer to a first pose, which is the current pose of the tip, and information for moving to a second pose for effectively photographing the identified body part.

[0092] Referring to Figure 5, the endoscope can identify brightness differences in an image at step S510 based on the acquired image and brightness information related to illumination. The endoscope can analyze the difference between the stored brightness information related to illumination and the brightness information from the image captured by the image sensor and the illumination. The endoscope can also identify brightness patterns indicated by the characteristics of internal tissue, illumination intensity, and direction.

[0093] An endoscope according to one embodiment can calculate relative pose information in step S520 based on the identified brightness difference. Based on the identified brightness difference, the endoscope can estimate the distance of the tip to a specific body part in the current pose, and calculate positional information for the movement of the tip and directional information for the image sensor equipped on the tip to take an image, based on the estimated distance, the center coordinates in the image, and the coordinates of the identified body part. The relative pose information also includes the path, direction, and necessary angular changes of the tip as it passes through the inside of the body.

[0094] An endoscope according to one embodiment can project a specific pattern of light based not only on brightness differences but also on structured light, and calculate relative pose information based on changes in the pattern. For example, the endoscope can estimate distance based on pattern changes, and calculate positional information for the tip to move and directional information for the image sensor equipped at the tip to take images, based on the estimated distance, the center coordinates in the image, and the coordinates of the identified body part.

[0095] An endoscope according to one embodiment can use multiple image sensors to analyze the pixel difference between two images and calculate the distance from the tip to the lesion.

[0096] Figure 6 is a flowchart illustrating the operation of calculating relative pose information of an endoscope device according to another embodiment of the present invention. The operation of the endoscope device in Figure 6 may correspond to step S430b in Figure 4B.

[0097] Referring to Figure 6, the endoscopic device can acquire the first image at the first location in step S610.

[0098] In other embodiments of the endoscopic device, a second image at a second location can be acquired in step S620.

[0099] In other embodiments of the endoscope device, relative pose information can be calculated in step S630 based on the change in the angle of the tip and the change in pixels on the screen while moving from the first point to the second point.

[0100] For example, an endoscope may use convolutional operations to analyze an input image. The endoscope may extract features from the input image through various convolutional layers and calculate the position of identified body parts based on a pre-trained model, using bounding box coordinates. These coordinates consist of left, top, right, and bottom, and the output bounding box coordinates may be converted to the center coordinates of the bounding box through a post-processing process.

[0101] The coordinates of the center of such a bounding box can be expressed as shown in Equation 2.

number

[0102] The endoscope device can calculate the change in the angle of its tip (Δθ) while moving from a first point to a second point, based on encoder information, and use the change in the image pixels (ΔL) to calculate the distance between the identified body part and the tip.

[0103] This can be shown as in equation 3.

number

[0104] The operation of the endoscopic device described above can be realized not only through a single image sensor, but also through stereo vision based on multiple image sensors. ΔB is also the distance between the image sensors.

[0105] The endoscope device can estimate distance and, based on the estimated distance, the center coordinates in the image, and the coordinates of the identified body part, calculate positional information for the tip to move and directional information for the image sensor equipped on the tip to take an image.

[0106] Figure 7 is a flowchart illustrating the operation of calculating relative pose information of an endoscope device according to yet another embodiment of the present invention. The operation of the endoscope device in Figure 7 may correspond to step S430b in Figure 4B.

[0107] Referring to Figure 7, the endoscope can acquire tip movement trajectory data in step S710. Such movement trajectory data is data of the tip's movement inside the body and includes at least one of the following: sensor-based tracking data such as magnetic field sensors, gyroscopes, and accelerometers, and image-based tracking data based on image sensors.

[0108] Furthermore, in other embodiments of the endoscopic device, relative pose information between the body part and the tip can be calculated in step S720 based on the movement trajectory of the tip and the acquired image. For example, the endoscopic device can calculate relative pose information for the tip to move from a first pose, which is its current position, to a second pose, which is its target position, based on position information based on the encoder value of the motor and a depth estimation algorithm.

[0109] Figure 8 is a flowchart illustrating the operation of an endoscope device for image acquisition according to one embodiment of the present invention. The operation of the endoscope device in Figure 8 may correspond to steps S440b to S460b in Figure 4B and steps S420c to S440c in Figure 4c.

[0110] Referring to Figure 8, the endoscopic device can identify at least one pre-set imaging location for the identified body part in step S810. Such at least one pre-set imaging location is a predetermined position for imaging or treating the body part, and may refer to a position set in advance by the user based on the user's clinical skills, or a position from which structural information of the identified body part can be obtained.

[0111] An endoscope according to one embodiment may generate a control signal for controlling the rotation of the tip based on at least one shooting location and relative pose information in step S820. Specifically, the endoscope may calculate relative pose information between a body part and the endoscope, and generate a control signal to position the tip at at least one shooting location based on the calculated relative pose information and the distance and direction to at least one pre-set shooting location.

[0112] In one embodiment, the endoscope device transmits a control signal to a drive unit in step S830, and the drive unit, based on the control signal, can adjust the tension of at least one wire to bend the curved section and control the rotation angle of the tip.

[0113] In one embodiment, when the tip of the endoscope is positioned at at least one shooting location in step S840, an image can be captured at the position of at least one shooting location based on the image sensor.

[0114] Figure 9 is a flowchart illustrating the operation related to torque feedback in an endoscope according to one embodiment of the present invention. Such torque feedback is also a physical signal based on a control signal from a drive unit that simulates the force generated when the tip comes into contact with internal human tissue or is based on a pre-learned model.

[0115] Referring to Figure 9, the endoscope device may generate torque feedback related to the movement of its tip in step S910. Such torque feedback may be generated based on control signals related to image acquisition. This torque feedback is generated based on information regarding the direction and amount of rotation required for the tip of the endoscope device, and there may be a positive correlation between the distance traveled and the magnitude of the torque feedback.

[0116] The endoscope device can transmit the torque feedback to the operating unit in step S920. For example, the endoscope device can control the tip by a rotation angle that must be controlled for image acquisition, and the bending steering unit can transmit torque feedback in the direction in which the tip is being steered, thereby providing this feedback to the user.

[0117] Figure 10 is a block diagram illustrating the block configuration of a computer device according to one embodiment of the present invention.

[0118] The computer device 1000 also includes a memory 1010 and a processor 1020. The computer device 1000 may be a separate device from the endoscope device or may be included in the control unit. It can execute one or more sets of instructions that perform any one or more of the methodologies described herein.

[0119] Memory 1010 may store a set of instruction words, including instruction words related to the system and instruction words related to the user interface, which perform any one or more of the methodological functions described herein. Memory 1010 temporarily or permanently stores data such as basic programs, applications, and configuration information for device operation. Memory 1010 may also include, but is not limited to, permanent mass storage devices such as RAM, ROM, and disk drives. Such software components may be loaded from memory 1010 and a separate computer-readable recording medium using a drive mechanism. Such separate computer-readable recording media may include computer-readable recording media such as floppy drives, disks, tapes, DVD (digital versatile disc) / CD-ROM (compact disc read only memory) drives, and memory cards. In one embodiment, software components may also be loaded into memory 1010 via a communication unit, rather than from a computer-readable recording medium. Furthermore, the memory 1010 may provide stored data at the request of the processor 1020. The memory 1010 according to one embodiment of the present invention may store configuration information.

[0120] The processor 1020 controls the overall operation of the computer device. The processor 1020 may also be configured to process instructions by performing basic arithmetic, logic, and input / output operations. These instructions may be provided to the processor 1020 by the memory 1010. For example, the processor 1020 may be configured to execute instructions received by program code stored in a recording device such as the memory 1010. For example, the processor 1020 may control the device to perform the operations described in the various embodiments above.

[0121] A processor 1020 according to one embodiment of the present invention can examine endoscopic images and tag or label specific body parts. Based on user input or the like, the computer device can label or classify parts of the upper gastrointestinal tract in the image with bounding boxes. For example, such upper gastrointestinal tract may also include at least one of the oral cavity, pharynx, esophagus, stomach, and duodenum.

[0122] A processor 1020 according to one embodiment of the present invention may generate a dataset containing labeled images. Such a dataset may also include diverse lighting conditions, viewing angles, and the state of body parts.

[0123] A processor 1020 according to one embodiment of the present invention can use a generated dataset to train a neural network model. A computer device can select a neural network model and train the model based on the dataset. For example, such a model may be at least one of an object sensing model and an object classification model, and may also include a convolutional neural network.

[0124] A processor 1020 according to another embodiment of the present invention may generate a learning model using endoscopic images and control history information corresponding to the user's clinical skills. Such control history information may refer to control history information used to observe endoscopic images and adjust the direction, angle, depth, etc., of the endoscope. The processor 1020 may acquire endoscopic images relating to the upper gastrointestinal tract and control history information. For example, such an upper gastrointestinal tract may also include at least one of the oral cavity, pharynx, esophagus, stomach, and duodenum.

[0125] In other embodiments, the processor 1020 may generate a dataset including endoscopic images and control history information.

[0126] In other embodiments, the processor 1020 may use the generated dataset to train a neural network model. The computer device may select the neural network model and train the model based on the dataset. For example, such a model may include a classification model such as a CNN (convolutional neural network) and image processing, and may include at least one of sequence processing models such as an RNN (recurrent neural network) and an LSTM (long short-term memory) network to perform sequence processing of control history information.

[0127] As described above, even though this embodiment has been described by limited embodiments and drawings, a person with ordinary skill in the art will be able to make various modifications and variations from the above description. For example, the described technique may be performed in a different order than described, and / or components such as the described system, structure, apparatus, and circuit may be combined or combined in a different manner than described, or replaced or substituted by other components or equivalents, and the appropriate results may still be achieved.

[0128] Accordingly, other manifestations, other embodiments, and those equivalent to the claims also fall within the scope of the claims. [Explanation of Symbols]

[0129] 100: Endoscope equipment 110: Output section 120: Control Unit 130: Drive unit 140: Scope 141: Insertion part 142: Curved section 143:Tip 151: Image sensor 152: Nozzle 153: Lighting 154: Lens 155: Working Channel 160:Operation unit 161: Curved steering section 200: Motor 210: First wire 220: Second wire 1000: Computer device 1010: Memory 1020: Processor

Claims

1. In a control method for an endoscope device, The control unit acquires an image related to the upper gastrointestinal tract from the image sensor. The control unit processes image data and detects at least one first body part from the image, wherein the detection is performed by a pre-trained machine learning model. The control unit processes the image data to calculate relative position information between the at least one detected first body part and the tip of the endoscope device, The control unit processes the image data and generates a first control signal for maneuvering the tip portion to correspond to the at least one first body part based on the relative position information, The control unit transmits the first control signal to the drive unit, The control unit processes the image data to identify at least one second body part from the image, wherein the identification is performed by a pre-trained machine learning model. The control unit processes the image data to calculate relative pose information between the identified at least one second body part and the tip, wherein the relative pose information represents a change in the tip's pose from a first pose to a second pose. The control unit processes the image data and generates a second control signal related to the imaging of at least one pre-set imaging location corresponding to the at least one second body part, based on the relative pose information. The control unit transmits the second control signal to the drive unit, Methods that include...

2. The step of acquiring an image related to the upper gastrointestinal tract is, The stage of acquiring the first image at the first location, This includes the step of acquiring a second image at a second location, The step of calculating the relative position information is: The method according to claim 1, comprising the steps of: i) calculating a target rotation angle of the tip based on the change in angle of the tip between the first point and the second point, and ii) calculating a target rotation angle of the tip based on the change between the first image and the second image.

3. The step of calculating the relative pause information is as follows: The control unit performs the steps of identifying brightness differences in the image based on the acquired image and brightness information related to illumination, The control unit estimates the distance between the tip and the second body part based on the identified brightness difference, The method according to claim 1, comprising the step of the control unit calculating the relative pose information based on the estimated distance.

4. The step of acquiring an image related to the upper gastrointestinal tract is, The stage of acquiring the first image at the first location, This includes the step of acquiring a second image at a second location, The step of calculating the relative pause information is as follows: i) the angle change of the tip between the first point and the second point, and ii) the distance between the tip and the second body part, which the control unit estimates based on the change between the first image and the second image. The method according to claim 1, wherein the control unit comprises the step of calculating the relative pause information.

5. The method according to claim 1, wherein the pre-trained machine learning model includes a classification model and a sensing model trained on a dataset of labeled images relating to the first and second body parts of the upper gastrointestinal tract.

6. The step of generating the second control signal is: The control unit includes the step of identifying at least one imaging location corresponding to the identified second body part, The control unit generates the second control signal based on the at least one shooting location and the relative pose information, The method according to claim 1, comprising the step of capturing an image when the position of the tip corresponds to the at least one shooting location of the control unit.

7. The method according to claim 1, further comprising the step of the control unit adjusting the tension of at least one wire and controlling the rotation angle of the tip based on the first control signal.

8. The method according to claim 1, further comprising the step of the control unit generating torque feedback and transmitting the torque feedback to the operating unit.

9. The method according to claim 8, wherein the torque feedback has a positive correlation with the target rotation angle to which the tip must move in response to the first control signal.

10. In endoscopes, The tip portion is equipped with an image sensor, The rotation angle control drive unit of the aforementioned tip portion, Includes a control unit, The control unit, Based on a pre-trained model, at least one first body part is detected from the image. The relative position information between the sensed first body part and the tip is calculated, Based on the relative position information, a first control signal is generated for steering in correspondence with the first body part. The first control signal is transmitted to the drive unit, Based on a pre-trained model, at least one second body part is identified from the image. The relative pose information between the identified at least one second body part and the tip is calculated, where the relative pose information represents the change in pose of the tip from a first pose to a second pose. Based on the relative pose information, a second control signal is generated for capturing images of at least one pre-set shooting location corresponding to the at least one second body part. An endoscope device configured to transmit the second control signal to the drive unit.

11. The control unit, First image obtained at location 1, second image obtained at location 2, i) the change in angle of the tip between the first point and the second point, and ii) the change between the first image and the second image, which is used to calculate the target rotation angle of the tip, according to claim 10.

12. The control unit, The endoscope device according to claim 10, wherein, based on the acquired image and brightness information related to illumination, a brightness difference in the image is identified, the distance between the tip and the second body part is estimated based on the identified brightness difference, and relative pose information is calculated based on the estimated distance.

13. The control unit, First image obtained at location 1, second image obtained at location 2, i) the angle change of the tip between the first point and the second point, and ii) the distance between the tip and the second body part, and the relative pose information, based on the change between the first image and the second image, according to claim 10.

14. The endoscope according to claim 10, wherein the pre-trained model includes a classification model and a sensing model trained on a dataset of labeled images relating to the first and second body parts of the upper gastrointestinal tract.

15. The control unit, The endoscope device according to claim 10, which identifies at least one shooting location corresponding to the identified second body part, generates the second control signal based on the at least one shooting location and the relative pose information, and captures an image when the position of the tip corresponds to the at least one shooting location.

16. It further includes a curved portion connected to the aforementioned tip, The control unit, The endoscope device according to claim 10, wherein the drive unit is controlled based on the first control signal or the second control signal, the tension of at least one wire connected to the drive unit is adjusted, and the rotation angle of the tip is adjusted based on the curvature of the curved portion.

17. It further includes an operating section equipped with a curved steering section, The control unit, The endoscope apparatus according to claim 10, wherein torque feedback is generated based on the drive unit and transmitted to the bending steering unit.

18. The endoscope apparatus according to claim 17, wherein the torque feedback has a positive correlation with the target rotation angle to which the tip must move in response to the first control signal.

Images