Endoscope device and control method for obtaining image of upper gastrointestinal tract

Abstract

The invention provides an endoscope device for obtaining an upper gastrointestinal tract image and a control method. The invention belongs to the technical field of medical imaging equipment and automatic control, and particularly relates to an endoscope device for shooting an image of an upper gastrointestinal tract by using an endoscope. A control method of an endoscope device according to an embodiment of the present specification may include: acquiring an image of an upper gastrointestinal tract from an image sensor; sensing at least one first body part from the image according to a pre-trained model; calculating relative position information between the sensed first body part and the front end part; generating a first control signal related to the position of the first body part according to the relative position information; and transmitting the first control signal to a driving unit.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to and the benefit of Korean Patent Application No. 10-24-26748, filed on February 23, 2024, which is hereby incorporated by reference for all purposes as if fully set forth herein. Technical Field

[0003] The present invention belongs to the field of medical imaging equipment and automation control technology, and in particular relates to an endoscope device for taking images of the upper gastrointestinal tract using an endoscope. Background Art

[0004] An endoscope is a general term for medical devices that use a viewing scope inserted into the body to observe organs without surgery or autopsy. Endoscopes are inserted into the body and illuminated with light, allowing the light reflected from the inner surface to be visualized. Endoscopes come in different types depending on their purpose and the body part they are intended for. They can be broadly categorized as rigid endoscopes (with metal tubes) and flexible endoscopes, typified by digestive organ endoscopes.

[0005] Today, when an endoscopist discovers a lesion during endoscopic treatment, they should perform other operations, such as inserting an instrument for biopsy or pressing a button on the scope. At this time, if they loosen their grip on the scope, the scope may shake, causing the lesion to fall out of the imaging field of view.

[0006] These endoscopic technologies rely primarily on manual manipulation by medical professionals to adjust the tip of the endoscope and obtain images of the patient's internal body parts. This process is highly dependent on the operator's proficiency and experience, which can make it difficult to obtain accurate and clear images of the desired body parts. In particular, unnecessary movement during endoscopic operation or difficulties in precise control can prevent adequate images of specific areas required for an accurate diagnosis.

[0007] Furthermore, endoscopic technology has limitations in precisely adjusting the position and angle of the endoscope's tip, making it difficult to obtain images, especially in areas with many curves, such as the upper gastrointestinal tract. These limitations reduce the efficiency and accuracy of endoscopic diagnosis in early disease detection and precise localization. Summary of the Invention

[0008] In order to solve the above-mentioned problems, the embodiments of this specification are proposed, and provide a technology for controlling the position and direction of the distal end portion of an endoscope based on an artificial neural network.

[0009] A control method for an endoscopic device according to an embodiment of the present specification for solving the above-mentioned problem may include: obtaining an image of the upper gastrointestinal tract from an image sensor; sensing at least one first body part from the image based on a pretrained model; calculating relative position information between the sensed first body part and the front end portion; generating a first control signal for turning corresponding to the first body part based on the relative position information; and transmitting the first control signal to a drive unit.

[0010] Obtaining images about the upper gastrointestinal tract may include: obtaining a first image at a first point; and obtaining a second image at a second point, and calculating the relative position information may include: calculating the target rotation angle of the front end portion based on i) the angular change of the front end portion between the first point and the second point and ii) the change between the first image and the second image.

[0011] The method may further include: identifying at least one second body part from the image based on the pre-trained model; calculating relative posture information between the identified second body part and the front end portion; generating a second control signal related to photographing the second body part based on the relative posture information; and transmitting the second control signal to the drive unit.

[0012] Calculating the relative posture information may include: identifying brightness differences on the image based on the obtained image and brightness information about the lighting; estimating the distance between the front end portion and the second body part based on the identified brightness differences; and calculating the relative posture information based on the estimated distance.

[0013] Obtaining images related to the upper gastrointestinal tract may include: obtaining a first image at a first point; and obtaining a second image at a second point, and calculating the relative posture information may include: calculating the distance between the front end portion and the second body part based on i) the change in the angle of the front end portion between the first point and the second point and ii) the change between the first image and the second image; and calculating the relative posture information.

[0014] The pre-trained model may include a classification model and a sensing model, which are trained by using labeled images of a first body part and a second body part related to the upper gastrointestinal tract as a dataset.

[0015] Generating the second control signal may include: identifying at least one shooting point corresponding to the identified second body part; generating the second control signal based on the at least one shooting point and the relative position information; and capturing an image when the position of the front end portion corresponds to the shooting point.

[0016] The method may further include adjusting tension of at least one wire based on the first control signal and controlling a rotation angle of the front end portion.

[0017] The invention may further include: the driving unit generating torque feedback to transmit the torque feedback to the operating unit.

[0018] The torque feedback may have a positive correlation with a target rotation angle to which the front end portion should move based on a first control signal related to the image capturing.

[0019] An endoscopic device according to an embodiment of the present specification for solving the above-mentioned problems may include: a front end portion having an image sensor; a drive unit that controls the rotation angle of the front end portion; and a control unit that senses at least one first body part from the image based on a pretrained model, calculates relative position information between the sensed first body part and the front end portion, generates a first control signal related to the position of the first body part based on the relative position information, and transmits the first control signal to the drive unit.

[0020] The control unit can obtain a first image at a first point and a second image at a second point, and calculate the target rotation angle of the front end portion based on i) the angular change of the front end portion between the first point and the second point and ii) the change between the first image and the second image.

[0021] The control unit can identify at least one second body part from the image based on the pre-trained model, calculate the relative posture information between the identified second body part and the front end portion, generate a second control signal related to the shooting of the second body part based on the relative posture information, and transmit the second control signal to the drive unit.

[0022] The control unit can identify the brightness difference on the image based on the obtained image and the brightness information of the relevant lighting, estimate the distance between the front end portion and the second body part based on the identified brightness difference, and calculate the relative posture information based on the estimated distance.

[0023] The control unit can obtain a first image at a first point and a second image at a second point, and based on i) the angle change of the front end portion between the first point and the second point and ii) the change between the first image and the second image, estimate the distance between the front end portion and the second body part and calculate the relative posture information.

[0024] The pre-trained model may include a classification model and a sensing model, which are trained by using labeled images of a first body part and a second body part related to the upper gastrointestinal tract as a dataset.

[0025] The control unit may identify at least one shooting point corresponding to the identified body part, generate the second control signal based on the at least one shooting point and the relative position information, and capture an image when the position of the front end portion corresponds to the shooting point.

[0026] The endoscopic device may further include a bending portion connected to the front end portion, and the control unit may control the driving unit based on the control signal, adjust the tension of at least one wire connected to the driving unit, and adjust the rotation angle of the front end portion based on the bending of the bending portion.

[0027] The endoscope device may further include an operation unit having a bending steering portion, and the control unit may generate torque feedback based on the driving unit and transmit the torque feedback to the bending steering portion.

[0028] The torque feedback may have a positive correlation with a target rotation angle to which the front end portion should move based on a first control signal related to the image capturing. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] The accompanying drawings, which are included to provide a further understanding of the invention and are incorporated in and constitute a part of this specification, illustrate exemplary embodiments of the invention and together with the description serve to explain the inventive concept.

[0030] Figure 1 FIG. 1 is a schematic diagram of an endoscope device according to an embodiment of the present invention.

[0031] Figure 2 FIG. 1 is a diagram for explaining a process of controlling a bending portion by a driving unit and a wire according to an embodiment of the present invention.

[0032] Figure 3 FIG. 1 is a flowchart of an operation of generating a learning model according to an embodiment of the present invention.

[0033] Figure 4A FIG. 1 is a flow chart of the operation of an endoscope device according to an embodiment of the present invention.

[0034] Figure 4B FIG. 1 is a flow chart of the operation of an endoscope device according to another embodiment of the present invention.

[0035] Figure 4C FIG. 1 is a flow chart of the operation of an endoscope device according to another embodiment of the present invention.

[0036] Figure 5 FIG. 4 is a flowchart of an operation of calculating relative posture information of an endoscopic device according to an embodiment of the present invention.

[0037] Figure 6 FIG. 4 is a flowchart of an operation of calculating relative posture information of an endoscopic device according to another embodiment of the present invention.

[0038] Figure 7 FIG. 4 is a flowchart of an operation of calculating relative posture information of an endoscopic device according to another embodiment of the present invention.

[0039] Figure 8 FIG. 1 is a flowchart of operations related to image capturing by an endoscope device according to an embodiment of the present invention.

[0040] Figure 9 FIG. 1 is a flowchart of operations related to torque feedback of an endoscopic device according to an embodiment of the present invention.

[0041] Figure 10 FIG. 4 is a block diagram of a computer device according to an embodiment of the present invention.

[0042] Explanation of symbols

[0043] 100: Endoscopic device

[0044] 110: Output unit

[0045] 120: Control unit

[0046] 130: Drive unit

[0047] 140: Observation mirror

[0048] 141: Insertion

[0049] 142: Bend

[0050] 143: Front end

[0051] 151: Image Sensor

[0052] 152: Nozzle

[0053] 153: Lighting

[0054] 154: Lens

[0055] 155: Working channel

[0056] 160: Operation unit

[0057] 161: Curved steering part

[0058] 200: Motor

[0059] 210: Frontline

[0060] 220: Second Line

[0061] 1010: Memory

[0062] 1020: Processor DETAILED DESCRIPTION

[0063] The terms used in the present invention are only used to describe specific embodiments and are not intended to limit the scope of other embodiments. Singular expressions may include plural expressions, unless the context clearly indicates otherwise. The terms used herein, including technical or scientific terms, may have the same meanings as those generally understood by those of ordinary skill in the technical field described in the present invention. Among the terms used in the present invention, the terms defined in general dictionaries may be interpreted as having the same or similar meanings as they have in the context of the relevant technology, and unless clearly defined in the present invention, should not be interpreted in an idealized or overly formalized sense. In some cases, even the terms defined in the present invention cannot be interpreted as excluding embodiments of the present invention.

[0064] Various embodiments will be described in detail below with reference to the accompanying drawings so that a person skilled in the art can easily implement the present invention. However, since the technical ideas of the present invention can be modified and implemented in various forms, they are not limited to the embodiments described in this specification. When describing the embodiments disclosed in this specification, if it is determined that the detailed description of the related known technologies may obscure the gist of the technical ideas of the present invention, the detailed description of the related known technologies will be omitted. The same reference numerals are given to the same or similar components, and their repeated descriptions will be omitted.

[0065] At this time, the term "unit" used in this embodiment refers to a component that performs a specific function performed by software or hardware such as a field programmable gate array (FPGA) or an application specific integrated circuit (ASIC). However, the "unit" is not limited to being executed by software or hardware. The "unit" can exist in the form of data stored in an addressable storage medium, or it can be configured to be implemented by instructions to enable one or more processors to perform a specific function.

[0066] Software may include a computer program, code, instruction, or a combination of one or more thereof, which may configure a processing device to operate as needed, or may issue instructions to a processing device independently or collectively. Software and / or data may be permanently or temporarily embodied in any type of machine, component, physical device, virtual device, computer storage medium or device, or transmitted signal wave, to be interpreted by the processing device or to provide instructions or data to the processing device. Software may be distributed across networked computer systems and stored or executed in a decentralized manner. Software and data may be stored on one or more computer-readable recording media. Software may be read into a main memory from other computer-readable media such as a data storage device, or from other devices via a communication interface. Software instructions stored in the main memory may cause the processor to perform the processes or steps described below. In contrast, hard-wired circuits may be used instead of or in combination with software instructions to perform processes consistent with the principles of the present invention. Therefore, embodiments consistent with the principles of the present invention are not limited to any specific combination of hardware circuits and software.

[0067] The terms used in this application are only used to describe specific embodiments and are not intended to limit the present invention. Singular expressions include plural expressions unless the context clearly indicates otherwise. In this application, terms such as "including" or "having" are intended to indicate the presence of features, numbers, steps, operations, components, parts, or combinations thereof described in the specification, and it should be understood that this does not preclude the possibility of the presence or addition of one or more other features, numbers, steps, operations, components, parts, or combinations thereof. Terms such as first and second can be used to describe various components, but components should not be limited by these terms. The terms are only used to distinguish one component from another.

[0068] The "learning model" mentioned in the present invention may include any type of algorithm or method for learning or understanding a specific pattern or structure from data. That is, the learning model may include not only machine learning models, such as regression models, decision trees, random forests, support vector machines, K-nearest neighbors, naive Bayes, clustering algorithms, etc., but also deep learning models, such as neural networks, convolutional neural networks, recurrent neural networks, transfer (Transformer) neural networks, generative adversarial networks (GANs), and autoencoders, etc. The "learning model" may indicate a learning parameter or weight combination that is used to predict or classify the output for a specific input, and the model may be trained by supervised learning, unsupervised learning, semi-supervised learning, reinforcement learning, and the like. In addition, this may include not only a single model, but also various learning methods and structures, such as integrated models, multimodal models, and models learned by transfer, etc. These learning models may also be pre-trained in a computer device separate from the computer device that predicts the output for the input and used in other computer devices.

[0069] The learning model according to an embodiment of the present invention may include at least one of models related to object classification (Object classification), object detection (Object Detection) and position estimation (Position Estimation).

[0070] Figure 1 FIG. 1 is a schematic diagram of an endoscope device according to an embodiment of the present invention.

[0071] Reference Figure 1 The endoscope device 100 according to one embodiment of the present invention may be a flexible endoscope, specifically a digestive organ endoscope. The endoscope device 100 may include a component capable of obtaining and photographing medical images of the interior of the digestive organ, and, if necessary, may include a component capable of inserting a tool to perform treatment or manipulation while viewing the medical images.

[0072] The endoscope device 100 may include an output portion 110 , a control unit 120 , a driving unit 130 , a viewing scope 140 , and an operating unit 160 .

[0073] The output unit 110 may include a display for displaying medical images. The output unit 110 may include a display module that can output visual information or implement a touch screen and support image expression, zoom in and out functions, etc., such as a liquid crystal display (LCD), a thin film transistor-liquid crystal display (TFT LCD), an organic light-emitting diode (OLED) or a flexible display (flexible display), a three-dimensional display (3D display), etc. In addition, the user can operate the image and obtain necessary information through the touch screen function. These output units 110 can display medical images obtained by the viewing mirror 140 or medical images processed by the control unit 120.

[0074] The drive unit 130 can provide the power required for inserting the scope 140 into the body or for moving it within the body. For example, the drive unit 130 can include multiple motors connected to wires within the scope 140 and a tension adjustment unit for adjusting the tension of the wires. The drive unit 130 can control the scope 140 in multiple directions by controlling the power of each of the multiple motors. Specifically, the drive unit 130 can adjust the wire tension by controlling the power of each of the multiple motors, and adjust the rotation angle of the front end portion 143 by bending the curved portion 142.

[0075] The viewing scope 140 may include an insertion portion 141, a bending portion 142, and a front end portion 143. The insertion portion 141 is a portion inserted into the body and may be turned by the bending portion 142 to reach an internal organ.

[0076] The bending portion 142 is connected to the insertion portion 141 to control the direction in which the scope 140 enters the body. The rotation angle of the bending portion 142 can be adjusted by a user's instruction or a control signal of a control unit.

[0077] The front end portion 143 is located at the front end of the viewing mirror 140 and can perform various operations according to the user's instructions or the control signal of the control unit. The front end portion 143 may include an image sensor 151, a nozzle 152, an illumination 153, a lens 154 and a working channel 155.

[0078] The image sensor 151 can capture images of the endoscope device. For example, the image sensor 151 can be a complementary metal-oxide-semiconductor (CMOS) or a charge-coupled device (CCD) sensor.

[0079] The nozzle 152 can clean the lens 154 by spraying a solution, medicine, etc. In addition, the nozzle 152 can spray medicine required for biopsy or treatment into the body.

[0080] The illumination 153 may illuminate a light source so that the image sensor 151 can capture an image at a certain illuminance. Brightness information about the illumination 153 may be pre-stored in the control unit 120.

[0081] The lens 154 can focus light so that the image sensor 151 can capture an appropriate image. These lenses 154 can also include wide-angle or zoom functions.

[0082] The working channel 155 may indicate a channel for delivering a separate instrument or sampling tool into the interior of the human body.

[0083] The operation unit 160 may indicate a user interface for actually operating the endoscope. The operation unit 160 may include a plurality of buttons, dials, and levers for controlling the bending and steering portion 161 and various functions of the endoscope. The user may input user instructions based on the elements provided in the operation unit 160.

[0084] The bending steering portion 161 can be used to adjust the bending portion 142. The bending steering portion 161 can be implemented in the form of a knob, a joystick, or a lever, and the user can steer the direction of the bending portion 142 by rotating or moving it. The bending steering portion 161 can receive torque feedback from the drive unit 130. For example, the torque feedback can be a physical signal generated by simulating the force generated when the front end portion 143 contacts the internal tissue of the human body or a control signal of the drive unit 130 based on a pre-trained model.

[0085] The control unit 120 can control the overall operation of the endoscope device 100 and can perform the operation of the endoscope device according to one embodiment. The control unit 120 can control the movement of the scope 140 via the drive unit 130 connected to the scope 140. The control unit 120 can perform various control operations for imaging the interior of the digestive organs through the scope 140. The control unit 120 can also perform various processing on the medical images obtained through the scope 140.

[0086] According to one embodiment, the control unit 120 can obtain an image of the upper gastrointestinal tract from an image sensor, sense at least one body part from the image based on a pre-trained model, calculate relative position information between the body part and the front end portion, generate a control signal related to the captured image based on the identified body part and the relative position information, and transmit the control signal to the drive unit. This relative position information can indicate steering information for moving from a first position, which is the current position of the front end portion, to a second position. For example, the control unit 120 can use the relative position information to calculate a target rotation angle that can be used to steer the front end portion toward a body part separated by x and y coordinates on the image. In addition, the control unit 120 controls the drive unit by generating a control signal based on the target rotation angle, and according to the control, the pitch and yaw of the front end portion can be adjusted to steer the front end portion toward a body part that is as far away as the x and y coordinates on the image.

[0087] According to one embodiment, the control unit 120 can obtain an image of the upper gastrointestinal tract from an image sensor, identify at least one body part from the image based on a pre-trained model, calculate the relative posture information between the body part and the front end, generate a control signal for capturing the image based on the identified body part and the relative posture information, and transmit the control signal to the drive unit. These relative posture information can indicate information for moving from a first posture to a second posture, where the first posture is the current posture of the front end, and the second posture is for effectively capturing the identified body part. The pose can include the position and direction of the object in space. For example, the position can be represented in the form of x, y, and z coordinates in a coordinate system, and the direction can be represented in the form of pitch (rolling around the x-axis), yaw (rolling around the y-axis), and roll (rolling around the z-axis).

[0088] The control unit 120 may include a central processing unit (CPU), a random access memory (RAM), a read-only memory (ROM), and a system bus, etc. The control unit 120 may be implemented with a single CPU or multiple CPUs (or digital signal processing (DSP), system on chip (SoC)). In one embodiment, the control unit 120 may be implemented with a digital signal processor (DSP), a microprocessor, or a time controller (TCON) that processes digital signals. However, the control unit 120 is not limited thereto and may include one or more of a central processing unit (CPU), a microcontroller unit (MCU), a microprocessing unit (MPU), a controller, an application processor (AP), a communication processor (CP), an ARM processor, or may be defined by such terms. In addition, the control unit 120 may be implemented as a system on chip (SoC) with a built-in processing algorithm, a large scale integration (LSI), or in the form of a field programmable gate array (FPGA). Furthermore, the control unit 120 may include a neural processing unit (NPU), a graphics processing unit (GPU), and a tensor processing unit (TPU).

[0089] Figure 2 This figure is a diagram for explaining the process of controlling the bending portion by a drive unit and wires according to one embodiment of the present invention. For ease of description, the operation of controlling the direction of the bending portion 142 using two motors 200 is shown. However, it will be clear to those skilled in the art that, by expanding this, multiple motors can be used to adjust the tension of each wire and the rotation angle of the front end portion.

[0090] Reference Figure 2The endoscope device 100 may include a motor 200, a first wire 210, and a second wire 220 included in a driving unit 130. The endoscope device may increase the tension of the first wire 210 and reduce the tension of the second wire 220 by controlling the motor 200 to bend the bending portion 142 to one side. Alternatively, the endoscope device may reduce the tension of the first wire 210 and increase the tension of the second wire 220 by controlling the motor 200 to bend the bending portion 142 to the other side.

[0091] The endoscope device 100 can adjust the rotation angle of the distal end portion 143 in this manner.

[0092] Figure 3 FIG. 1 is a flowchart of an operation of generating a learning model related to object sensing according to an embodiment of the present invention. Figure 3 The learning model in may correspond to a pre-trained model. For ease of description, these operations are disclosed as being learned by a separate computer device, but it will be clear to those skilled in the art that they may be operated by an endoscopic device or a separate computer device.

[0093] refer to Figure 3 In step S310, the computer device may annotate or mark a specific body part by viewing the endoscopic image. Based on user input, etc., the computer device may mark the part of the upper gastrointestinal tract in the image with a bounding box. For example, the upper gastrointestinal tract may include at least one of the oral cavity, pharynx, esophagus, stomach, and duodenum. In addition, the endoscopic image may include an internal cavity image of the oral cavity, pharynx, esophagus, stomach, and duodenum.

[0094] The computer device according to an embodiment may generate a data set including images with designated tags in step S320. These data sets may include data sets of various lighting conditions, viewing angles, states of body parts, and the like.

[0095] According to one embodiment, the computer device can use the data set generated in step S330 to train a neural network model. The computer device can select a neural network model and train the model based on the data set. For example, such a model can include a convolutional neural network as an object sensing model.

[0096] According to another embodiment, the computer device may generate a learning model related to object classification. Such a learning model related to object classification may be a model for recognizing an object used in a captured image.

[0097] In step S310, a computer device according to another embodiment may annotate or mark a specific body part by viewing an endoscopic image. Based on user input, the computer device may mark a portion of the upper gastrointestinal tract in the image. These portions of the upper gastrointestinal tract may indicate the body part used to capture the image.

[0098] The computer device according to an embodiment may generate a data set including images with designated tags in step S320. These data sets may include data sets of various lighting conditions, viewing angles, states of body parts, and the like.

[0099] According to one embodiment, the computer device can use the data set generated in step S330 to train a neural network model. The computer device can select a neural network model and train the model based on the data set. For example, such a model can include a convolutional neural network as an object classification model.

[0100] According to another embodiment, a computer device can use endoscopic images and control history information corresponding to the user's surgical technique to generate a learning model. These control history information can indicate the control history information used when observing the endoscopic image and adjusting the direction, angle, depth, etc. of the endoscope. According to another embodiment, a computer device can obtain an endoscopic image and control history information about the upper gastrointestinal tract in step S310. For example, the upper gastrointestinal tract may include at least one of the oral cavity, pharynx, esophagus, stomach, and duodenum. In addition, as an endoscopic image, an inner cavity image of the oral cavity, pharynx, esophagus, stomach, and duodenum can be included.

[0101] The computer device according to yet another embodiment may generate a data set including an endoscopic image and control history information in step S320 .

[0102] According to another embodiment, the computer device can use the data set generated in step S330 to train a neural network model. The computer device can select a neural network model and train the model based on the data set. For example, these models can include classification models such as convolutional neural networks (CNN) to perform image processing, and can include at least one of sequence processing models such as recurrent neural networks (RNN) and long short-term memory (LSTM) networks to perform sequence processing of control history information.

[0103] Figure 4A FIG. 1 is a flow chart of the operation of an endoscope device according to an embodiment of the present invention.

[0104] Reference Figure 4A In step S410a, the endoscope device may obtain an image of the upper gastrointestinal tract from an image sensor. For example, the front end of the endoscope device may enter the upper gastrointestinal tract and obtain an image from the image sensor by converting optical signals into electrical signals.

[0105] In step S420a, the endoscopic apparatus according to one embodiment may sense at least one body part from the image based on a pre-trained model. The endoscopic apparatus may sense the body part based on the pre-trained model that marks the upper gastrointestinal tract with a bounding box and confirm the position of the at least one body part within the image on the image.

[0106] In step S430a, the endoscope device according to one embodiment may calculate relative position information between the body part and the front end portion of the endoscope device. The endoscope device may calculate the relative position information based on image changes caused by changes in the angle of the front end portion. The endoscope device may calculate this relative position information based on disparity and changes in the angle of the front end portion.

[0107] In step S440a, the endoscopic device according to an embodiment may generate a control signal for steering relative to its position relative to the body part based on the relative position information.

[0108] In step S450a, the endoscope device according to an embodiment may transmit a control signal to the driving unit. The endoscope device according to an embodiment may adjust the tension of at least one wire and control the rotation angle of the front end portion to turn in response to the body part sensed in step S450a.

[0109] As a specific example, an endoscopic apparatus according to an embodiment may obtain a first image at a first point and obtain a second image at a second point rotated by a certain Δθ. If the parallax between the first image and the second image is ΔL, the center coordinate L of the bounding box of the sensed body part is used to be target The front end is turned to the sensed body part θ target It can be expressed as mathematical formula 1.

[0110] [Mathematical formula 1]

[0111]

[0112] Since Δθ can be calculated from the encoder information of the driving unit and ΔL can be calculated from the parallax between images, a relationship between the target distance on the image and the rotation target angle of the front end portion moved by the driving unit can be derived.

[0113] These equations (1) can be expressed in a coordinate system. Specifically, since the rotation of the endoscope device in the direction of travel within the body is called roll, the drive unit controls pitch and yaw, allowing the endoscope device to be turned to a body part as far as the x and y coordinates on the image.

[0114] Figure 4B FIG. 1 is a flow chart of the operation of an endoscope device according to another embodiment of the present invention.

[0115] Reference Figure 4B In step S410b, the endoscope device may obtain an image of the upper gastrointestinal tract from the image sensor. For example, the front end of the endoscope device may enter the upper gastrointestinal tract and obtain an image from the image sensor by converting optical signals into electrical signals.

[0116] In step S420b, the endoscopic device according to another embodiment may identify at least one body part from the image based on a pre-trained model. The endoscopic device may classify body parts based on a pre-trained model that labels the upper gastrointestinal tract and identify at least one body part within the image.

[0117] In step S430b, the endoscope device according to another embodiment may calculate relative posture information between the body part and the front end of the endoscope device, including distance, etc. For example, the endoscope device may calculate relative posture information based on the brightness of the lighting, the image change of the angle change of the front end, and the movement trajectory of the front end. The operation of the endoscope device to calculate these relative posture information may correspond to Figure 5 、 Figure 6 and Figure 7 .

[0118] According to another embodiment, the endoscope apparatus may generate control signals for capturing images based on the body part and relative posture information identified in step S440b. These control signals may include signals for adjusting the front end of the endoscope.

[0119] In step S450b, the endoscope device according to another embodiment may transmit a control signal to the drive unit. The endoscope device according to one embodiment may adjust the tension of at least one wire and control the rotation angle of the front end portion to correspond to at least one preset shooting point associated with the body part identified in step S450b.

[0120] According to another embodiment, the endoscope device can capture images based on the body part identified in step S460b and the relative posture information. When the front end is positioned at at least one preset capturing point related to the identified body part, the endoscope device can capture images.

[0121] Figure 4C FIG. 1 is a flow chart of the operation of an endoscope device according to another embodiment of the present invention.

[0122] Reference Figure 4C In step S410c, the endoscope device may obtain an image of the upper gastrointestinal tract from the image sensor. For example, the front end of the endoscope device may enter the upper gastrointestinal tract and obtain an image from the image sensor by converting optical signals into electrical signals.

[0123] In step S420c, the endoscope apparatus according to another embodiment may generate control signals for capturing images based on a pre-trained model using endoscopic images and control history information corresponding to the user's surgical technique. The endoscope apparatus may generate control signals by inputting endoscopic images into a learning model, such as a classification model and a sequence processing model. These control signals may include signals for adjusting the distal end of the endoscope.

[0124] In step S430c, the endoscope device according to another embodiment may transmit a control signal to the drive unit to adjust the tension of at least one wire and control the rotation angle of the front end portion to correspond to at least one preset shooting point associated with the body part identified in step S430c.

[0125] According to another embodiment, the endoscope device can capture images based on the body part and relative posture information identified in step S440c. When the front end is positioned at at least one preset capturing point related to the identified body part, the endoscope device can capture images.

[0126] Figure 5 FIG. 4 is a flowchart of an operation of calculating relative posture information of an endoscopic device according to an embodiment of the present invention. Figure 5 The operation of the endoscopic device in may correspond to Figure 4B The relative posture information may indicate information for moving from a first posture, which is the current posture of the front end, to a second posture, which is for effectively photographing the recognized body part.

[0127] Reference Figure 5 Based on the image and the brightness information about the illumination obtained in step S510, the endoscopic apparatus can identify brightness differences in the image. The endoscopic apparatus can analyze the difference between the stored brightness information about the illumination and the brightness information resulting from the image captured by the image sensor and the illumination. The endoscopic apparatus can also identify brightness patterns that appear based on the characteristics of the internal tissue and the intensity and direction of the illumination.

[0128] According to one embodiment, the endoscopic device can calculate relative posture information based on the brightness difference identified in step S520. Based on the identified brightness difference, the endoscopic device estimates the distance between the front end portion and the specific body part in the current posture. Based on the estimated distance, the center coordinates on the image, and the coordinates of the identified body part, the endoscopic device can calculate the position information required for the front end portion to move and the direction information required for the image sensor provided on the front end portion to capture the image. The relative posture information may include the path and direction of the front end portion through the body, as well as the required angular change.

[0129] According to one embodiment, the endoscopic device can also project a specific pattern of light based on brightness differences and structured light, and calculate relative posture information based on changes in this pattern. For example, the endoscopic device estimates distance based on changes in the pattern. Based on the estimated distance, the center coordinates on the image, and the coordinates of the identified body part, the device can calculate the position information required for moving the front end and the direction information required for the image sensor disposed on the front end to capture an image.

[0130] An endoscope apparatus according to an embodiment may calculate a distance from a leading end portion to a lesion by analyzing a pixel difference between two images using a plurality of image sensors.

[0131] Figure 6 FIG. 4 is a flowchart of an operation of calculating relative posture information of an endoscopic device according to another embodiment of the present invention. Figure 6 The operation of the endoscopic device in may correspond to Figure 4B Step S430b in .

[0132] Reference Figure 6 In step S610, the endoscope device may obtain a first image at a first point.

[0133] In step S620 , the endoscope device according to another embodiment may obtain a second image at a second point.

[0134] In step S630 , the endoscope apparatus according to another embodiment may calculate relative posture information based on an angle change of the leading end portion and a pixel change on a screen when moving from a first point to a second point.

[0135] For example, an endoscopic device can use convolution operations to analyze an input image. The endoscopic device can calculate the location of the identified body part as bounding box coordinates based on a pre-trained model that extracts and learns features from the input image through multiple convolutional layers. These coordinates consist of left, top, right, and bottom coordinates, and the output bounding box coordinates can be converted to the center coordinates of the bounding box through post-processing.

[0136] The center coordinates of these bounding boxes can be expressed as Equation 2.

[0137] [Mathematical formula 2]

[0138]

[0139] x^ indicates the center coordinate of the x-axis, which is the average of the left (l) and right (r) boundaries, and y^ indicates the center coordinate of the y-axis, which is the average of the upper (t) and lower (b) boundaries.

[0140] The endoscope device can calculate the angle change Δθ of the front end portion when moving from the first point to the second point based on the encoder information, and calculate the distance between the recognized body part and the front end portion using the pixel change (ΔL) of the image.

[0141] This can be expressed as mathematical formula 3.

[0142] [Mathematical formula 3]

[0143]

[0144] d^ indicates the distance between the lesion and the front end, R indicates the rotation radius of the front end, Δθ indicates the angular change of the endoscope's front end, f indicates the focal length, ΔB indicates the displacement of the image sensor, and ΔL is the pixel change on the screen, indicating the disparity between the two images of the same object. The endoscopic device can use a polynomial trajectory based on the calculated rotation angle to calculate the target angle to which the endoscope's front end should move. The endoscopic device can generate a control signal by analyzing the difference between the calculated target movement angle and the current angle of the front end.

[0145] The operation of the endoscope device can be achieved not only by a single image sensor but also by stereoscopic vision based on multiple image sensors. ΔB can be the distance between the image sensors.

[0146] The endoscope device estimates the distance, and based on the estimated distance, the center coordinates on the image, and the coordinates of the identified body part, can calculate the position information required for the front end to move and the direction information required for the image sensor set on the front end to take pictures.

[0147] Figure 7 FIG. 4 is a flowchart of an operation of calculating relative posture information of an endoscopic device according to another embodiment of the present invention. Figure 7 The operation of the endoscopic device in may correspond to Figure 4B Step S430b in .

[0148] Reference Figure 7 The endoscope device may obtain movement trajectory data of the front end portion in step S710. This movement trajectory data is data of movement of the front end portion inside the body, and may include, for example, at least one of tracking data based on sensors such as a magnetic field sensor, a gyroscope, and an accelerometer, and tracking data based on images such as an image sensor.

[0149] In step S720, the endoscope apparatus according to another embodiment may calculate relative posture information between the body part and the distal end portion based on the movement trajectory of the distal end portion and the obtained image. For example, the endoscope apparatus may calculate relative posture information for moving the distal end portion from a first posture, which is the current position, to a second posture, which is the target position, based on position information based on encoder values ​​of the motor and a depth estimation algorithm.

[0150] Figure 8 FIG. 1 is a flowchart of operations related to image capturing by an endoscope device according to an embodiment of the present invention. Figure 8 The operation of the endoscopic device in may correspond to Figure 4B Steps S440b to S460b in , and Figure 4C Steps S420c to S440c in .

[0151] Reference Figure 8 In step S810, the endoscope apparatus may identify at least one preset photographing point for the identified body part. The at least one preset photographing point is a predetermined position for photographing or treating the body part, and may indicate a position preset by the user based on the user's surgical technique or a position where structural information of the identified body part can be obtained.

[0152] In step S820, the endoscopic device according to one embodiment may generate a control signal for controlling rotation of the distal end portion based on the at least one imaging point and the relative posture information. Specifically, the endoscopic device may calculate relative posture information between the body part and the endoscopic device, and generate a control signal for positioning the distal end portion at the at least one imaging point based on the distance and direction between the calculated relative posture information and the at least one preset imaging point.

[0153] In step S830 , the endoscope device according to an embodiment transmits a control signal to the driving unit, and the driving unit adjusts the tension of at least one wire based on the control signal to control the rotation angle of the front end portion by bending the bending portion.

[0154] In step S840 , when the front end portion is disposed at each of the at least one photographing point, the endoscope apparatus according to an embodiment may photograph an image based on the image sensor at each of the at least one photographing point.

[0155] Figure 9 This is a flowchart of torque feedback-related operations of an endoscopic device according to an embodiment of the present invention. The torque feedback can be a physical signal generated by simulating the force generated when the front end portion contacts internal tissue of the human body or a control signal of a drive unit based on a pre-trained model.

[0156] Reference Figure 9 In step S910, the endoscope device may generate torque feedback related to the movement of the distal end portion. This torque feedback may be generated based on a control signal related to image capture. The torque feedback may be generated based on information indicating the direction and degree of rotation of the distal end portion of the endoscope device, and the movement distance and the magnitude of the torque feedback may be positively correlated.

[0157] In step S920, the endoscope device may transmit the torque feedback to the operating unit. For example, the endoscope device may control the front end according to the rotation angle to be controlled when capturing an image, while the bending steering unit provides feedback to the user by transmitting torque feedback in the direction in which the front end is turned.

[0158] Figure 10 FIG. 4 is a block diagram of a computer device according to an embodiment of the present invention.

[0159] The computer device may include a memory 1010 and a processor 1020. The computer device may be a separate device from the endoscopic device or may be included in a control unit. One or more sets of instructions may be executed to perform any one or more of the methods described herein.

[0160] The memory 1010 may store a set of instructions, including system-related instructions for executing any one or more of the functions of the methods described herein and user interface-related instructions. The memory 1010 temporarily or permanently stores data such as basic programs, applications, and setting information for device operation. The memory 1010 may include a non-volatile mass storage device (permanent mass storage device), such as a random access memory (RAM), a read-only memory (ROM), and a disk drive, but the present invention is not limited thereto. These software components may be loaded from a computer-readable recording medium separate from the memory 1010 using a drive mechanism. These separate computer-readable recording media may include computer-readable recording media such as a floppy disk drive, a magnetic disk, a magnetic tape, a DVD / CD-ROM drive, and a memory card. According to an embodiment, the software components may be loaded into the memory 1010 via a communication unit rather than a computer-readable recording medium. In addition, the memory 1010 may provide stored data upon request from the processor 1020. According to an embodiment of the present invention, the memory 1010 may store setting information.

[0161] The processor 1020 controls the overall operation of the computer device. Furthermore, the processor 1020 may be configured to process instructions by performing basic arithmetic, logical, and input / output operations. The memory 1010 may provide instructions to the processor 1020. For example, the processor 1020 may be configured to execute received instructions based on program code stored in a recording device such as the memory 1010. For example, the processor 1020 may control the device to perform operations according to the various embodiments described above.

[0162] According to one embodiment of the present invention, the processor 1020 can view an endoscopic image and annotate or label specific body parts. Based on user input, the computer device can use a bounding box to label or classify the upper gastrointestinal tract part in the image. For example, the upper gastrointestinal tract may include at least one of the oral cavity, pharynx, esophagus, stomach, and duodenum.

[0163] The processor 1020 according to an embodiment of the present invention may generate a data set including images to which tags are assigned, and the data set may include various lighting conditions, viewing angles, states of body parts, and the like.

[0164] According to an embodiment of the present invention, the processor 1020 can use the generated data set to train a neural network model. The computer device can select a neural network model and train the model based on the data set. For example, these models can include a convolutional neural network as at least one of an object sensing model and an object classification model.

[0165] According to another embodiment of the present invention, the processor 1020 can use endoscopic images and control history information corresponding to the user's surgical technique to generate a learning model. This control history information can indicate the control history information used when observing the endoscopic image and adjusting the direction, angle, depth, etc. of the endoscope. The processor 1020 can obtain endoscopic images and control history information about the upper gastrointestinal tract. For example, the upper gastrointestinal tract may include at least one of the oral cavity, pharynx, esophagus, stomach, and duodenum.

[0166] The processor 1020 according to another embodiment may generate a data set including an endoscopic image and control history information.

[0167] According to another embodiment, the processor 1020 can use the generated data set to train a neural network model. The computer device can select a neural network model and train the model based on the data set. For example, these models can include classification models such as convolutional neural networks (CNN) to perform image processing, and can include at least one of sequence processing models such as recurrent neural networks (RNN) and long short-term memory (LSTM) networks to perform sequence processing of control history information.

[0168] As described above, although the embodiments have been described using limited embodiments and figures, those skilled in the art may make various modifications and variations based on the above description. For example, the described techniques may be performed in a different order than the described method, and / or the components of the described systems, structures, devices, circuits, etc. may be combined or combined in a different manner than the described method, or even replaced or substituted with other components or equivalents, and appropriate results may be achieved.

[0169] Therefore, other embodiments, other examples, and equivalents of the claims are also within the scope of the following claims.

[0170] According to one embodiment of the present invention, the control unit of the endoscope device can accurately and effectively obtain images of specific body parts by controlling the position of the front end portion, thereby reducing the difficulty of operating the endoscope and improving the accuracy of medical diagnosis.

[0171] Effects of the present embodiment are not limited to the above-mentioned effects, and other effects not mentioned can be clearly understood by those skilled in the art from the description of the claims.

[0172] Although certain exemplary embodiments and implementations have been described herein, other embodiments and modifications will be apparent from this description. Therefore, the inventive concept is not limited to these embodiments, but rather to the broader scope of the appended claims, and various obvious modifications and equivalent arrangements will be apparent to those skilled in the art.

Claims

1. A method for controlling an endoscope device, comprising: obtaining an image related to the upper gastrointestinal tract from an image sensor; sensing at least one first body part from the image based on a pre-trained model; calculating the sensed relative position information between the first body part and the front end portion; generating a first control signal for turning correspondingly to the first body part based on the relative position information; and The first control signal is transmitted to the driving unit.

2. The method according to claim 1, wherein Obtaining the upper gastrointestinal tract related images includes: obtaining a first image at a first point; and Obtain a second image at a second point, Calculating the relative position information includes: A target rotation angle of the front end portion is calculated based on i) a change in the angle of the front end portion between the first point and the second point and ii) a change between the first image and the second image.

3. The method according to claim 1, further comprising: identifying at least one second body part from the image based on the pre-trained model; calculating relative posture information between the recognized second body part and the front end portion; generating a second control signal related to photographing the second body part based on the relative posture information; and The second control signal is transmitted to the driving unit.

4. The method according to claim 3, wherein: Calculating the relative posture information includes: identifying brightness differences on the image based on the obtained image and brightness information about the lighting; estimating a distance between the front end portion and the second body part based on the identified brightness difference; and The relative posture information is calculated based on the estimated distance.

5. The method according to claim 3, wherein Obtaining the upper gastrointestinal tract related images includes: obtaining a first image at a first point; and Obtain a second image at a second point, Calculating the relative posture information includes: estimating a distance between the front end portion and the second body part based on i) a change in the angle of the front end portion between the first point and the second point and ii) a change between the first image and the second image; and The relative posture information is calculated.

6. The method according to claim 3, wherein: The pre-trained model includes a classification model and a sensing model, which are trained by using images labeled with a first body part and a second body part related to the upper gastrointestinal tract as a dataset.

7. The method according to claim 3, wherein: Generating the second control signal includes: identifying at least one photographic point corresponding to the identified second body part; generating the second control signal based on the at least one shooting point and the relative posture information; and When the position of the front end portion corresponds to the at least one photographing point, an image is photographed.

8. The method according to claim 1, further comprising: The tension of at least one wire is adjusted based on the first control signal, and the rotation angle of the front end portion is controlled.

9. The method according to claim 1, further comprising: The driving unit generates torque feedback to transmit the torque feedback to the operating unit.

10. The method according to claim 9, wherein: The torque feedback has a positive correlation with a target rotation angle to which the front end portion should move according to the first control signal.

11. An endoscopic device comprising: a front end portion having an image sensor; a driving unit configured to control a rotation angle of the front end portion; as well as A control unit is configured to sense at least one first body part from the image based on a pretrained model, is configured to calculate relative position information between the sensed first body part and the front end portion, is configured to generate a first control signal for turning corresponding to the first body part based on the relative position information, and is configured to transmit the first control signal to the drive unit.

12. The endoscopic device according to claim 11, wherein The control unit is configured as follows: A first image is obtained at a first point, and a second image is obtained at a second point, and a target rotation angle of the front end portion is calculated based on i) an angular change of the front end portion between the first point and the second point and ii) a change between the first image and the second image.

13. The endoscopic device according to claim 11, wherein The control unit is configured as follows: Based on the pre-trained model, at least one second body part is identified from the image, and relative posture information between the identified second body part and the front end is calculated. Based on the relative posture information, a second control signal related to the shooting of the second body part is generated, and the second control signal is transmitted to the drive unit.

14. The endoscopic device according to claim 13, wherein: The control unit is configured as follows: Based on the obtained image and brightness information about the lighting, a brightness difference on the image is identified, a distance between the front end portion and the second body part is estimated based on the identified brightness difference, and the relative posture information is calculated based on the estimated distance.

15. The endoscopic device according to claim 13, wherein The control unit is configured as follows: A first image at a first point is obtained, and a second image at a second point is obtained, and based on i) a change in the angle of the front end portion between the first point and the second point and ii) a change between the first image and the second image, a distance between the front end portion and the second body part is estimated and the relative posture information is calculated.

16. The endoscopic device according to claim 13, wherein The pre-trained model includes a classification model and a sensing model, which are trained by using labeled images of a first body part and a second body part related to the upper gastrointestinal tract as a dataset.

17. The endoscopic device according to claim 13, wherein: The control unit is configured as follows: At least one shooting point corresponding to the identified body part is identified, and the second control signal is generated based on the at least one shooting point and the relative posture information, and an image is captured when the position of the front end portion corresponds to the at least one shooting point.

18. The endoscopic device according to claim 11, further comprising: a curved portion connected to the front end portion, The control unit is configured as follows: The driving unit is controlled based on the control signal, the tension of at least one wire connected to the driving unit is adjusted, and the rotation angle of the front end portion is adjusted based on the bending of the bending portion.

19. The endoscopic device according to claim 11, further comprising: an operating unit having a curved steering portion, The control unit is configured as follows: A torque feedback is generated based on the driving unit and transmitted to the curved steering portion.

20. The endoscopic device according to claim 19, wherein The torque feedback has a positive correlation with a target rotation angle to which the front end portion should move according to the first control signal.

Images