Three-dimensional stomach navigation method and system for capsule endoscope

Abstract

In order to accomplish the objective, the present invention comprises the steps of: obtaining a captured image from a capsule endoscope inserted into a human body; detecting a landmark from the image obtained from the capsule endoscope, and identifying position and pose information of the capsule endoscope; and generating a moving path of the capsule endoscope on the basis of at least one from among the position information and pose information of the capsule endoscope, the landmark, and a three-dimensional gastroscopy map.

Description

3D upper navigation method and system for capsule endoscopy

[0001] The present invention was made with the support of the Ministry of Science and ICT, the Ministry of Trade, Industry and Energy, the Ministry of Health and Welfare, and the Ministry of Food and Drug Safety, under the project identification number RS-2021-KD000001 and project number 1415184155. The research management specialized organization of the project is the Inter-Ministry Full-Cycle Medical Device Research and Development Project, the research project name is "Inter-Ministry Full-Cycle Medical Device Research and Development Project", the research project name is "Development of a medical device for active precision delivery of therapeutic substances based on microcarriers for knee cartilage regeneration", the main organization is the Korea Micro Medical Robot Research Institute, and the research period is from January 1, 2023 to December 31, 2023.

[0002] The present invention relates to a three-dimensional gastric navigation method and system for a capsule endoscope, and more particularly, to a three-dimensional gastric navigation method and system for a capsule endoscope that guides a three-dimensional path so that a capsule endoscope positioned inside a human body can explore the inside of the stomach.

[0003] An endoscope is generally a medical device used to examine the inside of the body for medical purposes. Depending on the area being examined, endoscopes are referred to as bronchoscopes, gastroscopes, laparoscopy, colonoscopes, and transnasal endoscopes. The most common gastroscopy method uses a catheter, a hose or tube equipped with a camera. A catheter equipped with a 2D camera and light is inserted into the esophagus to observe the stomach wall and other internal structures.

[0004] This conventional gastroscopy method involves the physician directly manipulating a catheter and diagnosing the subject based on the image of the stomach's interior displayed on a screen. Conventional gastroscopy methods, when performed without anesthesia, can cause vomiting, abdominal distension, pain, and discomfort, making the procedure distressing for the subject. Furthermore, when performed under anesthesia, the anesthesia can take at least 30 minutes to an hour to wear off, with an additional recovery period of at least one to two days.

[0005] In particular, conscious sedation endoscopy carries additional risks, such as falls from the floor or stairs due to weakness following anesthesia, or accidents while driving or operating equipment. Furthermore, propofol, the drug used in conscious sedation endoscopy, has no known effective antidote, so there is a small chance of death due to failure to awaken. Midazolid, a drug with an antidote, has limited depth of sleep and analgesic efficacy, and can cause hallucinations in subjects.

[0006] To address the shortcomings of conventional gastrointestinal endoscopes, capsule endoscopes were developed. Capsule endoscopes are devices that move along the digestive tract, guided by the peristaltic motion of the digestive system, to photograph the internal organs and transmit the captured data to an external device via communication. Medical capsule endoscopes are shaped like small capsules and can photograph the internal organs of a living body and wirelessly transmit the captured images to an external storage device. The captured images are then stored on the storage device. The images stored on the storage device are then converted and displayed on a display device, allowing the reader to observe the internal organs by viewing the image data displayed on the display device.

[0007] These existing capsule endoscopes have difficulty viewing the entire gastrointestinal tract, are difficult to confirm the current location in real time, and cannot be controlled externally because they move due to gastrointestinal peristalsis.

[0008] In this regard, Korean Patent Publication No. 10-2020-0114844 discloses an electromagnetic field device for operating an intra-body microrobot from outside the body. While such devices address the inherent limitation of external control inherent in conventional capsule endoscopes, they present limitations in manually or automatically controlling the capsule endoscope from outside, as the capsule endoscope's path cannot be tracked in real time.

[0009] Furthermore, conventional endoscopes have limitations in image information, such as low visibility due to the use of a 2D camera; distortion in the image due to the use of a wide-angle lens with a wide field of view within a narrow area; differences in the clarity of the examination and diagnosis depending on the ability of the person operating the capsule endoscope; and the inability to re-examine specific areas after the examination. These limitations in image information make it difficult to find the path that the capsule endoscope will move.

[0010] In this regard, Korean Patent Publication No. 10-2022-0064464 discloses a magnetically controlled pH sensor-assisted navigation capsule endoscope that controls its battery to uniformly acquire normal images from the colon. The prior art document is characterized by determining whether the capsule endoscope is in the colon based on the captured image and sensed pH, and turning the battery on if it is in the colon and off if it is not, to efficiently use power.

[0011] The above prior art documents aim to consume power efficiently, so they only need to determine the approximate location of whether it is a colon or not, and since the criteria for determining the location include not only the photograph taken but also the pH, an additional pH sensing device is required, which has limitations in that it is disadvantageous in terms of production cost and miniaturization.

[0012] That is, the above prior literature does not present a technical feature for real-time searching for the path that a capsule endoscope located inside an organ will move for diagnosis or treatment, and still does not resolve the problem that it is difficult to search for the path that a capsule endoscope will move due to the limitations of image information.

[0013] Accordingly, in order to search for a path along which a capsule endoscope will move, a three-dimensional navigation method and system for a capsule endoscope are required, which reconstructs images captured by the capsule endoscope to be advantageous for searching for a path along which the capsule endoscope will move, and searches for and guides the path along which the capsule endoscope will move based on the reconstructed images.

[0014] The purpose of the present invention is to provide a three-dimensional stomach navigation method and system for a capsule endoscope that guides a three-dimensional path in real time so that a capsule endoscope positioned inside a human body can explore the inside of the stomach.

[0015] In addition, the present invention aims to restore a capsule endoscope image captured in real time to 3D data so as to enable 3D navigation path search, and to visualize landmarks in the 3D data by aligning the restored 3D data with 3D landmark data.

[0016] In addition, the present invention aims to guide the capsule endoscope along a path to the next landmark location to be moved, thereby allowing the capsule endoscope to sequentially explore the inside of an organ.

[0017] In order to achieve the above object, the present invention is characterized by including a step of obtaining an image captured from a capsule endoscope inserted into a human body, a step of detecting a landmark from the image obtained from the capsule endoscope and determining position and pose information of the capsule endoscope, and a step of generating a movement path of the capsule endoscope based on at least one of the position information and pose information of the capsule endoscope, the landmark, or a three-dimensional upper gastrointestinal endoscope map.

[0018] Preferably, the step of acquiring the above-described captured image may include a step of aligning the three-dimensionally restored above data with the generated three-dimensional landmark data to form three-dimensional above data with the landmarks aligned.

[0019] Preferably, the three-dimensionally restored data can be formed in real time by extracting feature points of the current frame and the previous frame of the captured image and then matching the extracted feature points.

[0020] Preferably, the step of forming the three-dimensional data above with the landmarks aligned may be performed by moving and rotating the generated three-dimensional landmark data so that the Euclidean distance between the points constituting the generated three-dimensional landmark data and the three-dimensionally restored data is the closest, and overlapping the generated three-dimensional landmark data with the three-dimensionally restored data.

[0021] Preferably, the step of forming the three-dimensional data with the landmarks aligned can visualize the aligned landmark portion in the three-dimensional data with the landmarks aligned so as to be distinguished from other portions.

[0022] Preferably, the step of calculating the position and pose information of the capsule endoscope can be performed by extracting feature points of the current frame and the previous frame of the captured image, and then determining the pose information based on the degree of movement and rotation of the capsule endoscope extracted through matching the extracted feature points.

[0023] Preferably, the step of detecting the landmark may include a step of inputting the captured image into an artificial intelligence trained with a two-dimensional landmark data set.

[0024] Preferably, when one of several landmarks is detected, a step of extracting current position information and current pose information of the capsule endoscope from the identified position and pose information of the capsule endoscope may be included.

[0025] Preferably, the step of generating a movement path of the capsule endoscope may include, if any one of the landmarks is not the first landmark, a step of generating a path for moving to the first landmark based on current location information and current pose information of the capsule endoscope, and if any one of the landmarks is the first landmark, a step of generating a path for moving from the first landmark to a second landmark based on current location information and current pose information of the capsule endoscope.

[0026] Preferably, the step of generating a movement path of the capsule endoscope can generate a movement path of the capsule endoscope so that the capsule endoscope moves sequentially according to the order of preset landmarks.

[0027] Preferably, the landmark may include an area within the stomach of at least one of the cardia, fundus, greater curvature, angulus, or pylorus.

[0028] In addition, the present invention is characterized in that it includes an image receiving unit that acquires an image captured from a capsule endoscope inserted into a human body; a calculation unit that detects a landmark from the image acquired from the capsule endoscope and determines position and pose information of the capsule endoscope from three-dimensionally restored stomach data; a search unit that generates a movement path of the capsule endoscope based on at least one of the position information and pose information of the capsule endoscope, the landmark, or a three-dimensional gastrointestinal endoscope map; and a control unit that controls the capsule endoscope to move along the movement path generated by the search unit.

[0029] Preferably, the control unit can control the operation of the capsule endoscope using an external magnetic field.

[0030] The present invention has the advantage of enabling convenient and accurate diagnosis and examination by guiding the movement path of a capsule endoscope in real time so as to explore the inside of the stomach.

[0031] In addition, the present invention has the advantage of being able to confirm the current position of the capsule endoscope by constructing an image captured by the capsule endoscope into virtual 3D data and providing visually effective path guidance.

[0032] FIG. 1 shows a flowchart of a three-dimensional upper navigation method of a capsule endoscope according to an embodiment of the present invention.

[0033] FIG. 2 illustrates an embodiment of a three-dimensional navigation method of a capsule endoscope according to an embodiment of the present invention.

[0034] Figure 3 illustrates a process in which parasitic 3D landmark data according to an embodiment of the present invention is aligned with the above data restored in 3D.

[0035] FIG. 4 is a drawing illustrating an example of determining the current position of a capsule endoscope based on internal major landmark information according to an embodiment of the present invention.

[0036] Figure 5 illustrates a path search process when the first landmark is detected according to an embodiment of the present invention.

[0037] Figure 6 illustrates a capsule endoscope that moves sequentially along the order of preset landmarks according to an embodiment of the present invention.

[0038] Figure 7 shows a configuration diagram of a three-dimensional navigation system, which is another embodiment of the present invention.

[0039] A step of acquiring images taken from a capsule endoscope inserted into the human body;

[0040] A step of detecting a landmark from an image acquired from the capsule endoscope and determining the position and pose information of the capsule endoscope, and

[0041] A step of generating a movement path of the capsule endoscope based on at least one of the position information and pose information of the capsule endoscope, the landmark, or a 3D upper endoscope map.

[0042] A three-dimensional upper navigation method of a capsule endoscope including a .

[0043] Hereinafter, the present invention will be described in detail with reference to the contents described in the attached drawings. However, the present invention is not limited or restricted by the exemplary embodiments. The same reference numerals in each drawing indicate components that perform substantially the same functions.

[0044] The purpose and effects of the present invention can be naturally understood or made clearer by the following description, and the purpose and effects of the present invention are not limited solely by the following description. Furthermore, in describing the present invention, if a detailed description of known technologies related to the present invention is deemed to unnecessarily obscure the gist of the present invention, such detailed description will be omitted.

[0045] The terminology used herein is merely used to describe specific embodiments and is not intended to limit the present invention. The singular expression includes the plural expression unless the context clearly indicates otherwise. In this application, it should be understood that the terms "comprise" or "have" indicate the presence of a feature, number, step, operation, component, part, or combination thereof described in the description of the invention, but do not preclude the possibility of the presence or addition of one or more other features, numbers, steps, operations, components, parts, or combinations thereof.

[0046] While terms like "first" and "second" may be used to describe various components, these components should not be limited by these terms. These terms are used solely to distinguish one component from another. For example, without departing from the scope of the present invention, a first component could be referred to as a "second component," and similarly, a second component could also be referred to as a "first component."

[0047] Unless otherwise defined, all terms used herein, including technical or scientific terms, have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. Terms defined in commonly used dictionaries should be interpreted as having a meaning consistent with their meaning in the context of the relevant technology, and shall not be interpreted in an idealized or overly formal sense unless explicitly defined herein.

[0048] When interpreting components, even if there is no explicit description, it is interpreted as including the margin of error. When describing temporal relationships, for example, when temporal continuity is described with phrases such as "after," "following," "next to," or "before," this also includes cases where the relationship is not continuous, unless "immediately" or "directly" is used.

[0049] Hereinafter, the technical configuration of the present invention will be described in detail with reference to the attached drawings.

[0050] Fig. 1 is a flowchart of a three-dimensional upper navigation method of a capsule endoscope according to an embodiment of the present invention. Referring to Fig. 1, the three-dimensional upper navigation method of a capsule endoscope may include a step of acquiring a captured image (S101), a step of determining position and pose information of a capsule endoscope (S103), a step of generating a movement path of the capsule endoscope (S105), a step of controlling the capsule endoscope (S107), and a data storage step (S109).

[0051] In FIG. 1, each step of the three-dimensional navigation method of the capsule endoscope according to the present embodiment is sequentially performed, but it is not necessarily limited thereto, and each step illustrated in FIG. 1 may be performed simultaneously or separately in the process of generating a movement path and driving the capsule endoscope (300) to move along the movement path.

[0052] The 3D upper navigation method of a capsule endoscope can provide an environment in which an examiner or doctor can conveniently and accurately perform a diagnosis and examination by guiding a 3D path in real time so that a capsule endoscope (300) located inside a human body can explore the inside of the stomach. The 3D upper navigation method of a capsule endoscope can restore a capsule endoscope (300) image captured in real time into 3D data so that a 3D path can be explored, and the 3D restored stomach data and pre-generated 3D landmark data are aligned so that landmarks can be visualized in the 3D restored stomach data, thereby providing visually effective path guidance.

[0053] The three-dimensional upper navigation method of a capsule endoscope can identify the current position of the capsule endoscope (300) and calculate pose information to provide a three-dimensional movement path on the three-dimensionally restored upper data. The three-dimensional navigation path search method can sequentially search multiple landmarks within an organ according to a set order.

[0054] FIG. 2 illustrates an embodiment of a three-dimensional (3D) navigation method of a capsule endoscope according to an embodiment of the present invention. Referring to FIG. 2, the three-dimensional (3D) navigation method of a capsule endoscope can reconstruct a capsule endoscope image captured in real time into three-dimensional (3D) data based on a Visual SLAM algorithm (S201). At the same time, the three-dimensional (3D) navigation method of a capsule endoscope can detect a landmark in the captured capsule endoscope image. When a landmark is detected, the three-dimensional (3D) navigation method of a capsule endoscope can visualize the landmark by aligning the generated three-dimensional landmark data with the three-dimensionally reconstructed stomach data (S203). The three-dimensional (3D) navigation method of a capsule endoscope can determine the position and pose information of a capsule endoscope (300) in real time during the process of restoring the three-dimensional (3D) data, and when a landmark is detected, the current position and pose information of the capsule endoscope (300) can be extracted (S205). At this time, the extracted current location and current pose information can be used as a basis for generating a movement path.

[0055] The 3D upper navigation method of the capsule endoscope can generate a movement path based on the current landmark when a landmark is first detected (S207). If the current landmark is the first landmark, the 3D upper navigation method of the capsule endoscope can guide a movement path to the location of the second landmark, which is the next search order (S209). If the current landmark is not the first landmark, the 3D upper navigation method of the capsule endoscope can guide a movement path to the location of the first landmark (S211). The 3D upper navigation method of the capsule endoscope can end the upper navigation of the capsule endoscope (300) when it searches landmarks in a preset order and arrives at the last landmark (S213).

[0056] Below, each step of the 3D navigation path search method is described in detail.

[0057] The step of acquiring a captured image (S101) can acquire an image captured from a capsule endoscope inserted into the human body.

[0058] The step (S101) of acquiring the captured image can restore the image captured by the capsule endoscope (300) into 3D data in real time. The step (S101) of acquiring the captured image can prevent problems such as reduced visibility and image distortion of the capsule endoscope (300) image by guiding the movement path using the 3D restored data.

[0059] The step (S101) of acquiring a photographed image can be a preliminary task for 3D stomach data restoration, and can convert an image captured by a capsule endoscope (300) into a virtually stained image. The step (S101) of acquiring a photographed image can be a step of separating images by R, G, and B channels from consecutive frames, and then converting the image captured by the capsule endoscope (300) into a virtually stained image. The step (S101) of acquiring a photographed image can use a deep learning model trained with a data set consisting of an endoscopic image in which an organ is not stained and an endoscopic image in which an organ is stained. Although the present embodiment describes that the preliminary task for 3D stomach data restoration is performed in the step (S101) of acquiring a photographed image, it is not limited thereto.

[0060] The step (S101) of acquiring the captured image can improve the performance of the subsequent 3D stomach data restoration by converting the image captured by the capsule endoscope (300) into a virtually stained image. That is, when the 3D stomach data is restored using the virtually stained image, more feature points can be extracted compared to the unstained image, thereby restoring the 3D stomach data similar to the actual inside of the stomach.

[0061] The above data restored in 3D can be formed in real time by extracting feature points of the current frame and the previous frame from a captured image or a virtually dyed image, and then matching the extracted feature points.

[0062] Conventionally, the Structure from Motion (SfM) algorithm has been used for feature point extraction and feature point matching. The SfM algorithm recovers 3D data by detecting common feature points and finding correlations based on images acquired for a single object at multiple viewpoints. However, given the nature of the present invention, which reconstructs images captured in real time by a capsule endoscope (300) into 3D data, the SfM algorithm is not suitable.

[0063] In an embodiment of the present invention, the step (S101) of acquiring a photographed image uses a method of restoring a real-time photographed and transmitted image into 3D data. In one embodiment, the step (S101) of acquiring a photographed image can restore a real-time photographed image into 3D data using a Visual SLAM algorithm. The Visual SLAM algorithm requires less computation than the SfM algorithm, and is therefore advantageous for constructing real-time 3D data.

[0064] FIG. 3 illustrates a process in which parasitic 3D landmark data according to an embodiment of the present invention is registered with 3D restored data. Referring to FIG. 3, the step (S101) of acquiring a photographed image may include a step of registering the 3D restored data with the parasitic 3D landmark data to form 3D data with landmark registration. Here, image registration refers to a processing technique for transforming different images and displaying them in a single coordinate system.

[0065] Looking specifically at the alignment process, in the step of forming the 3D data with the landmarks aligned, when a landmark is detected in the captured image, the same landmark as the landmark detected in the pre-generated 3D landmark data can be extracted and aligned to the 3D reconstructed data. In the step of forming the 3D data with the landmarks aligned, the pre-generated 3D landmark data can be moved and rotated so that the Euclidean distance between the points constituting the pre-generated 3D landmark data and the 3D reconstructed data is the closest, and the pre-generated 3D landmark data can be overlapped with the 3D reconstructed data.

[0066] In one embodiment, the step of forming landmark-aligned 3D data can be performed using the Iterative Closest Point (ICP) algorithm. The ICP algorithm calculates the distance between corresponding points when two point clouds are given, and then finds a transformation matrix that minimizes this distance.

[0067] The step of forming landmark-aligned 3D data can visualize the landmark-aligned portion of the landmark-aligned 3D data so that it can be distinguished from other portions.

[0068] The step of forming 3D landmark-aligned data on the virtual 3D data aligns 3D landmark data and visualizes the landmarks, thereby enabling the location of representative landmarks within the 3D-constructed data to be identified, and providing a visually effective 3D navigation function.

[0069] The step of forming 3D stomach data with aligned landmarks can be a preliminary step of landmark detection to generate 3D landmark data. The step of forming 3D stomach data with aligned landmarks can utilize a deep learning model trained with a data set consisting of multiple frames of images corresponding to representative landmark areas inside the stomach photographed with a capsule endoscope (300). The detection step (S103) can generate 3D landmark data by inputting images corresponding to representative landmark areas inside the stomach into the trained deep learning model.

[0070] The step of forming landmark-aligned 3D stomach data is a preliminary task of generating a movement path performed in the step (S105) of generating a movement path of a capsule endoscope to be described later, and can generate a 3D upper endoscopy map that serves as a landmark of the movement path. The step of forming landmark-aligned 3D stomach data can generate a 3D upper endoscopy map by converting a general or universal human capsule endoscopy image into 3D stomach data and aligning the pre-generated 3D landmark data to the converted 3D stomach data. Although the present embodiment describes that the preliminary task of landmark detection and movement path generation is performed in the step of forming landmark-aligned 3D stomach data, it is not limited thereto.

[0071] FIG. 4 illustrates an example of identifying the current position of a capsule endoscope based on major landmark information inside the stomach according to an embodiment of the present invention. Referring to FIG. 4, the step (S103) of identifying the position and pose information of the capsule endoscope may detect a landmark from an image acquired by the capsule endoscope (300). Here, the landmark refers to an area that can identify a specific position or specific part inside the stomach. For example, a representative landmark of the stomach may include at least one area inside the stomach among the cardia, fundus, greater curvature, angulus, and pylorus.

[0072] Referring to the left picture of Fig. 4, four landmarks are set inside the stomach, consisting of the cardia & fundus (Ex. 1), the greater curvature (Ex. 2), the angulus (Ex. 3), or the pylorus (Ex. 4). Referring to the right picture of Fig. 4, it can be confirmed that the landmarks on the 3D stomach data with aligned landmarks have better visibility than the landmarks on the 3D restored stomach data.

[0073] The step (S103) of determining the position and pose information of the capsule endoscope may include a step of inputting the captured image into an artificial intelligence trained with a two-dimensional landmark data set. By inputting the image captured by the capsule endoscope (300) into the artificial intelligence trained with the two-dimensional landmark data set, landmark detection can be performed in real time. In this case, the artificial intelligence used may be a classification algorithm.

[0074] The step (S103) of identifying the position and pose information of the capsule endoscope can identify the position and pose information of the capsule endoscope (300). Here, the position refers to the approximate position of the capsule endoscope (300) inside the stomach, and the pose information refers to the position, angle (rotation angle), or movement direction of the camera built into the capsule endoscope (300). The step (S103) of identifying the position and pose information of the capsule endoscope can identify the pose information based on the degree of movement and degree of rotation of the capsule endoscope extracted by matching the extracted feature points after extracting the feature points of the current frame and the previous frame of the captured image. The identified position and pose information can be used to generate a movement path, etc.

[0075] The step of extracting the current position information and current pose information of the capsule endoscope can extract the current position information and current pose information of the capsule endoscope from the identified position and pose information of the capsule endoscope when any one landmark among multiple landmarks is detected.

[0076] The step of generating a movement path of the capsule endoscope (S105) may generate a movement path of the capsule endoscope based on at least one of position information and pose information of the capsule endoscope, a landmark, or a 3D upper endoscope map.

[0077] The step (S105) of generating a movement path of the capsule endoscope can generate a movement path in which the capsule endoscope (300) can explore the surroundings of the detected landmark when a landmark is detected. The step (S105) of generating a movement path of the capsule endoscope can generate a movement path in which the capsule endoscope (300) moves to the location of the next landmark to be explored when a landmark is detected. The step (S105) of generating a movement path of the capsule endoscope can generate a movement path in which the capsule endoscope (300) explores the surroundings of the detected landmark when a landmark is detected and then moves to the location of the next landmark to be explored.

[0078] The step (S105) of generating a movement path of the capsule endoscope may generate a movement path of the capsule endoscope so that the capsule endoscope moves sequentially according to the order of preset landmarks. In summary, assuming that there are first to third landmarks and that the search is set to proceed in the order of the first landmark to the third landmark, the step (S105) of generating a movement path of the capsule endoscope may generate a movement path so that the capsule endoscope (300) searches around the first landmark and then moves to the position of the second landmark. Subsequently, the step (S105) of generating a movement path of the capsule endoscope may generate a movement path so that, when the capsule endoscope (300) reaches the position of the second landmark, it searches around the second landmark and then moves to the position of the third landmark. The step (S105) of generating a movement path of the capsule endoscope may end the search when the capsule endoscope (300) reaches the position of the third landmark.

[0079] In the step (S105) of generating a movement path of the capsule endoscope, the generated movement path can be provided three-dimensionally so that the capsule endoscope (300) can be controlled in the space inside the stomach. That is, the movement path does not mean a path of movement on a two-dimensional plane, but can mean a path of movement in space three-dimensionally. The step (S105) of generating a movement path of the capsule endoscope can generate a movement path that takes into account the position, angle, and movement direction inside the stomach of the capsule endoscope (300) so that the capsule endoscope (300) can move in the space inside the stomach.

[0080] The step (S105) of generating a movement path of the capsule endoscope can provide a movement path that serves as a guide when an examiner or doctor operates the capsule endoscope (300) from the outside. The step (S105) of generating a movement path of the capsule endoscope can provide a movement path to a separate control device that automatically controls the capsule endoscope (300) if there is one.

[0081] The step (S105) of generating a movement path of the capsule endoscope may include a step of generating a path to move to the first landmark based on current location information and current pose information of the capsule endoscope, if any landmark is not the first landmark, and a step of generating a path to move from the first landmark to the second landmark based on current location information and current pose information of the capsule endoscope, if any landmark is the first landmark.

[0082] FIG. 5 illustrates a path search process when a landmark is first detected according to an embodiment of the present invention. Referring to FIG. 5, when a landmark is first detected (S301) in an image captured by a capsule endoscope (300), a step (S105) of generating a movement path of the capsule endoscope can determine whether the detected landmark is the first landmark, which is the starting point of the search (S303). The step (S105) of generating a movement path of the capsule endoscope can generate a movement path that moves to the location of the first landmark if the first detected landmark is not the first landmark, which is the starting point of the path search (S305). At this time, in preparation for a case of moving to a location other than the first landmark, after moving to the location of the first landmark, it is determined once again whether the detected landmark is the first landmark.

[0083] The step of generating a movement path of a capsule endoscope (S105) may generate a path to move to a location of a second landmark, which is a point following the first landmark, if the landmark is a first landmark, which is a starting point of the path search (S307). The step of generating a movement path of a capsule endoscope (S105) may generate a path to move to a location of a second landmark, which is a point following the first landmark, after generating a path to move to a location of the second landmark (S309). The step of generating a movement path of a capsule endoscope (S105) may end the path guidance when the last landmark is reached (S311).

[0084] Fig. 6 illustrates a capsule endoscope that moves sequentially according to the order of preset landmarks according to an embodiment of the present invention. Referring to Fig. 6, the step (S105) of generating a movement path of the capsule endoscope may generate a movement path of the capsule endoscope (300) in the order of Cardia & Fundus (Ex. 1), Greater Curvature (Ex. 2), Angulus (Ex. 3), and Pylorus (Ex. 4) according to the order of preset landmarks. At this time, it can be confirmed that each landmark is aligned with the pre-generated 3D landmark data, thereby improving visibility.

[0085] The step (S105) of generating a movement path of a capsule endoscope uses the position and pose information of the capsule endoscope (300) extracted when generating the movement path, but when generating a movement path to an internal location of the stomach that has not been restored from the 3D restored stomach data, the movement path can be generated based on a 3D gastrointestinal endoscope map.

[0086] The step (S105) of generating a movement path of the capsule endoscope can generate a movement path in a three-dimensional space so that the capsule endoscope (300) can be manipulated from the three-dimensionally restored above data. Therefore, the step (S105) of generating a movement path of the capsule endoscope can generate a movement path that takes into account the position, angle, and movement direction of the capsule endoscope (300) inside the stomach so that the capsule endoscope (300) can move in the space inside the stomach by taking into account the position and pose information of the capsule endoscope (300).

[0087] Meanwhile, according to an embodiment of the present invention, the step (S101) of acquiring a photographed image can restore the image photographed by the capsule endoscope (300) into 3D stomach data in real time, so when the capsule endoscope (300) reaches the first landmark, 3D stomach data for generating a movement path to move to another landmark may not be constructed. In this case, the step (S105) of generating a movement path of the capsule endoscope can generate a movement path to move to the next landmark based on a 3D gastrointestinal endoscope map constructed in advance. Since the shape of the inside of the stomach of most people and the location and shape of landmarks are almost similar, the 3D gastrointestinal endoscope map can be utilized.

[0088] The step (S105) of generating a movement path of a capsule endoscope is to generate a movement path using a 3D upper endoscope map, and then, when the 3D upper data for the movement path is restored, the movement path can be corrected based on the 3D restored upper data.

[0089] The step (S107) of controlling the capsule endoscope can control the capsule endoscope (300) to move along the movement path created in the step (S105) of creating the movement path of the capsule endoscope. The control step (S107) can control the capsule endoscope (300) by changing the magnetic field of a control device that controls the operation of the capsule endoscope (300) using an external magnetic field.

[0090] The storage step (not shown) can store various results generated in performing the present invention, such as 3D stomach data restored in real time, detected landmark images, and data in which landmarks and 3D restored stomach data are aligned. Conventional gastrointestinal endoscopes have a problem in that a specific part cannot be checked again after examination, but the storage step can solve the problem that there may be parts that are missed due to not being able to check depending on the doctor's individual ability, since it stores the images captured and reconstructed images by the capsule endoscope (300).

[0091] Fig. 7 illustrates a configuration diagram of a three-dimensional navigation system (100), which is another embodiment of the present invention. Referring to Fig. 7, the three-dimensional navigation system (100) may include an image receiving unit (110), a calculation unit (130), a search unit (150), and a control unit (170).

[0092] The image receiving unit (110) can acquire images captured from a capsule endoscope inserted into the human body. The image receiving unit (110) can receive images captured from the capsule endoscope (300) via wireless communication. The capsule endoscope (300) can capture several to several tens of still or moving images per second and transmit data to the image receiving unit (110) in real time. The image receiving unit (110) can perform the operation of the step (S101) of acquiring the captured images described above.

[0093] The operation unit (130) can detect landmarks from images acquired from a capsule endoscope and determine the position and pose information of the capsule endoscope from the three-dimensionally restored data. The operation unit (130) can perform the operation of the step (S103) of determining the position and pose information of the capsule endoscope described above.

[0094] The search unit (150) can detect landmarks from images acquired from a capsule endoscope and determine the position and pose information of the capsule endoscope from the three-dimensionally restored data. The search unit (150) can perform the operation of the step of generating the movement path of the capsule endoscope as described above.

[0095] The control unit (170) can control the capsule endoscope to move along the movement path generated by the search unit. The control unit (170) can perform the operation of the step (S107) of controlling the capsule endoscope described above.

[0096] The storage unit (not shown) can store various results generated in performing the present invention, such as 3D data restored in real time, detected landmark images, and data obtained by matching landmarks with 3D restored data. The storage unit (not shown) can perform the operations of the storage step described above.

[0097] While the present invention has been described in detail through representative examples above, those skilled in the art will understand that various modifications to the above-described embodiments are possible without departing from the scope of the present invention. Therefore, the scope of the present invention should not be limited to the described embodiments, but should be defined not only by the claims described below but also by all changes or modifications derived from the claims and equivalent concepts.

[0098] The purpose of the present invention is to provide a three-dimensional stomach navigation method and system for a capsule endoscope that guides a three-dimensional path in real time so that a capsule endoscope positioned inside a human body can explore the inside of the stomach.

[0099] In addition, the present invention aims to restore a capsule endoscope image captured in real time to 3D data so as to enable 3D navigation path search, and to visualize landmarks in the 3D data by aligning the restored 3D data with 3D landmark data.

[0100] In addition, the present invention aims to guide the capsule endoscope along a path to the next landmark location to be moved, thereby allowing the capsule endoscope to sequentially explore the inside of an organ.

Claims

1. A step of acquiring images taken from a capsule endoscope inserted into the human body. A step of detecting a landmark from an image acquired from the capsule endoscope and determining the position and pose information of the capsule endoscope, and A step of generating a movement path of the capsule endoscope based on at least one of the position information and pose information of the capsule endoscope, the landmark, or a 3D upper endoscope map. A three-dimensional upper navigation method of a capsule endoscope including a .

2. In paragraph 1, The step of acquiring the above-mentioned captured image is: A step of forming landmark-aligned 3D data by aligning the 3D restored data with the generated 3D landmark data. A three-dimensional upper navigation method of a capsule endoscope including a .

3. In paragraph 2, The above three-dimensionally restored data is, A three-dimensional navigation method for a capsule endoscope, characterized in that the feature points of the current frame and the previous frame of the above-described captured image are extracted and then the extracted feature points are matched to form a three-dimensional image in real time.

4. In paragraph 2, The step of forming the three-dimensional data above with the above landmarks aligned is: A 3D stomach navigation method for a capsule endoscope, characterized in that the parasitic 3D landmark data is moved and rotated so that the Euclidean distance between the points constituting the parasitic 3D landmark data and the 3D restored stomach data is the closest, and the parasitic 3D landmark data is overlapped with the 3D restored stomach data.

5. In paragraph 2, The step of forming the three-dimensional data above with the above landmarks aligned is: A three-dimensional navigation method for a capsule endoscope, characterized in that the three-dimensional data in which the landmarks are aligned is visualized so that the aligned landmark portion is distinguished from other portions.

6. In paragraph 1, The step of calculating the position and pose information of the above capsule endoscope is: A three-dimensional navigation method for a capsule endoscope, characterized in that pose information is determined based on the degree of movement and rotation of the capsule endoscope, which are extracted by extracting feature points of the current frame and the previous frame of the captured image, and then matching the extracted feature points.

7. In paragraph 1, The step of calculating the position and pose information of the above capsule endoscope is: A step of inputting the above-described captured image into an artificial intelligence trained with a 2D landmark data set. A three-dimensional upper navigation method of a capsule endoscope, characterized in that it includes.

8. In paragraph 1, A step of extracting current position information and current pose information of the capsule endoscope from the identified position and pose information of the capsule endoscope when one of several landmarks is detected. A three-dimensional upper navigation method of a capsule endoscope including a .

9. In paragraph 8, The step of creating the movement path of the above capsule endoscope is: If any of the above landmarks is not the first landmark, a step of generating a path to the first landmark based on the current location information and current pose information of the capsule endoscope, and If any one of the above landmarks is a first landmark, a step of generating a path for moving from the first landmark to the second landmark based on the current location information and current pose information of the capsule endoscope. A three-dimensional upper navigation method of a capsule endoscope including a .

10. In paragraph 1, The step of creating the movement path of the above capsule endoscope is: A three-dimensional navigation method for a capsule endoscope, characterized in that a movement path of the capsule endoscope is generated so that the capsule endoscope moves sequentially according to the order of preset landmarks.

11. In paragraph 1, The above landmark is, A three-dimensional stomach navigation method using a capsule endoscope, characterized in that it includes at least one region inside the stomach among the cardia, fundus, greater curvature, angulus, and pylorus.

12. An image receiving unit that acquires images taken from a capsule endoscope inserted into the human body; A computation unit that detects landmarks from images acquired from the capsule endoscope and determines position and pose information of the capsule endoscope from the three-dimensionally restored data; A navigation unit that generates a movement path of the capsule endoscope based on at least one of the position information and pose information of the capsule endoscope, the landmark, or a 3D upper endoscope map; and A three-dimensional navigation system, characterized by including a control unit that controls the capsule endoscope to move along the movement path generated by the search unit.

13. In paragraph 12, The above control unit, A three-dimensional navigation system characterized in that it controls the operation of the capsule endoscope using an external magnetic field.

Images