Method and apparatus for reconstructing images of inside of body obtained through endoscope device

Abstract

To provide a method and apparatus for reconstructing images of the inside of a body obtained through an endoscope device that three-dimensionally reconstructs an imaged organ on the basis of images obtained through an endoscope and information about an endoscopic scope.SOLUTION: A method of reconstructing images of the inside of the body that is performed by a computing device 200 including at least one processor 210 includes the steps of: obtaining basic data including the pose information of an end of an endoscopic scope and a captured image of the inside of the body; and generating a three-dimensional final image of the inside of the body by reconstructing the basic data using feature points including the image information of the surface of the inside of the body for the basic data.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] The present disclosure relates to a technique for restoring an image, and more specifically, to a method and an apparatus for restoring an image of the interior of a body acquired from an endoscope device.

Background Art

[0002] An endoscope is a general term for a medical instrument that observes organs by inserting a scope into the body without performing surgery or autopsy. An endoscope inserts a scope into the interior of the human body, irradiates light, and visualizes the light reflected from the surface of the inner wall. The types of endoscopes are classified according to the purpose and the body part, and can be roughly classified into a rigid endoscope in which the endoscope tube is formed of metal and a flexible endoscope typified by a gastrointestinal endoscope.

[0003] On the other hand, a technique for restoring an organ in three dimensions using a two-dimensional image has been actively applied in the medical field, and the same applies to endoscope images. In the case of an endoscope, since the body part to be imaged has a three-dimensional complex and irregular shape, specific information can be obtained from a three-dimensional image as compared with a two-dimensional image, and thus there are advantages in three-dimensional restoration. The technique for restoring a three-dimensional image from a two-dimensional image has evolved from a traditional non-learning method of aligning images taken at multiple time points and calculating time differences to a technique using a deep neural network based on depth information.

[0004] In the case of a gastrointestinal endoscope, the shape of the organ to be imaged is complex and there is a large deviation for each patient, so the movement of the scope to be imaged has to be very irregular. In particular, in the case of a colon endoscope, since an image of a long and thin tube is taken, even if the movement of the scope is fine, a large change in the captured image occurs, and it is difficult to obtain a high-quality image for three-dimensional restoration. Further, for restoration, characteristic shapes must exist in each region of the organ, but only information about the surface of the organ can be obtained from the characteristics of the endoscope image, and in the case of the stomach and the colon, since the surface colors are similar, it is difficult to perform highly accurate restoration.

Summary of the Invention

Problems to be Solved by the Invention

[0005] The present disclosure is for solving the above-described problems of the prior art, and relates to a method and an apparatus for restoring an internal image of the body obtained from an endoscope apparatus that three-dimensionally restores an organ photographed based on an image obtained through an endoscope and information about the endoscope scope.

[0006] However, the technical problems to be achieved by this embodiment are not limited to the technical problems as described above, and other technical problems may exist.

Means for Solving the Problems

[0007] According to an embodiment of the present disclosure for realizing the above-described problems, a method for restoring an internal image of the body is disclosed. The method includes: obtaining basic data including attitude information of an end of an endoscope scope and a photographed image of the inside of the body; and generating a three-dimensional final image of the inside of the body by restoring the basic data using feature points including image information of an internal surface of the body with respect to the basic data.

[0008] As an alternative, the feature points can be generated by a feature point generation model learned to represent a pattern or texture formed on the internal surface of the body.

[0009] As an alternative, the pattern or texture can include at least one of blood vessels, unevenness, wrinkles, and lesions.

[0010] As an alternative, the feature point generation model can generate the feature points by representing a shape in which a drug is applied to the internal surface of the body.

[0011] As an alternative, the basic data may further include depth information from the end of the endoscopic scope to the inner surface of the virtual body.

[0012] As an alternative, the step of generating the three-dimensional final image may include generating point cloud data using the basic data and the feature points, and restoring the point cloud data to generate the final image.

[0013] As an alternative, the step of generating the three-dimensional final image may include generating corrected point cloud data by removing points existing in a region outside a predetermined reference range from the point cloud data, and restoring the corrected point cloud data to generate the final image.

[0014] As an alternative, the method may further include determining a blind spot region where the surface is smoothly restored in the final image, and the blind spot region may include a site where imaging through the endoscopic device is missed inside the body.

[0015] According to an embodiment of the present disclosure for realizing the above-described problems, a method for restoring an image inside a body is disclosed. The method includes obtaining depth information from the end of the endoscopic scope to the inner surface of the body using a learned depth estimation model based on a captured image of the inside of the body, obtaining pose information of the end of the endoscopic scope using a learned pose estimation model based on the captured image, and generating a three-dimensional final image of the inside of the body using the depth information and the pose information.

[0016] As an alternative, the depth estimation model can be learned by first learning data including a first learning image of the inside of the body and a depth image corresponding to the first learning image.

[0017] As an alternative, the posture estimation model can be trained using a second training image including feature points for the internal surface information of the body and second training data including posture information corresponding to the second training image.

[0018] As an alternative, the step of obtaining the posture information of the end of the endoscope scope using the trained posture estimation model may include generating a corrected captured image using the feature points generated using a feature point generation model trained to represent a handle or texture formed on the internal surface of the body and the captured image, and obtaining the posture information of the end of the endoscope scope using the trained posture estimation model based on the corrected captured image.

[0019] As an alternative, the feature point generation model can generate the feature points by representing the shape of the drug applied to the internal surface of the body.

[0020] According to an embodiment of the present disclosure for realizing the above-described problems, a computing device for restoring an image of the inside of the body is disclosed. The device includes a memory storing basic data including the posture information of the end of the endoscope scope and a captured image of the inside of the body, and a processor generating a three-dimensional final image of the inside of the body by restoring the basic data using the basic data and feature points including the image information of the internal surface of the body for the basic data.

[0021] According to an embodiment of the present disclosure for realizing the problems described above, a computing device for restoring an image inside the body is disclosed. The device includes a memory for storing a captured image of the inside of the body, and based on the captured image, depth information from the end of the endoscope scope to the inner surface of the body is obtained using a depth estimation model that has been learned, and based on the captured image, attitude information of the end of the endoscope scope is obtained using an attitude estimation model that has been learned, and a processor that generates a three-dimensional final image of the inside of the body using the depth information and the attitude information.

Effect of the Invention

[0022] According to an embodiment of the present disclosure, the accuracy of restoration can be improved by restoring using feature points that reflect the inner surface information of the body while considering the characteristics of the body part.

[0023] Further, according to an embodiment of the present disclosure, since it is possible to grasp a blind spot area that the endoscope did not photograph based on the surface information in the restored image, the accuracy of endoscopic surgery can be improved, and convenience can be provided to medical staff who have to judge the blind spot area themselves.

Brief Description of the Drawings

[0024]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

DETAILED DESCRIPTION OF THE INVENTION

[0025] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the accompanying drawings so that those having ordinary knowledge in the technical field of the present disclosure (hereinafter referred to as those skilled in the art) can easily implement them. The embodiments presented in the present disclosure are provided so that those skilled in the art can use or implement the content of the present disclosure. Therefore, various modifications to the embodiments of the present disclosure will be apparent to those skilled in the art. That is, the present disclosure can be embodied in various different forms and is not limited to the following embodiments.

[0026] Throughout the specification of the present disclosure, the same or similar reference numerals refer to the same or similar components. Also, for the purpose of clearly explaining the present disclosure, the reference numerals of the parts not related to the description of the present disclosure can be omitted from the drawings.

[0027] The term "or" used in the present disclosure is not an exclusive "or" but is intended to mean an inclusive "or". That is, in the present disclosure, unless otherwise specified or the meaning is not clear from the context, "x uses a or b" should be understood to mean one of the natural inclusive substitutions. For example, in the present disclosure, unless otherwise specified or the meaning is not clear from the context, "x uses a or b" can be interpreted as either x uses a, x uses b, or x uses both a and b.

[0028] The term "and / or" used in the present disclosure should be understood to include all possible combinations of one or more of the related concepts listed.

[0029] The terms "comprising" and / or "including" as used in this disclosure should be understood to mean that a particular feature and / or component is present. However, the terms "comprising" and / or "including" should be understood not to exclude the presence or addition of one or more other features, other components, and / or combinations thereof.

[0030] In this disclosure, unless otherwise specified or indicated in the singular form and where the context is not clear, the singular should generally be construed to include "one or more".

[0031] The term "the Nth (N is a natural number)" as used in this disclosure can be understood as an expression used to distinguish the components of this disclosure from each other according to a predetermined criterion such as a functional perspective, a structural perspective, or for the convenience of explanation. For example, in this disclosure, components that perform different functional roles can be distinguished as the first component or the second component. However, components that are substantially the same within the technical concept of this disclosure but need to be distinguished for the convenience of explanation can also be distinguished as the first component or the second component.

[0032] On the one hand, the terms "module" or "unit" used in the present disclosure can be understood as terms indicating an independent functional unit that processes computing resources, such as a computer-related entity, firmware, software or a part thereof, hardware or a part thereof, a combination of software and hardware, etc. Here, a "module" or "unit" may be a unit composed of a single element, or may be a unit represented as a combination or set of multiple elements. For example, as a concept of negotiation, a "module" or "unit" can indicate a hardware element or a set thereof of a computing device, an application program that performs a specific function of software, a processing procedure (procedure) embodied by the execution of software, or a set of instruction words for the execution of a program. Also, in a broad sense, a "module" or "unit" may indicate the computing device itself that constitutes a system, or an application executed on the computing device. However, since the above concepts are only examples, the "module" or "unit" concept can be defined in various ways within the scope understandable by those skilled in the art based on the content of the present disclosure.

[0033] The term "model" used in the present disclosure can be understood as a system embodied using mathematical concepts and language to solve a specific problem, a set of software units for solving a specific problem, or an abstract model for a processing procedure for solving a specific problem. For example, a neural network "model" can indicate the entire system embodied as a neural network having problem-solving ability through learning. Here, the neural network can have problem-solving ability by optimizing parameters (parameters) that connect nodes or neurons through learning. A neural network "model" can also include a single neural network, or can include a set of neural networks combined with multiple neural networks.

[0034] The term "video" as used in this disclosure may refer to multi-dimensional data composed of discrete image elements. In other words, "video" can be understood as a term indicating a digital representation of an object visible to the human eye. For example, "video" may refer to multi-dimensional data composed of elements corresponding to pixels in a two-dimensional image. "Video" may refer to multi-dimensional data composed of elements corresponding to voxels in a three-dimensional image.

[0035] The above explanations of the terms are for helping to understand this disclosure. Therefore, it must be noted that when the above terms are not explicitly described as matters limiting the content of this disclosure, the content of this disclosure is not used in the sense of limiting the technical idea.

[0036] FIG. 1 is a block diagram of an endoscope system including an endoscope device and a computing device according to an embodiment of this disclosure.

[0037] Referring to FIG. 1, the endoscope system 10 includes an endoscope device 100 that acquires various information including medical video of the inside of the body, and a computing device 200 that receives and analyzes information about the inside of the body from the endoscope device 100 and restores the medical video of the inside of the body in three dimensions.

[0038] The endoscope device 100 can be a flexible endoscope, specifically, a gastrointestinal endoscope. The endoscope device 100 can include a configuration capable of acquiring medical video of the inside of the digestive tract, and, if necessary, a configuration capable of inserting instruments and performing treatment or procedures while viewing the medical video. The endoscope device 100 can include a control unit that controls the overall operation of the endoscope device 100, a scope inserted into the inside of the body, a drive unit that provides the power necessary for the movement of the scope, and the like.

[0039] At least a part of the scope is inserted into the body, and various cables, tubes, treatment tools, etc. can be inserted into the body through the scope, and medical imaging and procedures can be performed at the end of the scope. On the other hand, the end of the scope can be further provided with an attitude sensor or a depth sensor. The attitude sensor can acquire the 6D attitude information of the scope and transmit it to the control unit of the endoscope device 100. The depth sensor can acquire the depth information from the end of the scope to the inner surface of the body and transmit it to the control unit of the endoscope device 100. The attitude sensor or the depth sensor can be detachable depending on the situation, for example, it is installed when the scope is inserted into a model body, and can be separated when the scope is actually inserted into the body.

[0040] The endoscope device 100 can acquire medical images, that is, basic data including captured images, and provide them to the computing device 200. The basic data includes a captured image of the inside of the body, and can include the attitude information of the end of the endoscope scope or the depth information from the end of the endoscope scope to the inner surface of the body.

[0041] The basic data can be acquired from the endoscope device 100. Here, the endoscope device 100 can be an endoscope device 100 inserted into an actual patient or a body model (dummy, mannequin, etc.), or an endoscope device 100 embodied in a virtual environment. The scope of the endoscope device 100 can be inserted into an actual body or a model body. Alternatively, by simulating the operation of the endoscope device 100 in a virtual space, basic data about the virtual body can be acquired.

[0042] The computing device 200 of the present disclosure can restore a three-dimensional final image using the basic data received from the endoscope device 100. The final image can correspond to the inside of the body photographed by the endoscope device 100.

[0043] According to an embodiment of the present disclosure, the computing device 200 may be a hardware device or a part of a hardware device that performs comprehensive processing and operations of data, and may also be a software-based computing environment connected via a communication network. For example, the computing device 200 may perform the function of intensive data processing, and may be a server that is a subject sharing resources, or may be a client that shares resources through interaction with the server. Further, the computing device 200 may be a cloud system that enables a plurality of servers and clients to interact with each other to comprehensively process data. The above description is only an example related to the type of the computing device 200, and the type of the computing device 200 can be configured in various ways within the scope understandable by those skilled in the art based on the content of the present disclosure.

[0044] Referring to FIG. 1, the computing device 200 according to an embodiment of the present disclosure may include a processor 210, a memory 220, and a network unit 230. However, since FIG. 1 is only an example, the computing device 200 may include other configurations for implementing a computer environment. Further, only a part of the above-described configurations may be included in the computing device 200. The processor 210 according to an embodiment of the present disclosure can be understood as a component unit including hardware and / or software for executing computing operations. For example, the processor 210 can read a computer program and execute data processing for machine learning. The processor 210 can process operation processes such as processing of input data for machine learning, feature extraction for machine learning, and error calculation based on backpropagation. The processor 210 for executing such data processing can include a central processing unit (CPU), a general-purpose graphics processing unit (GPGPU), a tensor processing unit (TPU), an application specific integrated circuit (ASIC), or a field programmable gate array (FPGA), etc. Since the types of the processor 210 described above are only examples, the types of the processor 210 can be configured in various ways within the scope understandable by those skilled in the art based on the content of the present disclosure.

[0045] The processor 210 can restore the organ photographed by the endoscope device 100 in three dimensions using the information acquired through the endoscope device 100. Here, the processor 210 can perform three-dimensional restoration using the trajectory information of the end of the endoscope scope that acquires the photographed image, or can infer the posture information of the end of the endoscope scope and the depth information about the inside of the body from the photographed image through an artificial neural network model, and use this to execute three-dimensional restoration.

[0046] Specifically, the processor 210 can generate point cloud data based on the information acquired through the endoscope. A point cloud means a set of data points belonging to a three-dimensional space, and each point gathers to represent a three-dimensional shape or an object.

[0047] The processor 210 can restore a final image including a surface in three dimensions using point cloud data that is a set of points. The processor 210 can determine whether a site that must be photographed during the endoscopic procedure has been missed using the final image restored in three dimensions. In this specification, a site where photographing has been missed is referred to as a blind spot. The processor 210 can determine a blind spot area based on the restored form of the final image. For example, the processor 210 can determine a portion where the surface is smoothly restored in the final image as the blind spot area and a portion where the surface is roughly restored as the photographed area.

[0048] On the other hand, the basic data used by the processor 210 to generate the final image can vary depending on the data collection environment or the restoration method.

[0049] For example, the basic data can include a photographed image and the attitude information of the end of the endoscope scope, and can further include depth information from the end of the endoscope scope to the inner surface of the body.

[0050] On the other hand, during the restoration process of the three-dimensional final image, the processor 210 can use the feature points included in the basic data or generate feature points based on the basic data lacking feature points and use the generated feature points. That is, the processor 210 can restore the final image using, in addition to the basic data, feature points including image information for the inner surface information of the body. In the present disclosure, the feature points mean image elements indicating the inner surface shape of the body, and can mean elements including fine forms and hues observed on the inner surface of the body, such as wrinkles of organs, bends, unevenness, blood vessels, and lesions formed on the surface of the organs.

[0051] The processor 210 can train a feature point generation model to represent a pattern or texture formed on the inner surface of the body, and can add feature points to the basic data using the trained feature point generation model.

[0052] Alternatively, the processor 210 can train a feature point generation model to represent the shape to which the drug is applied based on the experimental image obtained by applying and photographing the drug inside the body, and add feature points to the basic data using the trained feature point generation model. The drug is a staining agent that stains a specific site in an endoscopic procedure, and can be, for example, indigo carmine solution. On the other hand, the detailed content of the feature point generation method will be described later.

[0053] The memory 220 according to an embodiment of the present disclosure can be understood as a component unit including hardware and / or software for storing and managing data processed by the computing device 200. That is, the memory 220 can store any form of data generated or determined by the processor 210 and any form of data received by the network unit 230. For example, the memory 220 can include at least one type of storage medium such as a flash memory type, a hard disk type, a multimedia card micro type, a card type memory, a RAM (random access memory), an SRAM (static random access memory), a ROM (read-only memory), an EEPROM (electrically erasable programmable read-only memory), a PROM (programmable read-only memory), a magnetic memory, a magnetic disk, and an optical disk. Further, the memory 220 can also include a database system for controlling and managing data in a predetermined system. Since the types of the memory 220 described above are merely examples, the types of the memory 220 can be variously configured within the scope understandable by those skilled in the art based on the content of the present disclosure.

[0054] The memory 220 can structure and organize data, combinations of data, program code executable by the processor 210, etc., which are necessary for the processor 210 to execute operations. Further, the memory 220 can store program code that causes the processor 210 to generate learning data.

[0055] The memory 220 can store basic data, point cloud data, final images, etc., and can store information about feature points generated by the feature point generation model. Further, the memory can store depth information and pose information inferred by the processor 210 via the artificial neural network model. The memory 220 can also store the artificial neural network model for the processor 210 to learn and the learning data of each model.

[0056] The network unit 230 according to an embodiment of the present disclosure can be understood as a component for transmitting and receiving data via any form of known wired or wireless communication system. For example, the network unit 230 can execute data transmission and reception using a wired or wireless communication system such as a local area network (LAN), wideband code division multiple access (WCDMA), long term evolution (LTE), wireless broadband internet (WiBro), fifth generation mobile communication (5G), ultra-wideband wireless communication, ZigBee, radio frequency (RF) communication, wireless LAN, wireless fidelity (WiFi), near field communication (NFC), or Bluetooth (registered trademark). Since the above-described communication systems are merely examples, the wired or wireless communication system for data transmission and reception of the network unit 230 can be variously applied in addition to the above-described examples.

[0057] The network unit 230 can receive the data necessary for the processor 210 to execute calculations via wired or wireless communication with any system or any client, etc. Also, the network unit 230 can transmit the data generated by the calculations of the processor 210 via wired or wireless communication with any system or any client, etc. For example, the network unit 230 can receive basic data including captured images through communication with a cloud server or an endoscope device 100 that performs tasks such as a medical video storage and transmission system or standardization of medical data. The network unit 230 can transmit various data generated by the calculations of the processor 210 through communication with the aforementioned systems, servers, or endoscope device 100, etc.

[0058] The data to be processed by the processor 210 can be stored in the memory 220 or received via the network unit 230, and the data generated by the processor 210 can be stored in the memory 220 or transmitted externally via the network unit 230.

[0059] On the other hand, in FIG. 1, the endoscope device 100 and the computing device 200 are shown as being separated, but this is exemplary, and the computing device 200 can also be provided inside the endoscope device 100 and constitute a part of the endoscope device 100.

[0060] According to the present disclosure, the computing device 200 can restore the body part photographed by the endoscope in three dimensions based on data obtained in various situations, so that the quality of restoration can be improved while being less restricted by the type of data obtained and the situation in which the data is obtained.

[0061] In addition, by adding and restoring feature points for the internal surface information of the body, the accuracy of restoration can be improved, and the blind spot area can be grasped based on the surface information. Therefore, the accuracy of endoscopic surgery can be provided, and convenience can be provided to medical staff who have to judge the blind spot area by themselves.

[0062] FIG. 2 is a configuration diagram of a computing device according to an embodiment of the present disclosure, FIG. 3 is an exemplary diagram showing an image generated by a feature point generation model according to an embodiment of the present disclosure, FIG. 4 is an exemplary diagram showing the generation of feature points according to an embodiment of the present disclosure, and FIG. 5 is an exemplary diagram showing a final image according to an embodiment of the present disclosure.

[0063] Referring to FIG. 2, the computing device 300 can generate a three-dimensional final image 500 using the basic data 400 acquired from the endoscopic device.

[0064] The basic data 400 can include at least one of the attitude information 410 of the end of the endoscopic scope, the captured image 420 of the inside of the body, and the depth information 430 indicating the distance from the end of the scope to the internal surface of the body. Exemplarily, the attitude information 410 can include coordinates indicating a position and a rotation value in a three-dimensional space. The captured image 420 is captured via a camera provided in the endoscopic scope and can be acquired in a virtual environment. The captured image 420 is a two-dimensional RGB image and can be a still image captured at a specific time point or a video captured continuously.

[0065] Here, the endoscopic device may mean an actual endoscopic device or an endoscopic device embodied in a virtual environment. In the case of an actual endoscopic device, the scope of the endoscopic device is inserted into the inside of an actual body or a phantom body, and the posture information 410 obtained via the posture sensor provided at the end of the scope can be acquired, and the captured image 420 can be acquired via the camera. On the other hand, when a depth sensor is provided at the end of the scope, depth information up to the inner surface of the body can be further acquired. In the case of a virtual endoscopic device, the posture information 410, the captured image 420, and the depth information 430 with respect to the inside of the virtual body embodied in the virtual environment can be acquired.

[0066] The computing device 300 can generate three-dimensional point cloud data using the basic data 400. The point cloud data can include points corresponding to the body part photographed by the endoscopic device. Here, the part photographed by the endoscopic device has a high point density, and the part corresponding to the blind spot area, that is, the part that the endoscopic device cannot photograph, may have a low point density.

[0067] The computing device 300 can restore the point cloud data to generate the final image 500. The final image 500 can include the three-dimensional surface inside the body. Here, the computing device 300 can add feature points to the basic data 400 using the feature point generation model 310 in order to generate the final image 500. The feature points can indicate the inner surface information of the body, and specifically, can include patterns or textures formed on the inner surface of the body. Exemplarily, when the body part is the large intestine, it can include blood vessels, unevenness, wrinkles formed on the inner wall of the large intestine, lesions such as polyps, etc. observed on the surface of the large intestine.

[0068] Referring to FIG. 3, the feature point generation model 310 can generate an image as shown in FIG. 3(c) including feature points by adding the feature points as shown in FIG. 3(b) to the basic data 400 with insufficient feature points as shown in FIG. 3(a).

[0069] The feature point generation model 310 can be trained through supervised learning using pairs of images with few feature points and images with many feature points as training data, or through unsupervised learning that does not require training data, or semi-supervised learning using training data that includes some labeled data. For example, the feature point generation model 310 can be trained through a generative artificial intelligence model. In particular, when considering the difficulties in securing a large amount of training data for endoscopic images, the need to generate labeled data based on clinical results, and the extremely low frequency of training data corresponding to abnormalities, the present disclosure can implement the feature point generation model 310 through a generative artificial intelligence. Exemplarily, the feature point generation model 310 can include a Generative Adversarial Networks (GAN) or a cycleGAN.

[0070] The feature point generation model 310 can add elements including fine forms and hues observed on the inner surface of the body to an image representing the slippery interior of the body. For example, it can add feature points representing lesions formed on the mucosa of the stomach or large intestine, wrinkles formed on the inner wall of the stomach or large intestine, blood vessels visible on the mucosa, bends formed by the curved shape of the organ, unevenness, unevenness due to lesions, polyps, etc. The feature point generation model 310 can be trained using data with similar hues and similar shapes, including data with many feature points and data with few feature points.

[0071] For example, the feature point generation model 310 generates an image with many feature points using two generators and discriminates the authenticity of the image with many feature points using two discriminators. The generator generates an image for deceiving the discriminator, and the discriminator determines the authenticity of the generated image. By repeating this process, the feature point generation model 310 can generate an image containing many feature points using an image with few feature points.

[0072] On the one hand, the feature point generation model 310 can generate feature points based on experimental images obtained by applying a pigment to an actual body or a model body and then taking a photograph.

[0073] Referring to FIG. 4, the experimental image may mean an image obtained by applying a pigment to the inside of the body and photographing the shape in which the pigment is distributed in order to emphasize the unevenness of the mucosal surface inside the body. This is because the degree of staining varies depending on the presence or absence of mucosal elevations or depressions, the state of the mucosa, and the presence or absence of lesions. The feature point generation model 310 can generate feature points using the distribution shape of the pigment from the experimental image and add this to the basic data 400.

[0074] The feature point generation model 310 can generate feature points by using the above-described methods in combination. For example, the feature point generation model 310 can generate feature points using an adversarial generation network and adjust the distribution of the feature points according to the shape to which the drug is applied.

[0075] On the other hand, there may already be sufficient feature points in the basic data 400. Exemplarily, in the case of a large intestine image, it includes image feature points due to the bending shape of the large intestine, the light and shadow generated by the shape, the liquid accumulating in the bend, and the like. On the other hand, in the case of a stomach image, there are very few feature points compared to the large intestine image due to the smooth surface of the stomach, the shape in which blood vessels are formed, and the like.

[0076] In this case, the feature point generation model 310 can generate feature points by using the above-described method with the basic data 400 for the stomach. And the feature point generation model 310 can extract the feature points already included in the basic data 400 by using the basic data 400 for the large intestine. That is, the feature point generation model 310 can generate feature points by generating feature points that are not in or insufficient in the basic data 400 using the basic data 400, or by extracting the feature points existing in the basic data from the basic data 400 with rich feature points. Alternatively, restoration can also be performed using the basic data 400 itself with rich feature points.

[0077] Referring to FIG. 5, the computing device 300 can generate point cloud data using the basic data 400 and the feature points generated via the feature point generation model 310. To generate the point cloud data, the captured image included in the basic data 400 can be matched with the pose information 410 or the depth information 430. Here, according to the present disclosure, the accuracy of the matching can be increased by adding feature points. For example, in the case of a captured image with a smooth surface or little change in hue, there are few feature points serving as the matching reference, so the matching accuracy between the captured image and the pose information 410 and the depth information 430 may be low. Therefore, according to the present disclosure, by adding feature points to the captured image, the number of reference points increases, so the accuracy of the point cloud data can be increased. The accuracy of the point cloud data may mean the degree of similarity between the three-dimensional final image 500 generated during restoration and the data acquisition environment.

[0078] The computing device 300 can restore the point cloud data to generate a three-dimensional final image 500 of the inside of the body. Here, the computing device 300 can generate corrected point cloud data by removing points existing in a region outside a predetermined reference range from the point cloud data. The computing device 300 can restore the corrected point cloud data to generate the final image 500.

[0079] On the one hand, the computing device 300 can distinguish between a portion of the final image 500 where the surface is smoothly restored and a portion where it is roughly restored. The computing device 300 can determine that the portion where the surface is smoothly restored is a blind spot area. If it is a part that has not been photographed by the endoscope scope, there may be less basic data 400 corresponding to that part, and the number of feature points added to the basic data 400 may also be reduced. Therefore, the number of points corresponding to that part on the point cloud data can be reduced, so that it can be restored to a plane. Here, the degree of roughness or smoothness can be determined according to a predetermined standard, and can be set according to the degree of height on the reference area.

[0080] FIG. 6 is a flowchart showing a method for restoring an image of the interior of a body according to an embodiment of the present disclosure.

[0081] Referring to FIG. 6, the computing device 300 can acquire basic data including the attitude information of the end of the endoscope scope and the captured image of the interior of the body (S110). The basic data can further include depth information from the end of the endoscope scope to the inner surface of the body. The interior of the body can be an actual body, a model body, or a virtual body, and is not limited to a single form.

[0082] The computing device 300 can generate a three-dimensional final image of the interior of the body by restoring the basic data using feature points including the image information of the inner surface of the body with respect to the basic data (S120). Here, the feature points can mean the feature points included in the basic data itself, the feature points extracted from the basic data, or the feature points generated by the feature point generation model. The feature point generation model can generate feature points by expressing the pattern or texture formed on the inner surface of the body. Here, the pattern or texture can include at least one of blood vessels, unevenness, wrinkles, and lesions.

[0083] Alternatively, the feature point generation model can generate feature points that represent patterns or textures formed on the inner surface of an actual or model body using experimental images taken by applying a drug inside the actual or model body and then photographing it.

[0084] The computing device 300 can generate point cloud data using the basic data and feature points, and restore the point cloud data to generate a final image.

[0085] The computing device 300 can generate corrected point cloud data by removing points existing in regions outside a predetermined reference range from the point cloud data. That is, the corrected point cloud can be one from which outlier points have been removed. The computing device 300 can restore the corrected point cloud data to generate a final image.

[0086] After that, the computing device 300 can determine a blind spot region as a portion where the surface is smoothly restored in the final image. The blind spot region can include a site where imaging by the endoscope device has leaked inside the body.

[0087] FIG. 7 is a configuration diagram of a computing device according to an embodiment of the present disclosure.

[0088] Since the computing device 600 in FIG. 7 is similar to the computing device 300 in FIG. 2 described above, the description of overlapping content will be omitted below.

[0089] Referring to FIG. 7, the computing device 600 can generate a three-dimensional final image using basic data including a captured image 420 of the inside of the body. The captured image 420 can be taken through a camera provided in the endoscope scope or obtained in a virtual environment.

[0090] Unlike the computing device 300 in FIG. 2, the computing device 600 in FIG. 7 can infer and obtain depth information and pose information from the captured image 420. For this purpose, the computing device 600 can use a depth estimation model 610 and a pose estimation model 620.

[0091] Based on the captured image 420 of the interior of the body, the depth estimation model 610 can infer the depth information from the end of the endoscope scope to the inner surface of the body. Here, the endoscope scope can be directly inserted into the interior of an actual body or a phantom body, or can operate in a virtual environment.

[0092] The depth estimation model 610 can be trained using the captured image 420 corresponding to the depth information as the first training data. The depth estimation model 610 can include, but is not limited to, a generative artificial intelligence model, specifically a paired image-to-image translation (Pix2Pix) model.

[0093] Based on the captured image 420 of the interior of the body, the pose estimation model 620 can infer the 6D pose information of the end of the endoscope scope inserted into the interior of the body. The pose estimation model 620 can be trained using an image including feature points for the inner surface information of the body and second training data including the corresponding pose information.

[0094] On the other hand, in the inference process of the pose estimation model 620, a captured image 420 with many feature points can be input, or a captured image 420 with few feature points can also be input. For example, an image captured after applying a drug to the interior of the body can be input. In this case, since the drug is distributed by the bending of the interior of the body and the unevenness of the surface, the number of feature points can be increased.

[0095] When a captured image 420 with few feature points is input, the feature point generation model 630 can add feature points to the captured image 420 in order to improve the accuracy of pose estimation. As described above based on FIG. 2, the feature point generation model 630 can add feature points via a generative artificial intelligence model. Alternatively, the feature point generation model 630 can add feature points so as to represent the shape coated with the drug. Alternatively, as described above, the computing device 600 can use the captured image 420 itself with rich feature points or extract feature points from the captured image 420.

[0096] FIG. 8 is a flowchart showing a method for restoring an image of the inside of a body according to an embodiment of the present disclosure.

[0097] Referring to FIG. 8, the computing device 600 can obtain depth information from the end of the endoscope scope to the inner surface of the body based on a captured image of the inside of the body using a learned depth estimation model (S210). The depth estimation model can be learned by first learning data including a first learning image of the inside of the body and a depth image corresponding to the first learning image.

[0098] The computing device 600 can obtain pose information of the end of the endoscope scope based on the captured image using a learned pose estimation model (S220). The pose estimation model can be learned by second learning data including a second learning image including feature points for the inner surface information of the body and pose information corresponding to the second learning image.

[0099] Here, the computing device 600 can generate a corrected captured image using the feature points and the captured image generated using a feature point generation model learned to represent a pattern or texture formed on the inner surface of the body. The feature point generation model can be a model learned to represent a pattern or texture formed on the inner surface of the body using experimental images captured after applying a chemical inside the body. Alternatively, illustratively, as described above based on FIG. 3, it can be a model learned to generate a pattern or texture formed on the inner surface of the body using an adversarial generation network.

[0100] Based on the corrected captured image, the computing device 600 can acquire the posture information of the end of the endoscope scope using the learned posture estimation model.

[0101] The computing device 600 can generate a three-dimensional final image of the inside of the body using the depth information and the posture information (S230).

[0102] Thereafter, the computing device 600 can determine a portion where the surface is restored smoothly in the final image as a blind spot area.

[0103] The above description of the present disclosure is for illustrative purposes, and it will be understandable to those with ordinary knowledge in the technical field to which the present disclosure belongs that they can be easily modified into other specific forms without changing the technical idea and essential features of the present disclosure. Therefore, it should be understood that the embodiments described above are illustrative in all aspects and not restrictive. For example, each component described as a single type can also be implemented in a distributed manner, and similarly, components described as distributed can also be implemented in a combined form.

[0104] The scope of the present disclosure is determined by the claims described below rather than the above detailed description, and all changes or modifications derived from the meaning, scope, and equivalent concept of the claims should be construed as being included in the scope of the present disclosure.

Explanation of Symbols

[0105] 10 Endoscope system 100 Endoscope device 200, 300, 600 Computing devices 310, 630 Feature point generation models 610 Depth estimation model 620 Pose estimation model 210 Processor 220 Memory 230 Network section 400 Basic data 410 Pose information 420 Captured image 430 Depth information 500 Final image

Claims

1. 1. A method for reconstructing an image of an interior of a body, performed by a computing device including at least one processor, comprising: obtaining basic data including posture information of an end of an endoscope and an image of the inside of the body; generating a three-dimensional final image of the interior of the body by restoring the basic data using feature points including image information of the interior surface of the body for the basic data; A method comprising:

2. The method of claim 1 , wherein the feature points are generated by a feature point generation model trained to represent patterns or textures formed on the interior surface of the body.

3. The method of claim 2 , wherein the pattern or texture includes at least one of blood vessels, irregularities, wrinkles, and lesions.

4. The method of claim 2 , wherein the feature point generation model generates the feature points by representing a shape of a drug applied to the internal surface of the body.

5. The method of claim 1 , wherein the baseline data further includes depth information from an end of the endoscope to an internal surface of the body.

6. The step of generating a three-dimensional final image comprises: generating point cloud data using the basic data and the feature points; reconstructing the point cloud data to generate the final image; The method of claim 1 , comprising:

7. The step of generating a three-dimensional final image comprises: generating modified point cloud data by removing points that are present in an area outside a predetermined reference range from the point cloud data; recovering the modified point cloud data to generate the final image; The method of claim 6, comprising:

8. The method further includes determining a portion of the final image where the surface is smoothly restored as a blind spot area, The method according to claim 1 , wherein the blind spot area includes a portion of the inside of the body that is not imaged through the endoscope.

9. 1. A method for reconstructing an image of an interior of a body, performed by a computing device including at least one processor, comprising: acquiring depth information from an end of the endoscope to an internal surface of the body based on an image of the inside of the body using a trained depth estimation model; acquiring posture information of the end of the endoscope based on the captured image by using a trained posture estimation model; generating a final three-dimensional image of the interior of the body using the depth information and the pose information; A method comprising:

10. The method of claim 9 , wherein the depth estimation model is trained with first training data including first training images for an interior of a body and depth images corresponding to the first training images.

11. The method of claim 9 , wherein the pose estimation model is trained with second training images including feature points for internal surface information of the body and second training data including pose information corresponding to the second training images.

12. The step of acquiring pose information of the end of the endoscope using a trained pose estimation model includes: generating a corrected photographed image using the photographed image and feature points generated using a feature point generation model trained to represent a pattern or texture formed on an internal surface of a body; acquiring orientation information of the end of the endoscope using a learned orientation estimation model based on the corrected captured image; The method of claim 11 , comprising:

13. The method of claim 12 , wherein the feature generation model generates the feature points by representing a shape of a drug applied to the internal surface of the body.

14. 1. A computing device for reconstructing an image of an interior of a body, comprising: a memory for storing basic data including posture information of the end of the endoscope and an image of the inside of the body; a processor for generating a three-dimensional final image of the interior of the body by reconstructing the basic data using feature points including image information of the internal surface of the body relative to the basic data; 13. An apparatus comprising:

15. 1. A computing device for reconstructing an image of an interior of a body, comprising: A memory for storing photographed images of the inside of the body; a processor that acquires depth information from an end of the endoscope to an internal surface of the body based on the captured image using a trained depth estimation model, acquires orientation information of the end of the endoscope based on the captured image using a trained orientation estimation model, and generates a final three-dimensional image of the inside of the body using the depth information and the orientation information; 13. An apparatus comprising:

Images