Information processing device, and information processing program

Abstract

To autonomously stop tracking of a viewpoint of an endoscope on the basis of an estimation result of the viewpoint of the endoscope.SOLUTION: An information processing device 3 estimates whether or not tracking of a viewpoint of an endoscope 2 is possible from an endoscopic image 4, tracks the viewpoint of the endoscope 2 using a pseudo virtual endoscopic image 7 and a virtual endoscopic image 6 generated from the endoscopic image 4 when it is estimated that the viewpoint of the endoscope 2 can be tracked, determines a tracking result of the viewpoint of the endoscope 2 using a tracking destination virtual endoscopic image 6C and a tracking destination virtual depth image 17 generated from a bronchial model 5 and the endoscopic image 4 on the basis of the viewpoint of the endoscope 2 after tracking, and suspends tracking of the viewpoint of the endoscope 2 until a tracking start instruction is issued when it is determined that tracking has failed.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to an information processing device and an information processing program. [Background technology]

[0002] Patent Document 1 discloses an endoscope system that performs image matching using an endoscope image and a virtual endoscope image generated from three-dimensional image data, and estimates the position of the endoscope.

[0003] Patent Document 2 discloses a medical image observation support device that acquires an endoscopic image, generates a virtual image from three-dimensional image data, matches the endoscopic image with the virtual image by comparing branching section characteristic information in the endoscopic image with branching section characteristic information in the virtual image, and estimates the position of the endoscope when the endoscopic image was captured.

[0004] Patent Document 3 discloses an endoscope insertion assistance device that acquires an endoscopic image and three-dimensional image data, generates a virtualized endoscopic image from the three-dimensional image data, determines the similarity between the endoscopic image and the virtualized endoscopic image, and, if they are not similar, updates the position and posture information of the virtualized endoscope so that the virtualized endoscopic image matches the endoscopic image.

[0005] Patent Document 4 discloses a navigation system that determines whether or not a branch duct of a tubular organ has been observed, and if it is determined that the target branch duct has not been observed, determines whether or not the position of the target branch duct is further distal than the current viewpoint position, and if it is determined that the position is further distal than the current viewpoint position, waits until the target branch duct has been observed. [Prior art documents] [Patent documents]

[0006] [Patent Document 1] International Publication No. 2014 / 141968 [Patent Document 2] International Publication No. 2007 / 129493 [Patent Document 3] Japanese Patent Application Laid-Open No. 2012-165838 [Patent Document 4] International Publication No. 2016 / 009701 Summary of the Invention [Problem to be solved by the invention]

[0007] In endoscopic examinations and the like, an information processing device is sometimes used that estimates the viewpoint of an endoscope in a living body from an endoscopic image and notifies a medical professional of the direction in which the endoscope should travel so that the endoscope reaches the target location.

[0008] For example, if an endoscopic image contains an unknown structure area, which makes it difficult to identify the shape or pattern of the subject, compared to when it is not present, the information obtained from the endoscopic image will be reduced, and the accuracy of estimating the endoscopic viewpoint may decrease.

[0009] In this case, if the direction of travel of the endoscope is determined using the estimated viewpoint of the endoscope as is, a situation may arise in which an incorrect direction of travel is notified to a medical professional.

[0010] The present disclosure has been made in consideration of the above circumstances, and aims to provide an information processing device and an information processing program that can autonomously stop tracking the viewpoint of an endoscope based on the estimation result of the viewpoint of the endoscope. [Means for solving the problem]

[0011] An information processing device of a first aspect of the technology disclosed herein includes an estimation unit that estimates from an endoscopic image of a living body whether or not it is possible to track the viewpoint of the endoscope that captured the endoscopic image; a tracking unit that tracks the viewpoint of the endoscope using the endoscopic image and an image generated from a three-dimensional model of the living body if the estimation unit estimates that it is possible to track the viewpoint of the endoscope; a determination unit that determines the tracking result of the viewpoint of the endoscope by the tracking unit using a three-dimensional model image generated from the three-dimensional model based on the viewpoint of the endoscope after tracking and the endoscopic image; and a control unit that, if the determination unit determines that tracking has failed, controls to stop tracking the viewpoint of the endoscope until an instruction to start tracking is notified.

[0012] An information processing device of a second aspect according to the technology of the present disclosure is an information processing device according to the first aspect, wherein the three-dimensional model image is at least one of a tracked virtual endoscopic image of the living body, which is an image of the three-dimensional model viewed from the viewpoint of the endoscope after tracking, and a tracked virtual depth image, which is a depth image of the three-dimensional model corresponding to the position where the tracked virtual endoscopic image was obtained, and the determination unit determines the tracking result of the endoscope's viewpoint using a determination model that has been machine-learned in advance to output the tracked virtual endoscopic image, the tracked virtual depth image, and whether tracking of the endoscope's viewpoint from the endoscope was successful.

[0013] An information processing device of a third aspect according to the technology of the present disclosure is an information processing device according to the first aspect, wherein the three-dimensional model image is at least one of a tracked virtual endoscopic image of the living body, which is an image of the three-dimensional model viewed from the viewpoint of the endoscope after tracking, and a tracked virtual depth image, which is a depth image of the three-dimensional model corresponding to the position at which the tracked virtual endoscopic image was obtained, and the determination unit determines the tracking result of the endoscope's viewpoint by comparing the endoscopic image with structural information representing the structure of the living body obtained from at least one of the tracked virtual endoscopic image and the tracked virtual depth image.

[0014] An information processing device of a fourth aspect according to the technology of the present disclosure is an information processing device according to any one of the first to third aspects, wherein, when the estimation unit estimates that the endoscope's viewpoint cannot be tracked from the endoscopic image, the estimation unit temporarily suspends tracking of the endoscope's viewpoint until an endoscopic image in which it is estimated that the endoscope's viewpoint can be tracked is acquired.

[0015] In an information processing device of a fifth aspect according to the technology of the present disclosure, when the determination unit determines that tracking has failed in the information processing device of any one of the first to third aspects, the control unit controls the display device to display a virtual endoscopic image, which is an image that a medical professional refers to in aligning the endoscope and is an image of the three-dimensional model viewed from a viewpoint where tracking was successful.

[0016] An information processing device of a sixth aspect according to the technology of the present disclosure is the information processing device of the fifth aspect, in which the control unit controls the display device to display a virtual endoscopic image of the viewpoint that was last successfully tracked before tracking of the endoscope viewpoint failed.

[0017] An information processing device of a seventh aspect according to the technology of the present disclosure is an information processing device according to the fifth aspect, in which the control unit controls the display device to display a virtual endoscopic image including a bronchial bifurcation point that is closer to the viewpoint that was last successfully tracked before tracking of the viewpoint of the endoscope failed.

[0018] An information processing device of an eighth aspect of the technology disclosed herein controls the information processing device of the fourth aspect to notify whether the tracking status of the endoscope viewpoint is running, paused, or stopped.

[0019] An information processing device of a ninth aspect according to the technique of the present disclosure is the information processing device according to the eighth aspect, wherein the control unit notifies the tracking state of the viewpoint of the endoscope by changing the display form of the three-dimensional model image.

[0020] An information processing device of a 10th aspect of the technology disclosed herein is an information processing device of the 9th aspect, in which when the tracking state of the endoscope's viewpoint is paused, the control unit performs control to notify the user of the reason why tracking of the endoscope's viewpoint has been paused.

[0021] An information processing program of an eleventh aspect related to the technology of the present disclosure is a program for causing a computer to execute a process of estimating, from an endoscopic image of a living body, whether or not it is possible to track the viewpoint of the endoscope that captured the endoscopic image; if it is estimated that it is possible to track the viewpoint of the endoscope, tracking the viewpoint of the endoscope using the endoscopic image and an image generated from a three-dimensional model of the living body; determining the tracking result of the viewpoint of the endoscope using a three-dimensional model image generated from the three-dimensional model based on the viewpoint of the endoscope after tracking and the endoscopic image; and if it is determined that tracking has failed, stopping tracking of the viewpoint of the endoscope until an instruction to start tracking is notified. [Effects of the Invention]

[0022] According to the present disclosure, tracking of the viewpoint of the endoscope can be autonomously stopped based on the estimation result of the viewpoint of the endoscope. [Brief explanation of the drawings]

[0023] [Figure 1] FIG. 1 illustrates an example of the configuration of an information processing system. [Figure 2] FIG. 10 is a diagram showing an example of an endoscopic image. [Figure 3] FIG. 10 is a diagram showing an example of an endoscopic image including a structure-unknown region. [Figure 4] FIG. 1 illustrates an example of machine learning for a traceability estimation model. [Figure 5] This is an example of a bronchial model. [Figure 6] FIG. 10 is a diagram illustrating an example of a virtual endoscopic image. [Figure 7] FIG. 10 is a diagram illustrating an example of a method for estimating a viewpoint difference between images. [Figure 8] FIG. 10 is a diagram illustrating an example of machine learning of a viewpoint difference estimation model. [Figure 9] FIG. 10 is a diagram showing an example of tracking the viewpoint of an endoscope. [Figure 10] 10A and 10B are diagrams illustrating an example of a determination method for determining the tracking result of the viewpoint of the endoscope. [Figure 11] FIG. 1 is a diagram illustrating an example of the configuration of an information processing device configured by a computer. [Figure 12] 10 is a flowchart illustrating an example of the flow of a tracking process. [Figure 13] 10 is a flowchart illustrating an example of the flow of a viewpoint tracking process. [Figure 14] 10A and 10B are diagrams illustrating examples of display forms of a tracked virtual endoscopic image. [Figure 15] 10A and 10B are diagrams illustrating examples of display forms of a tracked-destination virtual endoscopic image when the tracking state is shifted. [Figure 16] FIG. 10 is a diagram showing an example of displaying a structure-unknown region in an endoscopic image. [Figure 17] 10 is a flowchart showing a modified example of the flow of the tracking process. [Figure 18] FIG. 10 is a diagram illustrating a modified example of the tracking result determination model. [Figure 19] FIG. 10 is a diagram illustrating an example of a method for estimating similarity between images. [Figure 20] 10 is a flowchart showing an example of the flow of a viewpoint tracking process according to the second embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0024] Hereinafter, the present embodiment will be described with reference to the drawings. The same components and processes are denoted by the same reference numerals throughout the drawings, and duplicated explanations will be omitted. The dimensional proportions in the drawings are exaggerated for the sake of explanation, and may differ from the actual proportions.

[0025] First Embodiment FIG. 1 is a diagram illustrating an example of the configuration of an information processing system 1 according to a first embodiment. As shown in FIG. 1, the information processing system 1 includes an endoscope 2 and an information processing device 3. The tip of the endoscope 2 is provided with an imaging device such as a camera. The information processing system 1 transmits an image captured by the endoscope 2, i.e., an endoscopic image 4, to the information processing device 3. FIG. 2 is a diagram illustrating an example of the endoscopic image 4 captured by the endoscope 2.

[0026] There are no restrictions on the location of the living body into which the endoscope 2 is inserted, and it may be, for example, an endoscope 2 for the digestive organs or a ureteroscope, but as an example, the endoscope 2 of the present disclosure will be described as a bronchoscope inserted into the bronchi of the subject.

[0027] When inserting the endoscope 2 into the bronchi and advancing it to the position to be examined, the bronchi have many branching points and similar shapes. Therefore, a navigation device may be used to notify the medical staff operating the endoscope 2 of support information such as the position in the bronchi where the endoscope 2 is pointing and the direction in which the endoscope 2 should be advanced to reach the position to be examined (hereinafter referred to as the "target position").

[0028] The information processing device 3 is an example of a navigation device that notifies a medical professional of the traveling direction of the endoscope 2. The information processing device 3 includes functional units, such as an acquisition unit 3A, an estimation unit 3B, a model acquisition unit 3C, a storage unit 3D, a generation unit 3E, a tracking unit 3F, a determination unit 3G, a control unit 3H, and a notification unit 3J.

[0029] The acquisition unit 3A acquires an endoscopic image 4 from the endoscope 2.

[0030] The endoscopic image 4 displays an image of the inside of the subject's bronchi, but may contain an area of ​​unknown structure. The area of ​​unknown structure refers to an area in the endoscopic image 4 that cannot be used to estimate the position and posture of the endoscope 2 when the information processing device 3 uses the endoscopic image 4 to estimate the position and posture of the endoscope 2.

[0031] For example, the structure-unknown region includes a region with bubbles attached, a region where the endoscope 2 is out of focus, a region where the image is blocked by a deposit attached to the lens of the endoscope 2, a reflective region where light is reflected by lighting, a blown-out highlight region, a crushed-black region, etc. In other words, the structure-unknown region is a general term for a region that makes it difficult to identify the subject.

[0032] Fig. 3 is a diagram showing an example of an endoscopic image 4 including an area of ​​unknown structure. In the area indicated by the display frame 28 in endoscopic image 4-1 shown in Fig. 3, bubbles adhering to the bronchi, which are an example of an area of ​​unknown structure, can be seen. In the area indicated by the display frame 28 in endoscopic image 4-2, reflection of light due to illumination can be seen. Furthermore, endoscopic image 4-3 shows an example of an out-of-focus image, in which the bronchi are not in focus because the endoscope 2 is too close to them.

[0033] When the information processing device 3 estimates the position and posture of the endoscope 2, if an endoscopic image 4 is used that includes an area of ​​unknown structure that does not accurately represent the pattern of the bronchial wall (referred to as "bronchial texture") due to the shape of the bronchi or the state of the blood vessels and tissues, the estimation accuracy may be reduced compared to when an endoscopic image 4 that does not include an area of ​​unknown structure is used.

[0034] Therefore, the estimation unit 3B estimates whether or not it is possible to track the position and posture of the endoscope 2 that captured the endoscopic image 4 from the endoscopic image 4, based on attribute information of the structure-unknown region, such as the presence, type, and range of the structure-unknown region included in the endoscopic image 4. In other words, when the estimation unit 3B estimates the position and posture of the endoscope 2 from the endoscopic image 4, it estimates whether or not the error in the estimated position and posture of the endoscope 2 with respect to the actual position and posture of the endoscope 2 is within an allowable range.

[0035] The attitude of the endoscope 2 refers to the direction in which the endoscope 2 is facing. Since the shooting range of the endoscopic image 4 captured by the endoscope 2 is determined by the position and attitude of the endoscope 2, hereinafter the position and attitude of the endoscope 2 will be referred to as the "viewpoint of the endoscope 2." Furthermore, whether or not the viewpoint of the endoscope 2 can be tracked will be expressed using the expression "trackability of the endoscope 2."

[0036] The traceability of the viewpoint of the endoscope 2 is determined by the score output from the traceability estimation model 21 when, for example, an endoscopic image 4 is input to the traceability estimation model 21. The score is a value that represents the degree of traceability of the endoscope 2, and for example, the larger the score, the higher the probability that the viewpoint of the endoscope 2 can be tracked from the input endoscopic image 4. In the present disclosure, as an example, the score represents the probability that the viewpoint of the endoscope 2 can be tracked.

[0037] The traceability estimation model 21 is a model generated in advance by, for example, machine learning, and is stored in advance in the storage unit 3 D. Although the information processing device 3 shown in Fig. 1 includes the storage unit 3 D, the information processing device 3 does not necessarily need to include the storage unit 3 D, and an external device other than the information processing device 3, such as a data server, may be used as the storage unit 3 D.

[0038] 4 is a diagram showing an example of machine learning of the traceability estimation model 21. For example, a medical professional prepares in advance a plurality of pieces of training data (hereinafter referred to as "estimated training data") in which a score is associated with each of endoscopic images 4 captured in advance. The medical professional associates a score with each endoscopic image 4 based on the state of the structure-unknown region, such as the presence, type, and range of the structure-unknown region in the endoscopic image 4.

[0039] Then, machine learning of the traceability estimation model 21 is performed so that when an endoscopic image 4 included in the estimated learning data is input, the score associated with the input endoscopic image 4 is output from the traceability estimation model 21. That is, in the estimated learning data, the endoscopic image 4 is the input data, and the score is the training data.

[0040] For example, a CNN (Convolutional Neural Network) is used for the traceability estimation model 21. For example, deep learning is used as the machine learning method.

[0041] The image used by the CPU 10A when estimating the traceability of the endoscope 2 is not limited to the endoscopic image 4. The CPU 10A may estimate the traceability of the endoscope 2 using an image generated from the endoscopic image 4 or an image other than the endoscopic image 4 in which the shape of the bronchi can be recognized. The CPU 10A may also estimate the traceability of the endoscope 2 by combining a plurality of types of images.

[0042] For example, the CPU 10A may estimate the traceability of the endoscope 2 by using a depth image corresponding to the endoscopic image 4. The depth image corresponding to the endoscopic image 4 is an image that represents the shape of the bronchus at the position where the tip of the endoscope 2 is located as distance data from the bronchial entrance, using distance data obtained from a depth sensor that measures the distance from the bronchial entrance, for example. As the depth sensor, for example, a stereo camera, a TOF (Time of Flight) sensor, a LiDAR (Light Detection and Ranging), a photoelectric sensor, or the like is used. Furthermore, the CPU 10A may estimate the traceability of the endoscope 2 by combining the endoscopic image 4 and the depth image, or may estimate the traceability of the endoscope 2 by using position information of a structure-unknown region or brightness information of the endoscopic image 4.

[0043] In this case, the image used to estimate the traceability of the endoscope 2 is used as input data, and machine learning of the traceability estimation model 21 is performed using estimated learning data with the score as training data.

[0044] 1 compares the score output from the traceability estimation model 21 with a predetermined reference score. The reference score is a value that represents the minimum score at which the estimation error of the viewpoint of the endoscope 2 is considered to be within an acceptable range when tracking the viewpoint of the endoscope 2, and is set in advance by, for example, a medical professional and stored in advance in the storage unit 3D.

[0045] If the score is less than the standard score, even if the viewpoint of the endoscope 2 is tracked from the endoscopic image 4 acquired by the acquisition unit 3A, there is a high probability that the tracked viewpoint of the endoscope 2 will be incorrect. Therefore, the estimation unit 3B temporarily suspends tracking of the viewpoint of the endoscope 2 by not requesting the tracking unit 3F to track the viewpoint of the endoscope 2 until the acquisition unit 3A acquires an endoscopic image 4 that is estimated to enable tracking of the viewpoint of the endoscope 2. On the other hand, if the score is equal to or greater than the standard score, the estimation unit 3B requests the tracking unit 3F to track the viewpoint of the endoscope 2.

[0046] The tracking unit 3F, which has received a request to track the viewpoint of the endoscope 2 from the estimation unit 3B, tracks the viewpoint of the endoscope 2 using the endoscopic image 4 and an image generated from a three-dimensional model of the subject's living body.

[0047] Specifically, the tracking unit 3F estimates the viewpoint difference (sometimes simply referred to as "viewpoint difference between images") between the endoscopic image 4 and a virtual endoscopic image 6 generated from a bronchial model 5 of the subject, which will be described later. The tracking unit 3F tracks the viewpoint of the endoscope 2 using the estimated viewpoint difference between the images. That is, the tracking unit 3F tracks the viewpoint of the endoscope 2 when the score is equal to or greater than a reference score. The method of estimating the viewpoint difference between images in the tracking unit 3F will be described in detail later.

[0048] Meanwhile, a bronchial model 5 representing the shape of the bronchi of the subject is used to notify the medical staff of the traveling direction of the endoscope 2. The bronchial model 5 is an example of a three-dimensional model that is generated in advance by, for example, CT (Computed Tomography) before an endoscopic examination is performed. Fig. 5 shows an example of the bronchial model 5. The bronchial model 5 is generated in advance for each subject and is stored in advance in a storage unit 3D.

[0049] The model acquisition unit 3C in FIG. 1 acquires a bronchial model 5 of a subject to be examined from the storage unit 3D.

[0050] Using the bronchial model 5 acquired by the model acquisition unit 3C, the generation unit 3E generates an image of the bronchial model 5 viewed from a predetermined position inside the bronchial model 5, i.e., a virtual endoscopic image 6. As the predetermined position inside the bronchial model 5, for example, a position where the tip of the endoscope 2 is estimated to be located is used. FIG. 6 is a diagram showing an example of the virtual endoscopic image 6 generated by the generation unit 3E using the bronchial model 5.

[0051] The virtual endoscopic image 6 does not necessarily have to be a two-dimensional image, but may be a three-dimensional image generated using three-dimensional imaging data obtained by CT, MRI (Magnetic Resonance Imaging), or the like.

[0052] Because the bronchial model 5 is a three-dimensional model generated from data representing the shape of the subject's bronchi, it does not have bubbles or foreign objects attached, as is the case with the subject's actual bronchi. Furthermore, because the virtual endoscopic image 6 is generated by data processing using the bronchial model 5, it does not suffer from out-of-focus, reflections of lighting, or overexposure or underexposure. In other words, the virtual endoscopic image 6 is an image that faithfully reproduces how the shape of the bronchial model 5 appears when viewed from a specified viewpoint. Because the viewpoint of the endoscope 2 relative to the bronchial model 5 can be freely set, the generation unit 3E can generate virtual endoscopic images 6 from any viewpoint.

[0053] The tracking unit 3F in Figure 1 estimates the viewpoint difference between the viewpoint at which the endoscopic image 4 was obtained and the viewpoint at which the virtual endoscopic image 6 was obtained, based on the deviation in the shape and texture of the bronchi contained in the endoscopic image 4, which is estimated to be capable of tracking the viewpoint of the endoscope 2, and any one of the virtual endoscopic images 6 generated by the generation unit 3E.

[0054] Specifically, a method for estimating the viewpoint difference between images in the tracking unit 3F will be described. FIG. 7 is a diagram showing an example of a method for estimating the viewpoint difference between images. The tracking unit 3F estimates the viewpoint difference between the endoscopic image 4 and a virtual endoscopic image 6 obtained when a bronchial model 5 is viewed from a specified viewpoint using a viewpoint difference estimation model 22 that has been machine-learned in advance so as to output the viewpoint difference between the endoscopic image 4 and the virtual endoscopic image 6. Note that instead of the endoscopic image 4, an image generated from the endoscopic image 4 may be input to the viewpoint difference estimation model 22. An image generated from the endoscopic image 4 is also an example of the endoscopic image 4.

[0055] As can be seen from the fact that the viewpoint difference estimation model 22 outputs the viewpoint difference between the input endoscopic image 4 and the virtual endoscopic image 6, the virtual endoscopic image 6 input to the viewpoint difference estimation model 22 may be a virtual endoscopic image 6 obtained from a viewpoint different from the viewpoint at which the endoscopic image 4 was captured.

[0056] The tracking unit 3F inputs the endoscopic image 4 into the image transformation model 20 to transform the endoscopic image 4 into a pseudo virtual endoscopic image 7.

[0057] The pseudo virtual endoscopic image 7 is an image obtained by converting the endoscopic image 4 into an expression format that makes it look like the virtual endoscopic image 6. Specifically, the pseudo virtual endoscopic image 7 is an image obtained by removing the texture of the bronchi and regions with unknown structures from the endoscopic image 4, making the shape of the bronchi easier to grasp than the endoscopic image 4.

[0058] The image conversion model 20 is a model generated in advance using a GAN (Generative Adversarial Network) that has been trained to output a pseudo-virtual endoscopic image 7 that resembles a virtual endoscopic image 6 from an endoscopic image 4, for example, and is pre-stored in the memory unit 3D.

[0059] The tracking unit 3F inputs the virtual endoscopic image 6 and pseudo virtual endoscopic image 7 obtained as described above into the viewpoint difference estimation model 22, and estimates the viewpoint difference between the endoscopic image 4, which is the image from which the pseudo virtual endoscopic image 7 is generated, and the virtual endoscopic image 6. In other words, the pseudo virtual endoscopic image 7 is also an example of the endoscopic image 4 input to the viewpoint difference estimation model 22.

[0060] The reason why an image obtained by approximating the endoscopic image 4 to the virtual endoscopic image 6, such as the pseudo-virtual endoscopic image 7, was used as input to the viewpoint difference estimation model 22, rather than the endoscopic image 4, is that it takes into consideration the ease of preparing the training data (hereinafter referred to as "viewpoint difference training data") used for machine learning of the viewpoint difference estimation model 22.

[0061] To perform machine learning of the viewpoint difference estimation model 22, for example, two endoscopic images 4 with different viewpoints and information about the viewpoint difference between each of the endoscopic images 4 are used. However, due to their characteristics, endoscopes 2 are being made smaller so that they can be inserted into as many bronchi as possible, making it difficult to attach separate sensors for measuring the viewpoint of the endoscope 2, such as gyro sensors or motion sensors. Furthermore, because the endoscope 2 is inserted inside the body, the viewpoint of the endoscope 2 can only be ascertained through endoscopic images 4 with a limited angle of view, making it difficult to obtain the viewpoint difference from the endoscopic images 4.

[0062] On the other hand, as already explained, the viewpoint of the endoscope 2 relative to the bronchial model 5 can be freely set, and furthermore, since the bronchial model 5 is a three-dimensional model based on examination data such as CT, the correct viewpoint difference can be obtained by numerical calculation.

[0063] Therefore, using a combination of two virtual endoscopic images 6 with shifted viewpoints and information regarding the viewpoint difference between each virtual endoscopic image 6 as viewpoint difference learning data makes it possible to prepare a larger amount of viewpoint difference learning data more easily than using a combination of two endoscopic images 4 with shifted viewpoints and information regarding the viewpoint difference between each endoscopic image 4.

[0064] For the above reasons, two virtual endoscopic images 6 with different viewpoints are used as input data for the viewpoint difference learning data of the viewpoint difference estimation model 22. Therefore, if the endoscopic image 4 is input directly to the viewpoint difference estimation model 22, the estimation accuracy of the viewpoint difference in the viewpoint difference estimation model 22 may decrease. Therefore, the endoscopic image 4 is converted into a pseudo virtual endoscopic image 7 and then input to the viewpoint difference estimation model 22.

[0065] Naturally, the endoscopic image 4 may be input directly to the viewpoint difference estimation model 22. In this case, machine learning of the viewpoint difference estimation model 22 can be performed using viewpoint difference learning data that combines the endoscopic image 4, a virtual endoscopic image 6 whose viewpoint is shifted from that of the endoscopic image 4, and information on the viewpoint difference between each image.

[0066] The viewpoint difference estimation model 22 is a model generated in advance by such machine learning, and is stored in advance in the storage unit 3D.

[0067] FIG. 8 is a diagram showing an example of machine learning of the viewpoint difference estimation model 22. As shown in FIG.

[0068] For two virtual endoscopic images 6 each having a different viewpoint, a plurality of viewpoint difference learning data is prepared in advance, which associates the viewpoint difference of each virtual endoscopic image 6. In Fig. 8, a virtual endoscopic image 6A and a virtual endoscopic image 6B represent two virtual endoscopic images 6 each having a different viewpoint.

[0069] Then, machine learning of the viewpoint difference estimation model 22 is performed so that when the virtual endoscopic images 6A and 6B included in the viewpoint difference training data are input, the viewpoint difference associated with each input is output from the viewpoint difference estimation model 22. For example, CNN or the like is used for the viewpoint difference estimation model 22. For example, deep learning or the like is used as the machine learning method.

[0070] That is, in the viewpoint difference training data, the virtual endoscopic images 6A and 6B are input data, and the viewpoint difference is training data.

[0071] Note that machine learning of the viewpoint difference estimation model 22 may be performed using the range of the bronchial model 5 visible from each viewpoint (referred to as a "partial bronchial model") instead of one of the virtual endoscopic images 6A and 6B. That is, one of the virtual endoscopic images 6 may be expressed as three-dimensional data. In this case, the partial bronchial model at the specified viewpoint is used instead of the virtual endoscopic image 6 in the method of estimating the viewpoint difference between images shown in FIG. 7.

[0072] Furthermore, supplemental information related to the position of the endoscope 2 may be added as input data for the viewpoint difference learning data of the viewpoint difference estimation model 22. In this case, the information processing device 3 estimates the viewpoint difference of the endoscope 2 using the endoscopic image 4, the virtual endoscopic image 6, the supplemental information, and the viewpoint difference estimation model 22. For example, the supplemental information may be a depth image of the bronchial model 5 at the viewpoint position where the virtual endoscopic images 6A and 6B were obtained.

[0073] By adding supplemental information to the input of the viewpoint difference estimation model 22, the estimation accuracy of the viewpoint difference may be improved compared to when the viewpoint difference of the endoscope 2 is estimated without using the supplemental information.

[0074] Since the virtual endoscopic image 6 is an image generated from the bronchial model 5, the viewpoint of the virtual endoscopic image 6 is known to the tracking unit 3F. Therefore, the tracking unit 3F can identify the viewpoint of the endoscope 2 from the viewpoint of the virtual endoscopic image 6 and the viewpoint difference between the estimated endoscopic image 4 and the virtual endoscopic image 6.

[0075] Fig. 9 is a diagram showing an example of tracking the viewpoint of the endoscope 2. In Fig. 9, mark 25 represents a predetermined position inside the bronchial model 5 (for example, an approximate position where the endoscope 2 is thought to be located), and mark 26 represents the actual position of the endoscope 2. Based on the viewpoint difference between the images, the tracking unit 3F moves the position of mark 25 to the position of mark 26, and is able to assume the same posture on the bronchial model 5 as the endoscope 2 whose position is represented by mark 26. In this way, the tracking unit 3F tracks the viewpoint of the endoscope 2 based on the estimated viewpoint difference.

[0076] If the viewpoint of the endoscope 2 represented by the endoscopic image 4 is known, a tracked virtual endoscopic image 6C, which is a virtual endoscopic image 6 of the bronchial model 5 viewed from the viewpoint after tracking, can be generated from the bronchial model 5.

[0077] The tracked virtual endoscopic image 6C is an example of a three-dimensional model image generated from a three-dimensional model based on the viewpoint of the endoscope 2 after tracking. In addition, the tracked virtual endoscopic image 6C is an image obtained as a result of tracking the viewpoint of the endoscope 2, and is therefore also a tracking result of the viewpoint of the endoscope 2. The tracking unit 3F notifies the determination unit 3G of the generated tracked virtual endoscopic image 6C.

[0078] The determination unit 3G shown in Figure 1, which has received the tracked virtual endoscopic image 6C, determines whether the endoscopic image 4 and the tracked virtual endoscopic image 6C are similar images, using the endoscopic image 4 used to track the viewpoint of the endoscope 2 by the tracking unit 3F and the tracked virtual endoscopic image 6C received from the tracking unit 3F.

[0079] The endoscopic image 4 and the tracked virtual endoscopic image 6C being similar means that the difference in the shapes of the bronchi shown in the endoscopic image 4 and the tracked virtual endoscopic image 6C falls within a predetermined tolerance range within which the images can be considered similar. The fact that the tracked virtual endoscopic image 6C is similar to the endoscopic image 4 means that the bronchial model 5 is viewed from the same viewpoint as the current viewpoint of the endoscope 2, and therefore indicates that tracking of the viewpoint of the endoscope 2 has been successful. On the other hand, the fact that the tracked virtual endoscopic image 6C is not similar to the endoscopic image 4 means that the bronchial model 5 is viewed from a viewpoint different from the current viewpoint of the endoscope 2, and therefore indicates that tracking of the viewpoint of the endoscope 2 has failed. In other words, the determination unit 3G determines whether the tracking unit 3F has successfully tracked the viewpoint of the endoscope.

[0080] 10 is a diagram showing an example of a determination method for determining the tracking result of the viewpoint of the endoscope 2. The determination unit 3G determines the tracking result of the viewpoint of the endoscope 2 using a tracking result determination model 23 that has been machine-learned in advance to output whether tracking of the viewpoint of the endoscope 2 has been successful for the endoscopic image 4, the tracked virtual endoscopic image 6C, and the tracked virtual depth image 17, which is a depth image of the bronchial model 5 corresponding to the position where the tracked virtual endoscopic image 6C was obtained (also referred to as a "depth image corresponding to the position of the viewpoint of the endoscope 2").

[0081] The tracked virtual depth image 17 is an image that represents the distance from, for example, the entrance of the bronchial model 5 to the vicinity of the viewpoint where the tracked virtual endoscopic image 6C is generated. In other words, the tracked virtual depth image 17 is a depth image of the bronchial model 5 that corresponds to the position where the tracked virtual endoscopic image 6C is obtained.

[0082] As already explained, the determination of the tracking result of the viewpoint of the endoscope 2 is performed based on the similarity between the endoscopic image 4 and the tracking target virtual endoscopic image 6C, and therefore it is not necessarily necessary to use the tracking target virtual depth image 17 to determine the tracking result of the viewpoint of the endoscope 2. However, because the tracking target virtual depth image 17 serves as supplementary information related to the position of the endoscope 2, adding the tracking target virtual depth image 17 to the determination of the tracking result of the viewpoint of the endoscope 2 may improve the determination accuracy compared to determining the tracking result of the viewpoint of the endoscope 2 from only the endoscopic image 4 and the tracking target virtual endoscopic image 6C.

[0083] Furthermore, the tracking result of the viewpoint of the endoscope 2 may be determined based on the endoscopic image 4 and the tracking target virtual depth image 17. However, adding the tracking target virtual endoscopic image 6C to the determination of the tracking result of the viewpoint of the endoscope 2 may improve the accuracy of the determination compared to determining the tracking result of the viewpoint of the endoscope 2 based only on the endoscopic image 4 and the tracking target virtual depth image 17.

[0084] Therefore, in the following explanation, an example will be described in which the endoscopic image 4, the tracked virtual endoscopic image 6C, and the tracked virtual depth image 17 are input to the tracking result determination model 23 to determine the tracking result of the viewpoint of the endoscope 2.

[0085] The tracking result determination model 23 may be a model that outputs the tracking result of the viewpoint of the endoscope 2 in response to the input of the endoscopic image 4 and the tracking target virtual depth image 17.

[0086] The tracked target virtual endoscopic image 6C and the tracked target virtual depth image 17 generated from the bronchial model 5 are examples of three-dimensional model images.

[0087] The judgment learning data is used for machine learning of the tracking result judgment model 23 that outputs the tracking result of the viewpoint of the endoscope 2 from the endoscopic image 4, the tracked virtual endoscopic image 6C, and the tracked virtual depth image 17.

[0088] To generate the judgment learning data, a plurality of paired data are prepared in advance, which associates an endoscopic image 4 captured in advance with a virtual endoscopic image 6 generated from a viewpoint of the bronchial model 5 corresponding to the viewpoint at which the endoscopic image 4 was captured. The viewpoint at which the endoscopic image 4 in the paired data was captured and the viewpoint at which the virtual endoscopic image 6 was generated do not necessarily have to match, as long as they fall within an acceptable range at which they can be considered to be the same viewpoint.

[0089] For each of the prepared paired data, the medical professional randomly shifts the viewpoint of the bronchial model 5 used to generate the virtual endoscopic image 6 included in the paired data within a first range. The first range refers to a range in which the deviation between the viewpoint after shifting the bronchial model 5 and the viewpoint of the endoscope 2 that captured the endoscopic image 4 included in the same paired data as the virtual endoscopic image 6 generated from the bronchial model 5 before the viewpoint was shifted falls within an allowable range and the respective viewpoints can be considered to be the same. In other words, being within the first range refers to a range in which it can be considered that tracking of the viewpoint of the endoscope 2 has been successful.

[0090] The medical professional uses the bronchial model 5 to generate a virtual endoscopic image 6 obtained when the bronchial model 5 is viewed from a viewpoint after shifting within the first range, and generates a depth image corresponding to the viewpoint after shifting within the first range, and associates the virtual endoscopic image 6 and depth image with the endoscopic image 4 of the pair data from which the virtual endoscopic image 6 and depth image with the viewpoint shifted are generated. Furthermore, the medical professional associates a label of "tracking successful" with the endoscopic image 4 and the combination of the virtual endoscopic image 6 and depth image generated by shifting the viewpoint within the first range, and generates judgment learning data indicating that the endoscope 2 has been successfully tracked.

[0091] Meanwhile, for each of the prepared paired data, the medical professional randomly shifts the viewpoint of the bronchial model 5 used to generate the virtual endoscopic image 6 included in the paired data to a range (referred to as the "second range") beyond the first range. The second range refers to the range in which the deviation between the viewpoint after shifting the bronchial model 5 and the viewpoint of the endoscope 2 that captured the endoscopic image 4 included in the same paired data as the virtual endoscopic image 6 generated from the bronchial model 5 before the viewpoint was shifted exceeds the allowable range, and the respective viewpoints can be considered to be different. In other words, the second range refers to the range in which it can be considered that tracking of the viewpoint of the endoscope 2 has failed.

[0092] The medical professional uses the bronchial model 5 to generate a virtual endoscopic image 6 obtained when the bronchial model 5 is viewed from a viewpoint after shifting to the second range, and generates a depth image corresponding to the viewpoint after shifting to the second range, and associates the virtual endoscopic image 6 with the shifted viewpoint and the depth image with the endoscopic image 4 of the pair data from which the virtual endoscopic image 6 with the shifted viewpoint and the depth image were generated. Furthermore, the medical professional associates a label of "tracking failed" with the endoscopic image 4 and the combination of the virtual endoscopic image 6 with the viewpoint shifted to the second range and the depth image, and generates judgment learning data indicating that tracking of the endoscope 2 has failed.

[0093] As a result of the above, judgment learning data in which the endoscope 2 is successfully tracked and judgment learning data in which the endoscope 2 is unsuccessfully tracked are generated and used for machine learning of the tracking result judgment model 23. In the judgment learning data, the endoscopic image 4, the virtual endoscopic image 6, and the depth image are input data, and the labels "tracking successful" and "tracking unsuccessful" are training data.

[0094] In addition, when the tracking result determination model 23 is a model that outputs a tracking result of the viewpoint of the endoscope 2 in response to the input of the endoscopic image 4 and the virtual depth image 17 of the tracking target, the medical professional can generate determination learning data in which a label of "tracking successful" or "tracking failed" is associated with a combination of the endoscopic image 4 and a depth image obtained by shifting the viewpoint, depending on the degree of shift of the viewpoint. In addition, when the tracking result determination model 23 is a model that outputs a tracking result of the viewpoint of the endoscope 2 in response to the input of the endoscopic image 4 and the virtual endoscopic image 6C of the tracking target, the medical professional can generate determination learning data in which a label of "tracking successful" or "tracking failed" is associated with a combination of the endoscopic image 4 and the virtual endoscopic image 6 obtained by shifting the viewpoint, depending on the degree of shift of the viewpoint. Needless to say, the endoscopic image 4 may be replaced with a depth image corresponding to the endoscopic image 4.

[0095] The determination unit 3G notifies the control unit 3H of the tracking result of the viewpoint of the endoscope 2 determined in this way.

[0096] 1, when the tracking result of the viewpoint of the endoscope 2 received from the determination unit 3G is "tracking successful," the control unit 3H controls other functional units to continue tracking of the viewpoint of the endoscope 2. Furthermore, the control unit 3H determines the traveling direction for the endoscope 2 to reach the target location using the bronchial model 5 based on the viewpoint of the endoscope 2 after tracking, and controls the notification unit 3J to notify the medical staff of the determined traveling direction of the endoscope 2.

[0097] On the other hand, if the tracking result of the viewpoint of the endoscope 2 received from the judgment unit 3G is "tracking failed," the control unit 3H performs control to stop tracking of the viewpoint of the endoscope 2 until an instruction to start tracking is received from the medical professional.

[0098] The control unit 3H also controls the notification unit 3J to notify the medical staff of the tracking status of the viewpoint of the endoscope 2, such as whether the tracking process of the viewpoint of the endoscope 2 is being executed, stopped, or paused. The control unit 3H also controls the notification unit 3J to display various images used in the tracking process of the viewpoint of the endoscope 2 and various images generated by the tracking process of the viewpoint of the endoscope 2, such as the endoscopic image 4 and the tracking target virtual endoscopic image 6C.

[0099] The notification unit 3J, under the control of the control unit 3H, notifies medical personnel of various information useful for navigating the endoscope 2, for example, by displaying the tracking status of the viewpoint of the endoscope 2 and various images on a display, or by notifying the direction of travel of the endoscope 2 by voice.

[0100] The machine learning of various models, such as the image transformation model 20, the tracking possibility estimation model 21, the viewpoint difference estimation model 22, and the tracking result determination model 23, used in the processing of tracking the viewpoint of the endoscope 2, and the generation of the bronchial model 5 do not necessarily have to be performed by the information processing device 3, but may be performed by another computer with a higher processing capacity than the information processing device 3. In this case, the information processing device 3 will acquire from the other computer and use the various models generated by the other computer.

[0101] The information processing device 3 having such functions and shown in Fig. 1 is configured by, for example, a computer. Fig. 11 is a diagram showing an example of the configuration of the information processing device 3 configured by a computer.

[0102] 11, the information processing device 3 includes a control unit 10, an interface (I / F) unit 12, an operation unit 13, a notification unit 14, and a storage unit 15. The control unit 10, the I / F unit 12, the operation unit 13, the notification unit 14, and the storage unit 15 are connected via a bus 16 so as to be able to exchange various information with one another.

[0103] The control unit 10 controls the operation of the information processing device 3 based on instructions from a medical professional. The control unit 10 is an example of a processor, and includes a CPU (Central Processing Unit) 10A, a ROM (Read Only Memory) 10B, and a RAM (Random Access Memory) 10C, which are responsible for processing the various functional units of the information processing device 3 shown in Fig. 1. The ROM 10B stores in advance various programs including a control program 11 that the CPU 10A reads to determine the tracking result of the viewpoint of the endoscope 2, and various parameters that the CPU 10A references when controlling the operation of the information processing device 3. The RAM 10C is used as a temporary work area for the CPU 10A.

[0104] The I / F unit 12 uses wireless communication or wired communication to exchange various information with the endoscope 2. The information processing device 3 acquires the endoscopic image 4 from the endoscope 2 through the I / F unit 12.

[0105] The operation unit 13 is used by a medical professional to input, for example, instructions and various information regarding tracking of the viewpoint of the endoscope 2. There are no restrictions on the type of operation on the operation unit 13, and operations can be accepted using, for example, a switch, a touch panel, a touch pen, a keyboard, a mouse, etc.

[0106] The notification unit 14 notifies the medical staff of information processed by the control unit 10, such as the endoscopic image 4, the tracked virtual endoscopic image 6C, and information related to tracking of the viewpoint of the endoscope 2.

[0107] To notify information means to make the information recognizable to medical personnel. Therefore, for example, displaying information on a display (not shown), which is an example of a display device, printing information on a recording medium such as paper using an image forming device (not shown), and notifying information by voice using a speaker (not shown) are all examples of information notification by the notifying unit 14. Note that transmitting information through the I / F unit 12 is also an example of information notification. The information processing device 3 in the present disclosure notifies information by displaying on a display and by voice notification, as examples.

[0108] The storage unit 15 stores various models such as the bronchial tube model 5, an image transformation model 20, a traceability estimation model 21, a viewpoint difference estimation model 22, and a tracking result determination model 23. The storage unit 15 is an example of a storage device that maintains stored information even if the power supplied to the storage unit 15 is cut off, and is, for example, a semiconductor memory such as an SSD (Solid State Drive), but a hard disk may also be used.

[0109] Next, the processing of the information processing device 3 for tracking the viewpoint of the endoscope 2 will be described in detail.

[0110] 12 is a flowchart showing an example of the flow of tracking processing executed by the information processing device 3 when an instruction to start tracking the viewpoint of the endoscope 2 is received by a medical professional operating the operation unit 13. The CPU 10A of the information processing device 3 reads the control program 11 from the ROM 10B and executes the tracking processing. The control program 11 is an example of an information processing program disclosed herein.

[0111] It is assumed that the medical professional has set in advance an initial viewpoint of the endoscope 2 with respect to the bronchial model 5. The initial viewpoint of the endoscope 2 set by the medical professional may be a viewpoint different from the actual viewpoint of the endoscope 2. Alternatively, the CPU 10A may randomly set the initial viewpoint with respect to the bronchial model 5.

[0112] The storage unit 15 is also assumed to have stored in advance a subject's bronchial model 5, an image transformation model 20, a traceability estimation model 21, a viewpoint difference estimation model 22, a tracking result determination model 23, and a reference score.

[0113] First, in step S10, the CPU 10A acquires the endoscopic image 4 from the endoscope 2 through the I / F unit 12. The endoscopic image 4 may be a color image or a monochrome image.

[0114] In step S20, CPU 10A inputs endoscopic image 4 acquired by the processing in step S10 to traceability estimation model 21, and estimates the score of endoscopic image 4.

[0115] In step S30, CPU 10A determines whether or not the score of endoscopic image 4 estimated by the process of step S20 is less than the reference score.

[0116] If the score of the endoscopic image 4 is equal to or greater than the reference score, the process proceeds to step S40. In this case, since it is possible to track the viewpoint of the endoscope 2, in step S40, CPU 10A executes viewpoint tracking processing to identify the viewpoint of the endoscope 2 that captured the endoscopic image 4.

[0117] FIG. 13 is a flowchart showing an example of the flow of the viewpoint tracking process in step S40.

[0118] In step S42, CPU 10A generates a virtual endoscopic image 6 in which bronchial model 5 is viewed from the set viewpoint.

[0119] In step S44, CPU 10A inputs endoscopic image 4 acquired by the process of step S10 in FIG. 12 to image transformation model 20, and generates pseudo virtual endoscopic image 7 corresponding to endoscopic image 4.

[0120] In step S46, CPU 10A inputs the virtual endoscopic image 6 generated by the processing of step S42 and the pseudo-virtual endoscopic image 7 generated by the processing of step S44 into the viewpoint difference estimation model 22, and estimates the viewpoint difference between the endoscopic image 4 acquired by the processing of step S10 in Figure 12 and the virtual endoscopic image 6 generated by the processing of step S42.

[0121] In step S48, CPU 10A estimates the viewpoint of endoscope 2 by reflecting the viewpoint difference estimated by the processing in step S46 in the viewpoint set for bronchial model 5.

[0122] In this way, the viewpoint of the endoscope 2 is identified, and the viewpoint tracking process shown in FIG. 13 is completed.

[0123] 12, CPU 10A generates a tracked virtual endoscopic image 6C from bronchial model 5 based on the viewpoint of endoscope 2 after tracking obtained by the viewpoint tracking process. CPU 10A also generates a tracked virtual depth image 17 corresponding to the position of the viewpoint of endoscope 2 after tracking from bronchial model 5. Then, CPU 10A inputs endoscopic image 4, tracked virtual endoscopic image 6C, and tracked virtual depth image 17 obtained by the process of step S10 to tracking result determination model 23, and obtains the tracking result of the viewpoint of endoscope 2.

[0124] In step S60, if the tracking result of the viewpoint of endoscope 2 acquired by the processing of step S50 is "tracking successful", CPU 10A proceeds to step S70.

[0125] In step S70, CPU 10A sets the viewpoint of the endoscope 2 identified by the processing of step S40 to the bronchial model 5, and updates the viewpoint of the endoscope 2 on the bronchial model 5. CPU 10A performs a route search from the updated viewpoint of the endoscope 2 to the target location using the bronchial model 5. A known search method such as a binary search method can be used to search for a route to the target location.

[0126] Based on the route search result, the CPU 10A notifies the medical staff through a speaker with a voice indicating the traveling direction of the endoscope 2, such as "to the right" or "downward." In this way, the CPU 10A notifies the medical staff of the navigation information for the endoscope 2 to reach the target location, and proceeds to step S80.

[0127] On the other hand, if it is determined in the determination process of step S30 that the score of the endoscopic image 4 is less than the standard score, the process proceeds to step S80 without executing the processes of steps S40 to S70. That is, CPU 10A temporarily suspends tracking of the viewpoint of the endoscope 2 until an endoscopic image 4 having a score equal to or greater than the standard score is acquired.

[0128] Furthermore, if the result of tracking the viewpoint of the endoscope 2 is determined to be "tracking failure" by the determination process of step S60, the process proceeds to step S80 without executing the process of step S70. That is, if tracking of the viewpoint of the endoscope 2 has failed, CPU 10A stops tracking of the viewpoint of the endoscope 2.

[0129] In step S80, CPU 10A notifies the medical staff of the tracking status of the viewpoint of endoscope 2, which is in progress, paused, or stopped. Specifically, if the tracking result of the viewpoint of endoscope 2 is "tracking successful," CPU 10A notifies the medical staff that the tracking status is "in progress." Furthermore, if the tracking result of the viewpoint of endoscope 2 is "tracking failed," CPU 10A notifies the medical staff that the tracking status is "stopped." Furthermore, if the score of endoscopic image 4 is determined to be less than the reference score by the determination process in step S30, CPU 10A notifies the medical staff that the tracking status is "paused."

[0130] When the tracking state is "in progress", CPU 10A generates a tracked virtual endoscopic image 6C of the bronchial model 5 viewed from the updated viewpoint of endoscope 2 on bronchial model 5, and displays the generated tracked virtual endoscopic image 6C on the display alongside, for example, the endoscopic image 4 acquired by the processing of step S10. As a result, the tracked virtual endoscopic image 6C that follows the change in the viewpoint of endoscope 2 is displayed.

[0131] When the tracking state is "stopped," tracking of the viewpoint of the endoscope 2 is stopped, and therefore it is not possible to generate a tracked-target virtual endoscopic image 6C as would be the case when the tracking state is "ongoing." Therefore, the CPU 10A displays on the display a tracked-target virtual endoscopic image 6C at the viewpoint that was last successfully tracked before tracking of the viewpoint of the endoscope 2 failed.

[0132] If tracking of the viewpoint of the endoscope 2 has stopped, it means that the medical professional has moved the endoscope 2 beyond the allowable range that can be tracked by the information processing device 3. Therefore, in order to start tracking the viewpoint of the endoscope 2 again, the medical professional must temporarily return the position of the endoscope 2 to the position corresponding to the viewpoint that was successfully tracked.

[0133] When a medical professional performs an operation to return the endoscope 2, if the virtual endoscopic image 6C of the tracked destination at the last viewpoint where tracking was successful is displayed on the display, it is easier for the medical professional to understand how far to return the endoscope 2 compared to when the virtual endoscopic image 6C of the tracked destination is not displayed on the display.

[0134] It is easier for medical personnel to align the position of the endoscope 2 if a virtual endoscopic image 6 including a distinctive bronchial shape such as a bronchial bifurcation is displayed on the display rather than a tracking target virtual endoscopic image 6C including only the bronchial wall. Therefore, the CPU 10A may generate a tracking target virtual endoscopic image 6C from the bronchial model 5, which is an image generated from a viewpoint that was successfully tracked at a point closer to the viewpoint that was last successfully tracked before tracking of the viewpoint of the endoscope 2 failed, and which includes a bronchial bifurcation, and display it on the display.

[0135] In this way, the CPU 10A displays the tracking target virtual endoscopic image 6C, which the medical staff refers to when aligning the endoscope 2, on the display.

[0136] On the other hand, if the tracking state is "paused", CPU 10A displays, for example, a tracked virtual endoscopic image 6C of the tracking target at the last viewpoint that was successfully tracked on the display until tracking of the viewpoint of endoscope 2 is resumed.

[0137] 14 is a diagram showing an example of the display form of the tracked virtual endoscopic image 6C to the medical staff. As shown in FIG. 14, the CPU 10A notifies the medical staff of the tracking state of the viewpoint of the endoscope 2 by changing the display form of the tracked virtual endoscopic image 6C.

[0138] Specifically, the CPU 10A changes the display form of the frame of the tracked target virtual endoscopic image 6C in accordance with the tracking state of the viewpoint of the endoscope 2. For example, the CPU 10A changes one or more of the color of the frame, the color density of the frame, and the thickness of the frame in accordance with the tracking state of the viewpoint of the endoscope 2. There are no restrictions on the shape of the tracked target virtual endoscopic image 6C displayed on the display. Figure 14 shows examples of shapes using a rectangle and a circle.

[0139] Furthermore, the CPU 10A may superimpose characters or symbols representing the tracking status on the tracked-target virtual endoscopic image 6C. In the example shown in Fig. 14, no characters or symbols are superimposed on the tracked-target virtual endoscopic image 6C in which the tracking status of the viewpoint of the endoscope 2 is in progress, but for example, the character string "in progress" or a symbol representing "in progress" (for example, a circle symbol such as "○") may be superimposed on the tracked-target virtual endoscopic image 6C.

[0140] After notifying the navigation information and tracking status in this way, in step S90 of Fig. 12, CPU 10A determines whether or not the tracking status of the viewpoint of endoscope 2 is "stopped." If the tracking status of the viewpoint of endoscope 2 is "stopped," the process proceeds to step S100.

[0141] When the tracking state of the viewpoint of the endoscope 2 is "stopped," the medical professional aligns the position of the endoscope 2 while referring to the tracking target virtual endoscopic image 6C displayed on the display, and after completing the alignment of the endoscope 2, presses, for example, a tracking start button displayed on the display. Therefore, in step S100, the CPU 10A determines whether or not an instruction to start tracking of the viewpoint of the endoscope 2, notified by pressing the tracking start button, has been received. If the tracking start instruction has not been received, the process proceeds to step S90. In other words, when the tracking state of the viewpoint of the endoscope 2 is "stopped," the CPU 10A suspends the tracking process until a tracking start instruction is received.

[0142] The CPU 10A may notify the medical staff of the tracking status of the viewpoint of the endoscope 2 by the display format of the tracking start button. For example, if the tracking status of the viewpoint of the endoscope 2 is "in progress", the CPU 10A displays the tracking start button in a predetermined color (e.g., green). If the tracking status of the viewpoint of the endoscope 2 is "paused", the CPU 10A alternately displays (i.e., blinks) the tracking start button in two different colors (e.g., green and gray). If the tracking status of the viewpoint of the endoscope 2 is "stopped", the CPU 10A displays the tracking start button in a color (e.g., gray) different from when the tracking status of the viewpoint of the endoscope 2 is "in progress".

[0143] On the other hand, if the judgment process of step S90 determines that the tracking state of the viewpoint of the endoscope 2 is not "stopped", and if the judgment process of step S100 determines that a tracking start instruction has been received, the process proceeds to step S110.

[0144] In step S110, CPU 10A determines whether or not an instruction to end the tracking process has been received from the medical professional. If an instruction to end the process has not been received, the process proceeds to step S10, where a new endoscopic image 4 is acquired from the endoscope 2. That is, until an instruction to end the process is received from the medical professional, the information processing device 3 notifies the tracking status of the viewpoint of the endoscope 2, tracks the viewpoint of the endoscope 2 in accordance with the movement of the endoscope 2, and determines the tracking result of the viewpoint of the endoscope 2, and if tracking is successful, notifies the medical professional of navigation information of the endoscope 2.

[0145] On the other hand, if an end instruction is received, the tracking process shown in FIG. 12 ends.

[0146] As described above, the information processing device 3 according to the first embodiment judges the tracking result of the viewpoint of the endoscope 2, and if tracking of the viewpoint of the endoscope 2 fails, autonomously stops tracking of the viewpoint of the endoscope 2. Therefore, the information processing device 3 can prevent erroneous navigation information from being reported based on a viewpoint different from the actual viewpoint of the endoscope 2.

[0147] 12, if the paused state continues for a specified time or longer after the tracking of the viewpoint of the endoscope 2 is paused, it is possible that there is a deposit on the lens of the endoscope 2 or that the lighting of the endoscope 2 that illuminates the inside of the bronchi is broken. Therefore, if the paused state for tracking the viewpoint of the endoscope 2 continues for a specified time or longer, the CPU 10A may transition the tracking state of the viewpoint of the endoscope 2 to "paused." The specified time for transitioning the tracking state to "paused" is set, for example, by a medical professional and stored in advance in the storage unit 15.

[0148] 15 is a diagram showing an example of the display form of a tracking-target virtual endoscopic image 6C when tracking of the viewpoint of the endoscope 2 has been paused for a specified time or longer and the tracking state of the viewpoint of the endoscope 2 has transitioned to "stopped." As shown in FIG. 15, the CPU 10A changes the display form of the frame of the tracking-target virtual endoscopic image 6C in accordance with the length of time that has elapsed since tracking of the viewpoint of the endoscope 2 was paused, so that the current tracking state can be seen. When the elapsed time since the pause has exceeded a specified time, the CPU 10A fixes the display form of the frame to a display form that represents the tracking state of "stopped." The display form of the frame as the tracking state changes from "paused" to "stopped" may change either continuously or in stages.

[0149] Note that the fact that tracking of the viewpoint of the endoscope 2 is temporarily stopped means that the endoscopic image 4 includes a structure-unknown area that makes it difficult to track the viewpoint of the endoscope 2. Therefore, when the tracking status of the viewpoint of the endoscope is "temporarily stopped", the CPU 10A may notify the reason why tracking of the viewpoint of the endoscope 2 has been temporarily stopped.

[0150] Fig. 16 is a diagram showing a display example of an unknown structure region in an endoscopic image 4. As shown in Fig. 16, CPU 10A notifies a medical professional of the unknown structure region by, for example, surrounding the unknown structure region with a display frame 28. CPU 10A may also display the type of the unknown structure region along with displaying the position of the unknown structure region with the display frame 28. In the example shown in Fig. 16, it is displayed that the unknown structure region surrounded by the display frame 28 is a "bubble."

[0151] To detect an area with an unknown structure from the endoscopic image 4, for example, a model for detecting an area with an unknown structure is used.

[0152] The unknown structure region detection model is a model that, when an endoscopic image 4 is input, detects an unknown structure region from the endoscopic image 4 and outputs position information and the type of the unknown structure region in the endoscopic image 4. The unknown structure region detection model is generated by machine learning using training data that uses, for example, the endoscopic image 4 as input data and the position information and the type of the unknown structure region in the endoscopic image 4 as training data. The correspondence between the endoscopic image 4 in the training data and the position information and the type of the unknown structure region in the endoscopic image 4 is performed, for example, by a medical professional.

[0153] When a medical professional is notified of the presence of an unknown structure region, they may be able to prevent the occurrence of the unknown structure region by investigating the cause of the detection. If the unknown structure region is caused by the adhesion of bubbles or foreign matter, the unknown structure region can be removed, for example, by using forceps or water supply. Furthermore, if the unknown structure region is caused by the reflection of light from an illumination source, the unknown structure region can be removed, for example, by reducing the amount of light from the illumination source.

[0154] The information processing device 3 can track the viewpoint of the endoscope 2 even if the endoscopic image 4 includes an area of ​​unknown structure, but the fewer areas of unknown structure there are, the more accurate the tracking will be. Therefore, the information processing device 3 notifies the medical staff of the position of the detected area of ​​unknown structure, thereby supporting the medical staff in eliminating the cause of the area of ​​unknown structure.

[0155] It should be noted that the smaller the area of ​​the unknown structure region, the less impact it has on tracking the viewpoint of the endoscope 2. If the position of the unknown structure region is notified regardless of the area of ​​the detected unknown structure region, even if measures are taken to remove the cause of the unknown structure region, this will not contribute to improving the estimation accuracy of the viewpoint of the endoscope 2 and may impose an unnecessary burden on medical personnel. Therefore, the CPU 10A may be configured to notify the position of the unknown structure region in the endoscopic image 4 when the proportion of the unknown structure region in the endoscopic image 4 is equal to or greater than a specified proportion.

[0156] 12, CPU 10A estimates whether or not it is possible to track the viewpoint of the endoscope 2 by the determination process of step S30, and then performs the viewpoint tracking process and determines the tracking result of the viewpoint of the endoscope 2. However, as long as the tracking state of the viewpoint of the endoscope 2 can be correctly identified, there are no restrictions on the order of the processes in the tracking process, and it may be possible to estimate whether or not it was possible to track the viewpoint of the endoscope 2 after performing the viewpoint tracking process.

[0157] FIG. 17 is a flowchart showing a modified example of the flow of the tracking process executed by the information processing device 3 when an instruction to start tracking the viewpoint of the endoscope 2 is received by a medical professional operating the operation unit 13.

[0158] The flowchart shown in Figure 17 differs from the flowchart of the tracking process shown in Figure 12 in that after executing the process of step S10, the processes of steps S40 to S60 are executed, and if the tracking result of the viewpoint of the endoscope 2 is "tracking successful" by the determination process of step S60, the processes of steps S20 and S30 are executed.

[0159] That is, the CPU 10A first executes a viewpoint tracking process using the endoscopic image 4 acquired from the endoscope 2. If the tracking result of the viewpoint of the endoscope 2 is "tracking failed", the CPU 10A sets the tracking status to "stopped". On the other hand, if the tracking result of the viewpoint of the endoscope 2 is "tracking successful", the CPU 10A estimates the tracking feasibility of the endoscope 2 for the endoscopic image 4 used in the viewpoint tracking process. If the score of the endoscopic image 4 is equal to or greater than the standard score, the CPU 10A sets the tracking status to "in progress", and if the score of the endoscopic image 4 is less than the standard score, the CPU 10A sets the tracking status to "paused".

[0160] In this way, there are no restrictions on the order of the processes in the tracking process as long as it is possible to correctly identify the tracking state of the viewpoint of the endoscope 2. Therefore, for example, if the estimation of the traceability of the endoscope 2 shown in step S20 is performed in parallel with the processes shown in steps S40 to S60 in Fig. 17, the time from acquiring the endoscopic image 4 to notifying the navigation information can be shortened compared to when each process is performed sequentially.

[0161] <Modification of tracking result determination model 23> The above describes an example in which the tracking result determination model 23 generated by machine learning using the determination learning data is used to determine the tracking result of the viewpoint of the endoscope 2. However, the tracking result determination model 23 can also be constructed using a method other than machine learning.

[0162] 18 is a diagram showing a modified example of the tracking result determination model 23. The tracking result determination model 23 according to the modified example includes a structure estimation model 23A and a comparison model 23B.

[0163] The structure estimation model 23A estimates structural information representing the structure of a living body from an input image. There are no restrictions on the structural information estimated by the structure estimation model 23A, and it is possible to estimate, for example, at least one of the number of holes, the texture of the bronchi, and the shape of a tumor. As an example, the number of holes formed by the bronchi is estimated. To extract the structure of a living body from an image, a known image processing method based on, for example, shape recognition or feature extraction is used.

[0164] The comparison model 23B compares two pieces of structural information estimated from different images by the structure estimation model 23A, and determines that the two images represent the same location if the structural information is similar, and determines that the two images represent different locations if the structural information is not similar. There are no restrictions on the method for comparing structural information in the comparison model 23B, and any known comparison method can be used.

[0165] When an endoscopic image 4 and a tracked-target virtual endoscopic image 6C are input to the tracking result determination model 23 according to this modified example, the number of holes contained in the endoscopic image 4 is compared with the number of holes contained in the tracked-target virtual endoscopic image 6C. If the numbers of holes are the same, the tracking result determination model 23 according to the modified example outputs a determination result that the endoscopic image 4 and the tracked-target virtual endoscopic image 6C are photographing the same location, and if the numbers of holes are different, the tracking result determination model 23 according to the modified example outputs a determination result that the endoscopic image 4 and the tracked-target virtual endoscopic image 6C are photographing different locations.

[0166] The fact that the input endoscopic image 4 and the tracked virtual endoscopic image 6C capture the same location means that the viewpoint after tracking is viewing the bronchial model 5 from the same viewpoint as the endoscope 2, and therefore the tracking result for the viewpoint of the endoscope 2 is "tracking successful." On the other hand, the fact that the input endoscopic image 4 and the tracked virtual endoscopic image 6C capture different locations means that the viewpoint after tracking is viewing the bronchial model 5 from a viewpoint different from that of the endoscope 2, and therefore the tracking result for the viewpoint of the endoscope 2 is "tracking failed."

[0167] In this way, the information processing device 3 may determine the tracking result of the viewpoint of the endoscope 2 by comparing the structural information obtained from the endoscopic image 4 and the tracked virtual endoscopic image 6C using known image processing methods and comparison methods.

[0168] 18 has described an example in which the tracking result of the viewpoint of the endoscope 2 is determined from the endoscopic image 4 and the tracked virtual endoscopic image 6C. However, the depth image corresponding to the endoscopic image 4 and the tracked virtual depth image 17 may be input into the tracking result determination model 23 according to a modified example to determine the tracking result of the viewpoint of the endoscope 2. The endoscopic image 4 and the tracked virtual depth image 17 may be input into the tracking result determination model 23 according to a modified example to determine the tracking result of the viewpoint of the endoscope 2, or the endoscopic image 4, the tracked virtual endoscopic image 6C, and the tracked virtual depth image 17 may be input into the tracking result determination model 23 according to a modified example to determine the tracking result of the viewpoint of the endoscope 2. In this case, the comparison model 23B may determine that the images represent the same location if the structural information obtained from the endoscopic image 4, the structural information obtained from the tracked virtual endoscopic image 6C, and the structural information obtained from the tracked virtual depth image 17 are similar to each other, and may determine that the images represent different locations if the structural information is not similar to each other.

[0169] Furthermore, multiple images obtained from the endoscope 2 and multiple images obtained from the bronchial model 5, such as the endoscopic image 4 and the tracking target virtual endoscopic image 6C, and the depth image corresponding to the endoscopic image 4 and the tracking target virtual depth image 17, may be input into a tracking result determination model 23 relating to a modified example to determine the tracking result of the viewpoint of the endoscope 2.

[0170] Furthermore, instead of the tracked virtual endoscopic image 6C shown in Fig. 18, a partial bronchus model visible from the viewpoint after tracking may be input to the tracking result determination model 23 according to the modified example. In this case, the structure estimation model 23A estimates the number of holes visible when the partial bronchus model is viewed from the viewpoint after tracking for the input partial bronchus model.

[0171] Second Embodiment In the first embodiment, an example has been described in which a viewpoint difference is estimated using the viewpoint difference estimation model 22 generated by machine learning, and the viewpoint of the endoscope 2 is tracked based on the estimated viewpoint difference.

[0172] However, the method for tracking the viewpoint of the endoscope 2 is not limited to the method using the viewpoint difference estimation model 22 generated by machine learning. Any method may be used as long as it can track the viewpoint of the endoscope 2.

[0173] In the second embodiment, an information processing device 3 that tracks the viewpoint of the endoscope 2 based on the similarity of images will be described.

[0174] An example of the functional configuration of the information processing device 3 according to the second embodiment is the same as the example of the functional configuration of the information processing device 3 shown in FIG. 1, and is configured using the computer shown in FIG.

[0175] The information processing device 3 estimates the viewpoint of the endoscope 2 from the similarity between a depth image corresponding to the endoscopic image 4 (hereinafter referred to as "endoscopic depth image 18") and a depth image generated using the bronchial model 5. The similarity is expressed, for example, by a numerical value. As an example, a larger numerical value of the similarity indicates that the images are more similar.

[0176] Therefore, as will be described later, the information processing device 3 generates an endoscopic depth image 18 from the endoscopic image 4. The information processing device 3 also generates depth images (hereinafter referred to as "virtual depth images 19") at each position inside the bronchial model 5.

[0177] The information processing device 3 calculates the similarity between each generated virtual depth image 19 and the endoscopic depth image 18, and identifies the virtual depth image 19 that is most similar to the endoscopic depth image 18 from among the virtual depth images 19. Then, the information processing device 3 searches for a viewpoint that maximizes the similarity to the endoscopic image 4, which is the original image from which the endoscopic depth image 18 was generated, at the position represented by the identified virtual depth image 19, and estimates the viewpoint of the endoscope 2. In other words, the viewpoint that obtains an image that maximizes the similarity to the endoscopic image 4 at the position represented by the virtual depth image 19 that is most similar to the endoscopic depth image 18 is the viewpoint of the endoscope 2.

[0178] Note that a known image recognition method, such as calculating the similarity by comparing the feature amounts of the images, can be applied to calculate the similarity between the endoscopic depth image 18 and the virtual depth image 19 (sometimes simply referred to as "similarity between images"). Furthermore, a similarity estimation model, which is a model generated by machine learning and outputs the similarity between images, may be used to calculate the similarity between images.

[0179] 19 is a diagram showing an example of a method for estimating the similarity between images in the information processing device 3 according to the second embodiment. Prior to estimating the similarity, the generation unit 3E generates virtual depth images 19 at each position of the bronchial model 5.

[0180] The tracking unit 3F inputs the endoscopic image 4 to, for example, a depth estimation model 24, and converts the endoscopic image 4 into an endoscopic depth image 18. The depth estimation model 24 is a model generated in advance using, for example, a GAN trained to output the endoscopic depth image 18 from the endoscopic image 4, and is stored in advance in the storage unit 3D.

[0181] The tracking unit 3F estimates the viewpoint of the endoscope 2 by calculating the similarity between each virtual depth image 19 generated by the generating unit 3E and the endoscopic depth image 18, and tracks the viewpoint of the endoscope 2.

[0182] Next, the process of the information processing device 3 for tracking the viewpoint of the endoscope 2 based on the similarity between images will be described in detail.

[0183] The tracking process executed by the information processing device 3 according to the second embodiment follows the same flowchart as the tracking process shown in Fig. 12 or 17. However, the viewpoint tracking process in step S40 in Fig. 12 and Fig. 17 is different. Therefore, hereinafter, the viewpoint tracking process according to the second embodiment in step S40 in Fig. 12 and Fig. 17 will be described.

[0184] 20 is a flowchart showing an example of the flow of the gaze point tracking process according to the second embodiment. The storage unit 15 is assumed to have stored in advance a bronchial model 5 of the subject, virtual depth images 19 at various positions inside the bronchial model 5 of the subject, a traceability estimation model 21, a depth estimation model 24, a tracking result determination model 23, and a reference score.

[0185] In step S400, CPU 10A inputs endoscopic image 4 acquired by the process of step S10 in FIG. 12 or FIG.

[0186] In step S410, CPU 10A acquires one virtual depth image 19 from among the plurality of virtual depth images 19 stored in storage unit 15.

[0187] In step S420, CPU 10A calculates the similarity between the endoscopic depth image 18 generated by the processing of step S400 and the virtual depth image 19 acquired by the processing of step S410, and stores the calculated similarity in RAM 10C in association with the virtual depth image 19 used to calculate the similarity.

[0188] In step S430, CPU 10A determines whether or not all virtual depth images 19 stored in storage unit 15 have been acquired. If not all virtual depth images 19 have been acquired, the process proceeds to step S410, where CPU 10A acquires one unacquired virtual depth image 19. That is, CPU 10A repeats the processes of steps S410 to S430 until it is determined in the determination process of step S430 that all virtual depth images 19 have been acquired, thereby associating each virtual depth image 19 with a similarity to endoscopic depth image 18.

[0189] On the other hand, if it is determined in the determination process of step S430 that all virtual depth images 19 have been acquired, the process proceeds to step S440.

[0190] In step S440, the CPU 10A refers to the similarity associated with each virtual depth image 19 stored in the RAM 10C, and identifies the virtual depth image 19 associated with the greatest similarity among all the virtual depth images 19 as the virtual depth image 19 most similar to the endoscopic depth image 18.

[0191] In step S450, the CPU 10A searches for a viewpoint that provides an image that maximizes the similarity to the endoscopic image 4, which is the image from which the endoscopic depth image 18 was generated, at the position of the bronchial model 5 represented by the virtual depth image 19 identified by the processing in step S440, and estimates the viewpoint of the endoscope 2. This completes the viewpoint tracking processing according to the second embodiment shown in FIG.

[0192] According to the information processing device 3 of the second embodiment, the viewpoint of the endoscope 2 is estimated based on the similarity between the endoscopic depth image 18 and the virtual depth image 19, and the viewpoint of the endoscope 2 is tracked. In this way, the method of tracking the viewpoint of the endoscope 2 in the information processing device 3 is not limited to the method using the viewpoint difference estimation model 22, and other methods may be used. In other words, the process of tracking the viewpoint of the endoscope 2 in the information processing device 3 can be performed without being affected by the method of tracking the viewpoint of the endoscope 2.

[0193] While one form of the information processing system 1 has been described above using the embodiment, the disclosed form of the information processing system 1 is merely an example, and the form of the information processing system 1 is not limited to the scope described in the embodiment. Various changes or improvements can be made to the embodiment without departing from the gist of the present disclosure, and forms incorporating such changes or improvements are also included in the technical scope of the disclosure.

[0194] For example, the internal processing order in the flowcharts of each tracking process shown in Figures 12 and 17, and the internal processing order in the flowcharts of each viewpoint tracking process shown in Figures 13 and 20 may be changed without departing from the gist of the present disclosure.

[0195] In the above embodiments, the tracking processes are implemented by software as an example. However, the same processes as those shown in the flowcharts of the tracking processes may be implemented by hardware. In this case, the processing speed can be increased compared to when the tracking processes are implemented by software.

[0196] In each of the above embodiments, the term "processor" refers to a processor in a broad sense, and includes general-purpose processors (e.g., CPU 10A) and dedicated processors (e.g., GPU: Graphics Processing Unit, ASIC: Application Specific Integrated Circuit, FPGA: Field Programmable Gate Array, programmable logic device, etc.).

[0197] Furthermore, the operations of the processors in the above embodiments may not only be performed by a single processor, but may also be performed by multiple processors located at physically separate locations working together. Furthermore, the order of the operations of the processors is not limited to the order described in the above embodiments, and may be changed as appropriate.

[0198] In each of the above embodiments, an example has been described in which the control program 11 is stored in the ROM 10B. However, the storage destination of the control program 11 is not limited to the ROM 10B. The control program 11 can also be provided in a form recorded on a computer-readable storage medium.

[0199] For example, the control program 11 may be provided in a form recorded on an optical disk such as a CD-ROM (Compact Disk Read Only Memory), a DVD-ROM (Digital Versatile Disk Read Only Memory), or a Blu-ray disc. The control program 11 may also be provided in a form recorded on a portable semiconductor memory such as a USB (Universal Serial Bus) memory or a memory card. ROM 10B, a CD-ROM, a DVD-ROM, a Blu-ray disc, a USB, and a memory card are examples of non-transitory storage media.

[0200] Furthermore, the control unit 10 may download the control program 11 from an external device connected to a communication line via the I / F unit 12, and store the downloaded control program 11 in the ROM 10B of the control unit 10.

[0201] The following additional notes are further disclosed regarding the above embodiment.

[0202] (Appendix 1) an estimation unit that estimates, from an endoscopic image of a living body, whether or not it is possible to track the viewpoint of the endoscope that captured the endoscopic image; a tracking unit that tracks the viewpoint of the endoscope using the endoscopic image and an image generated from the three-dimensional model of the living body when the estimation unit estimates that the viewpoint of the endoscope can be tracked; a determination unit that determines a tracking result of the viewpoint of the endoscope by the tracking unit using a three-dimensional model image generated from the three-dimensional model based on the viewpoint of the endoscope after tracking and the endoscope image; a control unit that performs control to stop tracking of the viewpoint of the endoscope until an instruction to start tracking is issued when the determination unit determines that tracking has failed; An information processing device comprising:

[0203] (Appendix 2) the three-dimensional model image is at least one of a tracked virtual endoscopic image of the living body, which is an image of the three-dimensional model viewed from the viewpoint of the endoscope after tracking, and a tracked virtual depth image, which is a depth image of the three-dimensional model corresponding to a position where the tracked virtual endoscopic image is obtained; The determination unit determines the tracking result of the viewpoint of the endoscope using a determination model that has been machine-learned in advance to output whether tracking of the viewpoint of the endoscope from the tracking target virtual endoscopic image, the tracking target virtual depth image, and the endoscope has been successful. 10. The information processing device according to claim 1.

[0204] (Appendix 3) the three-dimensional model image is at least one of a tracked virtual endoscopic image of the living body, which is an image of the three-dimensional model viewed from the viewpoint of the endoscope after tracking, and a tracked virtual depth image, which is a depth image of the three-dimensional model corresponding to a position where the tracked virtual endoscopic image is obtained; The determination unit determines the tracking result of the viewpoint of the endoscope by comparing the endoscopic image with structural information representing the structure of the living body obtained from at least one of the tracking target virtual endoscopic image and the tracking target virtual depth image. 10. The information processing device according to claim 1.

[0205] (Appendix 4) When the estimation unit estimates that the viewpoint of the endoscope cannot be tracked from the endoscopic image, the estimation unit temporarily suspends tracking of the viewpoint of the endoscope until an endoscopic image in which the viewpoint of the endoscope can be tracked is acquired. 4. The information processing device according to claim 1.

[0206] (Appendix 5) When the determining unit determines that the tracking has failed, the control unit controls the display device to display a virtual endoscopic image, which is an image that a medical professional refers to in order to align the endoscope and is an image when the three-dimensional model is viewed from a viewpoint where tracking has been successful. 5. The information processing device according to any one of Supplementary notes 1 to 4.

[0207] (Appendix 6) The control unit controls the display device to display the virtual endoscopic image at the viewpoint that was last successfully tracked before the tracking of the viewpoint of the endoscope failed. 6. The information processing device according to claim 5.

[0208] (Appendix 7) The control unit controls the display device to display the virtual endoscopic image including a branch point of a bronchus located closer to the viewpoint that was last successfully tracked before the viewpoint of the endoscope failed to be tracked. 6. The information processing device according to claim 5.

[0209] (Appendix 8) Control is performed to notify whether the tracking state of the viewpoint of the endoscope is in progress, paused, or stopped. An information processing device according to any one of Supplementary notes 1 to 7.

[0210] (Appendix 9) The control unit notifies the tracking state of the viewpoint of the endoscope by changing the display form of the three-dimensional model image. 9. The information processing device according to claim 8.

[0211] (Appendix 10) When the tracking state of the viewpoint of the endoscope is temporarily stopped, the control unit performs control to notify the reason why the tracking of the viewpoint of the endoscope is temporarily stopped. 10. The information processing device according to claim 9.

[0212] (Appendix 11) From an endoscopic image of a living body, it is estimated whether or not it is possible to track the viewpoint of the endoscope that captured the endoscopic image; When it is estimated that tracking of the viewpoint of the endoscope is possible, tracking the viewpoint of the endoscope using the endoscopic image and an image generated from the three-dimensional model of the living body; determining a tracking result of the viewpoint of the endoscope using a three-dimensional model image generated from the three-dimensional model based on the viewpoint of the endoscope after tracking and the endoscope image; and causing the computer to execute a process of stopping tracking of the viewpoint of the endoscope until an instruction to start tracking is notified when it is determined that tracking has failed. Information processing program. [Explanation of symbols]

[0213] 1. Information Processing Systems 2 Endoscopy 3. Information processing equipment 3A Acquisition Department 3B Estimation part 3C Model Acquisition Department 3D storage 3E generation part 3F Tracking Department 3G judgment part 3H control section 3J Information Department 4 Endoscopic images 5 Bronchial Model 6, 6A, 6B Virtual endoscopic images 6C Tracking destination virtual endoscopy image 7. Pseudo-virtual endoscopic images 10. Control Unit 10A CPU 10B ROM 10C RAM 11 Control Program 12 I / F units 13 Operation unit 14. Announcement unit 15 Storage Unit 16 Bus 17 Tracking destination virtual depth image 18 Endoscopic depth images 19 Virtual Depth Images 20 Image Transformation Model 21 Traceability Estimation Model 22 Viewpoint difference estimation model 23 Tracking result judgment model 23A Structural estimation model 23B comparison model 24 Depth estimation model 25, 26 marks 28 display frame

Claims

1. an estimation unit that estimates, from an endoscopic image of a living body, whether or not it is possible to track the viewpoint of the endoscope that captured the endoscopic image; a tracking unit that tracks the viewpoint of the endoscope using the endoscopic image and an image generated from the three-dimensional model of the living body when the estimation unit estimates that tracking of the viewpoint of the endoscope is possible; a determination unit that determines a tracking result of the viewpoint of the endoscope by the tracking unit using a three-dimensional model image generated from the three-dimensional model based on the viewpoint of the endoscope after tracking and the endoscope image; a control unit that performs control to stop tracking of the viewpoint of the endoscope until an instruction to start tracking is issued when the determination unit determines that tracking has failed; An information processing device comprising:

2. the three-dimensional model image is at least one of a tracked virtual endoscopic image of the living body, which is an image of the three-dimensional model viewed from the viewpoint of the endoscope after tracking, and a tracked virtual depth image, which is a depth image of the three-dimensional model corresponding to a position where the tracked virtual endoscopic image is obtained; The determination unit determines the tracking result of the viewpoint of the endoscope using a determination model that has been machine-learned in advance to output whether tracking of the viewpoint of the endoscope from the tracking target virtual endoscopic image, the tracking target virtual depth image, and the endoscope has been successful. The information processing device according to claim 1 .

3. the three-dimensional model image is at least one of a tracked virtual endoscopic image of the living body, which is an image of the three-dimensional model viewed from the viewpoint of the endoscope after tracking, and a tracked virtual depth image, which is a depth image of the three-dimensional model corresponding to a position where the tracked virtual endoscopic image is obtained; The determination unit determines the tracking result of the viewpoint of the endoscope by comparing the endoscopic image with structural information representing the structure of the living body obtained from at least one of the tracking target virtual endoscopic image and the tracking target virtual depth image. The information processing device according to claim 1 .

4. When the estimation unit estimates that the viewpoint of the endoscope cannot be tracked from the endoscopic image, the estimation unit temporarily suspends tracking of the viewpoint of the endoscope until an endoscopic image in which the viewpoint of the endoscope can be tracked is acquired.

4. The information processing device according to claim 1.

5. When the determining unit determines that the tracking has failed, the control unit controls the display device to display a virtual endoscopic image, which is an image that a medical professional refers to in order to align the endoscope and is an image when the three-dimensional model is viewed from a viewpoint where tracking has been successful.

4. The information processing device according to claim 1.

6. The control unit controls the display device to display the virtual endoscopic image at the viewpoint that was last successfully tracked before the tracking of the viewpoint of the endoscope failed. The information processing device according to claim 5 .

7. The control unit controls the display device to display the virtual endoscopic image including a bronchial bifurcation point located closer to the viewpoint that was last successfully tracked before the tracking of the viewpoint of the endoscope failed. The information processing device according to claim 5 .

8. Control is performed to notify whether the tracking state of the viewpoint of the endoscope is in progress, paused, or stopped. The information processing device according to claim 4 .

9. The control unit notifies the tracking state of the viewpoint of the endoscope by changing the display form of the three-dimensional model image. The information processing device according to claim 8 .

10. When the tracking state of the viewpoint of the endoscope is temporarily stopped, the control unit performs control to notify the reason why the tracking of the viewpoint of the endoscope is temporarily stopped. The information processing device according to claim 9 .

11. From an endoscopic image of a living body, it is estimated whether or not it is possible to track the viewpoint of the endoscope that captured the endoscopic image; When it is estimated that the viewpoint of the endoscope can be tracked, the viewpoint of the endoscope is tracked using the endoscopic image and an image generated from the three-dimensional model of the living body; determining a tracking result of the viewpoint of the endoscope using a three-dimensional model image generated from the three-dimensional model based on the viewpoint of the endoscope after tracking and the endoscope image; and causing the computer to execute a process of stopping tracking of the viewpoint of the endoscope until an instruction to start tracking is notified when it is determined that tracking has failed. Information processing program.

Images