Image processing device, image processing method, and image processing program

Abstract

To provide an image processing device that detects an unobserved region of a subject with high accuracy.SOLUTION: An image processing device includes a processor. The processor acquires a plurality of images of a subject captured by an endoscope (S1), detects a predetermined region of interest of the subject included in the plurality of images (S4), generates a three-dimensional model that is a partial three-dimensional model of the subject based on some of the plurality of images, the three-dimensional model being of the predetermined region of interest (S5), and detects an unobserved region of the subject not captured by the endoscope on the basis of the partial three-dimensional model (S6).SELECTED DRAWING: Figure 7

Description

Technical Field

[0001] The present invention relates to an image processing apparatus, an image processing method, and an image processing program.

Background Art

[0002] Conventionally, a technique for reconstructing a three-dimensional model of a subject from a group of images during an endoscopic examination is known (see, for example, Patent Document 1). In Patent Document 1, an unobserved region in the three-dimensional model is detected, and the unobserved region is displayed so as to be visible. The unobserved region is a region that has not been observed by the endoscope. Such a technique is useful for detecting oversights during an endoscopic examination.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] The images input from the endoscope to the image processing apparatus include images of various scenes. In Patent Document 1, the input images are used for generating a three-dimensional model regardless of the scene. For example, images of a scene underwater or a scene where bubbles are attached to the lens are difficult to generate an accurate three-dimensional model of the subject, and may cause misdetection of unobserved regions.

Means for Solving the Problems

[0005] One aspect of the present invention includes a processor that acquires a plurality of images of a subject imaged by an endoscope, detects a predetermined region of interest of the subject included in the plurality of images, and generates a partial three-dimensional model of the subject based on a part of the plurality of images, the partial three-dimensional model including a three-dimensional model of the predetermined region of interest, and detects an unobserved region of the subject that is not imaged by the endoscope based on the partial three-dimensional model. The present invention relates to an image processing apparatus.

[0006] Another aspect of the present invention relates to an image processing method that acquires a plurality of images of a subject imaged by an endoscope, detects a predetermined region of interest of the subject included in the plurality of images, generates a partial three-dimensional model of the subject based on a part of the plurality of images, the partial three-dimensional model including a three-dimensional model of the predetermined region of interest, and detects an unobserved region of the subject that is not imaged by the endoscope based on the partial three-dimensional model.

[0007] Another aspect of the present invention relates to an image processing program that acquires a plurality of images of a subject imaged by an endoscope, detects a predetermined region of interest of the subject included in the plurality of images, generates a partial three-dimensional model of the subject based on a part of the plurality of images, the partial three-dimensional model including a three-dimensional model of the predetermined region of interest, and detects an unobserved region of the subject that is not imaged by the endoscope based on the partial three-dimensional model.

Brief Description of the Drawings

[0008]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5A

Figure 5B

Figure 6A

Figure 6B

Figure 7

Figure 8A

Figure 8B

Figure 8C

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13

Mode for Carrying Out the Invention

[0009] (First Embodiment) An image processing apparatus, an image processing method, an image processing program, and a recording medium according to the first embodiment of the present invention will be described with reference to the drawings. As shown in FIG. 1, an image processing apparatus 10 according to the present embodiment is applied to an endoscope system 100. The endoscope system 100 includes an image processing apparatus 10, an endoscope 20, a control apparatus 30, and a display apparatus 40.

[0010] The endoscope 20 is, for example, a flexible endoscope for the digestive tract such as the large intestine. The endoscope 20 has a two-dimensional camera 20a at its distal end, and the camera 20a captures a two-dimensional image of the subject. The control device 30 is connected to the endoscope 20 and controls illumination light and the like supplied to the endoscope 20. The image captured by the endoscope 20 is input to the display device 40 through the control device 30 and the image processing device 10, and is displayed on the display device 40. The display device 40 is a known display such as a liquid crystal display.

[0011] The image processing device 10 includes a processor 1 such as a central processing unit, a storage unit 2, a memory 3, an input unit 4, and an output unit 5. The image processing device 10 is composed of, for example, a personal computer. The storage unit 2 is a computer-readable non-transitory recording medium, and is, for example, a known magnetic disk, optical disk, or flash memory. The storage unit 2 stores an image processing program 2a that causes the processor 1 to execute an image processing method described later.

[0012] The memory 3 is composed of a volatile storage device such as a RAM (random access memory) and is used as a working area for the processor 1. The input unit 4 has a known input interface and is connected to the control device 30. The output unit 5 has a known output interface and is connected to the display device 40. The image is input to the image processing device 10 through the input unit 4 and output to the display device 40 through the output unit 5.

[0013] As shown in FIG. 2, the processor 1 includes, as functional units, an image acquisition unit 11, a preprocessing unit 12, a three-dimensional (3D) reconstruction unit 13, a fold detection unit 14, an extraction unit 15, an unobserved region detection unit 16, and a display control unit 17.

[0014] During the operation of the endoscope 20, a time-series of consecutive images constituting a moving image are input from the endoscope 20 to the image processing device 10. Figure 3 illustrates a colonoscopy. In a general colonoscopy, a user such as a doctor inserts the endoscope 20 from the anus to the cecum, and then examines each part of the large intestine based on the images while removing the endoscope 20 from the cecum to the anus. Therefore, during the examination, a series of images of the subject A such as the large intestine taken from different positions are input into the image processing device 10.

[0015] The image acquisition unit 11 obtains a plurality of images by sequentially acquiring the images input into the image processing device 10. The plurality of images are images capable of generating a 3D model of the subject A continuous along the moving direction of the field of view of the endoscope 20. The image acquisition unit 11 may acquire all the images input into the image processing device 10. In this case, the plurality of images are continuous images constituting a video. Alternatively, the image acquisition unit 11 may selectively acquire the images input into the image processing device 10. In this case, the plurality of images are intermittent images discretely arranged along the moving direction of the endoscope 20.

[0016] The preprocessing unit 12 performs preprocessing such as distortion correction on the images acquired by the image acquisition unit 11. The preprocessed images are at least temporarily stored in the storage unit 2, the memory 3, or other storage devices, thereby obtaining an image group composed of a plurality of preprocessed images.

[0017] As shown in Figure 4, the 3D reconstruction unit 13 generates a three-dimensional (3D) model D of the subject A from the image group. Specifically, the 3D reconstruction unit 13 uses a known 3D reconstruction technique such as SLAM to estimate the 3D shape of the subject A from the image group and reconstruct the 3D shape. The generated 3D model D is a 3D model of the entire subject A included in the image group. When the subject A is a lumen such as the large intestine, it has a tubular shape.

[0018] The fold detection unit 14 detects a predetermined region of interest of the subject included in the image group. Specifically, the predetermined region of interest is a fold B protruding from the inner wall of the large intestine A (see Fig. 3), and the fold detection unit 14 detects the fold B from each of the plurality of images G (see Figs. 5A and 5B). The predetermined region of interest is a region where the presence or absence of the unobserved region C is detected as described later. In a colonoscopy, a blind spot of the endoscope 20 is likely to occur on the back side of the fold B, and it is important to prevent the doctor from overlooking the back side of the fold B. Therefore, the fold B is set as the region of interest.

[0019] In one example, the fold detection unit 14 may recognize the fold B in the image G using a learning model. The learning model is generated by deep learning of images in which the regions of the folds are annotated and is stored in advance in the storage unit 2. In another example, the fold detection unit 14 may detect the edges in the image G and detect the fold B based on the edges.

[0020] The extraction unit 15 extracts a portion corresponding to the region of the fold B detected by the fold detection unit 14 from the overall 3D model D generated by the 3D reconstruction unit 13, thereby generating a partial 3D model. The partial 3D model generated in this way consists of the portion of the fold B and is based on a part of the image group.

[0021] The unobserved region detection unit 16 detects the unobserved region C based on the partial 3D model generated by the extraction unit 15. The unobserved region C is a region that has never been imaged by the endoscope 20. In the examples of Figs. 3 and 4, an unobserved region C occurs on the back side of the fold B. The unobserved region C forms a missing portion E that consists of a hole in which the shape of the subject A is not restored in the 3D model D. The unobserved region detection unit 16 may detect the missing portion E in the partial 3D model as the unobserved region.

[0022] The unobserved region detection unit 16 may detect the unobserved region based on the positional relationship between the fold B and the field of view F of the endoscope 20 in the overall or partial 3D model. Specifically, in the process of three-dimensional reconstruction, the position and orientation of the camera 20a in the overall 3D model D are calculated. The unobserved region detection unit 16 calculates the position of the field of view F in the overall or partial 3D model from the position and orientation of the camera 20a, and determines whether the back side of the fold B is included in the field of view F based on the positional relationship between the field of view F and the fold B in the overall or partial 3D model. Then, when the back side of the fold B is not included in the field of view F, the unobserved region detection unit 16 detects the back side of the fold B as an unobserved region, and when the back side of the fold B is included in the field of view F, determines that the back side of the fold B has been observed.

[0023] When the unobserved region detection unit 16 detects an unobserved region, the display control unit 17 generates a display H, and outputs the display H to the display device 40 through the output unit 5 together with the image G, thereby causing the display device 40 to display the display H in real time. The display H indicates that an unobserved region has been detected.

[0024] As shown in FIGS. 5A and 5B, when the detected unobserved region is included in the image G, the display control unit 17 may generate a display H that is superimposed on the image G and indicates the position of the unobserved region within the image G. The display H may be an arrow indicating the unobserved region (see FIG. 5A), or may be a marker superimposed at a position corresponding to the unobserved region (see FIG. 5B). The arrow H in FIG. 5A indicates the fold B in which an unobserved region has been detected on the back side, and the marker H in FIG. 5B is superimposed on the front side of the fold B in which an unobserved region has been detected on the back side.

[0025] As shown in FIGS. 6A and 6B, when the detected unobserved region does not appear in the image G, the display control unit 17 may display an alert notifying the existence of the unobserved region as the display H. The alert H may be displayed outside the image G. For example, the alert H may be a frame surrounding the image G (see FIG. 6A), or may be text (see FIG. 6B). The display H may also be a guide to the unobserved region. For example, it may be a display of the distance from the tip of the endoscope 20 to the unobserved region, or may be an operation procedure until reaching the unobserved region.

[0026] Next, an image processing method executed by the image processing apparatus 10 will be described. As shown in FIG. 7, the image processing method according to the present embodiment includes a step S1 of acquiring an image of a subject A captured by the endoscope 20, a step S2 of preprocessing the image, a step S3 of generating a 3D model D of the entire subject A, a step S4 of detecting a predetermined region of interest of the subject A from the image, a step S5 of generating a partial three-dimensional model of the subject A, a step S6 of detecting an unobserved region based on the partial three-dimensional model, and a step S7 of presenting information on the detected unobserved region to the user.

[0027] For example, in a colonoscopy, an image captured by the endoscope 20 is input to the image processing apparatus 10 as the endoscope 20 is withdrawn from the cecum toward the anus. The image acquisition unit 11 sequentially acquires the images input to the image processing apparatus 10 (step S1), and then the preprocessing unit 12 sequentially performs preprocessing on the images acquired by the image acquisition unit 11 (step S2). As a result, an image group composed of a plurality of preprocessed images is obtained. Next, the 3D reconstruction unit 13 generates a 3D model D of the entire subject A from the image group (step S3). Each time a new image is acquired, steps S2 and S3 are executed, whereby a 3D model D of the entire subject A captured by the endoscope 20 until then is generated in real time.

[0028] In parallel with steps S2 and S3, the fold detection unit 14 detects the fold B in the image acquired by the image acquisition unit 11 (step S4). Each time a new image is acquired by the image acquisition unit 11, step S4 is executed, whereby the regions of the fold B in the plurality of images used for generating the entire 3D model D are detected. Next, the extraction unit 15 extracts a portion corresponding to the region of the fold B detected from the plurality of images from the entire 3D model D, thereby obtaining a partial 3D model of the subject A (step S5). Next, the unobserved area detection unit 16 detects the presence or absence of an unobserved area based on the partial 3D model (step S6).

[0029] If no unobserved area is detected (NO in step S6), step S7 is not executed, and steps S1 to S6 are repeated. If an unobserved area is detected (YES in step S6), the display control unit 17 presents the information of the detected unobserved area to the user (step S7). Specifically, the display control unit 17 generates a display H indicating that an unobserved area has been detected, such as an arrow or a marker, and outputs the display H to the display device 40 together with the image G. The user can recognize the presence of the unobserved area C based on the display H displayed on the display device 40 in real time.

[0030] The video input to the image processing apparatus 10 during the endoscopy examination may include images of scenes that are not suitable for the detection of unobserved areas. For example, images of scenes where the subject is unclear or partially invisible, such as scenes underwater, scenes where bubbles are attached to the lens, and scenes containing residues, are not suitable because it is difficult to generate an accurate 3D model and may cause false detection of unobserved areas.

[0031] According to the present embodiment, based on the region of interest detected from the image G, a portion of the region of interest is extracted from the entire 3D model D. Thereby, even when the image group acquired by the processor 1 includes an image of an unsuitable scene, a partial 3D model of the region of interest is generated in which inaccurate portions based on the image of the unsuitable scene are excluded. By using such a partial 3D model for the detection of unobserved areas, it is possible to prevent false detection of unobserved areas and prevent incorrect information about unobserved areas from being presented to the user. In addition, it is possible to prevent excessive presentation of information about unobserved areas, such as information about unobserved areas other than the region of interest, to the user.

[0032] In general colonoscopy, an unobserved area C is likely to occur on the back side of the fold B, which is a blind spot. Since the region of interest is the region of the fold B, it is possible to effectively prevent the user from overlooking the back side of the fold B during colonoscopy.

[0033] In this embodiment, the fold detection unit 14 is configured to detect the fold B from the two-dimensional image G. Alternatively, or in addition to this, the fold B may be detected from the entire 3D model D. For example, as shown in FIGS. 8A and 8B, the fold detection unit 14 slices the entire 3D model D in a direction orthogonal to its longitudinal direction to generate a slice Ds of the 3D model D. The 3D model D includes a point cloud representing the surface of the subject A. In the slice Ds, the portion of the fold B has a higher point cloud density compared to other portions. FIG. 8C shows an image G corresponding to the slice Ds of FIG. 8B. The fold detection unit 14 detects the portion with a high point cloud density as the fold B. In this way, the fold B included in the image group can also be detected by using the 3D model D generated from the image group.

[0034] When combining the detection of the fold B from the image G and the detection of the fold B from the 3D model D, as shown in FIG. 9, the fold detection unit 14 may determine the point cloud of the fold B in the 3D model D based on the fold B detected from a plurality of images G. That is, the fold detection unit 14 extracts the point cloud corresponding to the region of the fold B detected from each of the plurality of images G from the 3D model D, and recognizes the point clouds corresponding to the regions of the same fold B in the plurality of images G as one fold. FIG. 9 shows the distribution of the point cloud in the three-dimensional space. The point clouds of squares, black circles, white circles, and triangles represent the point clouds of the fold B in different images G. In the 3D model D, in addition to the fold B, there may be regions where the point cloud exists with a high density. Based on both the image G and the 3D model D, the fold B can be detected more accurately.

[0035] (Second Embodiment) Next, an image processing apparatus, an image processing method, an image processing program, and a recording medium according to the second embodiment of the present invention will be described. This embodiment is different from the first embodiment in the method of generating a partial 3D model. In this embodiment, the configurations different from those of the first embodiment will be described, and the configurations common to the first embodiment will be denoted by the same reference numerals and the description thereof will be omitted.

[0036] Similar to the first embodiment, the image processing apparatus 10 according to this embodiment is applied to an endoscope system 100 including an endoscope 20, a control device 30, and a display device 40. The image processing apparatus 10 includes a processor 101, a storage unit 2, a memory 3, an input unit 4, and an output unit 5. As shown in FIG. 10, the processor 101 includes, as functional units, an image acquisition unit 11, a preprocessing unit 12, a 3D reconstruction unit 13, a fold detection unit 14, an unobserved region detection unit 16, and a display control unit 17, and also includes an image selection unit 18.

[0037] The image acquisition unit 11 obtains a plurality of images by sequentially acquiring the images input to the image processing apparatus 10. The fold detection unit 14 detects the folds B in each image acquired by the image acquisition unit 11. The image selection unit 18 selects only the images in which the folds B are detected by the fold detection unit 14 for 3D model generation.

[0038] The 3D reconstruction unit 13 generates a 3D model of the subject A from the plurality of images selected by the image selection unit 18. The 3D model generated in this way consists of the folds B and the surrounding parts and is a partial 3D model based on a part of the plurality of images. The unobserved region detection unit 16 detects an unobserved region based on the partial 3D model generated by the 3D reconstruction unit 13.

[0039] Next, the image processing method executed by the image processing apparatus 10 will be described. As shown in FIG. 11, the image processing method according to this embodiment includes steps S1, S2, S4, S6, S7, a step S8 of selecting an image in which an interest region is detected, and a step S9 of generating a partial 3D model from the selected image.

[0040] The image acquisition unit 11 sequentially acquires the images input to the image processing apparatus 10 (step S1). Subsequently, the fold detection unit 14 detects the presence or absence of the fold B in each image acquired by the image acquisition unit 11 (step S4). When the fold B is not detected (NO in step S4), steps S2, S6 to S9 are not executed, and steps S1 and S4 are repeated. On the other hand, when the fold B is detected (YES in step S4), the image selection unit 18 selects the image in which the fold B is detected for generating a 3D model, and the preprocessing unit 12 performs preprocessing on the selected image. The 3D reconstruction unit 13 generates a partial 3D model of the subject A, which is a 3D model of the fold B and its surrounding parts, from the plurality of preprocessed images (step S9). Next, steps S6 and S7 are executed in the same manner as in the first embodiment.

[0041] As described above, according to the present embodiment, a partial 3D model is generated only from the images of the scene including the region of interest. By using such a partial 3D model for detecting the unobserved region, it is possible to prevent the misdetection of the unobserved region and prevent the incorrect information of the unobserved region from being presented to the user. In addition, it is possible to prevent the information of the unobserved region from being excessively presented to the user. Further, since the region of interest is the region of the fold B, it is possible to effectively prevent the user from overlooking the back side of the fold B in the colonoscopy. In addition, since the number of images used for generating the 3D model is limited, compared with the first embodiment, the processing amount for generating the 3D model can be reduced and the processing speed can be improved.

[0042] (Third Embodiment) Next, an image processing apparatus, an image processing method, an image processing program, and a recording medium according to the third embodiment of the present invention will be described. This embodiment is different from the first embodiment in the method of generating a partial 3D model. In this embodiment, the configuration different from the first embodiment will be described, and the same reference numerals will be given to the configurations common to the first embodiment, and the description thereof will be omitted.

[0043] Similar to the first embodiment, the image processing apparatus 10 according to this embodiment is applied to an endoscope system 100 including an endoscope 20, a control device 30, and a display device 40. The image processing apparatus 10 has a processor 102, a storage unit 2, a memory 3, an input unit 4, and an output unit 5. As shown in FIG. 12, the processor 102 has, as functional units, an extraction unit 19 in addition to a preprocessing unit 12, a 3D reconstruction unit 13, a fold detection unit 14, an unobserved region detection unit 16, and a display control unit 17.

[0044] The preprocessing unit 12, the 3D reconstruction unit 13, and the fold detection unit 14 execute processing using the image G acquired by the image acquisition unit 11, similar to the first embodiment. The extraction unit 19 extracts, from the entire 3D model D generated by the 3D reconstruction unit 13, the portion generated from the image in which the fold B is detected by the fold detection unit 14, thereby generating a partial three-dimensional model including the fold B and its surrounding portions. The unobserved region detection unit 16 detects an unobserved region based on the partial 3D model generated by the extraction unit 19.

[0045] Next, an image processing method executed by the image processing apparatus 10 will be described. As shown in FIG. 13, the image processing method according to this embodiment includes steps S1 to S4, S6, S7, and a step S10 of generating a partial three-dimensional model of the subject A.

[0046] After steps S1 to S4 are executed in the same manner as in the first embodiment, the extraction unit 19 extracts, from the entire 3D model D, the portion generated from the image in which the fold B is detected, thereby obtaining a partial 3D model of the subject A (step S10). Next, steps S6 and S7 are executed in the same manner as in the first embodiment.

[0047] Thus, according to this embodiment, a partial 3D model consisting only of an image of a scene including the region of interest and corresponding portions is generated. By using such a partial 3D model for detecting the unobserved region, it is possible to prevent misdetection of the unobserved region and prevent incorrect information of the unobserved region from being presented to the user. Also, it is possible to prevent excessive presentation of information of the unobserved region to the user.

[0048] Also, since the region of interest is the region of the fold B, it is possible to effectively prevent the user from overlooking the back side of the fold B in colonoscopy. Also, according to this embodiment, the partial 3D model includes its peripheral portion in addition to the region of interest. Thereby, the accuracy of detecting the unobserved region in the region of interest and its peripheral portion can be improved.

[0049] In each of the above embodiments, the display control unit 17 may change the display H after the unobserved region C is imaged by the endoscope 20. After the user recognizes the existence of the unobserved region C based on the display H, the user moves the visual field F of the endoscope 20 to observe the unobserved region C. The processors 1, 101, 102 calculate, for example, the positional relationship between the visual field F and the unobserved region C in the entire or partial 3D model D, and determine that the unobserved region C has been imaged when the unobserved region C is disposed within the visual field F.

[0050] After imaging the unobserved region C, the display control unit 17 may cancel the display H (see FIGS. 5A to 6B) such as an arrow, a marker, a frame, or characters. Alternatively, the display control unit 17 may change the mode of the display H. For example, the color of the frame may be changed (see FIG. 6A), or the characters may be changed to "observation completed" or the like (see FIG. 6B).

[0051] When the captured unobserved area C appears again in the image G, the display control unit 17 may not display the display H. Alternatively, the display control unit 17 may display a display indicating that the unobserved area C has been observed. For example, an arrow, a marker, or a frame (see FIGS. 5A to 6B) may be displayed in another color, or characters such as "observed" may be displayed.

[0052] In each of the above embodiments, the processors 1, 101, and 102 do not necessarily perform processing such as preprocessing, 3D reconstruction, and detection of the fold B on the entire image G, and may perform the processing only on a part selected from each image G.

[0053] In each of the above embodiments, the predetermined region of interest is the fold B of the large intestine. However, the region of interest is not limited to this, and may be any region where detection of the presence or absence of an unobserved region is required. In order to prevent overlooking blind spots, the region of interest is preferably a region where blind spots of the visual field F of the endoscope 20 are likely to occur. For example, it may be a tissue protruding from the inner surface of the lumen, such as a polyp.

[0054] As described above, the embodiments of the present invention and their modifications have been described in detail with reference to the drawings. However, the specific configuration of the present invention is not limited to the above embodiments and modifications, and various design changes are possible without departing from the gist of the present invention. Also, the components shown in the above embodiments and modifications can be combined as appropriate. For example, the subject may be a lumen other than the large intestine, or an organ other than the lumen that can be the subject of an endoscopy. The region of interest may be set according to the subject.

Explanation of Reference Numerals

[0055] 1 Processor 2 Storage unit (recording medium) 2a Image processing program 10 Image processing apparatus 20 Endoscope A Subject C Unobserved region 3D model of the whole D E Missing part F Field of view G Image

Claims

1. A processor is provided. The processor: Obtaining a plurality of images of a subject captured by an endoscope; Detecting a predetermined region of interest of the subject contained within the plurality of images; generating a partial three-dimensional model of the subject based on a portion of the plurality of images, the three-dimensional model of the predetermined region of interest; and an image processing device that detects an unobserved region of the subject that is not imaged by the endoscope based on the partial three-dimensional model.

2. The image processing apparatus of claim 1 , wherein the predetermined region of interest is tissue protruding from an inner surface of a lumen.

3. The processor further generates from the plurality of images a three-dimensional model of the entire subject contained in the plurality of images; detecting the predetermined region of interest includes detecting the predetermined region of interest in at least one of each of the plurality of images and the overall three-dimensional model; The image processing apparatus of claim 1 , wherein generating the partial three-dimensional model comprises extracting a portion of the predetermined region of interest from the entire three-dimensional model.

4. detecting the predetermined region of interest includes detecting the predetermined region of interest in each of the plurality of images; The image processing device according to claim 1 , wherein generating the partial three-dimensional model comprises generating the partial three-dimensional model from only an image in which the predetermined region of interest is detected, among the plurality of images.

5. The processor further generates from the plurality of images a three-dimensional model of the entire object contained in the plurality of images; detecting the predetermined region of interest includes detecting the predetermined region of interest in each of the plurality of images; The image processing apparatus of claim 1 , wherein generating the partial three-dimensional model comprises extracting from the entire three-dimensional model a portion generated from the image in which the predetermined region of interest is detected.

6. The image processing device according to claim 1 , wherein detecting the unobserved region includes detecting the unobserved region based on a positional relationship between the specified region of interest in a partial or entire three-dimensional model of the subject and a field of view of the endoscope.

7. The image processing apparatus according to claim 1 , wherein detecting the unobserved region includes detecting a missing portion of the partial three-dimensional model as the unobserved region.

8. The processor further comprises: The image processing apparatus according to claim 1 , wherein when the unobserved area is detected, a display indicating that the unobserved area has been detected is displayed on a display device in real time.

9. The image processing apparatus of claim 8 , wherein the indication is an arrow superimposed on the image and indicates a location of the predetermined region of interest within the image.

10. The image processing apparatus according to claim 8 , wherein the indication is a marker superimposed on a position in the image corresponding to the unobserved region.

11. Obtaining a plurality of images of a subject captured by an endoscope; Detecting a predetermined region of interest of the subject contained within the plurality of images; generating a partial three-dimensional model of the subject based on a portion of the plurality of images, the three-dimensional model of the predetermined region of interest; and detecting an unobserved region of the subject that is not imaged by the endoscope based on the partial three-dimensional model.

12. An image processing program comprising: Obtaining a plurality of images of a subject captured by an endoscope; Detecting a predetermined region of interest of the subject contained within the plurality of images; generating a partial three-dimensional model of the subject based on a portion of the plurality of images, the three-dimensional model of the predetermined region of interest; An image processing program that detects an unobserved region of the subject that is not imaged by the endoscope based on the partial three-dimensional model.

Images