Image processing apparatus, image processing method, and recording medium

Abstract

The invention provides an image processing apparatus, an image processing method, and a recording medium. The image processing apparatus includes a processor. The processor acquires a plurality of images of a subject captured by the endoscope, detects a predetermined region of interest of the subject included in the plurality of images, generates a local three-dimensional model of the subject based on a part of the plurality of images, the local three-dimensional model being a three-dimensional model of the predetermined region of interest, and generates a local three-dimensional model of the subject based on the local three-dimensional model. An unobserved region of a subject that is not imaged by an endoscope is detected.

Description

Technical Field

[0001] The present invention relates to an image processing device, an image processing method and a recording medium. Background Art

[0002] In the past, there is a known technique for reconstructing a three-dimensional model of a subject from a group of images in an endoscopic examination (for example, see Patent Document 1). In Patent Document 1, an unobserved area in a three-dimensional model is detected and the unobserved area is displayed so as to be visually identifiable. The unobserved area is an area that is not observed by the endoscope. Such a technique is useful for detecting missed areas in an endoscopic examination.

[0003] Prior art literature

[0004] Patent Literature

[0005] Patent Document 1: Japanese Patent No. 6242543 Summary of the invention

[0006] Problems to be solved by the invention

[0007] The images input from the endoscope to the image processing device include images of various scenes. In Patent Document 1, the input images are used to generate a three-dimensional model regardless of the scene. For example, it is difficult to generate an accurate three-dimensional model of the subject from images of scenes in water or scenes with bubbles attached to the lens, which may cause erroneous detection of unobserved areas.

[0008] Means for solving problems

[0009] One embodiment of the present invention is an image processing device, which has a processor that performs the following processing: obtains multiple images of a subject photographed by an endoscope, detects a specified area of ​​interest of the subject contained in the multiple images, generates a local three-dimensional model of the subject based on a portion of the multiple images, the local three-dimensional model is a three-dimensional model of the specified area of ​​interest, and based on the local three-dimensional model, detects an unobserved area of ​​the subject that is not photographed by the endoscope.

[0010] Another embodiment of the present invention is an image processing method, wherein a plurality of images of a subject captured by an endoscope are obtained, a specified region of interest of the subject contained in the plurality of images is detected, a local three-dimensional model of the subject based on a portion of the plurality of images is generated, the local three-dimensional model is a three-dimensional model of the specified region of interest, and based on the local three-dimensional model, an unobserved region of the subject that is not captured by the endoscope is detected.

[0011] Another embodiment of the present invention is a computer-readable non-temporary recording medium having an image processing program recorded thereon, wherein the image processing program causes a computer to perform the following processing: obtaining multiple images of a subject captured by an endoscope, detecting a specified area of ​​interest of the subject contained in the multiple images, generating a local three-dimensional model of the subject based on a portion of the multiple images, the local three-dimensional model being a three-dimensional model of the specified area of ​​interest, and detecting an unobserved area of ​​the subject that is not captured by the endoscope based on the local three-dimensional model. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Figure 1 This is a block diagram showing the overall configuration of the endoscope system according to the first embodiment.

[0013] Figure 2 This is a functional block diagram of a processor of the image processing device according to the first embodiment.

[0014] Figure 3 The diagram explains the steps of colon endoscopy.

[0015] Figure 4 The figure shows the whole 3D model generated in the colon endoscopy.

[0016] Figure 5A 1 is a diagram showing an example of a display screen showing an unobserved area in an image.

[0017] Figure 5B FIG. 1 is a diagram showing another example of a display screen showing an unobserved area in an image.

[0018] Fig. 6A 1 is a diagram showing an example of a display screen indicating that an unobserved area has been detected.

[0019] Figure 6B FIG. 1 is a diagram showing another example of a display screen indicating that an unobserved area has been detected.

[0020] Figure 7 This is a flowchart of the image processing method according to the first embodiment.

[0021] Fig. 8A A diagram illustrating a method for detecting wrinkles from a 3D model.

[0022] Figure 8B A diagram illustrating a method for detecting wrinkles from a 3D model.

[0023] Figure 8C is shown with Figure 8B The slices of the image correspond to the graph.

[0024] Fig. 9It is a diagram showing the distribution of wrinkle point groups in a plurality of images.

[0025] Fig.10 This is a functional block diagram of a processor of the image processing device according to the second embodiment.

[0026] Fig.11 This is a flowchart of the image processing method according to the second embodiment.

[0027] Fig.12 This is a functional block diagram of a processor of the image processing device according to the third embodiment.

[0028] Fig.13 This is a flowchart of the image processing method according to the third embodiment. DETAILED DESCRIPTION

[0029] (First Embodiment)

[0030] An image processing device, an image processing method, an image processing program, and a recording medium according to a first embodiment of the present invention will be described with reference to the drawings.

[0031] like Figure 1 As shown, the image processing device 10 of the present embodiment is applied to an endoscope system 100 .

[0032] The endoscope system 100 includes an image processing device 10 , an endoscope 20 , a control device 30 , and a display device 40 .

[0033] The endoscope 20 is, for example, a flexible endoscope for use in digestive organs such as the large intestine. The endoscope 20 has a two-dimensional camera 20a at its front end, and captures a two-dimensional image of the subject through the camera 20a. The control device 30 is connected to the endoscope 20 to control the illumination light supplied to the endoscope 20. The image captured by the endoscope 20 is input to the display device 40 through the control device 30 and the image processing device 10, and is displayed on the display device 40. The display device 40 is a well-known display such as a liquid crystal display.

[0034] The image processing device 10 includes a processor 1 such as a central processing unit, a storage unit 2, a memory 3, an input unit 4, and an output unit 5. The image processing device 10 is configured by, for example, a personal computer.

[0035] The storage unit 2 is a computer-readable non-transitory recording medium, such as a known magnetic disk, optical disk, or flash memory, etc. The storage unit 2 stores an image processing program 2 a for causing the processor 1 to execute an image processing method described later.

[0036] The memory 3 is composed of a volatile storage device such as a RAM (random access memory), and is used as a work area of ​​the processor 1 .

[0037] The input unit 4 has a known input interface and is connected to the control device 30. The output unit 5 has a known output interface and is connected to the display device 40. The image is input to the image processing device 10 through the input unit 4 and is output to the display device 40 through the output unit 5.

[0038] like Figure 2 As shown, the processor 1 includes an image acquisition unit 11, a preprocessing unit 12, a three-dimensional (3D) reconstruction unit 13, a wrinkle detection unit 14, an extraction unit 15, an unobserved region detection unit 16, and a display control unit 17 as functional units.

[0039] During the operation of the endoscope 20 , continuous images in a time series constituting a dynamic image are input from the endoscope 20 to the image processing device 10 .

[0040] Figure 3 Colonoscopy is described. In a general colonoscopy, a user such as a doctor inserts an endoscope 20 from the anus to the cecum, and then pulls the endoscope 20 out from the cecum to the anus while inspecting various parts of the colon based on images. Therefore, during the inspection, continuous images obtained by photographing a subject A such as the inner wall surface of the colon lumen from different positions are input to the image processing device 10.

[0041] The image acquisition unit 11 obtains a plurality of images by sequentially acquiring images input to the image processing device 10. The plurality of images are images that can generate a 3D model of the subject A that is continuous along the moving direction of the field of view of the endoscope 20. The image acquisition unit 11 may also acquire all the images input to the image processing device 10, in which case the plurality of images are continuous images constituting a dynamic image. Alternatively, the image acquisition unit 11 may also selectively acquire images input to the image processing device 10, in which case the plurality of images are images discretely arranged along the moving direction of the endoscope 20. Since the doctor observes the inside of the lumen without missing any blind spots, image frames are acquired accordingly, and if these image frames are arranged, an image of the inner wall of the lumen covering the observed lumen can be obtained.

[0042] The preprocessing unit 12 performs preprocessing such as distortion correction on the image acquired by the image acquisition unit 11. The preprocessed image is at least temporarily stored in the storage unit 2, the memory 3 or another storage device, thereby obtaining an image group consisting of a plurality of preprocessed images.

[0043] like Figure 4As shown, the 3D reconstruction unit 13 generates a three-dimensional (3D) model D of the subject A based on the image group. Specifically, the 3D reconstruction unit 13 estimates the three-dimensional shape of the subject A based on the image group using a well-known three-dimensional reconstruction technology such as SLAM, and reconstructs the three-dimensional shape. The generated 3D model D is an entire 3D model of the subject A included in the image group, and becomes a tube shape when the subject A is a tubular cavity such as a large intestine.

[0044] The wrinkle detection unit 14 detects a predetermined region of interest of the subject included in the image group. Specifically, in the image obtained by photographing the wrinkle, there are anatomical features obtained from the image, which can be distinguished from the image obtained by photographing other than the wrinkle. In addition, the wrinkle portion is a portion protruding from the lumen of the digestive tract, etc., and protrudes in a direction perpendicular to the direction of travel of the lumen, so there is a surface and an inside, which is easy to be missed during observation. The predetermined region of interest is the anatomical feature of the wrinkle B protruding from the inner wall of the large intestine A (refer to Figure 3 ), the wrinkle detection unit 14 detects wrinkles B from the plurality of images G (when Figure 3 (When the front end of the endoscope is bent like the middle section of the endoscope so that the fold B enters the field of view of the camera unit, the portion standing up from the lumen becomes a shadow, or becomes darker as it goes deeper, or when distance data is obtained, it can be determined that the distance to the camera unit has changed). In such a state, if the user is notified that the area of ​​interest has been detected and the determination of the unobserved area has been focused on, a sense of security is also conveyed to the user. The specified area of ​​interest is an area that is given importance and the detection of the presence or absence of the unobserved area C is performed as described later. In colon endoscopy, blind spots of the endoscope 20 are likely to occur on the inner side of the fold B, and it is important to prevent the doctor from missing the inner side of the fold B. Therefore, fold B is set as the area of ​​interest. In addition, at this time, if it is obtained Figure 5A Such an endoscopic image can also be displayed as shown in the figure "Fold area focused detection". Figure 5A , Figure 5B This polygon is described in .

[0045] In one example, the wrinkle detection unit 14 may use a learning model to recognize wrinkles B in the image G. The learning model is generated by performing deep learning on an image with wrinkle regions labeled, and is stored in advance in the storage unit 2 .

[0046] In another example, the wrinkle detection unit 14 may detect edges unique to wrinkles in the image G and detect wrinkles B based on the edges. In this way, the wrinkle detection unit distinguishes images obtained by photographing wrinkles from other images by using an inference model obtained by learning using wrinkle images obtained in advance as training data and a similar image retrieval technique.

[0047] The extraction unit 15 extracts a portion corresponding to the area of ​​the wrinkle B detected by the wrinkle detection unit 14 from the entire 3D model D generated by the 3D reconstruction unit 13, thereby generating a local 3D model. The local 3D model generated in this way is a model based on a part of the image group, which is composed of the portion of the wrinkle B. Here, an example of detecting the wrinkle area based on the features of the image frames continuously captured is shown, but the wrinkle portion may also be detected based on the result of generating the 3D model.

[0048] The unobserved region detecting unit 16 detects the unobserved region C based on the local 3D model generated by the extracting unit 15. The unobserved region C is a region that has never been imaged by the endoscope 20.

[0049] A 3D model is obtained by overlapping similar feature parts in successively obtained, for example, temporally adjacent image frames, and stitching the images in a panoramic (image pasting) mode. Therefore, when there are no similar feature parts between the multiple images obtained, it is possible that a specific part is skipped during observation and the required image cannot be obtained, and it can be treated as an unobserved area.

[0050] In addition, the depth (distance distribution) information can be obtained from the 3D model to determine the wrinkle portion. There are also endoscopes that have a function of detecting distance distribution built into the imaging unit, and the information can also be used.

[0051] exist Figure 3 and Figure 4 In the example of FIG. 1 , an unobserved region C is generated on the back side of the wrinkle B. The unobserved region C forms a missing portion E consisting of a hole in which the shape of the object A is not restored in the 3D model D. The unobserved region detection unit 16 may detect the missing portion E in the partial 3D model as the unobserved region.

[0052] The unobserved region detecting unit 16 may detect the unobserved region based on the positional relationship between the wrinkles B in the entire 3D model or the partial 3D model and the field of view F of the endoscope 20 .

[0053] Specifically, in the three-dimensional reconstruction process, the position and posture of the camera 20a in the entire 3D model D are calculated. The unobserved region detection unit 16 calculates the position of the field of view F in the entire 3D model or the partial 3D model based on the position and posture of the camera 20a, and determines whether the back side of the fold B is included in the field of view F based on the positional relationship between the field of view F in the entire 3D model or the partial 3D model and the fold B. Then, the unobserved region detection unit 16 detects the back side of the fold B as an unobserved region when the back side of the fold B is not included in the field of view F, and determines that the back side of the fold B is observed when the back side of the fold B is included in the field of view F.

[0054] When the unobserved region detection unit 16 detects an unobserved region, the display control unit 17 generates a display screen H and outputs the display screen H together with the image G to the display device 40 via the output unit 5 , thereby causing the display device 40 to display the display screen H in real time. The display screen H indicates that the unobserved region is detected.

[0055] like Figure 5A and Figure 5B As shown in FIG. 1 , when the detected unobserved area is included in the image G, the display control unit 17 may generate a display screen H that overlaps with the image G and indicates the position of the unobserved area in the image G. The display screen H may be an arrow indicating the unobserved area (see FIG. 1 ). Figure 5A ), or a mark superimposed on a position corresponding to an unobserved area (see Figure 5B ). Figure 5A The arrow H indicates that wrinkles B are detected in the unobserved area on the inner side. Figure 5B The mark H overlaps with the surface side of the wrinkle B in which the unobserved area is detected on the back side.

[0056] like Fig. 6A and Figure 6B As shown in FIG. 1 , when the detected unobserved area is not reflected in the image G, the display control unit 17 may display an alarm notifying the existence of the unobserved area as a display screen H. The alarm H may also be displayed outside the image G. For example, the alarm H may be a frame surrounding the image G (see FIG. 1 ). Fig. 6A ), or text (see Figure 6B ).

[0057] The display screen H may be a guide to reach the unobserved area, for example, a display of the distance from the distal end of the endoscope 20 to the unobserved area, or an operation procedure to reach the unobserved area.

[0058] Next, an image processing method executed by the image processing device 10 will be described.

[0059] like Figure 7 As shown, the image processing method of this embodiment includes: step S1, obtaining an image of the subject A taken by the endoscope 20; step S2, preprocessing the image; step S3, generating an overall 3D model D of the subject A; step S4, detecting a specified area of ​​interest of the subject A from the image; step S5, generating a local three-dimensional model of the subject A; step S6, detecting an unobserved area based on the local three-dimensional model; and step S7, prompting the user with information about the detected unobserved area.

[0060] For example, in a colon endoscopy, an image captured by the endoscope 20 while the endoscope 20 is being pulled out from the cecum toward the anus is input to the image processing device 10 .

[0061] The image acquisition unit 11 sequentially acquires images input to the image processing device 10 (step S1), and then the preprocessing unit 12 sequentially preprocesses the images acquired by the image acquisition unit 11 (step S2). Thus, an image group consisting of a plurality of preprocessed images is obtained. Next, the 3D reconstruction unit 13 generates an entire 3D model D of the subject A based on the image group (step S3).

[0062] By executing steps S2 and S3 every time a new image is acquired, a whole 3D model D of the subject A that was previously imaged by the endoscope 20 is generated in real time.

[0063] In parallel with steps S2 and S3, wrinkle detector 14 detects wrinkles B in the image acquired by image acquisition unit 11 (step S4). Step S4 is executed every time image acquisition unit 11 acquires a new image, thereby detecting the region of wrinkles B in multiple images used to generate overall 3D model D.

[0064] Next, the extraction unit 15 extracts a portion corresponding to the region of the wrinkles B detected from the plurality of images from the entire 3D model D, thereby obtaining a local 3D model of the subject A (step S5 ).

[0065] Next, the unobserved region detecting unit 16 detects the presence or absence of an unobserved region based on the local 3D model (step S6 ).

[0066] When the unobserved area is not detected (step S6: No), step S7 is not executed, and steps S1 to S6 are repeated.

[0067] When the unobserved area is detected (step S6: Yes), the display control unit 17 presents information of the detected unobserved area to the user (step S7). Specifically, the display control unit 17 generates a display screen H, such as an arrow or a mark, indicating that the unobserved area is detected, and outputs the display screen H to the display device 40 together with the image G. The user can recognize the existence of the unobserved area C based on the display screen H displayed in real time on the display device 40.

[0068] The dynamic images input to the image processing device 10 during endoscopy may include images of scenes that are not suitable for detection of unobserved areas. For example, images of scenes where the subject is unclear or partially invisible, such as scenes in water, scenes where bubbles are attached to the lens, and scenes containing debris, are not suitable because it is difficult to generate an accurate 3D model and may cause erroneous detection of unobserved areas.

[0069] According to the present embodiment, based on the region of interest detected from the image G, a portion of the region of interest is extracted from the overall 3D model D. Thus, even when the image group acquired by the processor 1 includes an image of an inappropriate scene, a local 3D model of the region of interest is generated that excludes the inaccurate portion of the image based on the inappropriate scene. By using such a local 3D model for the detection of the unobserved region, it is possible to prevent erroneous detection of the unobserved region and prevent the user from being presented with erroneous information about the unobserved region. Furthermore, it is possible to prevent the user from being excessively presented with information about the unobserved region other than the region of interest, such as information about the unobserved region.

[0070] In general colon endoscopy, unobserved area C is easily generated behind the blind spot of fold B. By making the region of interest the region of fold B, the user can be effectively prevented from missing the back of fold B during colon endoscopy.

[0071] In the present embodiment, the wrinkle detection unit 14 detects the wrinkles B from the two-dimensional image G. However, the wrinkles B may be detected from the entire 3D model D instead of or in addition to this.

[0072] For example, Fig. 8A and Figure 8B As shown, the wrinkle detection unit 14 slices the entire 3D model D in a direction perpendicular to its longitudinal direction to generate a slice Ds of the 3D model D. The 3D model D includes a point group representing the surface of the object A. In the slice Ds, the density of the point group is higher in the wrinkle B portion than in other portions. Figure 8C Shown with Figure 8B The image G corresponding to the slice Ds is obtained. The wrinkle detection unit 14 detects a portion of the point group with high density as a wrinkle B. In this way, by using the 3D model D generated based on the image group, the wrinkle B included in the image group can also be detected.

[0073] In the case of combining the detection of wrinkles B from the image G and the detection of wrinkles B from the 3D model D, as Fig. 9 As shown, the wrinkle detection unit 14 may determine the point group of wrinkles B in the 3D model D based on the wrinkles B detected from the plurality of images G.

[0074] That is, the wrinkle detection unit 14 extracts, from the 3D model D, point groups corresponding to regions of wrinkles B detected from the plurality of images G, and recognizes point groups corresponding to regions of the same wrinkle B in the plurality of images G as one wrinkle. Fig. 9 1 and 2 show the distribution of point groups in a three-dimensional space. Point groups of quadrilaterals, black circles, white circles, and triangles represent point groups of wrinkles B in different images G.

[0075] In the 3D model D, there may be a region where a point group exists at a high density in addition to the wrinkles B. Based on both the image G and the 3D model D, the wrinkles B can be detected more accurately.

[0076] (Second Embodiment)

[0077] Next, an image processing device, an image processing method, an image processing program, and a recording medium according to a second embodiment of the present invention will be described.

[0078] This embodiment is different from the first embodiment in the method of generating a local 3D model. In this embodiment, the configurations different from the first embodiment are described, and the configurations identical to the first embodiment are denoted by the same reference numerals and description thereof is omitted.

[0079] Similar to the first embodiment, the image processing device 10 of this embodiment is applied to an endoscope system 100 including an endoscope 20, a control device 30, and a display device 40. The image processing device 10 includes a processor 101, a storage unit 2, a memory 3, an input unit 4, and an output unit 5.

[0080] like Fig.10 As shown, the processor 101 includes an image selection unit 18 as a functional unit in addition to the image acquisition unit 11 , the preprocessing unit 12 , the 3D reconstruction unit 13 , the wrinkle detection unit 14 , the unobserved region detection unit 16 , and the display control unit 17 .

[0081] The image acquisition unit 11 obtains a plurality of images by sequentially acquiring images input to the image processing device 10 .

[0082] The wrinkle detection unit 14 detects wrinkles B in each image acquired by the image acquisition unit 11 .

[0083] The image selection unit 18 selects only the image in which the wrinkle B is detected by the wrinkle detection unit 14 , and uses it to generate a 3D model.

[0084] The 3D reconstruction unit 13 generates a 3D model of the subject A based on the plurality of images selected by the image selection unit 18. The 3D model generated in this way is a local 3D model based on a part of the plurality of images, which is composed of the wrinkle B and its surroundings.

[0085] The unobserved region detecting unit 16 detects the unobserved region based on the local 3D model generated by the 3D reconstruction unit 13 .

[0086] Next, an image processing method executed by the image processing device 10 will be described.

[0087] like Fig.11As shown, the image processing method of this embodiment includes: steps S1, S2, S4, S6, S7, step S8 of selecting an image in which a focus area is detected, and step S9 of generating a local 3D model according to the selected image.

[0088] The image acquisition unit 11 sequentially acquires images input to the image processing device 10 (step S1 ), and then the wrinkle detection unit 14 detects the presence or absence of wrinkles B in each image acquired by the image acquisition unit 11 (step S4 ).

[0089] When wrinkles B are not detected (step S4: No), steps S2 and S6 to S9 are not performed, and steps S1 and S4 are repeated.

[0090] On the other hand, when wrinkle B is detected (step S4: Yes), the image selection unit 18 selects the image in which wrinkle B is detected for generating a 3D model, and the preprocessing unit 12 performs preprocessing on the selected image. The 3D reconstruction unit 13 generates a local 3D model of the subject A based on the preprocessed multiple images, and the local 3D model is a 3D model of wrinkle B and its surroundings (step S9).

[0091] Next, steps S6 and S7 are executed in the same manner as in the first embodiment.

[0092] Thus, according to this embodiment, a local 3D model is generated only from an image of a scene containing a region of interest. By using such a local 3D model for detecting an unobserved region, it is possible to prevent false detection of an unobserved region and prevent the user from being prompted with erroneous information about the unobserved region. Furthermore, it is possible to prevent the user from being prompted with excessive information about the unobserved region.

[0093] Furthermore, the region of interest is the region of the fold B, and thus, the user can be effectively prevented from missing the back side of the fold B during colon endoscopy.

[0094] Furthermore, since the number of images used for generating a 3D model is limited, the amount of processing for generating a 3D model can be reduced and the processing speed can be improved compared to the first embodiment.

[0095] (Third Embodiment)

[0096] Next, an image processing device, an image processing method, an image processing program, and a recording medium according to a third embodiment of the present invention will be described.

[0097] This embodiment is different from the first embodiment in the method of generating a local 3D model. In this embodiment, the configurations different from the first embodiment are described, and the configurations identical to the first embodiment are denoted by the same reference numerals and description thereof is omitted.

[0098] Similar to the first embodiment, the image processing device 10 of this embodiment is applied to an endoscope system 100 including an endoscope 20, a control device 30, and a display device 40. The image processing device 10 includes a processor 102, a storage unit 2, a memory 3, an input unit 4, and an output unit 5.

[0099] like Fig.12 As shown, the processor 102 includes an extraction unit 19 as a functional unit in addition to the preprocessing unit 12 , the 3D reconstruction unit 13 , the wrinkle detection unit 14 , the unobserved region detection unit 16 , and the display control unit 17 .

[0100] The preprocessing unit 12 , the 3D reconstruction unit 13 , and the wrinkle detection unit 14 perform processing using the image G acquired by the image acquisition unit 11 , similarly to the first embodiment.

[0101] The extraction unit 19 extracts a portion generated based on an image in which wrinkles B are detected by the wrinkle detection unit 14 from the entire 3D model D generated by the 3D reconstruction unit 13, thereby generating a local three-dimensional model consisting of the wrinkles B and their surroundings.

[0102] The unobserved region detecting unit 16 detects the unobserved region based on the local 3D model generated by the extracting unit 19 .

[0103] Next, an image processing method executed by the image processing device 10 will be described.

[0104] like Fig.13 As shown, the image processing method of this embodiment includes: steps S1 to S4, S6, S7, and step S10 of generating a local three-dimensional model of the subject A.

[0105] After executing steps S1 to S4 in the same manner as in the first embodiment, the extraction unit 19 extracts a portion generated based on the image in which the wrinkles B are detected from the entire 3D model D, thereby obtaining a partial 3D model of the subject A (step S10 ).

[0106] Next, steps S6 and S7 are executed in the same manner as in the first embodiment.

[0107] Thus, according to the present embodiment, a local 3D model consisting only of a portion corresponding to an image of a scene including a region of interest is generated. By using such a local 3D model for detecting an unobserved region, it is possible to prevent erroneous detection of an unobserved region and prevent the user from being presented with erroneous information about the unobserved region. Furthermore, it is possible to prevent the user from being presented with excessive information about the unobserved region.

[0108] Furthermore, the region of interest is the region of the fold B, and thus, the user can be effectively prevented from missing the back side of the fold B during colon endoscopy.

[0109] Furthermore, according to the present embodiment, the local 3D model includes not only the region of interest but also its surroundings, thereby improving the detection accuracy of the region of interest and the unobserved regions of its surroundings.

[0110] In each of the above-described embodiments, the display control unit 17 may change the display screen H after the unobserved region C is imaged by the endoscope 20 .

[0111] After the user recognizes the existence of the unobserved area C from the display screen H, the user moves the field of view F of the endoscope 20 to observe the unobserved area C. The processor 1, 101, 102 calculates the positional relationship between the field of view F and the unobserved area C in the overall 3D model D or the local 3D model D, and determines that the unobserved area C is imaged when the unobserved area C is arranged in the field of view F.

[0112] After the unobserved area C is photographed, the display control unit 17 may delete the display screen H such as arrows, marks, frames or characters (see FIG. 5A to FIG. 6B Alternatively, the display control unit 17 may change the form of the display screen H, for example, the color of the frame (see Fig. 6A ), or change the text to "Observation completed" etc. (refer to Figure 6B ).

[0113] When the captured unobserved area C is reflected again in the image G, the display control unit 17 may not display the display screen H. Alternatively, the display control unit 17 may display a display screen indicating that the unobserved area C has been observed, for example, by displaying an arrow, a mark, or a frame in another color (see FIG. 5A to FIG. 6B ), or display text such as "Observed".

[0114] In the above-mentioned embodiments, the processors 1 , 101 , and 102 do not necessarily need to perform preprocessing, 3D reconstruction, wrinkle B detection, and other processing on the entire image G, and may only process a portion selected from each image G.

[0115] In the above-mentioned embodiments, the predetermined region of interest is the fold B of the large intestine, but the region of interest is not limited thereto, and may be any region where it is necessary to detect whether there is an unobserved region. In order to prevent blind spots from being missed, the region of interest is preferably a region that is prone to blind spots in the field of view F of the endoscope 20, for example, it may be a tissue protruding from the inner surface of the lumen such as a polyp.

[0116] Above, the embodiments of the present invention and its variants are described in detail with reference to the accompanying drawings, but the specific structure of the present invention is not limited to the above embodiments and variants, and various design changes can be made within the scope of the gist of the present invention. In addition, the constituent elements shown in the above embodiments and variants can be appropriately combined.

[0117] For example, the subject may be a lumen other than the large intestine, or an organ other than the lumen that can be the subject of endoscopic examination. The region of interest may also be set according to the subject.

[0118] Description of Reference Numerals

[0119] 1 processor

[0120] 2 Storage unit (recording medium)

[0121] 2a Image processing program

[0122] 10 Image processing device

[0123] 20 Endoscope

[0124] ASubject

[0125] C Unobserved area

[0126] D Overall 3D Model

[0127] E missing part

[0128] F-Field of View

[0129] G-image

Claims

1. An image processing device comprising a processor, The processor performs the following processing: Acquire multiple images of the subject captured by the endoscope, detecting a predetermined region of interest of the subject included in the plurality of images, generating a local three-dimensional model of the subject based on a part of the plurality of images, the local three-dimensional model being a three-dimensional model of the predetermined area of ​​interest, Based on the local three-dimensional model, an unobserved area of ​​the object that is not captured by the endoscope is detected.

2. The image processing device according to claim 1, wherein: The prescribed region of interest is tissue protruding from the inner surface of the lumen.

3. The image processing device according to claim 1, wherein: The processor also generates an overall three-dimensional model of the object contained in the multiple images according to the multiple images. The process of detecting the specified area of ​​interest includes: detecting the specified area of ​​interest based on at least one of each of the plurality of images and the overall three-dimensional model, The process of generating the local three-dimensional model includes extracting a portion of the prescribed region of interest from the overall three-dimensional model.

4. The image processing device according to claim 3, wherein: When detecting the predetermined target area from each of the plurality of images, The processor recognizes the predetermined region of interest in the plurality of images using a learning model, wherein the learning model is generated by performing deep learning on images labeled with the predetermined region of interest and is pre-stored in a storage unit.

5. The image processing device according to claim 3, wherein: When detecting the predetermined target area from each of the plurality of images, The processor detects an edge specific to the prescribed region of interest in the image, and detects the prescribed region of interest based on the edge.

6. The image processing device according to claim 3, wherein: When the predetermined region of interest is detected based on both each of the plurality of images and the overall three-dimensional model, The processor determines the predetermined region of interest within the entire three-dimensional model based on the predetermined region of interest detected from each of the plurality of images.

7. The image processing device according to claim 1, wherein: The process of detecting the specified region of interest includes: detecting the specified region of interest from the plurality of images respectively; The process of generating the local three-dimensional model includes generating the local three-dimensional model based on only the image in which the prescribed target region is detected among the plurality of images.

8. The image processing device according to claim 7, wherein: The local three-dimensional model includes the predetermined region of interest and its surrounding area.

9. The image processing apparatus according to claim 1, wherein: The processor also generates an overall three-dimensional model of the object contained in the multiple images according to the multiple images. The process of detecting the specified region of interest includes: detecting the specified region of interest from the plurality of images respectively; The process of generating the local three-dimensional model includes extracting, from the overall three-dimensional model, a portion generated based on the image in which the prescribed area of ​​interest is detected.

10. The image processing apparatus according to claim 1, wherein: The process of detecting the unobserved area includes detecting the unobserved area according to a positional relationship between the prescribed region of interest in a local three-dimensional model or an entire three-dimensional model of the object and the field of view of the endoscope.

11. The image processing apparatus according to claim 10, wherein: According to the positional relationship between the specified region of interest and the field of view of the endoscope, detecting the unobserved region includes: The processor performs the following steps: Calculate the position of the endoscope's field of view in the local three-dimensional model or the overall three-dimensional model according to the position and posture of the endoscope's camera head, determining whether the inner side of the specified region of interest is included in the field of view according to the positional relationship between the specified region of interest and the field of view of the endoscope in the local three-dimensional model or the overall three-dimensional model, When the back side of the predetermined region of interest is not included in the field of view of the endoscope, the back side of the predetermined region of interest is detected as the unobserved region.

12. The image processing apparatus according to claim 1, wherein: The process of detecting the unobserved area includes detecting a missing portion of the local three-dimensional model as the unobserved area.

13. The image processing apparatus according to claim 1, wherein: Furthermore, when the unobserved area is detected, the processor causes the display device to display a display screen indicating that the unobserved area is detected in real time.

14. The image processing apparatus according to claim 13, wherein: The display screen includes an arrow and a mark superimposed on the image and indicating a position of the predetermined target area in the image, and a text frame indicating a detection status.

15. The image processing apparatus according to claim 13, wherein: The display screen is a mark superimposed on a position corresponding to the unobserved area in the image.

16. An image processing method, wherein: Acquire multiple images of the subject captured by the endoscope, detecting a predetermined region of interest of the subject included in the plurality of images, generating a local three-dimensional model of the subject based on a part of the plurality of images, the local three-dimensional model being a three-dimensional model of the predetermined area of ​​interest, Based on the local three-dimensional model, an unobserved area of ​​the object that is not captured by the endoscope is detected.

17. A computer-readable non-transitory recording medium having an image processing program recorded thereon, wherein: The image processing program causes the computer to perform the following processing: Acquire multiple images of the subject captured by the endoscope, detecting a predetermined region of interest of the subject included in the plurality of images, generating a local three-dimensional model of the subject based on a part of the plurality of images, the local three-dimensional model being a three-dimensional model of the predetermined area of ​​interest, Based on the local three-dimensional model, an unobserved area of ​​the object that is not captured by the endoscope is detected.

18. An image processing device comprising a processor, The processor performs the following processing: Acquire the images of the subject continuously captured by the endoscope, Detecting a specified area of ​​interest of the subject according to image features contained in the image frame, Based on a result of generating a three-dimensional model of the predetermined region of interest using continuous images corresponding to the predetermined region of interest, an unobserved region of the object that is not captured by the endoscope is detected.

19. The image processing apparatus according to claim 18, wherein: The processor indicates unobserved areas.

20. The image processing apparatus according to claim 18, wherein: The prescribed region of interest is a wrinkle on the inner surface of the lumen.

21. An image processing device comprising a processor, The processor performs the following processing: Acquire multiple images of the subject continuously captured by the endoscope, generating a local three-dimensional model of the subject based on a part of the plurality of images, the local three-dimensional model being a three-dimensional model of a predetermined area of ​​interest, A predetermined region of interest of the object is detected based on the local three-dimensional model, and an unobserved region of the object that is not captured by the endoscope is detected.

22. An image processing device comprising a processor, The processor performs the following processing: Acquire multiple images of the subject continuously captured by the endoscope, generating a local three-dimensional model of the subject based on the plurality of images, detecting a predetermined region of interest of the subject according to image features contained in an image frame or the generated local three-dimensional model, Using continuous images corresponding to the predetermined region of interest, an unobserved region of the object that is not captured by the endoscope is detected for the predetermined region of interest based on the local three-dimensional model.

23. An image processing device comprising a processor, The processor performs the following processing: Acquire multiple images of the subject captured by the endoscope, detecting a predetermined region of interest of the subject included in the plurality of images, generating a local three-dimensional model of the subject based on a part of the plurality of images, the local three-dimensional model being a three-dimensional model of the predetermined area of ​​interest, Based on the local three-dimensional model, detecting an unobserved area of ​​the object that is not photographed by the endoscope, When the unobserved area is detected, the display device is caused to display in real time a display screen indicating that the unobserved area is detected, the display screen including a graphic superimposed on the image and indicating the position of the specified area of ​​interest in the image and a text box indicating the detection status of the area of ​​interest.

Images