Image processing device, control method and program

The image processing device enhances endoscopy by generating three-dimensional reconstructions and matching them with standard models to clearly display imaged regions, addressing the challenge of distinguishing imaged and unimaged areas and guiding the examiner for comprehensive imaging.

JP7589738B2Inactive Publication Date: 2024-11-26NEC CORP
View PDF 5 Cites 0 Cited by

Patent Information

Application Number
JP2022524803
Authority / Receiving Office
JP · JP
Patent Type
Patents
Current Assignee / Owner
Filing Date
2020-05-21
Publication Date
2024-11-26
Estimated Expiration
Not applicable · inactive patent

AI Technical Summary

Technical Problem

In endoscopy, distinguishing between imaged and unimaged areas of the test subject is challenging, making it difficult to accurately identify and display the photographed regions of the inspection target.

Method used

An image processing device that generates three-dimensional reconstruction data based on endoscope images, matches this data with a standard three-dimensional model, and outputs a clear image on a display device indicating the photographed regions, with warnings for insufficiently imaged areas and guidance for the examiner.

Benefits of technology

Enables accurate and efficient display of imaged regions during endoscopy, guiding the examiner to complete thorough imaging by highlighting unimaged areas and providing navigation support.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 0007589738000001
    Figure 0007589738000001
  • Figure 0007589738000002
    Figure 0007589738000002
  • Figure 0007589738000003
    Figure 0007589738000003
Patent Text Reader

Abstract

This image processing device 1X comprises a three-dimensional reconstruction means 31X, a matching means 32X, and an output control means 33X. The three-dimensional reconstruction means 31X generates reconstruction data Mr in which an inspection target is three-dimensionally reconstructed, on the basis of a captured image Ic obtained by photographing the inspection target by a photographing unit provided in an endoscope. The matching means 32X matches a standard model Ms, which is a standard three-dimensional model of the inspection target, with the reconstruction data Mr. Based on the matching result, the output control means 33X causes a display device 2X to display a photographed area specifying image indicating the photographed area of the inspection target on the standard model.
Need to check novelty before this filing date? Find Prior Art

Description

[Technical field]

[0001] The present disclosure relates to the technical field of an image processing device, a control method, and a storage medium that process images acquired in an endoscopic examination. [Background technology]

[0002] Conventionally, an endoscope system that displays an image of the inside of an organ lumen has been known. For example, Patent Document 1 discloses a technique for generating 3D model data of an object to be examined based on an image captured by an endoscope and displaying the data as a 3D model image. In addition, Non-Patent Document 1 discloses a technique for restoring the 3D shape of the stomach from a captured image using SfM (Structure from Motion). Furthermore, Non-Patent Document 2 discloses a technique for non-rigid registration between 3D shapes. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] International Publication WO2017 / 203814 [Non-patent literature]

[0004] [Non-Patent Document 1] Aji Resindra Widya et al. “3D Reconstruction of Whole Stomach from Endoscope Video Using Structure-from-Motion”,2019 41st Annual International Conference of the IEEE Engineering in Medicine and Biology Society, pp. 3900-3904. [Non-Patent Document 2] Dai,H et al. “Non-rigid 3D Shape Registration using an Adaptive Template”, The European Conference on Computer Vision (ECCV) Workshops, 2018. Summary of the Invention [Problem to be solved by the invention]

[0005] During an endoscopic examination, the examiner needs to move the imaging unit of the endoscope to areas of the subject that have not yet been photographed, but it has been difficult to accurately distinguish between areas that have been photographed and areas that have not yet been photographed.

[0006] In view of the above-mentioned problems, an object of the present disclosure is to provide an image processing device, a control method, and a storage medium that are capable of preferably displaying an imaged area of ​​an examination subject in an endoscopic examination. [Means for solving the problem]

[0007] One aspect of the image processing device includes a 3D reconstruction unit that generates reconstruction data of an object to be inspected, based on an image of the object to be inspected captured by an imaging unit provided in an endoscope, a matching unit that performs matching between a standard model, which is a standard 3D model of the object to be inspected, and the reconstruction data, and an output control unit that displays, on a display device, an imaging area indicating image showing an imaging area of ​​the object to be inspected on the standard model, based on a result of the matching. The output control means outputs a warning regarding a partial area in which the photographed area has a photographing time less than a threshold value. . Another aspect of the image processing device includes a 3D reconstruction means for generating reconstruction data of the inspection object in 3D based on an image of the inspection object photographed by an imaging unit provided in an endoscope, a matching means for matching a standard model that is a standard 3D model of the inspection object with the reconstructed data, and an output control means for causing a display device to display an imaging area indication image showing the imaging area of ​​the inspection object on the standard model based on a result of the matching, and the output control means changes the display mode within the imaging area based on the difference in imaging time within the imaging area. Another aspect of the image processing device has a 3D reconstruction means for generating reconstruction data of the inspection object in 3D based on an image of the inspection object captured by an imaging unit provided in an endoscope, a matching means for matching a standard model that is a standard 3D model of the inspection object with the reconstructed data, and an output control means for causing a display device to display an imaging area indication image that shows the imaged area of ​​the inspection object on the standard model based on a result of the matching, and the output control means causes the display device to display information indicating the progress rate of the imaging.

[0008] One aspect of the control method includes a computer generating reconstruction data of an object to be inspected, based on an image of the object to be inspected captured by an imaging unit provided in an endoscope, matching the reconstruction data with a standard model, which is a standard three-dimensional model of the object to be inspected, and displaying, on a display device, an imaging area indicating image showing an imaged area of ​​the object to be inspected on the standard model, based on a result of the matching. If there is a partial area in the photographed area whose photographing time is lower than a threshold, a warning is output regarding the partial area. , a control method.

[0009] One aspect of the program causes a computer to function as a 3D reconstruction means for generating reconstruction data of an object of inspection 3D reconstructed based on an image of the object of inspection captured by an imaging unit provided in an endoscope, a matching means for matching a standard model, which is a standard 3D model of the object of inspection, with the reconstructed data, and an output control means for displaying, on a display device, an imaging area indicating image showing an imaging area of ​​the object of inspection on the standard model based on a result of the matching. the output control means outputs a warning regarding a partial area in the photographed area, the partial area having a photographing time lower than a threshold value. It is a program. Effect of the Invention

[0010] According to the present disclosure, in an endoscopic examination, an imaged area of ​​an examination subject can be suitably displayed. [Brief description of the drawings]

[0011] [Figure 1] 1 shows a schematic configuration of an endoscopic examination system. [Diagram 2] 2 shows the hardware configuration of an image processing device. [Diagram 3] FIG. 2 is a functional block diagram of the image processing device. [Figure 4] FIG. 2 is a diagram illustrating an overview of the processing performed by a three-dimensional reconstruction unit and a matching unit. [Diagram 5] 5 is an example of a flowchart showing an overview of a display process executed by the image processing device during an endoscopic examination in the first embodiment. [Figure 6] 13 is an example of a flowchart illustrating a procedure of a standard model generation process. [Figure 7] 13 is an example of a flowchart illustrating a procedure of a matching process. [Figure 8] 11 is an example of a flowchart illustrating a procedure for extracting features of reconstructed data. [Figure 9] (A) A close-up of a portion of the reconstructed data, (B) a close-up of a portion of the reconstructed data after removing isolated points, and (C) a close-up of a portion of the reconstructed data after Delaunay division. [Figure 10] 13 shows a first display example of an examiner confirmation screen. [Figure 11] 13 shows a second display example of the examiner confirmation screen. [Figure 12] 13 shows a third display example of the examiner confirmation screen. [Figure 13] 13 shows a fourth display example of the examiner confirmation screen. [Figure 14] 13 shows a fifth display example of the examiner confirmation screen. [Figure 15] 13 shows a sixth display example of the examiner confirmation screen. [Figure 16] FIG. 11 is a block diagram of an image processing device according to a second embodiment. [Figure 17] 13 is an example of a flowchart showing a processing procedure of an image processing device in a second embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0012] Hereinafter, embodiments of an image processing device, a control method, and a storage medium will be described with reference to the drawings.

[0013] First Embodiment (1) System Configuration Fig. 1 shows a schematic configuration of an endoscopic examination system 100. As shown in Fig. 1, the endoscopic examination system 100 mainly includes an image processing device 1, a display device 2, and an endoscope 3 connected to the image processing device 1. Hereinafter, as a representative example, processing in an endoscopic examination in which the stomach is the examination target will be described.

[0014] The image processing device 1 acquires images (also called "taken images Ic") taken by the endoscope 3 in a time series from the endoscope 3, and displays a screen (also called "examiner confirmation screen") based on the taken images Ic on the display device 2. The taken images Ic are images of the stomach wall of the subject, which is the subject of the examination, taken at a predetermined time interval during the examination. In this embodiment, the image processing device 1 generates data (also called "reconstructed data Mr") that reconstructs the three-dimensional shape of the subject's stomach from the taken images Ic, and matches it with a standard three-dimensional model of the stomach (also called "standard model Ms") that has been generated in advance. Then, based on the matching result, the image processing device 1 generates an image (also called "imaging area indicating image Im") that indicates the area already photographed by the endoscope 3 on the standard model Ms, and displays the imaging area indicating image Im on the display device 2. The photographed area indicating image Im is an image with a sense of depth (stereoscopic effect) in which a virtual examination object in a three-dimensional space and its photographed areas are converted into information on a two-dimensional plane using three-dimensional computer graphics.

[0015] The display device 2 is a display or the like that performs a predetermined display based on a display signal supplied from the image processing device 1.

[0016] The endoscope 3 is an instrument that is inserted into the large intestine of a subject to capture images of the lumen of the large intestine. The endoscope 3 mainly includes a connection section 35 for connecting to the image processing device 1, an operation section 36 through which the examiner performs predetermined input, a flexible shaft 37 that is inserted into the lumen, and a tip section 38 that incorporates an imaging section such as a micro-imaging element.

[0017] (2) Hardware Configuration 2 shows a hardware configuration of the image processing device 1. The image processing device 1 mainly includes a processor 11, a memory 12, an interface 13, an input unit 14, a light source unit 15, and a sound output unit 16. These elements are connected via a data bus 19.

[0018] The processor 11 executes a predetermined process by executing a program or the like stored in the memory 12. The processor 11 is a processor such as a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), or a quantum processor.

[0019] The memory 12 is composed of various volatile memories, such as RAM (Random Access Memory) and ROM (Read Only Memory), that are mainly used as working memories, and non-volatile memories that store information necessary for processing by the image processing device 1. The memory 12 may include an external storage device, such as a hard disk connected to or built into the image processing device 1, or may include a removable storage medium, such as a flash memory. The memory 12 stores programs for the image processing device 1 to execute each process in this embodiment. The memory 12 also includes a captured image storage unit 21 and a standard model storage unit 22.

[0020] The captured image storage unit 21 stores a series of captured images Ic captured by the endoscope scope 3 during an endoscopic examination under the control of the processor 11. These captured images Ic are images used to generate the reconstruction data Mr, and are stored in the captured image storage unit 21 in association with, for example, identification information of the subject (e.g., patient ID) and time stamp information. The standard model storage unit 22 stores a standard model Ms generated in advance using CT images of multiple subjects. The generation of standard model information will be described later with reference to FIG. 5. At least one of the captured image storage unit 21 and the standard model storage unit 22 may be provided in an external device capable of data communication with the image processing device 1 via a wired or wireless connection, instead of the memory 12. The external device may be one or more server devices capable of data communication with the image processing device 1 via a communication network.

[0021] The interface 13 performs an interface operation between the image processing device 1 and an external device. For example, the interface 13 supplies the display information "Id" generated by the processor 11 to the display device 2. The interface 13 also supplies light generated by the light source unit 15 to the endoscope 3. The interface 13 also supplies an electrical signal indicating the captured image Ic supplied from the endoscope 3 to the processor 11. The interface 13 may be a communication interface such as a network adapter for communicating with an external device by wire or wirelessly, or may be a hardware interface conforming to USB (Universal Serial Bus), SATA (Serial AT Attachment), or the like.

[0022] The input unit 14 generates an input signal based on an operation by the examiner. The input unit 14 is, for example, a button, a touch panel, a remote controller, a voice input device, etc. The light source unit 15 generates light to be supplied to the tip 38 of the endoscope 3. The light source unit 15 may also include a built-in pump for sending out water or air to be supplied to the endoscope 3. The sound output unit 16 outputs sound based on the control of the processor 11.

[0023] (3) Functional Blocks Fig. 3 is a functional block diagram of the image processing device 1. As shown in Fig. 3, the processor 11 of the image processing device 1 functionally includes a captured image acquisition unit 30, a three-dimensional reconstruction unit 31, a matching unit 32, and an output control unit 33.

[0024] The photographed image acquiring unit 30 acquires the photographed images Ic captured by the endoscope 3 at predetermined intervals via the interface 13. Then, the photographed image acquiring unit 30 supplies the acquired photographed images Ic to the 3D reconstruction unit 31. In addition, the photographed image acquiring unit 30 stores the acquired photographed images Ic in the photographed image storage unit 21 in association with a time stamp, a patient ID, etc.

[0025] The three-dimensional reconstruction unit 31 generates reconstructed data Mr indicating the three-dimensional shape of the photographed stomach based on the multiple photographed images Ic acquired during the endoscopic examination by the photographed image acquisition unit 30. The reconstructed data Mr includes, for example, point cloud data having three-dimensional position information.

[0026] In this case, for example, after acquiring the number of captured images Ic required for generating the reconstructed data Mr, the three-dimensional reconstruction unit 31 constructs the reconstructed data Mr using a method for restoring the three-dimensional shape of the subject and the relative position of the imaging unit from the multiple images. As such a method, for example, a method such as Structure from Motion (SfM) exists. Thereafter, the three-dimensional reconstruction unit 31 updates the generated reconstructed data Mr, for example, every time it acquires a predetermined number of captured images Ic. The above-mentioned predetermined number may be one or more, and is previously determined to a value taking into consideration, for example, the processing capacity of the image processing device 1. Then, the three-dimensional reconstruction unit 31 supplies the generated (including updated) reconstructed data Mr to the matching unit 32. A method for generating the reconstructed data Mr will be described later.

[0027] The matching unit 32 matches the reconstructed data Mr supplied from the 3D reconstruction unit 31 with the standard model Ms stored in the standard model storage unit 22, and supplies a matching result "Rm" that is the result of the matching to the output control unit 33. In this case, the matching unit 32 performs non-rigid alignment, and generates, as the matching result Rm, the reconstructed data Mr after non-rigid alignment and data that represents the reconstructed data Mr in a common 3D coordinate system.

[0028] The output control unit 33 generates an imaging area indicating image Im by projecting the reconstruction data Mr and the standard model Ms, which have been aligned on the three-dimensional coordinate system, onto a predetermined two-dimensional coordinate system based on the matching result Rm generated by the matching unit 32. Then, the output control unit 33 generates display information Id for displaying an examiner confirmation screen in which the imaging area indicating image Im and the latest captured image Ic acquired by the captured image acquisition unit 30 are arranged side by side, and supplies the generated display information Id to the display device 2. The output control unit 33 also outputs information for providing guidance or warning regarding imaging by the examiner using the endoscope 3. This information may be output on the examiner confirmation screen or by the sound output unit 16. Specific processing by the output control unit 33 will be described later.

[0029] FIG. 4 is a diagram showing an overview of the processing of the three-dimensional reconstruction unit 31 and the matching unit 32. As shown in FIG. 4, the three-dimensional reconstruction unit 31 generates reconstructed data Mr corresponding to the three-dimensional shape of the photographed region based on a plurality of photographed images Ic acquired up to now during endoscopic examination. Then, the matching unit 32 performs matching (non-rigid positioning) between the standard model Ms constructed by referring to the standard model storage unit 22 and the reconstructed data Mr. As a result, the matching unit 32 generates a matching result Rm in which the standard model Ms representing the entire stomach and the reconstructed data Mr corresponding to the photographed region are associated in three-dimensional space. Based on such a matching result Rm, the output control unit 33 can suitably generate the photographed region indicating image Im.

[0030] Each of the components of the photographed image acquisition unit 30, the three-dimensional reconstruction unit 31, the matching unit 32, and the output control unit 33 described in FIG. 3 and FIG. 4 can be realized, for example, by the processor 11 executing a program. More specifically, each of the components can be realized by the processor 11 executing a program stored in the memory 12. Also, each of the components may be realized by recording a necessary program in an arbitrary non-volatile storage medium and installing it as necessary. Also, each of these components may be realized by any combination of hardware, firmware, and software, without being limited to being realized by software by a program. Also, each of these components may be realized by using an integrated circuit programmable by a user, such as an FPGA (field-programmable gate array) or a microcomputer. In this case, a program composed of each of the above components may be realized by using this integrated circuit. In this way, each of the components may be realized by an arbitrary computer (controller) including hardware other than a processor. The above is the same in other embodiments described later.

[0031] (4) Processing Flow Next, the processing flow in the first embodiment will be described.

[0032] (4-1) Overview FIG. 5 is an example of a flowchart showing an overview of the display process executed by the image processing device 1 during endoscopic examination in the first embodiment.

[0033] First, the image processing device 1 acquires the captured image Ic (step S11). In this case, the captured image acquisition unit 30 of the image processing device 1 receives the captured image Ic from the endoscope scope 3 via the interface 13.

[0034] Next, the image processing device 1 generates reconstructed data Mr by 3D reconstructing the inspection target from the multiple captured images Ic acquired in step S11 (step S12). In this case, the 3D reconstruction unit 31 of the image processing device 1 generates reconstructed data Mr corresponding to the captured region using a technique such as SfM, based on the captured images Ic acquired from the start of the inspection to the current processing time.

[0035] Next, the image processing device 1 performs matching between the standard model Ms and the reconstructed data Mr (step S13). In this case, the matching unit 32 of the image processing device 1 generates a matching result Rm by performing non-rigid registration between the standard model Ms acquired from the standard model storage unit 22 and the reconstructed data Mr generated by the three-dimensional reconstruction unit 31. The process of step S13 will be described in detail with reference to FIG.

[0036] Then, the image processing device 1 displays the photographing area indicating image Im on the display device 2 (step S14). In this case, the output control unit 33 of the image processing device 1 generates display information Id for displaying an examiner confirmation screen in which the photographing area indicating image Im is arranged next to the latest photographed image Ic, as described later, and supplies the display information Id to the display device 2. In addition, the output control unit 33 performs a predetermined output for notifying the examiner of the presence of an area with a short photographing time within the photographed area of ​​the stomach to be examined, and displays information indicating the direction in which the endoscope scope 3 should be moved or the area to be photographed on the examiner confirmation screen.

[0037] Next, the image processing device 1 determines whether or not the endoscopic examination has ended (step S15). For example, when the image processing device 1 detects a predetermined input to the input unit 14 or the operation unit 36, the image processing device 1 determines that the endoscopic examination has ended. Then, when the image processing device 1 determines that the endoscopic examination has ended (step S15; Yes), the image processing device 1 ends the processing of the flowchart. On the other hand, when the image processing device 1 determines that the endoscopic examination has not ended (step S15; No), the image processing device 1 returns the processing to step S11. Then, in step S11, the image processing device 1 acquires a captured image Ic newly generated by the endoscope 3, and re-executes the processing of steps S12 to S15 by including the captured image Ic as a processing target.

[0038] (4-2) Standard model generation process 6 is a flowchart showing an example of a process for generating a standard model Ms to be stored in the standard model storage unit 22. For convenience of explanation, the image processing device 1 will be described below as the processing subject, but any device other than the image processing device 1 may be the processing subject. In this case, after the standard model Ms is generated by the any device, the generated standard model Ms is stored in the memory 12 via data communication or a removable storage medium or the like.

[0039] First, the image processing device 1 acquires a 3D-CT image of the abdomen (step S21). Then, the image processing device 1 extracts a stomach area from the 3D-CT image based on a user input (step S22). In this case, the image processing device 1 displays a 3D-CT image on the display device 2, and receives a user input specifying the stomach area via the input unit 14. Then, the image processing device 1 executes the processes of steps S21 and S22 on the 3D-CT images of multiple subjects, and extracts the stomach areas of the multiple subjects.

[0040] Next, the image processing device 1 averages the stomach volume data representing the stomach regions extracted from the 3D-CT images of the multiple subjects, and generates a three-dimensional standard stomach volume model (step S23). The stomach volume data is, for example, three-dimensional voxel data that represents the stomach region with two values ​​of 0 and 1. In step S23, the image processing device 1 aligns the stomach volume data of each subject based on user input, and then generates a three-dimensional standard stomach volume model by taking the arithmetic average of the aligned stomach volume data.

[0041] Next, the image processing device 1 creates a three-dimensional standard model Ms, which is a surface model, from the three-dimensional standard stomach volume model generated in step S23 (step S24). In this case, the image processing device 1 converts the standard stomach volume model into the standard model Ms using any algorithm that converts voxel data into polygon data. Examples of the above algorithm include the marching cubes method and the marching tetrahedra method. After step S24, the generated standard model Ms is stored in the memory 12 that the image processing device 1 can refer to.

[0042] (4-3) Matching process FIG. 7 is an example of a flowchart showing the procedure of the matching process between the standard model Ms and the reconstructed data Mr executed in step S13.

[0043] First, the matching unit 32 extracts feature points that serve as landmarks from each of the standard model Ms and the reconstructed data Mr (step S31). The feature point extraction process for the reconstructed data Mr will be described later with reference to Figs. 8 and 9. Note that the standard model Ms may have predetermined identification labels or the like attached to the feature points to be extracted.

[0044] Then, the matching unit 32 obtains correspondence between the feature points extracted from the standard model Ms and the reconstructed data Mr, respectively, and performs rigid body matching between the standard model Ms and the reconstructed data Mr (step S32). In this case, the matching unit 32 translates (including rotates) at least one of the standard model Ms and the reconstructed data Mr so as to minimize the distance between the corresponding feature points.

[0045] Next, the matching unit 32 morphs the standard model Ms using the reconstructed data Mr as a reference (step S33). In this case, the matching unit 32 matches the standard model Ms with the reconstructed data Mr by using a matching method between point groups such as ICPD (Iterative Coherent Point Drift), and moves points in the standard model Ms other than those considered to be feature points (landmarks).

[0046] Furthermore, the matching unit 32 performs an interpolation of points of the standard model Ms (step S34). In this case, for example, the matching unit 32 performs a projection of vertices using a Laplace-Beltrami operator to interpolate points of the standard model Ms after morphing.

[0047] FIG. 8 is an example of a flowchart showing the procedure of the feature extraction process of the reconstructed data Mr executed in step S31 of FIG.

[0048] First, the matching unit 32 performs three-dimensional smoothing on the reconstructed data Mr indicating the three-dimensional shape generated by the three-dimensional reconstruction unit 31 (step S41). This three-dimensional smoothing will be described later.

[0049] Next, the matching unit 32 extracts feature points that are characteristic points in the point cloud based on the point cloud constituting the smoothed reconstructed data Mr and its connection graph (step S42). In this case, the matching unit 32 extracts the above-mentioned feature points by using various point cloud feature extraction methods such as Principal Component Analysis (PCA) of the point cloud and Difference of Center of Gravity (DoCoG).

[0050] Next, the three-dimensional smoothing in step S41 will be described with reference to FIGS.

[0051] Fig. 9(A) is an enlarged view of a portion of the reconstructed data Mr generated by the three-dimensional reconstruction unit 31. As shown in Fig. 9(A), the reconstructed data Mr generated by the three-dimensional reconstruction unit 31 has an isolated vertex "P1" that is separated from other vertices that it connects to. Fig. 9(B) shows the state of a portion of the reconstructed data Mr after the vertex P1 is regarded as an isolated point and removed.

[0052] In this case, the matching unit 32 first calculates the average value "Lm" of the distance between two vertices connected to each vertex constituting the reconstructed data Mr. In this case, instead of calculating the average value of the distances between all vertices and all connected vertices for each vertex as the average value Lm, the matching unit 32 may calculate the average value of the distances between some vertices and all or a predetermined number of vertices connected to the vertex as the average value Lm.

[0053] Next, the matching unit 32 calculates the number of other vertices (also called the number of neighboring vertices) connected within a distance (αL) that is "α" times the calculated average value Lm for each vertex constituting the reconstructed data Mr. Then, the matching unit 32 regards a vertex having a neighboring vertex number less than a threshold number as an isolated point, and removes the isolated point. The above-mentioned coefficient α and the threshold number are, for example, compatibility values ​​stored in advance in the memory 12 or the like. In the example of FIG. 9(A), the distances between the vertex P1 and the other vertices P2 to P5 connected thereto are all longer than the distance αL, and the number of neighboring vertices is less than the threshold, so the matching unit 32 regards the vertex P1 as an isolated point. Then, the matching unit 32 removes the vertex P1 that has been regarded as an isolated point, as shown in FIG. 9(B).

[0054] Thereafter, the matching unit 32 performs Delaunay division on the vertices that were connected to the removed vertex. Fig. 9(C) shows a part of the reconstructed data Mr after Delaunay division. In Fig. 9(C), the matching unit 32 performs Delaunay division on the vertices P2 to P5 that were connected to the removed vertex P1, and newly connects vertex P2 and vertex P5. This enables the matching unit 32 to remove isolated points and suitably smooth the reconstructed data Mr that becomes a three-dimensional shape.

[0055] (5) Inspector confirmation screen Next, a detailed description will be given of the examiner confirmation screen that the output control unit 33 causes the display device 2 to display. The output control unit 33 generates display information Id for displaying the examiner confirmation screen in which the photographing area indicating image Im and the latest photographed image Ic are arranged side by side, and supplies the display information Id to the display device 2 to cause the display device 2 to display the examiner confirmation screen. The output control unit 33 also performs control to notify the examiner of the presence of an area with a short photographing time within the photographed area, and to guide the examiner in the direction in which the endoscope scope 3 should be moved or the area to be photographed.

[0056] 10 shows a first display example of the examiner confirmation screen. In the first display example, the output control unit 33 changes the display mode of the photographed area in the photographed area indicating image Im based on the difference in the photographed time in the photographed area. In addition, the output control unit 33 outputs a warning for the photographed area whose photographed time is lower than the threshold value.

[0057] In Figure 10, the output control unit 33 displays on the examiner confirmation screen the latest captured image Ic captured by the endoscope 3, a captured area indication image Im that indicates the captured area captured by the endoscope 3 during the examination, and a notification window 44 that displays the notification content to the examiner.

[0058] Here, the output control unit 33 displays a projected standard model 40 in which the standard model Ms after matching is projected onto two-dimensional coordinates on the shooting area indicating image Im. Then, the output control unit 33 displays a photographed projection area 41 indicating an area in which the reconstructed data Mr after matching is projected onto the same two-dimensional coordinates as the standard model Ms, superimposed on the projected standard model 40. Here, the photographed projection area 41 has a predetermined transmittance so that the superimposed projected standard model 40 can be seen. Furthermore, the output control unit 33 displays an endoscope icon 42 which typically represents a part of the endoscope scope 3 and a shooting area icon 43 which typically represents the shooting range of the endoscope scope 3, superimposed on the projected standard model 40 on the shooting area indicating image Im. In this case, the output control unit 33 estimates the position and shooting area of ​​the endoscope scope 3 in the standard model Ms based on the relative position and posture of the shooting unit of the endoscope scope 3 obtained as a result of executing SfM when generating the reconstruction data Mr. Then, the output control unit 33 displays an endoscope icon 42 and an imaging area icon 43 based on the estimation result.

[0059] The output control unit 33 may receive an input for moving the viewpoint position for projecting the reconstructed data Mr and the standard model Ms based on a touch operation or the like performed by the user on the photographing area indicating image Im. In this case, the output control unit 33 regenerates the photographing area indicating image Im by projecting the reconstructed data Mr, the standard model Ms, etc. onto two-dimensional coordinates based on the viewpoint position specified by the input, and redisplays the photographing area indicating image Im on the display device 2. This allows the output control unit 33 to allow the examiner to confirm the photographed area when the stomach of the examination target is observed from all viewpoints.

[0060] Furthermore, the output control unit 33 divides the photographed projection area 41 based on the photographing time, and displays the divided areas in different colors. In the example of Fig. 10, the output control unit 33 divides the photographed projection area 41 into a first area 50A where the photographing time is less than the first threshold and equal to or greater than the second threshold, a second area 50B where the photographing time is less than the second threshold, and the rest of the area, and displays these divided areas in different colors (including differences in shading or transmittance).

[0061] Here, the first threshold and the second threshold are two-stage thresholds for determining whether the shooting time is sufficient, and the first threshold is set to a value greater than the second threshold. In this case, the output control unit 33 counts the number of captured images Ic among the captured images Ic used to generate the reconstructed data Mr for each surface (which may be a vertex or a feature point, etc.) of the reconstructed data Mr, for example, and estimates the shooting time for each surface of the reconstructed data Mr according to the number of counted captured images Ic. After that, the output control unit 33 divides the captured projection area 41 based on the above-mentioned first threshold and second threshold.

[0062] The output control unit 33 determines that the first region 50A, where the shooting time is less than the first threshold and equal to or greater than the second threshold, is a region where the shooting time is slightly insufficient, and fills it in a relatively light or inconspicuous color. On the other hand, the output control unit 33 determines that the second region 50B, where the shooting time is less than the second threshold, is a region where the shooting time is significantly insufficient, and fills it in a relatively dark or inconspicuous color. In addition, the output control unit 33 displays the photographed projection region 41 other than the first region 50A and the second region 50B in a color that is less noticeable (or has a higher transmittance) than the first region 50A and the second region 50B.

[0063] Furthermore, in order to notify the examiner of the presence of the first region 50A and the second region 50B for which the imaging time is insufficient, the output control unit 33 displays text information stating "There is an area for which the observation time is insufficient. Please photograph the filled-in area" in the notification window 44. This allows the output control unit 33 to appropriately make the examiner aware of the presence of an area for which the imaging time is insufficient among the already-imaged areas, and to appropriately prompt the examiner to re-image the area.

[0064] In the first display example, the output control unit 33 may notify the presence of an area with insufficient observation time by voice or warning sound instead of or in addition to notifying the presence of an area with insufficient observation time in the notification window 44. In this case, the output control unit 33 supplies audio output information for causing the sound output unit 16 to output a predetermined voice or warning sound.

[0065] In addition, the manner in which the display mode in the photographed projection area 41 of the photographed area indicating image Im is changed based on the difference in the photographed projection area 41 is not limited to the manner shown in Fig. 10. For example, the output control unit 33 may continuously change the shade or color in the photographed projection area 41 according to the photographed time like a heat map. In another example, the output control unit 33 may use one or three or more thresholds instead of using the first and second thresholds, and gradually change the display mode of the area in the photographed projection area 41 according to the relationship with the used thresholds.

[0066] 11 shows a second display example of the examiner confirmation screen. In the second display example, the output control unit 33 displays information on a region that needs to be photographed in an endoscopic examination (also called a "region that needs to be photographed") on an imaging region indicating image Im as guidance information on the movement of the endoscope 3.

[0067] 11, the output control unit 33 displays, on the examiner confirmation screen, a photographed image Ic, a photographed area indicating image Im, a notification window 44, and a progress rate gauge 45. The output control unit 33 displays, in the photographed area indicating image Im, a projection standard model 40, a photographed projection area 41, an endoscope icon 42, and a photographed area icon 43, as in the first display example shown in FIG.

[0068] In the second display example, the output control unit 33 clearly indicates marks 51A-51E indicating the positions of the parts requiring imaging on the projection standard model 40 in the imaging region indicating image Im. Here, as an example, the output control unit 33 determines the sizes of the marks 51A-51E according to the sizes of the parts requiring imaging that correspond to them. Note that information regarding the positions and sizes of the parts requiring imaging of the standard model Ms is stored in advance in, for example, the memory 12. When the subject of examination is the stomach, the areas that need to be imaged include, for example, (A) posterior wall / anterior wall / lesser curvature of the upper body of the stomach, (B) posterior wall / greater curvature / anterior wall / lesser curvature of the middle body of the stomach, (C) posterior wall / greater curvature / anterior wall / lesser curvature of the lower body of the stomach, (D) lesser curvature of the angulus / greater curvature, (E) fundus, (F) vestibule, (G) lesser curvature of the angulus / posterior wall / anterior wall, (H) lesser curvature of the cardia, and (I) pylorus.

[0069] Furthermore, the output control unit 33 presents text information to the effect that imaging of the regions requiring imaging, indicated by the marks 51A to 51E, should be performed, and a recommended imaging order, in the notification window 44. Note that information regarding the recommended imaging order of the regions requiring imaging is stored in advance in, for example, the memory 12.

[0070] Furthermore, the output control unit 33 clearly indicates the progress rate, which indicates the proportion (percentage) of the parts requiring imaging that have been imaged by the endoscope 3 out of all the parts requiring imaging that have been set, by the progress rate gauge 45. In the example of Fig. 11, there are five parts requiring imaging, and the positions of two parts requiring imaging, which correspond to 40% of the five parts requiring imaging, overlap with the reconstructed data Mr, so the output control unit 33 displays the progress rate gauge 45, which indicates that the progress rate is 40%.

[0071] In this way, in the second display example, the output control unit 33 clearly indicates the region that needs to be imaged on the imaging region indicating image Im. This allows the examiner to comprehensively grasp the region that needs to be imaged and to suitably compare it with the current imaging position and the region that has already been imaged. In addition, the output control unit 33 allows the examiner to smoothly perform the examination by presenting a recommended imaging order for the region that needs to be imaged. In addition, the output control unit 33 allows the examiner to easily grasp the progress based on the region that needs to be imaged by displaying a progress rate gauge 45. Note that the output control unit 33 may determine the progress rate based on the ratio of the stomach wall that has already been imaged (i.e., the degree of overlap between the standard model Ms and the reconstructed data Mr) instead of determining the progress rate based on the ratio of the region that needs to be imaged that has already been imaged.

[0072] 12 shows a third display example of the examiner confirmation screen. In the third display example, the output control unit 33 displays route information indicating a recommended route of the imaging unit of the endoscope 3 within the examination target on the imaging area display image Im as guidance information regarding the movement of the endoscope 3.

[0073] In FIG. 12, the output control unit 33 displays a projection standard model 40, an already-photographed projection area 41, an endoscope icon 42, and an imaging area icon 43 in the imaging area indicating image Im, similar to the first display example and the like.

[0074] In the third display example, a path line 53 indicating a recommended path that serves as a model for the endoscope 3 (more specifically, the imaging unit) to take is displayed on the projected standard model 40. In this case, for example, path information indicating a three-dimensional recommended path of the imaging unit based on the standard model Ms created by aggregating the examination history of an expert doctor is stored in advance in the memory 12. Then, the output control unit 33 generates the path line 53 to be displayed on the imaging area indicating image Im by projecting the three-dimensional recommended path indicated by this path information onto the same two-dimensional coordinates as the standard model Ms.

[0075] Furthermore, the output control unit 33 outputs a predetermined warning when it determines that the position of the imaging unit estimated by SfM or the like is away from the recommended route by a predetermined distance or more. In the example of FIG. 12, the output control unit 33 compares the estimated current position and orientation of the imaging unit with the recommended route, determines that the position of the imaging unit needs to be moved back by 3 cm, and issues a warning by displaying text information "Please move back 3 cm" in the notification window 44. Note that the output control unit 33 may cause the sound output unit 16 to output a voice or warning sound instead of issuing the above warning on the examiner confirmation screen.

[0076] In this way, in the third display example, the output control unit 33 can provide guidance (navigation) to appropriately assist in the movement of the endoscope scope 3 by presenting a recommended route on the shooting area explicit image Im and issuing a warning if the user deviates from the recommended route.

[0077] Fig. 13 shows a fourth display example of the examiner confirmation screen. In the fourth display example, the output control unit 33 outputs information guiding the direction to an unphotographed area as guidance information related to photography by the endoscope 3. In Fig. 13, the output control unit 33 displays a projection standard model 40, a photographed projection area 41, an endoscope icon 42, and a photography area icon 43 in the photography area indicating image Im, similar to the first display example and the like.

[0078] In the fourth display example, the output control unit 33 specifies an uncaptured area where the standard model Ms and the reconstructed data Mr do not overlap. Then, the output control unit 33 indicates, on the capturing area indicating image Im, the direction of the uncaptured area (for example, the position of the nearest uncaptured area or the center position of the uncaptured area) relative to the current capturing position estimated by SfM or the like.

[0079] 13, since the middle and lower parts of the stomach are unphotographed regions, the output control unit 33 displays an arrow 54 indicating the direction from the photography position to the middle and lower parts of the stomach near the tip position of the endoscope icon 42 corresponding to the photography position. Similarly, the output control unit 33 displays an arrow 55 indicating the direction of the middle and lower parts of the stomach, which are unphotographed regions, superimposed on the photographed image Ic. Then, the output control unit 33 displays text information in the notification window 44 to inform the user that the photography unit should be moved in the direction of the arrow (here, arrows 54 and 55).

[0080] According to the fourth display example, the output control unit 33 appropriately guides the imaging unit to move to an unimaged region, and the endoscopic examination can be smoothly performed.

[0081] Fig. 14 shows a fifth display example of the examiner confirmation screen. In the fifth display example, the output control unit 33 displays, on the shooting area indicating image Im, landmark information indicating a landmark part (landmark part) of the test subject as guide information regarding shooting by the endoscope 3. In Fig. 14, the output control unit 33 displays, in the shooting area indicating image Im, a projection standard model 40 and an imaged projection area 41, as in the first display example and the like.

[0082] In the fifth display example, the output control unit 33 displays the mark line 56 indicating the mark site such as the main lymphatic vessel, blood vessel, and nerve superimposed on the projected standard model 40. In this case, for example, the relative three-dimensional position information of the mark site with respect to the standard model Ms is stored in the memory 12 in advance, and the output control unit 33 displays the mark line 56 at a position overlapping the mark site on the projected standard model 40 by projecting the three-dimensional information of the mark site to the same two-dimensional coordinates as the standard model Ms. In addition, the output control unit 33 refers to the subject's past examination history from the memory 12, etc., and displays the mark area 57 indicating the position of the lesion site detected in the past examination as the position of the mark site on the imaging area indicating image Im. In this case, for example, the past examination history includes information indicating the position and size (or the corresponding range) of the detected lesion site on the standard model Ms. Then, the output control unit 33 indicates the text information indicating the presence of the lesion site detected in the past and the mark area 57 indicating the lesion site in the notification window 44.

[0083] According to the fifth display example, the output control unit 33 suitably indicates on the shooting area indicating image Im the areas that serve as landmarks in the examination of the subject, thereby enabling the examiner to smoothly perform shooting with the endoscope scope 3.

[0084] 15 shows a sixth display example of the examiner confirmation screen. In the sixth display example, when an arbitrary position of the photographed projection area 41 indicating the photographed area on the photographed area indicating image Im is designated by a user input after the endoscopy is completed, the output control unit 33 displays the photographed image Ic corresponding to the designated position.

[0085] 15, the output control unit 33 displays the projection standard model 40 and the photographed projection area 41 in the photographed area indicating image Im, similarly to the first display example and the like. Then, the output control unit 33 accepts an input specifying an arbitrary position on the photographed projection area 41 by the input unit 14 such as a mouse. In FIG. 15, the output control unit 33 displays a cursor as an example, and accepts an input specifying an arbitrary position on the photographed projection area 41 by the cursor.

[0086] When an arbitrary position on the photographed projection area 41 is specified, the output control unit 33 extracts the photographed image Ic corresponding to the specified position from the photographed image storage unit 21, and displays the extracted photographed image Ic in association with the specified position. In FIG. 15, the output control unit 33 displays a balloon 58 indicating the specified position, and displays the photographed image Ic in the balloon. Note that, for example, during an endoscopic examination, the image processing device 1 stores information such as the photographing position of each photographed image Ic estimated when generating the reconstruction data Mr in association with the photographed image Ic in the photographed image storage unit 21. When performing the display of FIG. 15, the output control unit 33 retrieves the corresponding photographed image Ic from the photographed image storage unit 21 based on the position on the standard model Ms specified by the user, thereby acquiring the photographed image Ic to be displayed in the balloon 58. In addition, instead of acquiring a captured image Ic in which the position specified by the user is the shooting position of the shooting unit as the captured image Ic to be displayed in the speech bubble 58, the output control unit 33 may acquire a captured image Ic that includes the position specified by the user as a subject.

[0087] In this way, in the sixth display example, the image processing device 1 can accept input specifying any position in the photographed area indicated in the photographed area indicating image Im after the examination is completed, and can suitably present the photographed image Ic showing the detailed situation of that position to the examiner.

[0088] (6) Variations Next, a description will be given of suitable modifications of the first embodiment. The following modifications may be applied in combination to the above-mentioned embodiment.

[0089] (Variation 1) The image processing device 1 may record the examination time required for the endoscopic examination as history information (that is, log information) of the examination.

[0090] In this case, for example, the image processing device 1 measures the length of time from when acquisition of the captured image Ic starts in step S11 of the flowchart in Fig. 5 until when it is determined that the endoscopic examination has ended in step S15 as the examination time. Then, when it is determined that the endoscopic examination has ended in step S15, the output control unit 33 of the image processing device 1 stores the measured examination time in the memory 12 in association with information such as the subject's identification information (e.g., patient ID) and date. This allows the image processing device 1 to suitably store the time required for the endoscopic examination in the memory 12 (storage device) as log information.

[0091] (Variation 2) After the endoscopic examination, the image processing device 1 may generate report information indicating the implementation status or results of the completed endoscopic examination.

[0092] In this case, for example, after the endoscopy is determined to be completed, the output control unit 18 of the image processing device 1 generates, as report information, information on the examination time required for the endoscopy and / or information on the path that the imaging unit of the endoscope 3 actually moved during the endoscopy. The path information is, for example, time-series data of three-dimensional position information indicating the movement history of the imaging unit of the endoscope 3, based on at least one of the standard model Ms and the reconstructed data Mr. In this case, the report information may be information capable of displaying the path of the imaging unit of the endoscope 3 on a three-dimensional model of the stomach. In addition, the report information may include various information indicating the status or results of the performed endoscopy, such as the latest reconstructed data Mr generated up to the end of the examination. Then, the output control unit 18 stores the generated report information in the memory 12, for example, in association with the identification information of the subject (for example, a patient ID) and information such as a date.

[0093] According to this modification, the image processing device 1 can suitably generate useful information after an endoscopic examination.

[0094] <Second embodiment> 16 is a block diagram of an image processing device 1X in the second embodiment. The image processing device 1X includes a three-dimensional reconstruction means 31X, a matching means 32X, and an output control means 33X.

[0095] The three-dimensional reconstruction means 31X generates reconstruction data "Mr" by three-dimensionally reconstructing the inspection object based on a photographed image "Ic" of the inspection object photographed by an imaging unit provided in the endoscope. In this case, the three-dimensional reconstruction means 31X may directly receive the photographed image Ic from the imaging unit, or may obtain the photographed image Ic from a storage device that accumulates images photographed by the imaging unit. The "inspection object" may be the stomach or other organs such as the large intestine. The three-dimensional reconstruction means 31X may be the three-dimensional reconstruction unit 31 in the first embodiment.

[0096] The matching means 32X matches a standard model "Ms" which is a standard three-dimensional model of the inspection target with the reconstructed data Mr. Here, the matching means 32X may acquire the standard model Ms from a memory included in the image processing device 1X, or may acquire it from an external device different from the image processing device 1X. The matching means 32X may be the matching unit 32 in the first embodiment.

[0097] The output control means 33X causes the display device 2X to display an imaging area indicating image of the imaging area of ​​the test subject on the standard model based on the matching result. The output control means 33X can be the output control unit 33 in the first embodiment.

[0098] 17 is an example of a flowchart showing a processing procedure executed by the image processing device 1X in the second embodiment. First, the three-dimensional reconstruction means 31X generates reconstruction data Mr by three-dimensionally reconstructing the inspection object based on a captured image Ic of the inspection object captured by an imaging unit provided in the endoscope (step S51). The matching means 32X matches the standard model Ms, which is a standard three-dimensional model of the inspection object, with the reconstruction data Mr (step S52). Based on the matching result, the output control means 33X causes the display device 2X to display an imaging area indicating image showing the imaging area of ​​the inspection object on the standard model (step S53).

[0099] According to the second embodiment, the image processing device 1X can suitably indicate an imaged region on the standard model Ms, which is a standard three-dimensional model of the inspection target.

[0100] In addition, some or all of the above-described embodiments (including modified examples, the same applies below) can be described as, but are not limited to, the following notes.

[0101] [Appendix 1] a three-dimensional reconstruction unit that generates reconstruction data by three-dimensionally reconstructing the object of inspection based on an image of the object of inspection captured by an imaging unit provided in the endoscope; a matching means for matching a standard model, which is a standard three-dimensional model of the inspection object, with the reconstructed data; an output control means for displaying, on a display device, a photographed area indicating image showing the photographed area of ​​the test subject on the standard model based on a result of the matching; An image processing device comprising:

[0102] [Appendix 2] 2. The image processing device according to claim 1, wherein the output control means causes the display device to display the latest captured image acquired from the imaging unit and the imaging area indicating image side by side.

[0103] [Appendix 3] The image processing device according to claim 1 or 2, wherein when a partial area having an image capture time lower than a threshold exists in the image-captured area, the output control means outputs a warning regarding the partial area.

[0104] [Appendix 4] The image processing device according to any one of claims 1 to 3, wherein the output control means changes a display mode within the photographed area based on a difference in photographing time within the photographed area.

[0105] [Appendix 5] The image processing device according to claim 4, wherein the output control means divides the photographed area based on the photographing time, and displays the divided areas in different colors on the display device.

[0106] [Appendix 6] 6. The image processing device according to claim 1, wherein the output control means causes the display device to display guide information related to movement of the imaging unit.

[0107] [Appendix 7] The image processing device according to claim 6, wherein the output control means displays, as the guide information, information on a part of the subject to be photographed that indicates a part of the subject to be photographed, on the photographing area indicating image.

[0108] [Appendix 8] 8. The image processing device according to claim 6, wherein the output control means causes the display device to display information indicating a progress rate of the shooting.

[0109] [Appendix 9] The image processing device according to claim 8, wherein the output control means determines the progress rate based on a ratio of the parts of the examination target that have been photographed to the parts that require the photography.

[0110] [Appendix 10] The image processing device according to any one of appendices 6 to 9, wherein the output control means displays, as the guidance information, route information indicating a recommended route of the shooting unit on the standard model on the shooting area indicating image.

[0111] [Appendix 11] The image processing device according to claim 10, wherein the output control means outputs a warning when it is determined that the position of the photographing unit has deviated from the path.

[0112] [Appendix 12] The image processing device according to any one of claims 6 to 11, wherein the output control means displays, as the guide information, landmark information indicating a landmark part of the test subject on the shooting area indicating image.

[0113] [Appendix 13] The image processing device according to claim 12, wherein the landmark information is information indicating at least one of a previously detected lesion site, a lymphatic vessel, a blood vessel, or a nerve on the standard model.

[0114] [Appendix 14] The image processing device according to any one of claims 6 to 13, wherein the output control means outputs, as the guidance information, information that provides guidance on a direction to an unphotographed area of ​​the inspection target.

[0115] [Appendix 15] The image processing device according to any one of claims 1 to 14, wherein, when a position within the photographed area is specified in the photographed area indicating image, the photographed image corresponding to the specified position is displayed on the display device.

[0116] [Appendix 16] The image processing device according to any one of claims 1 to 15, wherein the matching means performs matching between the standard model and reconstructed data that has been smoothed by removing isolated points from vertices constituting the reconstructed data generated based on the captured image.

[0117] [Appendix 17] The image processing device according to any one of appendixes 1 to 16, wherein the output control means stores log information relating to the time required for the endoscopic examination in a storage device. Note that the "storage device" may be a memory within the image processing device, or may be an external device separate from the image processing device.

[0118] [Appendix 18] 18. The image processing device according to any one of claims 1 to 17, wherein the output control means generates report information relating to the examination after the endoscope examination is completed.

[0119] [Appendix 19] By computer, generating reconstruction data by three-dimensionally reconstructing the object of inspection based on a captured image of the object of inspection captured by an imaging unit provided in the endoscope; Matching the reconstructed data with a standard model, which is a standard three-dimensional model of the inspection object; displaying, on a display device, an imaging area indicating image showing an imaging area of ​​the test subject on the standard model based on a result of the matching; Control methods.

[0120] [Appendix 20] a three-dimensional reconstruction unit that generates reconstruction data by three-dimensionally reconstructing the object of inspection based on an image of the object of inspection captured by an imaging unit provided in the endoscope; a matching means for matching a standard model, which is a standard three-dimensional model of the inspection object, with the reconstructed data; An output control means for displaying, on a display device, a photographed area indicating image showing the photographed area of ​​the test object on the standard model based on the result of the matching. A storage medium that stores a program that causes a computer to function as a

[0121] Although the present invention has been described above with reference to the embodiments, the present invention is not limited to the above-mentioned embodiments. Various modifications that can be understood by a person skilled in the art can be made to the configuration and details of the present invention within the scope of the present invention. In other words, the present invention naturally includes various modifications and corrections that a person skilled in the art could make in accordance with the entire disclosure, including the claims, and the technical ideas. In addition, the disclosures of the above cited patent documents and the like are incorporated herein by reference. [Explanation of symbols]

[0122] 1. 1X image processing device 2, 2X display device 3 Endoscope 4. Server equipment 11 Processors 12. Memory 13. Interface 14 Input section 15 Light source section 16 Sound output section 100, 100A Endoscopy System

Claims

1. a three-dimensional reconstruction unit that generates reconstruction data by three-dimensionally reconstructing the object of inspection based on an image of the object of inspection captured by an imaging unit provided in the endoscope; a matching means for matching a standard model, which is a standard three-dimensional model of the inspection object, with the reconstructed data; an output control means for displaying, on a display device, a photographed area indicating image showing the photographed area of ​​the test subject on the standard model based on a result of the matching; having The output control means outputs a warning regarding a partial area in the photographed area whose photographing time is lower than a threshold value.

2. a three-dimensional reconstruction unit that generates reconstruction data by three-dimensionally reconstructing the object of inspection based on an image of the object of inspection captured by an imaging unit provided in the endoscope; a matching means for matching a standard model, which is a standard three-dimensional model of the inspection object, with the reconstructed data; an output control means for displaying, on a display device, a photographed area indicating image showing the photographed area of ​​the test subject on the standard model based on a result of the matching; having The output control means changes a display mode within the photographed area based on a difference in photographing time within the photographed area.

3. a three-dimensional reconstruction unit that generates reconstruction data by three-dimensionally reconstructing the object of inspection based on an image of the object of inspection captured by an imaging unit provided in the endoscope; a matching means for matching a standard model, which is a standard three-dimensional model of the inspection object, with the reconstructed data; an output control means for displaying, on a display device, a photographed area indicating image showing the photographed area of ​​the test subject on the standard model based on a result of the matching; having The output control means causes the display device to display information indicating a progress rate of the photographing.

4. 4. The image processing device according to claim 1, wherein the output control means causes the display device to display the latest captured image acquired from the imaging section and the imaging area indicating image side by side.

5. 3. The image processing apparatus according to claim 2, wherein said output control means divides said photographed area based on said photographing time, and causes said display device to display the divided areas in different colors.

6. 6. The image processing device according to claim 1, wherein the output control means causes the display device to display guide information relating to movement of the image capturing unit.

7. The image processing apparatus according to claim 6 , wherein the output control means displays, as the guide information, information on a region to be photographed, which indicates a region of the examination subject that needs to be photographed, on the photographing region indicating image.

8. By computer, generating reconstruction data by three-dimensionally reconstructing the object of inspection based on a captured image of the object of inspection captured by an imaging unit provided in the endoscope; Matching the reconstructed data with a standard model, which is a standard three-dimensional model of the inspection object; Based on the result of the matching, a photographed area indicating image showing the photographed area of ​​the test subject on the standard model is displayed on a display device; a control method for outputting a warning regarding a partial area, the partial area having an imaging time lower than a threshold, when the partial area includes the imaging time lower than a threshold.

9. a three-dimensional reconstruction unit that generates reconstruction data by three-dimensionally reconstructing the object of inspection based on an image of the object of inspection captured by an imaging unit provided in the endoscope; a matching means for matching a standard model, which is a standard three-dimensional model of the inspection object, with the reconstructed data; An output control means for displaying, on a display device, a photographed area indicating image showing the photographed area of ​​the test object on the standard model based on the result of the matching. The computer functions as The output control means outputs a warning regarding a partial area in which the photographed area has a photographing time less than a threshold value.

Citation Information

Patent Citations

  • Scanning mechanism for imaging probe

    JP2010516305A

  • Endoscope system and operation method of endoscope system

    WO2014148184A1

  • Endoscope system

    WO2015049962A1

  • Image-processing device

    WO2015194242A1

  • Endoscope device and operation method for endoscope device

    WO2017203814A1