Image processing device, image processing method and program

Abstract

This image processing device 1X mainly comprises a three-dimensional reconstruction means 31X, a matching means 32X, and a display control means 33X. The three-dimensional reconstruction means 31X generates reconstruction data in which an inspected object is three-dimensionally reconstructed, on the basis of an endoscope image obtained by capturing an image of the inspected object by an imaging unit provided to an endoscope. The matching means 32X performs matching between a three-dimensional model of the inspected object and the reconstruction data. The display control means 33x displays, on the basis of the result of the matching, information relating to metadata associated with the three-dimensional model.

Description

[Technical field]

[0001] The present disclosure relates to the technical field of an image processing device, an image processing method, and a storage medium that process images acquired in an endoscopic examination. [Background technology]

[0002] Conventionally, an endoscope system that displays an image of the inside of an organ lumen has been known. For example, Patent Document 1 discloses a technique for generating three-dimensional model data of an object to be examined based on an image captured by an endoscope, and displaying the data as a three-dimensional model image. Patent Document 2 discloses a technique for generating volume data representing the large intestine by capturing an image of a three-dimensional imaging region including the large intestine using an X-ray CT scanner. Non-Patent Document 1 discloses a technique for restoring the three-dimensional shape of the stomach from a captured image using Structure from Motion (SfM). Non-Patent Document 2 discloses a technique for non-rigid registration between three-dimensional shapes. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] International Publication WO2017 / 203814 [Patent Document 2] Patent Publication No. 2011-139797 [Non-patent literature]

[0004] [Non-Patent Document 1] Aji Resindra Widya et al. “3D Reconstruction of Whole Stomach from Endoscope Video Using Structure-from-Motion”, 2019 41st Annual International Conference of the IEEE Engineering in Medicine and Biology Society, pp. 3900-3904. [Non-Patent Document 2] Dai,H et al. “Non-rigid 3D Shape Registration using an Adaptive Template”, The European Conference on Computer Vision (ECCV) Workshops, 2018. Summary of the Invention [Problem to be solved by the invention]

[0005] Prior to endoscopic examination, a preliminary examination may be performed using CT, MRI, etc. In such cases, it is preferable to be able to use information identified in the preliminary examination during endoscopic examination.

[0006] In view of the above-mentioned problems, one object of the present disclosure is to provide an image processing device, an image processing method, and a storage medium that can make optimal use of information obtained in advance in an endoscopic examination. [Means for solving the problem]

[0007] One aspect of the image processing device is a three-dimensional reconstruction means for generating reconstruction data by three-dimensionally reconstructing the object of inspection based on an endoscopic image obtained by photographing the object of inspection by an imaging unit provided in the endoscope; a matching means for matching the three-dimensional model of the inspection object with the reconstructed data; a display control means for displaying information about metadata associated with the three-dimensional model based on a result of the matching; The image processing device has the following features.

[0008] One aspect of the image processing method includes: By computer, generating reconstruction data that reconstructs the object of inspection in three dimensions based on an endoscopic image of the object of inspection captured by an imaging unit provided in the endoscope; matching the three-dimensional model of the object to the reconstructed data; displaying information about metadata associated with the three-dimensional model based on the results of the matching; An image processing method.

[0009] program One aspect of the present invention is generating reconstruction data that reconstructs the object of inspection in three dimensions based on an endoscopic image of the object of inspection captured by an imaging unit provided in the endoscope; matching the three-dimensional model of the object to the reconstructed data; a program for causing a computer to execute a process of displaying information regarding metadata associated with the three-dimensional model based on a result of the matching; M It is. Effect of the Invention

[0010] One example of the effect of the present invention is that information obtained in advance can be suitably utilized in an endoscopic examination. [Brief description of the drawings]

[0011] [Figure 1] 1 shows a schematic configuration of an endoscopic examination system. [Diagram 2] 2 shows the hardware configuration of an image processing device. [Diagram 3] FIG. 2 is a functional block diagram of the image processing device. [Figure 4]FIG. 2 is a diagram illustrating an overview of the processing performed by a three-dimensional reconstruction unit and a matching unit. [Diagram 5] 5 is an example of a flowchart showing an overview of a display process executed by the image processing device during an endoscopic examination in the first embodiment. [Figure 6] 13 shows a first display example of an examiner confirmation screen. [Figure 7] 13 shows a second display example of the examiner confirmation screen. [Figure 8] 13 shows a third display example of the examiner confirmation screen. [Figure 9] 13 shows a fourth display example of the examiner confirmation screen. [Figure 10] 13 shows a fifth display example of the examiner confirmation screen. [Figure 11] 13 shows a sixth display example of the examiner confirmation screen. [Figure 12] FIG. 11 is a block diagram of an image processing device according to a second embodiment. [Figure 13] 13 is a flowchart illustrating an example of a processing procedure of an image processing device according to a second embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0012] Hereinafter, embodiments of an image processing device, an image processing method, and a storage medium will be described with reference to the drawings.

[0013] First Embodiment (1) System Configuration FIG. 1 shows a schematic configuration of an endoscopic examination system 100. In an examination (including treatment) using an endoscope, the endoscopic examination system 100 presents information on a site suspected of a lesion (lesion site) detected based on a preliminary examination performed before the endoscopic examination, together with an endoscopic image obtained in real time. In this way, the endoscopic examination system 100 supports an examiner such as a doctor who performs an endoscopic examination. Note that the preliminary examination is an examination in which scan data of an organ to be examined is generated by CT or MRI, etc., and a diagnosis is made based on the generated data. The above-mentioned diagnosis may be made by a doctor or a computer.

[0014] As shown in FIG. 1, an endoscopic examination system 100 mainly includes an image processing device 1, a display device 2, and an endoscope 3 connected to the image processing device 1.

[0015] The image processing device 1 acquires images (also called "endoscopic images Ic") captured by the endoscope 3 in a time series from the endoscope 3, and displays a screen (also called "examiner confirmation screen") for the examiner of the endoscopic examination to confirm on the display device 2. The endoscopic images Ic are images captured at a predetermined time interval during at least one of the insertion process and the ejection process of the endoscope 3 into the subject. In this embodiment, the image processing device 1 generates data (also called "reconstructed data Mr") that reconstructs the three-dimensional shape of the subject's organ (digestive system) to be examined from the endoscopic images Ic, and matches the data with a three-dimensional model (also called "pre-examination model Mp") of the subject's organ to be examined that is generated based on the results of a pre-examination using CT, MRI, or the like. Then, the image processing device 1 displays on the display device 2 an examiner confirmation screen in which the positions of the lesion sites and the like detected in the pre-examination are highlighted on the endoscopic images Ic based on the matching results.

[0016] The display device 2 is a display or the like that performs a predetermined display based on a display signal supplied from the image processing device 1.

[0017] The endoscope scope 3 mainly comprises an operation unit 36 ​​through which the examiner makes predetermined inputs, a flexible shaft 37 that is inserted into the subject's organ to be examined, a tip portion 38 that incorporates an imaging unit such as an ultra-miniature imaging element, and a connection portion 39 for connecting to the image processing device 1.

[0018] In the following description, the processing in an endoscopic examination of the stomach is mainly assumed, but the subject of the examination is not limited to the stomach, and may be the digestive tract (digestive organs) such as the large intestine, esophagus, small intestine, duodenum, etc. Examples of endoscopes that are targets of the present disclosure include pharyngeal endoscopes, bronchoscopes, upper digestive tract endoscopes, duodenoscopes, small intestine endoscopes, colonoscopes, capsule endoscopes, thoracoscopes, laparoscopes, cystoscopes, cholangioscopes, arthroscopes, spinal endoscopes, vascular endoscopes, and epidural endoscopes.

[0019] Furthermore, the area to be detected in the preliminary inspection and the endoscopic inspection is not limited to the lesion area, but may be any area that the examiner needs to focus on (also referred to as the "area of ​​interest"). Such an area of ​​interest may be a lesion area, an area where inflammation has occurred, an area where a surgical scar or other cut has occurred, an area where a fold or protrusion has occurred, an area where the tip 38 of the endoscope 3 is likely to come into contact (be easily pinched) with the wall surface inside the lumen, and the like. Furthermore, when the area of ​​interest is a lesion area, the medical condition of the lesion area to be detected is exemplified as follows (a) to (f).

[0020] (a) Head and neck: pharyngeal cancer, malignant lymphoma, papilloma (b) Esophagus: Esophageal cancer, esophagitis, hiatal hernia, Barrett's esophagus, esophageal varices, achalasia, esophageal submucosal tumor, benign esophageal tumor (c) Stomach: stomach cancer, gastritis, stomach ulcer, stomach polyps, stomach tumors (d) Duodenum: duodenal cancer, duodenal ulcer, duodenitis, duodenal tumor, duodenal lymphoma (e) Small intestine: small intestine cancer, small intestine neoplastic disease, small intestine inflammatory disease, small intestine vascular disease (f) Colon: colon cancer, colon neoplastic disease, colon inflammatory disease, colon polyps, colon polyposis, Crohn's disease, colitis, intestinal tuberculosis, hemorrhoids

[0021] (2) Hardware Configuration 2 shows a hardware configuration of the image processing device 1. The image processing device 1 mainly includes a processor 11, a memory 12, an interface 13, an input unit 14, a light source unit 15, and a sound output unit 16. These elements are connected via a data bus 19.

[0022] The processor 11 executes a predetermined process by executing a program or the like stored in the memory 12. The processor 11 is a processor such as a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), or a TPU (Tensor Processing Unit). The processor 11 may be composed of multiple processors. The processor 11 is an example of a computer.

[0023] The memory 12 is composed of various volatile memories used as working memories, such as RAM (Random Access Memory) and ROM (Read Only Memory), and non-volatile memories that store information necessary for processing by the image processing device 1. The memory 12 may include an external storage device, such as a hard disk connected to or built into the image processing device 1, or may include a removable storage medium, such as a flash memory. The memory 12 stores programs for the image processing device 1 to execute each process in this embodiment.

[0024] Moreover, the memory 12 functionally includes an endoscopic image storage unit 21 and a preliminary examination information storage unit 22 .

[0025] The endoscopic image storage unit 21 stores a series of endoscopic images Ic captured by the endoscope 3 during an endoscopic examination under the control of the processor 11. These endoscopic images Ic are images used to generate the reconstruction data Mr, and are stored in the endoscopic image storage unit 21 in association with, for example, identification information of the subject (e.g., a patient ID) and time stamp information.

[0026] The pre-examination information storage unit 22 stores pre-examination information, which is information on the results of a pre-examination using a CT, MRI, or the like on a subject. The pre-examination information includes scan data of an organ to be examined in the subject using a CT, MRI, or the like (also referred to as "pre-scan data"), a pre-examination model Mp which is a three-dimensional shape model of the organ to be examined generated from the pre-scan data, and metadata associated with the pre-scan data and the pre-examination model Mp.

[0027] The metadata is, for example, data added to the pre-scan data by a doctor in charge of the pre-examination through annotation work, or data obtained by applying CAD (Computer Aided Diagnosis) to the pre-scan data. The annotation work described above is, for example, a work in which the doctor in charge of the pre-examination refers to the pre-scan data displayed on a display or the like, specifies a point of interest in the pre-scan data, and inputs comments or the like related to the point of interest into a computer. The metadata includes information on a point of interest such as a lesion site detected in the pre-examination. For example, the metadata includes position information (for example, coordinate values ​​in a coordinate system representing the pre-scan data) specifying a point of interest that should be noted in an endoscopic examination, and content information indicating a diagnosis result or the like related to the position indicated by the position information (i.e., the point of interest). The metadata may also include information on the attributes of the doctor in charge of the pre-examination (including the name and affiliation information of the doctor in charge), as described later.

[0028] The pre-examination model Mp is generated by extracting a three-dimensional shape of an organ to be examined from pre-scan data, which is a three-dimensional CT image or MRI data. Here, the pre-examination model Mp is expressed in a predetermined three-dimensional coordinate system, for example. The pre-examination information storage unit 22 may further include coordinate conversion information between the three-dimensional coordinate system of the pre-examination model Mp and a coordinate system (two-dimensional or three-dimensional coordinate system) representing the pre-scan data. This coordinate conversion information is generated in the process of generating the pre-examination model Mp from the pre-scan data. Note that the process of generating the pre-examination model Mp from the pre-scan data may be performed by the image processing device 1 in advance before the endoscopic examination, or may be performed by a device other than the image processing device 1 before the endoscopic examination.

[0029] Here, at least one of the endoscopic image storage unit 21 and the preliminary examination information storage unit 22 may be provided in an external device capable of wired or wireless data communication with the image processing device 1, instead of the memory 12. In this case, the external device may be one or more server devices capable of data communication with the image processing device 1 via a communication network.

[0030] In addition to the above-mentioned information, the memory 12 may store various information necessary for the processing in this embodiment. For example, when the image processing device 1 executes CAD based on the endoscopic image Ic, the memory 12 may further store parameters of a lesion detection model necessary for executing CAD. In this case, the lesion detection model is, for example, a machine learning model such as a neural network or a support vector machine, and is configured to output, when the endoscopic image Ic is input, the presence or absence of a lesion site in the input endoscopic image Ic and position information (which may be area information) in the endoscopic image Ic when the lesion site exists. In addition, when the lesion detection model is configured by a neural network, the memory 12 stores various parameters such as a layer structure, a neuron structure of each layer, the number of filters and filter size in each layer, and the weight of each element of each filter.

[0031] The interface 13 performs an interface operation between the image processing device 1 and an external device. For example, the interface 13 supplies the display information "Id" generated by the processor 11 to the display device 2. The interface 13 also supplies light generated by the light source unit 15 to the endoscope 3. The interface 13 also supplies an electrical signal indicating an endoscopic image Ic supplied from the endoscope 3 to the processor 11. The interface 13 may be a communication interface such as a network adapter for communicating with an external device by wire or wirelessly, or may be a hardware interface conforming to USB (Universal Serial Bus), SATA (Serial AT Attachment), or the like.

[0032] The input unit 14 generates an input signal based on an operation by the examiner. The input unit 14 is, for example, a button, a touch panel, a remote controller, a voice input device, etc. The light source unit 15 generates light to be supplied to the tip 38 of the endoscope 3. The light source unit 15 may also include a built-in pump for sending out water or air to be supplied to the endoscope 3. The sound output unit 16 outputs sound based on the control of the processor 11.

[0033] (3) Functional Blocks Fig. 3 is a functional block diagram of the image processing device 1. As shown in Fig. 3, the processor 11 of the image processing device 1 functionally includes an endoscopic image acquisition section 30, a three-dimensional reconstruction section 31, a matching section 32, and a display control section 33. Note that in Fig. 3, blocks where data is exchanged are connected by solid lines, but the combination of blocks where data is exchanged is not limited to that shown in Fig. 3. The same applies to other functional block diagrams described later.

[0034] The endoscopic image acquiring unit 30 acquires the endoscopic images Ic captured by the endoscope 3 at predetermined intervals via the interface 13. Then, the endoscopic image acquiring unit 30 supplies the acquired endoscopic images Ic to the 3D reconstruction unit 31. The endoscopic image acquiring unit 30 also stores the acquired endoscopic images Ic in the endoscopic image storage unit 21 in association with a time stamp, a patient ID, etc. The endoscopic image acquiring unit 30 also supplies the most recently acquired endoscopic image Ic to the display control unit 33.

[0035] The 3D reconstruction unit 31 generates reconstruction data Mr indicating the 3D shape of the captured organ based on the multiple endoscopic images Ic acquired during the endoscopic examination by the endoscopic image acquisition unit 30. The reconstruction data Mr includes, for example, point cloud data having 3D position information.

[0036] In this case, for example, after acquiring the number of endoscopic images Ic required for generating the reconstructed data Mr, the three-dimensional reconstruction unit 31 constructs the reconstructed data Mr using a method for restoring the three-dimensional shape of the subject and the relative position of the imaging unit from the multiple images. As such a method, for example, a method such as Structure from Motion (SfM) exists. Thereafter, the three-dimensional reconstruction unit 31 updates the generated reconstructed data Mr, for example, every time it acquires a predetermined number of endoscopic images Ic. The above-mentioned predetermined number may be one or more, and is previously determined to a value taking into consideration, for example, the processing capacity of the image processing device 1. Then, the three-dimensional reconstruction unit 31 supplies the generated (including updated) reconstructed data Mr to the matching unit 32. The method for generating the reconstructed data Mr will be described later.

[0037] The matching unit 32 matches the reconstructed data Mr supplied from the three-dimensional reconstruction unit 31 with the pre-inspection model Mp stored in the pre-inspection information storage unit 22, and supplies a matching result "Rm" that is a result of the matching to the display control unit 33. In this case, for example, the matching unit 32 performs non-rigid alignment, and generates data that represents the reconstructed data Mr and the pre-inspection model Mp that have been non-rigidly aligned in a common three-dimensional coordinate system (also called a "common coordinate system"). Then, for example, the matching unit 32 generates the above-mentioned data and / or coordinate transformation information related to the common coordinate system as the matching result Rm. Here, the above-mentioned coordinate transformation information includes, for example, coordinate transformation information from the coordinate system adopted by the reconstructed data Mr to the common coordinate system, and coordinate transformation information from the coordinate system adopted by the pre-inspection model Mp to the common coordinate system.

[0038] The display control unit 33 generates display information Id for the examiner confirmation screen based on the matching result Rm generated by the matching unit 32, and supplies the generated display information Id to the display device 2, thereby causing the display device 2 to display the examiner confirmation screen. In this case, the display control unit 33 causes information on the area of ​​interest indicated by the metadata included in the pre-examination information storage unit 22 to be displayed on the examiner confirmation screen in association with the endoscopic image Ic supplied from the endoscopic image acquisition unit 30. The display control unit 33 may also output information for providing guidance or warning regarding the photographing of the endoscope 3 by the examiner. This information may be output on the examiner confirmation screen, or may be output by the sound output unit 16.

[0039] Fig. 4 is a diagram showing an overview of the processing of the three-dimensional reconstruction unit 31 and the matching unit 32. For convenience of explanation, Fig. 4 shows a case where the stomach is the examination target, but the outline of the processing shown in Fig. 4 is similarly applied to the large intestine and other digestive tracts.

[0040] 4, locations 80-83 that are the subject of metadata (also referred to as "metadata target locations") are clearly indicated. In other words, the metadata target locations 80-83 are locations identified by position information included in the metadata, and are locations that are subject to content information of the metadata. The metadata target locations 80-83 can be represented in the coordinate system of the pre-inspection model Mp, based on the metadata stored in the pre-inspection information storage unit 22.

[0041] In addition, the 3D reconstruction unit 31 generates reconstruction data Mr corresponding to the 3D shape of the area inside the digestive tract that has been photographed by the endoscope scope 3 (the photographed area) based on multiple endoscopic images Ic that have been acquired up to now during the endoscopic examination.

[0042] Then, the matching unit 32 performs matching (non-rigid registration) between the pre-inspection model Mp stored in the pre-inspection information storage unit 22 and the reconstructed data Mr. As a result, the matching unit 32 associates the pre-inspection model Mp, which represents the whole, with the reconstructed data Mr, which corresponds to the photographed region, in a common coordinate system, and generates a matching result Rm, which represents the association result (for example, coordinate conversion information from each data to the common coordinate system). Furthermore, for the metadata target locations 80 to 83 represented in the coordinate system of the pre-inspection model Mp, the corresponding positional relationship with the reconstructed data Mr (i.e., the endoscopic image Ic constituting the reconstructed data Mr) is specified by the coordinate conversion to the common coordinate system. In the diagram after matching shown in FIG. 4, the metadata target locations 81 and 82 overlapping with the reconstructed data Mr are clearly shown.

[0043] Then, based on the matching result Rm and the metadata, the display control unit 33 generates display information Id for the examiner confirmation screen including the endoscopic image Ic in which the metadata target portion is highlighted.

[0044] Each of the components of the endoscopic image acquisition unit 30, the three-dimensional reconstruction unit 31, the matching unit 32, and the display control unit 33 can be realized, for example, by the processor 11 executing a program. Also, each of the components may be realized by recording a necessary program in an arbitrary non-volatile storage medium and installing it as necessary. At least a part of each of these components may be realized by any combination of hardware, firmware, and software, without being limited to being realized by software by a program. Also, at least a part of each of these components may be realized by using an integrated circuit programmable by a user, such as an FPGA (Field-Programmable Gate Array) or a microcontroller. In this case, a program composed of each of the above components may be realized using this integrated circuit. Also, at least a part of each of the components may be configured by an ASSP (Application Specific Standard Produce), an ASIC (Application Specific Integrated Circuit), or a quantum processor (quantum computer control chip). In this way, each of the components may be realized by various hardware. The above is the same in other embodiments described later. Furthermore, each of these components may be realized by cooperation of multiple computers, for example, using cloud computing technology.

[0045] (4) Processing Flow FIG. 5 is an example of a flowchart showing an overview of the display process executed by the image processing device 1 during endoscopic examination in the first embodiment.

[0046] First, the image processing device 1 acquires an endoscopic image Ic (step S11). In this case, the endoscopic image acquisition unit 30 of the image processing device 1 receives the endoscopic image Ic from the endoscope 3 via the interface 13.

[0047] Next, the image processing device 1 generates reconstruction data Mr by 3D reconstructing the inspection target from the multiple endoscopic images Ic acquired in step S11 (step S12). In this case, the 3D reconstruction unit 31 of the image processing device 1 generates the reconstruction data Mr using a technique such as SfM based on the endoscopic images Ic acquired from the start of the inspection to the current processing time.

[0048] Next, the image processing device 1 performs matching between the pre-inspection model Mp and the reconstructed data Mr (step S13). In this case, the matching unit 32 of the image processing device 1 performs non-rigid registration between the pre-inspection model Mp acquired from the pre-inspection information storage unit 22 and the reconstructed data Mr generated by the 3D reconstruction unit 31, thereby generating a matching result Rm.

[0049] Then, the image processing device 1 determines whether or not a metadata target portion exists in the endoscopic image Ic to be displayed on the display device 2 based on the matching result Rm (step S14). In this case, the endoscopic image Ic to be displayed on the display device 2 is, for example, the latest endoscopic image Ic among the endoscopic images Ic used to generate the reconstruction data Mr.

[0050] Here, a method for determining whether or not a metadata target portion exists in the endoscopic image Ic displayed on the display device 2 will be described in more detail. For example, the image processing device 1 first recognizes the metadata target portion in the common coordinate system based on the matching result Rm and the metadata, and further converts the metadata target portion into the coordinate system of the endoscopic image Ic displayed on the display device 2. Then, the image processing device 1 determines that the metadata target portion exists in the endoscopic image Ic displayed on the display device 2 when a metadata target portion belonging to the display range of the endoscopic image Ic displayed on the display device 2 exists among the metadata target portions after the coordinate conversion. In general, when three-dimensional data is generated from multiple images using a method such as SfM, coordinate conversion information between the generated three-dimensional data and the original image is obtained, and the image processing device 1 can convert the metadata target portion in the common coordinate system into the coordinate system of each endoscopic image Ic by using this coordinate conversion information.

[0051] Then, when the image processing device 1 determines that a metadata target portion is present in the endoscopic image Ic to be displayed on the display device 2 (step S14; Yes), it causes the display device 2 to display the endoscopic image Ic showing information related to the metadata (step S15). In this case, the display control unit 33 of the image processing device 1 generates display information Id for displaying the endoscopic image Ic in which the metadata target portion is highlighted and an examiner confirmation screen showing details related to the metadata target portion, as described below, and supplies the display information Id to the display device 2. On the other hand, when the image processing device 1 determines that a metadata target portion is not present in the endoscopic image Ic to be displayed on the display device 2 (step S14; No), it causes the display device 2 to display the target endoscopic image Ic (step S16).

[0052] Next, the image processing device 1 determines whether or not the endoscopic examination has ended (step S17). For example, when the image processing device 1 detects a predetermined input to the input unit 14 or the operation unit 36, the image processing device 1 determines that the endoscopic examination has ended. Then, when the image processing device 1 determines that the endoscopic examination has ended (step S17; Yes), the image processing device 1 ends the processing of the flowchart. On the other hand, when the image processing device 1 determines that the endoscopic examination has not ended (step S17; No), the image processing device 1 returns the processing to step S11. Then, in step S11, the image processing device 1 acquires an endoscopic image Ic newly generated by the endoscope 3, and re-executes the processing of steps S12 to S17 by including the endoscopic image Ic as a processing target.

[0053] Here, a supplementary explanation will be given on the generation process of the pre-inspection model Mp to be stored in the pre-inspection information storage unit 22. Hereinafter, for convenience of explanation, the image processing device 1 will be described as the processing subject, but any device other than the image processing device 1 may be the processing subject. In that case, after the pre-inspection model Mp is generated by the any device, the generated pre-inspection model Mp is stored in the memory 12 (more specifically, the pre-inspection information storage unit 22) via data communication or a removable storage medium, etc.

[0054] First, the image processing device 1 acquires pre-scan data such as 3D-CT images or MRI data of an organ to be examined of a subject. Then, the image processing device 1 extracts the region of the organ to be examined from the pre-scan data based on a user input. In this case, the image processing device 1 displays the pre-scan data on the display device 2, for example, and receives a user input specifying the region of the organ to be examined by the input unit 14. Then, the image processing device 1 generates volume data representing the region of the organ to be examined extracted from the pre-scan data of the subject. This volume data is, for example, three-dimensional voxel data representing the region of the organ to be examined by two values ​​of 0 and 1. Next, the image processing device 1 creates a three-dimensional pre-examination model Mp, which is a surface model, from the above-mentioned volume data. In this case, the image processing device 1 converts the volume data into the pre-examination model Mp using any algorithm that converts voxel data into polygon data. The above algorithm includes, for example, the marching cubes method and the marching tetrahedra method. The generated pre-inspection model Mp is stored in the memory 12 (more specifically, the pre-inspection information storage unit 22) that the image processing device 1 can refer to.

[0055] Next, the matching process in step S13 will be explained in more detail.

[0056] First, the matching unit 32 extracts landmark feature points from the pre-inspection model Mp and the reconstructed data Mr, respectively. In this case, the matching unit 32 performs, for example, three-dimensional smoothing of the reconstructed data Mr, and extracts feature points that are characteristic points in the point cloud based on the point cloud constituting the smoothed reconstructed data Mr and its connection graph. In this case, the matching unit 32 extracts the above-mentioned feature points by using various point cloud feature extraction methods, such as Principal Component Analysis (PCA) of the point cloud and Difference of Center of Gravity (DoCoG). Note that the pre-inspection model Mp may have predetermined identification labels or the like attached to the feature points to be extracted.

[0057] Next, the matching unit 32 obtains correspondence between feature points extracted from the pre-inspection model Mp and the reconstructed data Mr, respectively, and performs rigid body matching between the pre-inspection model Mp and the reconstructed data Mr. In this case, the matching unit 32 translates (including rotates) at least one of the pre-inspection model Mp or the reconstructed data Mr so that the distance between corresponding feature points is minimized. Next, the matching unit 32 morphs the pre-inspection model Mp using the reconstructed data Mr as a reference. In this case, the matching unit 32 matches the pre-inspection model Mp and the reconstructed data Mr by using a matching method between point groups such as ICPD (Iterative Coherent Point Drift), and moves points in the pre-inspection model Mp other than those considered to be feature points (landmarks).

[0058] (5) Inspector confirmation screen Next, a detailed description will be given of each display example (first display example to sixth display example) of the examiner confirmation screen that the display control unit 33 causes the display device 2 to display. The display control unit 33 generates display information Id for displaying the examiner confirmation screen including the endoscopic image Ic in which the metadata target portion is highlighted, and supplies the display information Id to the display device 2, thereby causing the display device 2 to display the examiner confirmation screen.

[0059] 6 shows a first display example of the examiner confirmation screen. In the first display example, when a metadata target portion corresponding to the latest endoscopic image Ic generated by the endoscope 3 exists, the display control unit 33 displays a mark for highlighting the metadata target portion and information such as a diagnosis result linked to the metadata target portion.

[0060] In Figure 6, the display control unit 33 mainly displays, on the examiner confirmation screen, the latest endoscopic image Ic captured by the endoscope 3, a dashed circle mark 41 surrounding a metadata target portion in the endoscopic image Ic, and a metadata display field 42 showing the content of the metadata related to the metadata target portion highlighted by the mark 41.

[0061] Here, the display control unit 33 displays the diagnosis result of the pre-examination (here, a comment saying "May be a site of XX lesion"), the diagnosis date and time, and the doctor's attributes (here, the doctor's name "A" and the hospital name "XX Hospital") for the metadata target location indicated by the mark 41 in the metadata display field 42. In this case, information related to the above-mentioned diagnosis result, diagnosis date and time, the doctor's attributes, etc. are stored as metadata in the pre-examination information storage unit 22 in association with position information (e.g., coordinate information expressed in a coordinate system of the pre-scan data, etc.) indicating the metadata target location at the center position of the mark 41.

[0062] Thus, according to the first display example, the display control unit 33 can display information on the metadata generated in the preliminary examination in association with the endoscopic image Ic captured in the endoscopic examination. This allows the examiner performing the endoscopic examination to accurately grasp the lesion site and other points of interest detected in the preliminary examination during the endoscopic examination. Furthermore, in the first display example, the examiner performing the endoscopic examination can also conveniently grasp the attributes of the doctor who performed the preliminary examination on the examiner confirmation screen.

[0063] Note that the display control unit 33 may highlight the metadata target portion on the endoscopic image Ic with a mark of any shape other than a dashed circle (for example, a star or other figure) instead of highlighting the metadata target portion with the dashed circle mark 41 surrounding the metadata target portion. In addition, when the metadata target portion is recorded as an area in the metadata, the display control unit 33 may highlight the entire area or a line bordering the area on the endoscopic image Ic.

[0064] 7 shows a second display example of the examiner confirmation screen. In the second display example, when multiple doctors have performed the pre-examination, the display control unit 33 changes the display mode of the metadata for each doctor who has performed the pre-examination.

[0065] In Fig. 7, metadata target locations based on metadata added by different doctors A and B through annotation work are present within the imaging range of the latest endoscopic image Ic captured by the endoscope 3. Here, the display control unit 33 displays, on the latest endoscopic image Ic, a dashed circle mark 41A surrounding the metadata target location based on the metadata added by doctor A in the preliminary examination, and a dashed-dotted circle mark 41B surrounding the metadata target location based on the metadata added by doctor B in the preliminary examination. Furthermore, the display control unit 33 displays, on the examiner confirmation screen, a metadata display field 42A showing the content of the metadata corresponding to the metadata target location highlighted by the mark 41A, and a metadata display field 42B showing the content of the metadata corresponding to the metadata target location highlighted by the mark 41B. Here, metadata display field 42A displays the diagnosis result for the target metadata target area (here, the comment "This may be the site of a XX lesion"), the date and time of diagnosis, and the attributes of Doctor A who made the diagnosis, and metadata display field 42B displays the diagnosis result for the target metadata target area (here, the comment "This is a surgical scar"), the date and time of diagnosis, and the attributes of Doctor B who made the diagnosis.

[0066] As described above, the display control unit 33 indicates the mark 41A, which is a display object based on the metadata added by Doctor A, and the metadata display field 42A, by dashed lines, and indicates the mark 41B, which is a display object based on the metadata added by Doctor B, and the metadata display field 42B, by dashed and dotted lines. In this way, in the second display example, the display control unit 33 changes the line type of the mark surrounding the metadata target portion for each doctor who performed the pre-examination. In this case, any line type, such as dashed line, dashed line, or solid line, may be used.

[0067] In this way, in the second display example, the display control unit 33 changes the display mode of the information (display object) related to the metadata for each doctor who performed the pre-examination. This allows the display control unit 33 to clearly recognize the difference between the doctors who have added the metadata to the examiner of the endoscopic examination. Instead of or in addition to changing the line type of the mark surrounding the metadata target portion for each doctor who performed the pre-examination, the display control unit 33 may change the shape of the mark surrounding the metadata target portion for each doctor who performed the pre-examination. In this case, the shape of the mark may be any shape such as a circle, a triangle, or a square. Furthermore, the display mode in the case where the display mode related to the metadata is changed for each doctor who performed the pre-examination is not limited to the example given here. Any display mode that allows the examiner to distinguish and recognize the doctor in charge of the pre-examination corresponding to the metadata target portion may be used. This is also true in the case where the display mode of the information superimposed on the endoscopic image is changed in any display example other than the second display example (for example, the fourth display example and the sixth display example described later). This allows the display mode of the information that the examiner should distinguish and recognize to be different, thereby supporting the examiner in the examination.

[0068] Furthermore, the display control unit 33 may, for example, display the mark representing the metadata target location (marks 41A and 41B in FIG. 7) in a display mode in which the mark does not overlap with the target metadata target location. In another example, the display control unit 33 may increase the transparency of the mark representing the metadata target location and the mark in the overlapping portion of the target metadata target location (specifically, increase the transparency to such an extent that the metadata target location overlapping with the mark can be visually recognized). For example, in this case, the display control unit 33 may make the inside of the mark indicating the metadata target location white (transparent). This allows the display control unit 33 to easily visually recognize the metadata target location by the examiner.

[0069] 8 shows a third display example of the examiner confirmation screen. In the third display example, the display control unit 33 displays the matching degree indicating the degree of matching between the reconstructed data Mr and the pre-inspection model Mp by the matching unit 32 as the accuracy of the position of the metadata target location highlighted on the endoscopic image Ic (also called "position accuracy"). The above matching degree indicates a higher value as the degree of matching between the reconstructed data Mr and the pre-inspection model Mp increases. Furthermore, in the third display example, the display control unit 33 displays the predicted arrival distance from the tip 38 of the endoscope 3 to the metadata target location.

[0070] 8, the display control unit 33 displays a mark 41B surrounding the metadata target portion on the endoscopic image Ic, and also displays a metadata display field 42C showing the contents of the metadata related to the metadata target portion. Here, the display control unit 33 displays the diagnosis result of the preliminary examination (here, a comment "possibly a lesion site") in the metadata display field 42C.

[0071] Furthermore, the display control unit 33 displays an expected arrival distance display field 45 that indicates the expected arrival distance from the tip 38 to the location where the metadata currently being displayed exists. Note that, for example, in SfM or the like used to generate the reconstruction data Mr, a parameter indicating the shooting position is obtained, and the display control unit 33 calculates the parameter and the position of the metadata target location as the above-mentioned expected arrival distance.

[0072] Furthermore, the display control unit 33 displays a numerical value (here, 65 in a range of 0 to 100) indicating the position accuracy and a position accuracy display field 46 indicating a gauge corresponding to the numerical value on the examiner confirmation screen. In this case, for example, the display control unit 33 acquires a value corresponding to a solution (optimum solution) of an evaluation function (for example, the sum of distances between feature points of the pre-inspection model Mp and the reconstructed data Mr) used in matching by the matching unit 32 as the degree of match, and calculates a value obtained by normalizing the degree of match to be in a range of 0 to 100 by any normalization method as the position accuracy. Note that, when minimizing the evaluation function in matching, the display control unit 33 may increase the position accuracy as the value of the evaluation function in the solution is lower, and when maximizing the evaluation function in matching, the display control unit 33 may increase the position accuracy as the value of the evaluation function in the solution is higher.

[0073] As described above, the display control unit 33 displays the position accuracy of the position of the metadata target portion on the endoscopic image Ic and the endoscopic image Ic in which the metadata target portion is highlighted. According to the third display example, the display control unit 33 displays the position accuracy together with the endoscopic image Ic in which the metadata target portion is highlighted, thereby allowing the examiner to recognize the reliability of the position of the highlighted metadata target portion. In addition, the display control unit 33 can appropriately support the operation of the examiner in the endoscopic examination by displaying the expected arrival distance, which is information useful for operating the endoscope 3. Note that the calculation of the position accuracy may be performed by any processing block in the image processing device 1 other than the display control unit 33 based on the matching result by the matching unit 32.

[0074] 9 shows a fourth display example of the examiner confirmation screen. In the fourth display example, the display control unit 33 detects a lesion site in the endoscopic image Ic by executing CAD in real time, and displays information based on a comparison result between the position of the lesion site based on the CAD and the metadata target position.

[0075] 9, the display control unit 33 displays, on the endoscopic image Ic, a mark 41D that surrounds a position where the metadata target portion (i.e., the position of the lesion portion detected in the preliminary examination) and the position of the lesion portion detected by CAD using the endoscopic image Ic overlap, and a mark 41E that surrounds a position that is not the metadata target portion but is detected as the lesion portion by CAD. Furthermore, the display control unit 33 displays a comment display field 42D that displays a comment corresponding to the position highlighted by the mark 41D, and a comment display field 42E that displays a comment corresponding to the position highlighted by the mark 41E.

[0076] Here, the display control unit 33 displays in a comment display field 42D corresponding to the mark 41D that the position indicated by the mark 41D was detected as a lesion site by both the diagnosis in the preliminary examination and real-time CAD. In addition, the display control unit 33 displays in a comment display field 42E corresponding to the mark 41E that a lesion site not detected in the preliminary examination was detected by real-time CAD. Note that the display control unit 33 may also provide a mark and comment display field in the case where there is a site detected as a lesion site in the preliminary examination but not detected by real-time CAD, and present the existence of such a site to the examiner performing the endoscopic examination.

[0077] In this way, in the fourth display example, the display control unit 33 notifies the examiner of information based on the comparison result between the diagnosis in the preliminary examination and the real-time CAD diagnosis to the examiner performing the endoscopic examination. This allows the display control unit 33 to appropriately support the examiner in the endoscopic examination.

[0078] 10 shows a fifth display example of the examiner confirmation screen. In the fifth display example, the display control unit 33 performs a process of extracting a lesion area including a metadata target portion from the endoscopic image Ic, and highlights the extracted lesion area on the endoscopic image Ic.

[0079] In FIG. 10, the display control unit 33 highlights a lesion area 41F in an endoscopic image Ic that includes a metadata target area on the endoscopic image Ic, and displays a metadata display field 42F showing the contents of the metadata related to the lesion area 41F.

[0080] In the fifth display example, when a metadata target location exists in the endoscopic image Ic to be displayed, the display control unit 33 executes a process of extracting a lesion area 41F in the endoscopic image Ic including the metadata target location. In this case, the display control unit 33 extracts the above-mentioned lesion area 41F based on information output from a lesion area extraction model by inputting the endoscopic image Ic to be displayed and the metadata target location to the model. In this case, when the endoscopic image Ic and position information indicating a pathogenic site in the endoscopic image Ic are input, the lesion area extraction model is trained to output information indicating the entire area of ​​the lesion site including the position indicated by the position information, and the trained parameters are stored in advance in the memory 12 or the like. Note that, when an area has already been specified as a metadata target location in the metadata, the display control unit 33 may display the lesion area 41F based on the metadata.

[0081] In this way, according to the fifth display example, the lesion area can be accurately extracted based on the metadata generated by the pre-examination, and the lesion area can be displayed on the examiner confirmation screen together with the diagnosis results of the pre-examination, etc.

[0082] 11 shows a sixth display example of the examiner confirmation screen. In the sixth display example, when the metadata includes information on an arbitrary point of interest other than a lesion site, the display control unit 33 determines the display mode of the metadata target point according to the type of the point of interest indicated by the metadata.

[0083] The display control unit 33 displays, on the examiner confirmation screen, an endoscopic image Ic on which marks 41G and 41H having display modes corresponding to the types of the areas of interest are superimposed, a mark explanation display area 43 showing the correspondence between the types of marks that may be used and the types of the areas of interest, and a medical history display field 44 showing the medical history of the examinee. In the example of FIG. 10, the display control unit 33 recognizes that a lesion site and a surgical scar are present in the displayed endoscopic image Ic based on the metadata, and displays, on the endoscopic image Ic, a circular mark 41G surrounding the metadata target site corresponding to the lesion site and a triangular mark 41H surrounding the metadata target site corresponding to the surgical scar. This allows the display control unit 33 to easily allow the examiner to understand the type of the area of ​​interest detected in the preliminary examination.

[0084] The display control unit 33 also has a medical history display field 44 that shows the medical history of the subject (here, "subject Z"). In this case, the display control unit 33 receives, for example, information on the medical history of the subject from the memory 12 or an external device that stores the medical history, and displays the medical history display field 44 based on the received information. Such information is suitably referred to by the examiner as reference information during an endoscopic examination.

[0085] (6) Variations When detecting a lesion site based on the endoscopic image Ic in the fourth display example or the like, the image processing device 1 may detect the lesion site using both the pre-scan data and the endoscopic image Ic.

[0086] In this case, the image processing device 1 uses a lesion detection model that receives as input the endoscopic image Ic and an image of pre-scan data (e.g., a CT image) corresponding to the endoscopic image Ic (i.e., a region to be photographed is common to the endoscopic image Ic). In this case, when the endoscopic image Ic and the image of the pre-scan data are input, the lesion detection model is trained to output the presence or absence of a lesion site in the endoscopic image Ic and the position information of the lesion site if the lesion site exists. In this case, the endoscopic image Ic and the image of the pre-scan data may be superimposed in the channel direction and input to the lesion detection model, or these images may be connected vertically or horizontally as a single image and input to the lesion detection model. Note that the image processing device 1 specifies the image of the pre-scan data corresponding to each endoscopic image Ic, for example, based on the matching result Rm in the matching unit 32. Then, the image processing device 1 inputs the endoscopic image Ic and the image of the pre-scan data to the lesion detection model, and obtains the presence or absence of a lesion site in the endoscopic image Ic output by the lesion detection model and the position information of the lesion site if the lesion site exists.

[0087] According to this modification, the image processing device 1 can detect a lesion site with higher accuracy by using both the pre-scan data and the endoscopic image Ic.

[0088] <Second embodiment> 12 is a block diagram of an image processing device 1X in the second embodiment. The image processing device 1X mainly includes a three-dimensional reconstruction means 31X, a matching means 32X, and a display control means 33X. The image processing device 1X may be composed of a plurality of devices.

[0089] The three-dimensional reconstruction means 31X generates reconstruction data in which the inspection object is three-dimensionally reconstructed based on an endoscopic image of the inspection object captured by an imaging unit provided in the endoscope. In this case, the three-dimensional reconstruction means 31X may directly receive the endoscopic image from the imaging unit, or may acquire the endoscopic image from a storage device that accumulates the endoscopic image captured by the imaging unit. The "inspection object" may be the large intestine, or may be another organ such as the stomach. The three-dimensional reconstruction means 31X may be the three-dimensional reconstruction unit 31 in the first embodiment.

[0090] The matching means 32X matches the 3D model of the inspection target with the reconstructed data. Here, the matching means 32X may obtain the 3D model of the inspection target from a memory included in the image processing device 1X, or may obtain the model from an external device different from the image processing device 1X. The matching means 32X may be the matching unit 32 in the first embodiment.

[0091] The display control means 33X displays information about metadata associated with the three-dimensional model based on the matching result. The display control means 33X may display the above information on a display unit included in the image processing device 1X, or may display the above information on a display device separate from the image processing device 1X. The display control means 33X may be the display control unit 33 in the first embodiment.

[0092] 13 is an example of a flowchart showing a processing procedure executed by the image processing device 1X in the second embodiment. The three-dimensional reconstruction means 31X generates reconstruction data in which the inspection object is three-dimensionally reconstructed based on an endoscopic image of the inspection object captured by an imaging unit provided in the endoscope (step S21). The matching means 32X matches the three-dimensional model of the inspection object with the reconstruction data (step S22). The display control means 33X displays information about metadata associated with the three-dimensional model based on the matching result (step S23).

[0093] According to the second embodiment, the image processing device 1X can display information related to metadata associated in advance with a three-dimensional model of an inspection target.

[0094] In addition, some or all of the above-described embodiments (including modified examples, the same applies below) can be described as, but are not limited to, the following notes.

[0095] [Appendix 1] a three-dimensional reconstruction means for generating reconstruction data by three-dimensionally reconstructing the object of inspection based on an endoscopic image obtained by photographing the object of inspection by an imaging unit provided in the endoscope; a matching means for matching the three-dimensional model of the inspection object with the reconstructed data; a display control means for displaying information about metadata associated with the three-dimensional model based on a result of the matching; An image processing device comprising: [Appendix 2] The metadata includes location information indicating a metadata target location that the metadata targets, The image processing device described in Appendix 1, wherein the display control means, when it is determined that the metadata target area exists in the endoscopic image based on the matching result and the position information, highlights the metadata target area on the endoscopic image. [Appendix 3] 3. The image processing device according to claim 2, wherein the metadata target area includes at least an area diagnosed as a lesion area. [Appendix 4] the metadata includes information representing a diagnosis result regarding the metadata target portion, 4. The image processing device according to claim 2, wherein the display control means displays information representing the diagnosis result in association with the endoscopic image in which the metadata target area is highlighted. [Appendix 5] The metadata includes information representing a diagnosis result regarding the metadata target portion and an attribute of a diagnoser, The image processing device according to claim 4, wherein the display control means displays information representing the diagnosis result and the attributes in association with the endoscopic image in which the metadata target area is highlighted. [Appendix 6] 6. The image processing device according to claim 5, wherein, when there are a plurality of diagnosers, the display control means changes a display mode of the information on the metadata for each of the diagnosers. [Appendix 7] the display control means displays the accuracy of the position of the metadata target portion on the endoscopic image and the endoscopic image with the metadata target portion highlighted; The accuracy is calculated based on the result of the matching. 7. The image processing device according to claim 2, [Appendix 8] the metadata target area is an area diagnosed as a lesion area, The image processing device according to any one of claims 2 to 7, wherein the display control means displays information based on a result of comparing a detected position of a lesion site based on the endoscopic image with the metadata target location. [Appendix 9] The image processing device according to any one of appendices 2 to 8, wherein the display control means determines a display mode for the metadata object portion based on a type of the metadata object portion. [Appendix 10] The image processing device according to any one of claims 2 to 9, wherein the display control means, when the metadata target location is a location diagnosed as a lesion site, extracts a lesion area including the metadata target location and highlights the lesion area on the endoscopic image. [Appendix 11] the three-dimensional model is data generated based on scan data of the object of inspection obtained in a pre-inspection performed before the inspection using the endoscope; The image processing device according to any one of claims 1 to 10, wherein the metadata is data generated based on an inspection result of the pre-inspection. [Appendix 12] By computer, generating reconstruction data that reconstructs the object of inspection in three dimensions based on an endoscopic image of the object of inspection captured by an imaging unit provided in the endoscope; matching the three-dimensional model of the object to the reconstructed data; displaying information about metadata associated with the three-dimensional model based on the results of the matching; Image processing methods. [Appendix 13] generating reconstruction data that reconstructs the object of inspection in three dimensions based on an endoscopic image of the object of inspection captured by an imaging unit provided in the endoscope; matching the three-dimensional model of the object to the reconstructed data; A storage medium storing a program for causing a computer to execute a process for displaying information regarding metadata associated with the three-dimensional model based on the result of the matching.

[0096] Although the present invention has been described above with reference to the embodiments, the present invention is not limited to the above-mentioned embodiments. Various modifications that can be understood by a person skilled in the art can be made to the configuration and details of the present invention within the scope of the present invention. In other words, the present invention naturally includes various modifications and corrections that a person skilled in the art could make in accordance with the entire disclosure, including the claims, and the technical ideas. In addition, the disclosures of the above cited patent documents and the like are incorporated herein by reference. [Explanation of symbols]

[0097] 1. 1X image processing device 2 Display device 3 Endoscope 11 Processors 12. Memory 13. Interface 14 Input section 15 Light source section 16 Sound output section 100 Endoscopy System

Claims

1. a three-dimensional reconstruction unit that generates reconstruction data by three-dimensionally reconstructing the object of inspection based on an endoscopic image obtained by photographing the object of inspection by an imaging unit provided in the endoscope; a matching means for matching the three-dimensional model of the inspection object with the reconstructed data; a display control means for displaying information regarding metadata associated with the three-dimensional model based on a result of the matching; An image processing device comprising:

2. The metadata includes location information indicating a metadata target location that the metadata targets, 2. The image processing device according to claim 1, wherein when it is determined that the metadata target portion exists in the endoscopic image based on the matching result and the position information, the display control means highlights the metadata target portion on the endoscopic image.

3. The image processing device according to claim 2 , wherein the metadata target portion includes at least a portion diagnosed as a lesion portion.

4. the metadata includes information representing a diagnosis result regarding the metadata target portion, 4. The image processing apparatus according to claim 2, wherein the display control means displays information indicating the diagnosis result in association with the endoscopic image in which the metadata target portion is highlighted.

5. The metadata includes information representing a diagnosis result regarding the metadata target portion and an attribute of a diagnoser, The image processing apparatus according to claim 4 , wherein the display control means displays the information representing the diagnosis result and the attribute in association with the endoscopic image in which the metadata target portion is highlighted.

6. The image processing apparatus according to claim 5 , wherein, when there are a plurality of diagnosers, the display control means changes a display mode of the information on the metadata for each of the diagnosers.

7. the display control means displays the accuracy of the position of the metadata target portion on the endoscopic image and the endoscopic image with the metadata target portion highlighted; The accuracy is calculated based on the result of the matching. The image processing device according to any one of claims 2 to 6.

8. the metadata target area is an area diagnosed as a lesion area, 8. The image processing device according to claim 2, wherein the display control means displays information based on a result of comparing a detected position of a lesion site based on the endoscopic image with the metadata target location.

9. 9. The image processing device according to claim 2, wherein the display control means determines a display mode for the metadata target portion based on a type of the metadata target portion.

10. The image processing device according to any one of claims 2 to 9, wherein the display control means, when the metadata target location is a location diagnosed as a lesion site, extracts a lesion area including the metadata target location and highlights the lesion area on the endoscopic image.

11. the three-dimensional model is data generated based on scan data of the object to be examined obtained in a pre-examination performed before the examination using the endoscope; The image processing device according to any one of claims 1 to 10, wherein the metadata is data generated based on an inspection result of the pre-inspection.

12. By computer, generating reconstruction data by three-dimensionally reconstructing the object of inspection based on an endoscopic image obtained by photographing the object of inspection by an imaging unit provided in the endoscope; matching the three-dimensional model of the object to the reconstructed data; displaying information about metadata associated with the three-dimensional model based on the result of the matching; Image processing methods.

13. generating reconstruction data by three-dimensionally reconstructing the object of inspection based on an endoscopic image obtained by photographing the object of inspection by an imaging unit provided in the endoscope; matching the three-dimensional model of the object to the reconstructed data; A program that causes a computer to execute a process of displaying information about metadata associated with the three-dimensional model based on a result of the matching.

Images