Image display method, display control device, and computer-readable storage medium

Abstract

The present invention provides an image display method, a display control device, and a computer-readable storage medium, which visualize positions having three-dimensional coordinates associated with at least a part of a two-dimensional image of a subject. In a storage step, a storage control unit stores three-dimensional data of the subject in a storage medium. In a first selection step, a selection unit selects a reference image. In a second selection step, the selection unit selects a two-dimensional image, i.e., a selection image, which is used to generate the three-dimensional data, based on the reference image. In an estimation step, an estimation unit estimates second camera coordinates related to the reference image based on first camera coordinates related to the selection image. In a display step, a display control unit causes a display to display an image of the subject in which positions indicated by at least one of the second camera coordinates and three-dimensional coordinates of one or more points of the subject calculated based on the second camera coordinates are visualized.

Description

Technical Field

[0001] The present invention relates to an image display method, a display control device, and a program. Background Art

[0002] Industrial endoscope devices are used to inspect abnormalities (such as damage and corrosion) inside boilers, gas turbines, automobile engines, and pipes. During endoscope inspections, in order to leave evidence of the presence or absence of abnormalities and the severity of the abnormalities, the inspection operator records still images during the inspection. After the endoscope inspection is completed, an inspection report is generated. Generally, text indicating the state of abnormalities captured in the recorded still images is attached to the inspection report together with the still images.

[0003] Position information is included in additional matters related to abnormalities. The position information indicates the position in the inspection object where the recorded still image was obtained. The position information of the detected abnormality is important when replacing or repairing the abnormal part or when performing the next inspection. The inspection operator reads the position based on the marks printed on the insertion portion of the endoscope to obtain the position information. The marks indicate the length of the insertion portion inserted into the inspection object (insertion length). The operator copies this position into a memo.

[0004] However, in the above-described inspections, for the following four reasons and others, it is sometimes difficult for the operator to grasp the position information of the abnormality.

[0005] (1) The operator can know the approximate position of a part in a relatively simple structure such as a straight pipe. In the case of inspecting a structure with a complex shape (such as a turbine inspection), it is difficult for the operator to know the position of the abnormal part only based on the insertion length.

[0006] (2) Even in the case of inspecting a simple structure such as a straight pipe, the act of copying the insertion length into a memo is troublesome for the operator.

[0007] (3) In a harsh inspection environment where safety is threatened, there is no time, mental, or physical room to copy the insertion length into a memo during the inspection.

[0008] (4) The operator forgets to copy the insertion length into the memo.

[0009] As one method for solving the above problems, there is a method of associating the still images recorded during the inspection with three-dimensional data (3D data) representing the three-dimensional shape (3D shape) of the inspection object and visualizing the position where the still image was obtained. By using this method, the position where the still image of the inspection object was obtained becomes clear.

[0010] For example, the following method is disclosed in Patent Document 1: A schematic diagram is associated with a still image, and the schematic diagram and the still image are visualized. The schematic diagram schematically represents the internal structure of an organism. By comparing the sound or text associated with the schematic diagram with the sound or text associated with the still image, the position where the still image is acquired is determined.

[0011] The following method is disclosed in Patent Document 2: A moving image of an object to be examined is associated with a three-dimensional image acquired by a CT (Computed Tomography) device or an MRI (Magnetic Resonance Imaging) device, and the moving image or the still image is visualized. A virtual image of an organ photographed by an endoscope is generated. By comparing this image with the frames of the moving image, the position of the object to be examined corresponding to the frames of the moving image is determined.

[0012] The following method is disclosed in Patent Document 3: A three-dimensional point group corresponding to two or more two-dimensional images recorded during an examination is displayed on a three-dimensional image (3D image) of the object to be examined. The device disclosed in Patent Document 3 uses two or more still images recorded during the examination or a moving image recorded during the examination. The device reconstructs the 3D shape of the object to be examined and displays a 3D image of the 3D shape. The device receives an arbitrary position on the 3D image from the user and extracts a region including the position. The device displays a still image of the region.

[0013] Prior Art Documents

[0014] Patent Documents

[0015] Patent Document 1: Japanese Patent No. 5451718 Gazette

[0016] Patent Document 2: Japanese Patent No. 6608111 Gazette

[0017] Patent Document 3: Japanese Unexamined Patent Application Publication No. 2017 - 130049 Summary of the Invention

[0018] Problems to be Solved by the Invention

[0019] The technologies disclosed in Patent Document 1 and Patent Document 2 can associate 3D data of an object to be inspected with still images recorded during the inspection and visualize the 3D data and the still images. Therefore, an operator can grasp the position of the object to be inspected where the still image is acquired. However, the position that the operator can grasp may be limited to characteristic parts (e.g., branch structures) of the object to be inspected. In addition, in Patent Document 1, the operation of associating sound or text with a still image is troublesome for the operator.

[0020] The device disclosed in Patent Document 3 displays a still image of a specific area specified by a user on a 3D image. However, the device does not detect the area of the 3D shape corresponding to the still image that the user is interested in and does not display the position of that area in the 3D shape.

[0021] The images associated with 3D data for grasping the position of an object to be inspected are not limited to still images. The user can also label a specific frame with a tag indicating a frame of interest during the recording of a moving image, and the image of the area of interest (abnormality, etc.) captured in the specific frame is associated with the 3D data. It can also be that by using machine learning or the like to automatically detect an abnormality from the frame, the frame or the area within the frame is associated with the 3D data. Since the above-mentioned images are associated with the 3D data, the operator and the inspection approver can grasp the position on the 3D data. However, the related description is not recorded in any of Patent Documents 1 to 3.

[0022] An object of the present invention is to provide an image display method, a display control device, and a program capable of visualizing a position having three-dimensional coordinates associated with at least a part of a two-dimensional image of a subject.

[0023] Solution to the problem

[0024] The present invention is an image display method, comprising the following steps: a storage step, in which a storage control unit stores three-dimensional data generated based on two or more two-dimensional images of a subject in a storage medium, the three-dimensional data including three-dimensional coordinates of two or more points of the subject and first camera coordinates, the first camera coordinates being three-dimensional coordinates of a first camera that has acquired each of the two or more two-dimensional images and being associated with each of the two or more two-dimensional images; a first selection step, in which a selection unit selects a reference image that is a two-dimensional image of the subject; a second selection step, in which the selection unit selects at least one two-dimensional image from the two or more two-dimensional images as a selected image based on the reference image; an estimation step, in which an estimation unit estimates three-dimensional coordinates of a second camera, i.e., second camera coordinates, that has acquired the reference image based on the first camera coordinates associated with the selected image; and a display step, in which a display control unit causes a display to display an image of the subject in which a position indicated by at least one of the second camera coordinates and three-dimensional coordinates of one or more points of the subject calculated based on the second camera coordinates is visualized.

[0025] In the image display method of the present invention, the image of the subject is a three-dimensional image, in which at least one of the second camera coordinates and three-dimensional coordinates of one or more points of the subject calculated based on the second camera coordinates is visualized, and three-dimensional coordinates of one or more of the two or more points are visualized.

[0026] In the image display method of the present invention, the three-dimensional data further includes first pose information indicating a pose of the first camera, the first pose information being associated with each of the two or more two-dimensional images. In the estimation step, the estimation unit further estimates a pose of the second camera based on the first pose information associated with the selected image, generates second pose information indicating the estimated pose, and in the estimation step, the estimation unit further estimates three-dimensional coordinates of one or more points of a region of the subject captured in the reference image based on the second camera coordinates and the second pose information, and the three-dimensional coordinates of the one or more points of the region are visualized in the three-dimensional image.

[0027] In the image display method of the present invention, in the display step, the display control unit further causes information indicating a position of the region to be superimposed on the three-dimensional image.

[0028] The image display method of the present invention further includes a viewpoint change step, in which, after the three-dimensional image is displayed on the display, the display control unit changes a viewpoint of the three-dimensional image.

[0029] In the image display method of the present invention, in the first selection step, the selection unit selects two or more of the reference images, and in the display step, the display control unit further causes the display to display information indicating the position of the common area between the two or more reference images.

[0030] In the image display method of the present invention, the three-dimensional image is an image of shape data representing the three-dimensional shape of the subject, and the image display method further includes an association step in which the data processing unit associates the three-dimensional coordinates of two or more points of the subject with the three-dimensional shape shown by the shape data.

[0031] In the image display method of the present invention, in the display step, the display control unit further causes the display to display viewing angle information indicating the viewing angle of the reference image.

[0032] In the image display method of the present invention, in the display step, the display control unit further causes the display to display the three-dimensional distance between the reference point included in two or more points of the subject and the area.

[0033] In the image display method of the present invention, the subject includes two or more local areas arranged periodically, and in the display step, the display control unit further causes the display to display the number of at least a part of the local areas from the reference part to the concerned part, the reference part being one of the two or more local areas, and the concerned part being the local area corresponding to the area and different from the reference part.

[0034] In the image display method of the present invention, the area is the whole of the range captured in the reference image.

[0035] In the image display method of the present invention, the area is a part of the range captured in the reference image.

[0036] In the image display method of the present invention, in the estimation step, the estimation unit estimates the three-dimensional coordinates of one or more points in the region of the subject captured in the reference image based on the second camera coordinates, the second pose information, and the selected image. The one or more points in the region are different from the points having the three-dimensional coordinates included in the three-dimensional data. The three-dimensional coordinates of the one or more points in the region are visualized in the three-dimensional image, and the three-dimensional coordinates of one or more points included in the region among the three-dimensional coordinates included in the three-dimensional data are visualized. The density of the points having the three-dimensional coordinates visualized by the three-dimensional image is higher than the density of the points included in the region among the two or more points of the subject.

[0037] In the image display method of the present invention, in the first selection step, the selection unit selects the reference image including the reference region that satisfies a preset condition. In the estimation step, the estimation unit estimates the three-dimensional coordinates of one or more points in the reference region.

[0038] In the image display method of the present invention, in the estimation step, the estimation unit estimates the three-dimensional coordinates of the one or more points in the reference region based on the second camera coordinates, the second pose information, and the selected image. The one or more points in the reference region are different from the points having the three-dimensional coordinates included in the three-dimensional data. The three-dimensional coordinates of the one or more points in the reference region are visualized in the three-dimensional image, and the three-dimensional coordinates of one or more points included in the region among the three-dimensional coordinates included in the three-dimensional data are visualized. The density of the points having the three-dimensional coordinates visualized by the three-dimensional image is higher than the density of the points included in the region among the two or more points of the subject.

[0039] In the image display method of the present invention, the three-dimensional data further includes the two-dimensional coordinates of the first points included in each of the two or more two-dimensional images. In the estimation step, the estimation unit estimates the second camera coordinates based on the first camera coordinates, the two-dimensional coordinates of the first points included in one of the two or more two-dimensional images, and the two-dimensional coordinates of the second points corresponding to the first points in the reference image.

[0040] In the image display method of the present invention, in the first selection step, the selection unit selects one of the one or more selection target images as the reference image. The one or more selection target images are two-dimensional images of the subject and are not included in the two or more two-dimensional images.

[0041] In the image display method of the present invention, each of the two-dimensional images among the two or more two-dimensional images is temporally associated with other two-dimensional images included in the two or more two-dimensional images.

[0042] In the image display method of the present invention, each of the two-dimensional images among the two or more two-dimensional images is an image acquired by a monocular camera.

[0043] In the image display method of the present invention, the image of the subject is a two-dimensional image, and in the two-dimensional image, two-dimensional coordinates corresponding to at least one of the second camera coordinates and the three-dimensional coordinates of one or more points of the subject calculated based on the second camera coordinates are visualized, and two-dimensional coordinates corresponding to the three-dimensional coordinates of one or more points among the two or more points are visualized.

[0044] In the image display method of the present invention, the three-dimensional data further includes first pose information indicating the pose of the first camera, the first pose information is associated with each of the two or more two-dimensional images, and in the estimation step, the estimation unit further estimates the pose of the second camera based on the first pose information associated with the selected image, and generates second pose information indicating the estimated pose. In the estimation step, the estimation unit further estimates the three-dimensional coordinates of one or more points in the region of the subject captured in the reference image based on the second camera coordinates and the second pose information, and two-dimensional coordinates corresponding to the three-dimensional coordinates of the one or more points in the region are visualized in the two-dimensional image.

[0045] In the image display method of the present invention, in the display step, the display control unit further superimposes information indicating the position of the region on the two-dimensional image.

[0046] The image display method of the present invention further includes a generation step, in which a generation unit generates the two-dimensional image by using the three-dimensional data.

[0047] In the generation step, the generation unit generates a first image, which is a two-dimensional image of the subject observed from a first viewpoint, and generates a second image, which is a two-dimensional image of the subject observed from a second viewpoint different from the first viewpoint. In the display step, the display control unit causes the display to display the first image and the second image.

[0048] In the image display method of the present invention, the reference image is pre-associated with the information of the subject, and the image display method further includes an information display step. In this information display step, the display step is executed, and when an instruction for selecting the reference image is input, the display control unit causes the display to display the information associated with the reference image indicated by the instruction.

[0049] In the image display method of the present invention, the two-dimensional image is an image of shape data representing the two-dimensional shape of the subject, and the image display method further includes an association step. In this association step, the data processing unit associates the two-dimensional coordinates corresponding to the three-dimensional coordinates of two or more points of the subject with the two-dimensional shape shown by the shape data.

[0050] In the image display method of the present invention, in the second selection step, the selection unit uses time information that temporally correlates two or more of the two-dimensional images with each other to select at least one of the two or more two-dimensional images as the selected image.

[0051] The present invention is a display control device having: a storage control unit that stores three-dimensional data generated based on two or more two-dimensional images of a subject in a storage medium, the three-dimensional data including the three-dimensional coordinates of two or more points of the subject and first camera coordinates, the first camera coordinates being the three-dimensional coordinates of the first camera that acquired each of the two or more two-dimensional images and being associated with each of the two or more two-dimensional images; a selection unit that selects a reference image that is a two-dimensional image of the subject and, based on the reference image, selects at least one of the two or more two-dimensional images as a selected image; an estimation unit that estimates the three-dimensional coordinates of a second camera, i.e., second camera coordinates, that acquired the reference image based on the first camera coordinates associated with the selected image; and a display control unit that causes a display to display an image of the subject in which at least one of the positions indicated by the second camera coordinates and the three-dimensional coordinates of one or more points of the subject calculated based on the second camera coordinates is visualized.

[0052] The present invention is a program for causing a computer to execute the following steps: a storage step of storing three-dimensional data generated based on two or more two-dimensional images of a subject in a storage medium, the three-dimensional data including three-dimensional coordinates of two or more points of the subject and first camera coordinates, the first camera coordinates being three-dimensional coordinates of a first camera that has acquired each of the two or more two-dimensional images and being associated with each of the two or more two-dimensional images; a first selection step of selecting a reference image that is a two-dimensional image of the subject; a second selection step of selecting at least one two-dimensional image from the two or more two-dimensional images as a selected image based on the reference image; an estimation step of estimating three-dimensional coordinates of a second camera, i.e., second camera coordinates, that has acquired the reference image based on the first camera coordinates associated with the selected image; and a display step of displaying, on a display, an image of the subject in which at least one of the second camera coordinates and three-dimensional coordinates of one or more points of the subject calculated based on the second camera coordinates is visualized.

[0053] Effects of the Invention

[0054] According to the present invention, an image display method, a display control device, and a program can visualize a position having three-dimensional coordinates associated with at least a part of a two-dimensional image of a subject. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] Figure 1 is a block diagram showing the configuration of a PC according to a first embodiment of the present invention.

[0056] Figure 2 is a flowchart showing the process of processing executed by a PC according to a first embodiment of the present invention.

[0057] Figure 3 is a schematic diagram showing the state of image acquisition in a first embodiment of the present invention.

[0058] Figure 4 is a flowchart showing the process of processing for generating a three-dimensional model in a first embodiment of the present invention.

[0059] Figure 5 is a flowchart showing the process of processing for estimating the position and orientation of a camera in a first embodiment of the present invention.

[0060] Figure 6 is a schematic diagram showing the state of image acquisition in a first embodiment of the present invention.

[0061] Figure 7 is a schematic diagram showing the process of processing for estimating the position and orientation of a camera in a first embodiment of the present invention.

[0062] Figure 8 It is a diagram showing an example of a display screen of a display unit included in a PC according to the first embodiment of the present invention.

[0063] Figure 9 It is a diagram showing an example of a display screen of a display unit included in a PC according to the first embodiment of the present invention.

[0064] Figure 10 It is a diagram showing an example of a display screen of a display unit included in a PC according to the first embodiment of the present invention.

[0065] Figure 11 It is a diagram showing an example of a display screen of a display unit included in a PC according to the first embodiment of the present invention.

[0066] Figure 12 It is a diagram showing an example of a display screen of a display unit included in a PC according to the first embodiment of the present invention.

[0067] Figure 13 It is a diagram showing an example of a display screen of a display unit included in a PC according to the first embodiment of the present invention.

[0068] Figure 14 It is a perspective view showing the overall structure of an endoscope device according to the first embodiment of the present invention.

[0069] Figure 15 It is a block diagram showing the internal structure of an endoscope device according to the first embodiment of the present invention.

[0070] Figure 16 It is a block diagram showing the structure of a PC according to the first modification of the first embodiment of the present invention.

[0071] Figure 17 It is a flowchart showing the process of processing performed by a PC according to the first modification of the first embodiment of the present invention.

[0072] Figure 18 It is a diagram showing an example of a display screen of a display unit included in a PC according to the first modification of the first embodiment of the present invention.

[0073] Figure 19 It is a diagram showing an example of a two-dimensional image of a subject in the second modification of the first embodiment of the present invention.

[0074] Figure 20 It is a flowchart showing the process of processing performed by a PC according to the fourth modification of the first embodiment of the present invention.

[0075] Figure 21 This is a diagram showing an example of a display screen of a display unit included in a PC according to a fourth modification of the first embodiment of the present invention.

[0076] Figure 22 This is a flowchart showing the process of processing performed by a PC according to a fifth modification of the first embodiment of the present invention.

[0077] Figure 23 This is a diagram showing an example of a display screen of a display unit included in a PC according to a fifth modification of the first embodiment of the present invention.

[0078] Figure 24 This is a block diagram showing the configuration of a PC according to a sixth modification of the first embodiment of the present invention.

[0079] Figure 25 This is a flowchart showing the process of processing performed by a PC according to a sixth modification of the first embodiment of the present invention.

[0080] Figure 26 This is a block diagram showing the configuration of a PC according to the second embodiment of the present invention.

[0081] Figure 27 This is a flowchart showing the process of processing performed by a PC according to the second embodiment of the present invention.

[0082] Figure 28 This is a flowchart showing the process of processing performed by a PC according to the second embodiment of the present invention.

[0083] Figure 29 This is a diagram showing an example of a display screen of a display unit included in a PC according to the second embodiment of the present invention.

[0084] Figure 30 This is a block diagram showing the configuration of a PC according to the third embodiment of the present invention.

[0085] Figure 31 This is a flowchart showing the process of processing performed by a PC according to the third embodiment of the present invention.

[0086] Figure 32 This is a diagram showing an example of a display screen of a display unit included in a PC according to the third embodiment of the present invention.

[0087] Figure 33 This is a flowchart showing the process of processing performed by a PC according to the fourth embodiment of the present invention.

[0088] Figure 34 This is a diagram showing an example of a display screen of a display unit included in a PC according to the fourth embodiment of the present invention.

[0089] Figure 35 It is a flowchart showing the process of the processing executed by the PC showing a modification of the fourth embodiment of the present invention.

[0090] Figure 36 It is a diagram showing an example of the display screen of the display unit included in the PC showing a modification of the fourth embodiment of the present invention.

[0091] Figure 37 It is a flowchart showing the process of the processing executed by the PC according to the fifth embodiment of the present invention.

[0092] Figure 38 It is a flowchart showing the process of the processing executed by the PC according to the fifth embodiment of the present invention.

[0093] Figure 39 It is a block diagram showing the configuration of the image display system according to the sixth embodiment of the present invention.

[0094] Figure 40 It is a block diagram showing the configuration of the PC according to the sixth embodiment of the present invention.

[0095] Figure 41 It is a flowchart showing the process of the processing executed by the PC according to the sixth embodiment of the present invention.

[0096] Figure 42 It is a block diagram showing the configuration of the PC according to the seventh embodiment of the present invention.

[0097] Figure 43 It is a flowchart showing the process of the processing executed by the PC according to the seventh embodiment of the present invention.

[0098] Figure 44 It is a diagram showing an example of the display screen of the display unit included in the PC according to the seventh embodiment of the present invention.

[0099] Figure 45 It is a diagram showing an example of the display screen of the display unit included in the PC according to the seventh embodiment of the present invention.

[0100] Figure 46 It is a diagram showing an example of the display screen of the display unit included in the PC according to the seventh embodiment of the present invention.

[0101] Figure 47 It is a flowchart showing the process of the processing executed by the PC showing a modification of the seventh embodiment of the present invention.

[0102] Figure 48 It is a diagram showing an example of the inspection report in the modification of the seventh embodiment of the present invention.

[0103] Figure 49 This is a diagram showing the information of the region of interest attached to the inspection report in a modified example of the seventh embodiment of the present invention. Detailed implementation manners

[0104] Hereinafter, embodiments of the present invention will be described with reference to the accompanying drawings.

[0105] (First Embodiment)

[0106] Figure 1 This shows the structure of a PC (Personal Computer) 41 according to the first embodiment of the present invention. For example, the PC 41 is a desktop PC. The PC 41 may also be a laptop PC or a tablet terminal with portability. The PC 41 may also be a computer system operating in the cloud. Figure 1 The shown PC 41 has a CPU 43, an operation unit 44, a display unit 45, a communication unit 46, and a memory 47. The PC 41 functions as a display control device.

[0107] The operation unit 44 is a user interface. For example, the operation unit 44 is at least one of a button, a switch, a key, a mouse, a joystick, a touchpad, a trackball, and a touch panel. The operation unit 44 accepts operations performed by the user on the PC 41. The user can input various information into the PC 41 by operating the operation unit 44. The operation unit 44 accepts the information input by the user and outputs the information to the CPU 43.

[0108] The display unit 45 has a display screen and displays a 3D model or the like of a subject inside the object to be inspected on the display screen. The display unit 45 is a monitor (display) such as an LCD (Liquid Crystal Display). The display unit 45 may also be a touch panel. In this case, the operation unit 44 and the display unit 45 are integrated.

[0109] The communication unit 46 performs communication with an external device such as an endoscope device. For example, the communication unit 46 is connected to the external device by a cable or wirelessly. The communication between the communication unit 46 and the external device may also be performed via a LAN (Local Area Network) or the Internet.

[0110] The memory 47 is a volatile or non-volatile memory. For example, the memory 47 is at least one of a RAM (Random Access Memory), a DRAM (Dynamic Random Access Memory), an SRAM (Static Random Access Memory), a ROM (Read Only Memory), an EPROM (Erasable Programmable Read Only Memory), an EEPROM (Electrically Erasable Programmable Read-Only Memory), and a flash memory. The memory 47 stores images and the like.

[0111] In each embodiment of the present invention, the endoscope device acquires an image (image data) of a subject. However, the device for acquiring an image is not limited to the endoscope device. It may be that a device having a camera acquires an image, and the PC 41 acquires the image. It may also be that a device having a camera acquires an image, and the image is stored in a storage medium such as a memory card. The PC 41 may also acquire the image from the storage medium. It may also be that the PC 41 has a camera, and the camera acquires an image.

[0112] The CPU 43 controls the operation of the PC 41. The functions of the CPU 43 include an image acquisition unit 430, a condition acceptance unit 431, a 3D model generation unit 432, an image selection unit 433, an estimation unit 434, a display control unit 435, and a storage control unit 436. Figure 1 At least one module within the shown CPU 43 may also be configured by a circuit different from the CPU 43.

[0113] Each part within the CPU 43 can also be constituted by at least one of a processor and a logic circuit. For example, the processor is at least one of a CPU, a DSP (Digital Signal Processor), and a GPU (Graphics Processing Unit). For example, the logic circuit is at least one of an ASIC (Application Specific Integrated Circuit) and an FPGA (Field-Programmable Gate Array). Each part within the CPU 43 can include one or more processors. Each part within the CPU 43 can include one or more logic circuits.

[0114] The image acquisition unit 430 controls the communication unit 46 to acquire a group of still images including two or more still images from the endoscope device. A still image is a two-dimensional image (2D image) of the subject. Therefore, the image acquisition unit 430 acquires two or more 2D images. The group of still images can also be a moving image including two or more 2D images that are temporally related to each other.

[0115] The user operates the operation unit 44 to input information indicating conditions for generating a three-dimensional model (3D model) of the subject into the PC 41. The condition reception unit 431 receives the conditions for generating the 3D model of the subject based on the information input by the user. Specifically, the conditions include internal parameters of the camera, distortion correction parameters of the camera, setting values, and reference lengths, etc. The setting values are used in various processes for generating the 3D model. The reference length is used to match the scale of the 3D model with that of the actual subject.

[0116] The 3D model generation unit 432 generates (reconstructs) a 3D model of the subject based on two or more 2D images included in the group of still images. The 3D model generation unit 432 does not need to use all the 2D images included in the group of still images. When the group of still images includes three or more 2D images, the 3D model generation unit 432 generates a 3D model based on all or a part of the 2D images in the group of still images. The 3D model is three-dimensional data (3D data) of the subject. The 3D model includes three-dimensional coordinates (3D coordinates), camera coordinates, and pose information of two or more points (three-dimensional point group) of the subject.

[0117] The camera coordinates (first camera coordinates) are the 3D coordinates of the camera (first camera) for each 2D image among two or more acquired 2D images, and are associated with each 2D image among the two or more 2D images. The camera coordinates are the 3D coordinates of the viewpoint when the 2D image is acquired. For example, the camera coordinates are the 3D coordinates of the observation optical system that the camera has. The pose information (first pose information) represents the pose of the camera for each 2D image among two or more acquired 2D images, and is associated with each 2D image among the two or more 2D images. For example, the pose information represents the pose of the observation optical system that the camera has.

[0118] Two or more points of the subject are captured in the 2D image. In this specification, for convenience, the term "point" is used, but the point of the subject does not have to be one point corresponding to one pixel on the screen. The point of the subject may also include an area of any size. The point of the subject may also include an area that can be specified in sub-pixels.

[0119] The 3D model contains the 2D coordinates of the points included in each 2D image among two or more 2D images. In the 3D model, the 3D coordinates and the 2D coordinates are associated with each other. In the 3D model, the 3D coordinates and the 2D coordinates are associated with the 2D image containing the 2D coordinates.

[0120] The image selection unit 433 selects a reference frame (reference image) that has captured the region of interest from the group of still images. The region of interest is a specific region specified by the user or the computer in order to visualize the position on the 3D model. The position on the 3D model corresponding to the region of interest is visualized. The region of interest is mainly classified into the entire specific image and the region of a specific subject captured in the specific image.

[0121] An example where the region of interest is the entire specific image is described. For example, the region of interest is the entire region of the still image acquired during the inspection. The entire region is the entire range captured in the still image. The region of interest may also be the entire region of an image in which an abnormality is automatically detected by using technical means such as machine learning. The region of interest may also be the entire region of an image labeled by the user during the recording of the moving image.

[0122] An example where the region of interest is the region of a specific subject is described. For example, the region of interest is a specific region of the subject specified by the user in a specific image during or after the inspection. That is, the region of interest is a partial region of the still image. The partial region is a part of the range captured in the still image. The region of interest may also be a specific region included in an image in which an abnormality is automatically detected.

[0123] Therefore, the region of interest is the whole or a part of one still image included in the still image group. The region of interest is not limited to the above examples. The image selection unit 433 may also select two or more reference frames from the still image group.

[0124] In addition, the image selection unit 433 selects at least one image (selected image) from two or more images included in the still image group that are used to generate the 3D model. Hereinafter, two or more images used to generate the 3D model are referred to as key frames. In addition, the key frames selected by the image selection unit 433 are referred to as selected frames. The reference frames and the selected frames selected by the image selection unit 433 are used by the estimation unit 434.

[0125] As described above, the 3D model includes camera coordinates and pose information, and the camera coordinates and pose information are associated with the still images included in the still image group. The selected frames are included in the still image group. The estimation unit 434 estimates the 3D coordinates (second camera coordinates) of the camera (second camera) that captured the reference frame based on the camera coordinates associated with the selected frames.

[0126] The estimation unit 434 also estimates the pose of the camera that captured the reference frame and generates pose information (second pose information) representing the estimated pose. The estimation unit 434 estimates the 3D coordinates of the region of the subject captured in the reference frame based on the camera coordinates and the pose information of the camera that captured the reference frame. Thus, the estimation unit 434 estimates the position on the 3D model corresponding to the region of interest.

[0127] The display control unit 435 causes the display unit 45 to display an image by outputting the image to the display unit 45. In addition, the display control unit 435 superimposes information on the image and outputs the image to the display unit 45. Thus, the display control unit 435 causes the display unit 45 to display an image with the superimposed information. Specifically, the display control unit 435 causes the display unit 45 to display a three-dimensional image (3D image) in which the 3D coordinates included in the 3D model are visualized. In addition, the display control unit 435 displays at least one of the camera coordinates of the camera that captured the reference frame and the position of the region on the 3D model corresponding to the region of interest on the 3D image. In addition, the display control unit 435 causes the display unit 45 to display various information presented to the user.

[0128] The storage control unit 436 stores the still image group acquired from the endoscope device in the memory 47. The storage control unit 436 stores the conditions accepted by the condition acceptance unit 431 in the memory 47. The storage control unit 436 stores the 3D model generated by the 3D model generation unit 432 in the memory 47. In addition to storing the above information and data in the memory 47, the storage control unit 436 stores various information and various data obtained by the processing of the CPU 43 in the memory 47.

[0129] Alternatively, the program may be read into the CPU 43 and the read program may be executed. The program includes commands for defining the operations of the CPU 43. That is to say, the functions of the CPU 43 may also be implemented by software. For example, the program may be provided by a "computer-readable storage medium" such as a flash memory. The program may also be transmitted to the PC 41 from a computer that holds the program via a transmission medium or by a transmission wave in the transmission medium. The "transmission medium" for transmitting the program is a medium having a function of transmitting information. Media having a function of transmitting information include networks (communication networks) such as the Internet and communication lines (communication wires) such as telephone lines. The above-described program may also implement a part of the foregoing functions. Further, the above-described program may be a difference file (difference program). The foregoing functions may also be implemented by a combination of a program recorded in a computer and the difference program.

[0130] The PC 41 may also execute processing without receiving information from the user. Therefore, the PC 41 does not need to have the operation unit 44. The PC 41 may also use an external display unit of the PC 41 to display images and the like. Therefore, the PC 41 does not need to have the display unit 45. When the PC 41 has a camera for acquiring a group of still images, the PC 41 does not need to acquire the group of still images from the endoscope device. Therefore, the PC 41 does not need to have the communication unit 46. The memory 47 may also be configured to be installable on and removable from the PC 41. Therefore, the PC 41 does not need to have the memory 47.

[0131] Refer to Figure 2 to describe the process of the processing executed to generate the 3D model and display the region of interest. Figure 2 The figure shows the process of the processing executed by the CPU 43. The PC 41 executes processing based on the group of still images, the conditions for generating the 3D model, and the information of the region of interest. Thereby, the PC 41 visualizes the position on the 3D model corresponding to the region of interest.

[0132] In the first embodiment, the region of interest is the entire region of the still image recorded according to an instruction from the user during an examination using an endoscope. Further, the group of still images is a moving image, and the time when the still image is recorded is associated with the moving image file recorded during the examination. An example in which the time when the still image is recorded is not associated with the moving image file is described in the third modification of the first embodiment. An example in which the user does not set the region of interest but the device automatically sets the region of interest is described in the third embodiment.

[0133] The image acquisition unit 430 acquires a group of still images (2D images) including two or more still images from the endoscope device by controlling the communication unit 46. The storage control unit 436 stores the group of still images in the memory 47 (step S101).

[0134] After step S101, the user inputs information representing the conditions for generating a 3D model of the subject to the PC 41. The condition reception unit 431 receives the conditions for generating a 3D model of the subject based on the information input by the user to the PC 41. The storage control unit 436 stores the conditions in the memory 47 (step S102). As described above, the conditions include the internal parameters of the camera, the distortion correction parameters of the camera, the set values, the reference length, and the like. The user does not need to specify all of these conditions. At least one of these conditions can also be automatically set by the CPU 43.

[0135] After step S102, the 3D model generation unit 432 generates a 3D model of the subject based on two or more 2D images included in the group of still images and the conditions for generating the 3D model (step S103). The 3D model generation unit 432 does not need to use all the 2D images included in the group of still images.

[0136] Describe the process of the specific processing performed by the 3D model generation unit 432. The 3D model generation unit 432 uses the group of still images acquired by the image acquisition unit 430 and the conditions received by the condition reception unit 431. Below, an example in which the 3D model generation unit 432 uses two images (still images) included in the group of still images is described. When two images are taken, the two viewpoints of the camera are different from each other. Even when three or more images are used, the basic principle is the same as when two images are used. The method described below can also be applied to the case of using three or more images.

[0137] In the method described below, feature points in each of the two images acquired from two different viewpoints are detected, and a plurality of feature points are correlated with each other. In addition, in the method described below, the position of the camera, the pose of the camera, and the 3D coordinates of the feature points are estimated based on the plurality of feature points. The method using the information of the feature points is called the Indirect Method. The method applicable to each embodiment of the present invention is not limited to this method.

[0138] For example, there is the following method: directly using the pixel values of two images obtained from two different viewpoints. By using this method, the position of the camera, the pose of the camera, and the 3D coordinates corresponding to each pixel are estimated. This method is called the Direct Method. In each embodiment of the present invention, this method can also be used. Any method can be used as long as the position of the camera, the pose of the camera, and the 3D coordinates of the subject can be estimated by using two or more images obtained from two or more different viewpoints.

[0139] Figure 3 Schematically shows the situation of image acquisition when acquiring two images of a subject. In the following description, the term "camera" is used in a broad sense. When an endoscope acquires an image, the camera in the following description specifically refers to the observation optical system at the front end of the endoscope.

[0140] As Figure 3 shown, initially, an image I1 is acquired in the imaging state c1 of the camera. Next, an image I2 is acquired in the imaging state c2 of the camera. In the imaging state c1 and the imaging state c2, at least one of the imaging position and the imaging pose is different. In Figure 3 , both the imaging position and the imaging pose are different in the imaging state c1 and the imaging state c2.

[0141] In each embodiment of the present invention, it is assumed that the same endoscope is used to acquire the image I1 and the image I2. In addition, in each embodiment of the present invention, it is assumed that the parameters of the objective optical system of the endoscope do not change. The parameters of the objective optical system are the focal length, distortion aberration, and pixel size of the image sensor, etc. Hereinafter, for convenience, the parameters of the objective optical system are simply referred to as internal parameters. When such conditions are assumed, the internal parameters that describe the characteristics of the optical system of the endoscope can be used in common regardless of the position and pose of the camera (observation optical system). In each embodiment of the present invention, it is assumed that the internal parameters are acquired when the product is shipped from the factory. In addition, in each embodiment of the present invention, it is assumed that the internal parameters are known when the image is acquired.

[0142] In various embodiments of the present invention, it is contemplated to extract two or more images from a group of still images, and it is contemplated that the group of still images is acquired by one endoscope. However, the present invention is not limited thereto. For example, the present invention can also be applied even when a 3D model is restored using a plurality of groups of still images acquired by a plurality of endoscopes. In this case, as long as the images I1 and I2 are acquired using different endoscope devices, each endoscope can maintain independent internal parameters. Even if the internal parameters are unknown, the internal parameters can be set as variables to perform calculations. Therefore, the subsequent process will not change significantly depending on whether the internal parameters are known or not.

[0143] Refer to Figure 4 to describe the process of calculating the 3D coordinates of a subject based on two images and generating a 3D model. Figure 4 The figure shows the process of the process for generating a 3D model.

[0144] First, the 3D model generation unit 432 performs a feature point detection process (step S103a). In the feature point detection process, the 3D model generation unit 432 detects the feature points of each of the two images. Feature points refer to corners, edges, etc. in the subject information captured in the image where the image brightness gradient is large. As a method for detecting such feature points, SIFT (Scale-invariant Feature Transform) and FAST (Features from Accelerated Segment Test) are used. The 3D model generation unit 432 can detect the feature points in the image by using such a method.

[0145] Figure 3 The figure shows the detection of the feature point P from the image I1 11 and the detection of the feature point P from the image I2 12 as an example. In Figure 3 only one feature point of each image is shown, but actually multiple feature points can be detected in each image. The number of detected feature points may be different between images. Each feature point detected from each image is transformed into data of a feature quantity. The feature quantity is data representing the features of the feature point.

[0146] After step S103a, the 3D model generation unit 432 performs feature point correlation processing (step S103b). In the feature point correlation processing, the 3D model generation unit 432 compares the correlation of feature amounts between images for each feature point detected by the feature point detection processing (step S103a). When the correlation of feature amounts is compared and feature points with close feature amounts are found in each image, the 3D model generation unit 432 stores this information in the memory 47. Thus, the 3D model generation unit 432 correlates the feature points of each image with each other. On the other hand, when no feature points with close feature amounts are found, the 3D model generation unit 432 discards the information of the feature points.

[0147] After step S103b, the 3D model generation unit 432 reads out the coordinates of the mutually correlated feature points (feature point pairs) of two images from the memory 47. The 3D model generation unit 432 performs position and pose calculation processing based on the read coordinates (step S103c). In the position and pose calculation processing, the 3D model generation unit 432 calculates the relative position and relative pose between the imaging state c1 of the camera that acquired image I1 and the imaging state c2 of the camera that acquired image I2. More specifically, the 3D model generation unit 432 solves the following equation (1) using the epipolar constraint to calculate the matrix E.

[0148] [Equation 1]

[0149]

[0150] The matrix E is called an essential matrix. The essential matrix E is a matrix that preserves the relative position and relative pose between the imaging state c1 of the camera that acquired image I1 and the imaging state c2 of the camera that acquired image I2. In Equation (1), the matrix p1 is a matrix containing the coordinates of the feature points detected from image I1. The matrix p2 is a matrix containing the coordinates of the feature points detected from image I2. Since the essential matrix E contains information related to the relative position and relative pose of the cameras, it corresponds to the external parameters of the cameras. The 3D model generation unit 432 can solve the essential matrix E using a known algorithm.

[0151] As Figure 3 shown, when the position change amount (relative position) of the camera is t and the pose change amount (relative pose) of the camera is R, Equations (2) and (3) hold.

[0152] [Equation 2]

[0153] t = (t x , t y , t z )...(2)

[0154]

[0155] In Equation (2), the displacement in the x-axis direction is represented as t x , the displacement in the y-axis direction is represented as t y , and the displacement in the z-axis direction is represented as t z . In Equation (3), the rotation amount α about the x-axis is represented as R x (α), the rotation amount β about the y-axis is represented as R y (β), and the rotation amount γ about the z-axis is represented as R z (γ). After calculating the elementary matrix E, the optimization process of Bundle Adjustment can also be performed to improve the restoration accuracy of the 3D coordinates.

[0156] The 3D model generation unit 432 calculates the 3D coordinates (camera coordinates) in the coordinate system of the 3D model by using the calculated amount of change in the position of the camera. For example, the 3D model generation unit 432 defines the 3D coordinates of the camera that has acquired the image I1. The 3D model generation unit 432 calculates the 3D coordinates of the camera that has acquired the image I2 based on the 3D coordinates of the camera that has acquired the image I1 and the amount of change in the position of the camera that has acquired the image I2.

[0157] The 3D model generation unit 432 calculates the pose information in the coordinate system of the 3D model by using the calculated amount of change in the pose of the camera. For example, the 3D model generation unit 432 defines the pose information of the camera that has acquired the image I1. The 3D model generation unit 432 generates the pose information of the camera that has acquired the image I2 based on the pose information of the camera that has acquired the image I1 and the amount of change in the pose of the camera that has acquired the image I2.

[0158] The 3D model generation unit 432 generates data (3D shape data) of a three-dimensional shape (3D shape) that includes the 3D coordinates (camera coordinates) at the position of the camera and the pose information representing the pose of the camera by performing the calculation process of the position and the pose (step S103c). In addition, when methods such as Structure from Motion or visual-SLAM are applied to the calculation process of the position and the pose (step S103c), in step S103c, the 3D model generation unit 432 also calculates the 3D coordinates of each feature point. The 3D shape data generated in step S103c does not include the 3D coordinates of the points on the subject other than the feature points. Therefore, the 3D shape data shows a sparse 3D shape of the subject.

[0159] The 3D shape data includes the 3D coordinates of each feature point, the above-mentioned camera coordinates, and the above-mentioned pose information. The 3D coordinates of each feature point are defined in the coordinate system of the 3D model. The 3D coordinates of each feature point are associated with the two-dimensional coordinates (2D coordinates) of each feature point. The 2D coordinates of each feature point are defined in the coordinate system of the 2D image including each feature point. The 2D coordinates and 3D coordinates of each feature point are associated with the 2D image including each feature point.

[0160] After step S103c, the 3D model generation unit 432 performs three-dimensional shape restoration processing (step S103d) based on the relative position and relative pose (position change amount t and pose change amount R) of the camera calculated in step S103c. The 3D model generation unit 432 generates a 3D model of the subject in the three-dimensional shape restoration processing. As a method for restoring the three-dimensional shape of the subject, PMVS (Patch-based Multi-view Stereo) and matching processing based on parallel stereo can be cited. However, the means is not particularly limited.

[0161] In step S103d, the 3D model generation unit 432 calculates the 3D coordinates of the points other than the feature points on the subject. The 3D coordinates of each point other than the feature points are defined in the coordinate system of the 3D model. The 3D coordinates of each point are associated with the 2D coordinates of each point. The 2D coordinates of each point are defined in the coordinate system of the 2D image including each point. The 2D coordinates and 3D coordinates of each point are associated with the 2D image including each point. The 3D model generation unit 432 updates the 3D shape data. The updated 3D shape data includes the 3D coordinates of each feature point, the 3D coordinates of each point other than the feature points, the camera coordinates, and the pose information. The 3D shape data updated in step S103d includes the 3D coordinates of the points other than the feature points on the subject in addition to the 3D coordinates of the feature points. Therefore, the 3D shape data shows the dense 3D shape of the subject.

[0162] After step S103d, the 3D model generation unit 432 performs three-dimensional coordinate transformation processing (step S103e) based on the 3D shape data obtained by processing in the three-dimensional shape restoration processing (step S103d) and the reference length accepted by the condition acceptance unit 431. The 3D model generation unit 432 transforms the 3D shape data of the subject into three-dimensional coordinate data having a dimension of length in the three-dimensional coordinate transformation processing. When step S103e is executed, Figure 4 The processing shown ends.

[0163] In order to shorten the processing time, step S103d may be omitted. In this case, after step S103c is executed, step S103e is executed instead of step S103d.

[0164] Step S103e may also be omitted. In this case, after step S103d is executed, instead of executing step S103e, the process ends. Figure 4 The processing shown. In this case, the 3D model shows the relative shape of the subject without a dimension of length. Even when the 3D model shows the relative shape of the subject, the PC 41 can determine the region of the 3D model corresponding to the region of interest.

[0165] In order to Figure 3 generate a 3D model according to the principle shown, at least a part of the region of each 2D image needs to be common with at least a part of the region of each 2D image in at least one other 2D image. That is, the region of the first 2D image and the region of the second 2D image different from the first 2D image contain a common region. The region other than the common region in the first 2D image and the region other than the common region in the second 2D image are different from each other.

[0166] Refer to Figure 2 again to describe the process of the processing executed after generating the 3D model. After step S103, the storage control unit 436 stores the 3D model generated by the 3D model generation unit 432 in the memory 47 (step S104).

[0167] After step S104, the image selection unit 433 selects a reference frame including the region of interest from the group of still images. In the first embodiment, the time when the still image is recorded is associated with the moving image file recorded during the inspection. For example, this time is embedded in the header of the moving image file. The image selection unit 433 determines the time when the still image is recorded by reading the header of the moving image file. For example, when the frame rate of the moving image is 60 fps and a still image is recorded 20 seconds after the start of generating the moving image, the still image corresponding to the 1200th image of the moving image is recorded. The image selection unit 433 reads out the still image recorded at the determined time as the reference frame from the memory 47 (step S105).

[0168] In the first embodiment, the reference frame is the same as the still image specified by the user. The region of interest is the entire range captured in the reference frame.

[0169] The conditions for the reference frame are stored in the memory 47 in advance. In the above example, this condition indicates that the still image to be selected is associated with a specific time. In step S105, the image selection unit 433 selects the 2D image that satisfies this condition from among two or more 2D images included in the group of still images.

[0170] After step S105, the image selection unit 433 selects at least one key frame (selected frame) from among two or more key frames used to generate the 3D model in step S103. The selected frame is associated with the time closest in time to the time determined in step S105. The selected frame is different from the reference frame. The image selection unit 433 reads out the selected frame from the memory 47 (step S106).

[0171] The conditions for the selected frame are stored in the memory 47 in advance. In the above example, this condition means that the key frame to be selected is associated with the time closest in time to the time determined in step S105. In step S106, the image selection unit 433 selects the key frame that satisfies this condition from among two or more key frames.

[0172] In the above example, the image selection unit 433 selects the key frame closest in time to the reference frame as the selected frame. Thereby, the image selection unit 433 selects the key frame obtained with the composition most similar to the composition of the reference frame. Therefore, it is possible to expect that the same region of the subject is captured in the reference frame and the selected frame. However, each embodiment of the present invention is not limited to this process. In addition, the image selection unit 433 may select at least two key frames as the selected frame. For example, the image selection unit 433 may select the key frame closest in time to the reference frame and the key frame second closest to the reference frame.

[0173] At least a part of the region of the reference frame and at least a part of the region of the selected frame are common. That is, the region of the reference frame and the region of the selected frame include a common region. The region other than the common region in the reference frame and the region other than the common region in the selected frame are different from each other.

[0174] After step S106, the estimation unit 434 reads out the 3D model from the memory 47. The estimation unit 434 estimates the position and orientation of the camera that acquired the reference frame based on the reference frame, the selected frame, and the 3D model (step S107).

[0175] Refer to Figure 5 and Figure 6 to describe the process for estimating the position and orientation of the camera. Figure 5 Shows the process for estimating the position and orientation of the camera. Figure 6 Schematically shows the situation of acquiring the reference frame and the selected frame. The reference frame I is acquired in the imaging state C d of the camera, and the selected frame I, which is one of the key frames, is acquired in the imaging state C d of the camera KF and KF。In the first embodiment, the reference frame and the selected frame are included in the group of still images acquired by the endoscope. Therefore, the camera that acquires the reference frame is the same as the camera that acquires the selected frame. The camera that acquires the reference frame may also be different from the camera that acquires the selected frame.

[0176] The estimation unit 434 extracts a part of the three-dimensional point group (3D point group) that constitutes the 3D model. The region of the subject corresponding to the extracted 3D point group is within the imaging field of view of the camera that acquires the selected frame. Therefore, this region is captured in the selected frame. As described above, the 2D coordinates and 3D coordinates of the feature points are associated with the 2D image containing the feature points. The 3D model includes the 3D coordinates of the feature points included in each of two or more key frames. The estimation unit 434 extracts the 3D coordinates of the feature points associated with the selected frame. Thus, the estimation unit 434 extracts the 3D point group. For example, the estimation unit 434 extracts Figure 6 points P11, P12, P13, and P14 on the 3D model MD1 shown in (step S107a).

[0177] After step S107a, the estimation unit 434 re-projects the 3D point group extracted in step S107a onto the two-dimensional coordinate system of the selected frame, thereby obtaining the 2D coordinates of the 3D point group. The estimation unit 434 can re-project the 3D coordinates of the 3D point group onto the two-dimensional coordinate system of the selected frame by using a camera model held in advance. Generally, a pinhole camera model or the like is often used as the camera model. For example, the estimation unit 434 obtains Figure 6 the 2D coordinates of the points such as point P11 KF in the selected frame I KF , point P12 KF , point P13 KF , and point P14 KF . In Figure 6 , point P11 KF in the selected frame I KF corresponds to point P11 on the 3D model MD1, point P12 KF in the selected frame I KF corresponds to point P12 on the 3D model MD1, point P13 KF in the selected frame I KF corresponds to point P13 on the 3D model MD1, and point P14 KF in the selected frame I KF corresponds to point P14 on the 3D model MD1 (step S107b).

[0178] After step S107b, the estimation unit 434 searches for the 2D coordinates of the reference frame corresponding to the 2D coordinates of the selected frame obtained in step S107b. That is, the estimation unit 434 determines the same points in the reference frame as the points captured in the selected frame. For example, the estimation unit 434 determines Figure 6 the reference frame I shown d point P11 d point P12 d point P13 d and point P14 d . In Figure 6 the reference frame I d point P11 d corresponds to point P11 KF in the selected frame I KF , point P12 d in the reference frame I d corresponds to point P12 KF in the selected frame I KF , point P13 d in the reference frame I d corresponds to point P13 KF in the selected frame I KF , and point P14 d in the reference frame I d corresponds to point P14 KF in the selected frame I KF (step S107c).

[0179] In Figure 4 step S103a shown, the feature amounts at the feature points of each image are calculated. The 3D model may also include the feature amounts at the feature points of each 2D image in two or more 2D images included in the still image group. Also in step S107c, the estimation unit 434 may determine the points in the reference frame that are the same as the points in the selected frame based on the feature amounts of the points in the selected frame and the feature amounts of the points in the reference frame.

[0180] After step S107c, the estimation unit 434 estimates the position and orientation of the camera that captured the reference frame based on the 3D coordinates obtained in step S107a, the 2D coordinates obtained in step S107b, and the 2D coordinates obtained in step S107c. Specifically, the estimation unit 434 solves a perspective n point problem with the 3D coordinates on the 3D model, the 2D coordinates of the selected frame, and the 2D coordinates of the reference frame as inputs, thereby estimating the position and orientation of the camera. Thus, the estimation unit 434 estimates the relative position of the second camera with respect to the position of the first camera, and estimates the relative orientation of the second camera with respect to the orientation of the first camera. The first camera is the camera that captured the selected frame. The second camera is the camera that captured the reference frame. For example, the estimation unit 434 estimates Figure 6 the position t1 shown KF-d and the orientation R1 KF-d (step S107d).

[0181] The 3D model contains the camera coordinates and orientation information of the camera that captured two or more 2D images used to generate the 3D model. In the first embodiment, it is assumed that the 3D model generation unit 432 generates the 3D model without using the reference frame. In this case, the image selection unit 433 selects a 2D image different from all of the two or more key frames used to generate the 3D model as the reference frame. The 3D model does not contain the camera coordinates and orientation information of the camera that captured the reference frame. Therefore, in step S107d, the estimation unit 434 estimates the position and orientation of the camera that captured the reference frame.

[0182] After step S107d, the estimation unit 434 transforms the position and orientation of the camera obtained in step S107d into the position and orientation of the camera in the coordinate system of the 3D model (step S107e). When step S107e is executed, Figure 5 the processing shown ends.

[0183] Refer to Figure 7 to describe the details of step S107e. In step S103, the 3D model generation unit 432 estimates the position and orientation of the camera that captured Figure 7 the selected frame I in KF . That is, the 3D model generation unit 432 estimates the position t2 Figure 7 with respect to the origin O10 of the coordinate system of the 3D model MD10 in W-KF and the orientation R2 W-KF . The 3D data of the 3D model MD10 contains the camera coordinates of the camera that captured the selected frame I KF and the orientation information of this camera. That is, the 3D data contains the position t2W-KF 3D coordinates and pose R2 W-KF Pose information.

[0184] In step S107d, the estimation unit 434 estimates the relative position t1 d of the camera that has acquired the reference frame I KF-d and the relative pose R1 KF-d . In step S107e, the estimation unit 434 obtains the position t2 KF of the camera that has acquired the selected frame I W-KF (3D coordinates) and the pose R2 W-KF (pose information) of this camera. In step S107e, the estimation unit 434 estimates the acquisition of Figure 7 the position and pose of the camera of the reference frame I d in. Specifically, the estimation unit 434 is based on the position t2 W-KF and the position t1 KF-d to estimate the position t3 W-d with respect to the origin O10. W-KF In addition, the estimation unit 434 is based on the pose R2 KF-d and the pose R1 W-d to estimate the pose R3

[0185] Refer to Figure 2 again to describe the process of the processing performed after generating the 3D model. After step S107, the estimation unit 434 calculates the 3D coordinates of the region of the 3D model corresponding to the reference frame. Thereby, the estimation unit 434 estimates the position on the 3D model corresponding to the region of interest (step S108).

[0186] Describe the details of step S108. The estimation unit 434 projects the 2D coordinates of one or more points of the reference frame into the 3D model based on the position and pose of the camera estimated in step S107 and the conditions (intrinsic parameters and distortion correction parameters) received from the condition acceptance unit 431. Thereby, the estimation unit 434 determines one or more points on the 3D model. One or more points of the reference frame include points having the 2D coordinates determined in step S107c. That is, one or more points of the reference frame include one or more feature points. Or, one or more points of the reference frame include points having 2D coordinates other than the 2D coordinates determined in step S107c. That is, one or more points of the reference frame include one or more points other than feature points. One or more points of the reference frame may also include one or more feature points and one or more points other than feature points. The estimation unit 434 obtains the 3D coordinates of the determined points from the 3D model. The estimation unit 434 calculates the 3D coordinates of the subject that has entered the imaging field of view of the camera that has acquired the reference frame by performing the above processing.

[0187] After step S108, the estimation unit 434 associates the 3D coordinates calculated in step S108 with the 3D model. Specifically, the estimation unit 434 associates the 3D coordinates with the 3D model by executing annotation (step S109).

[0188] After step S109, the display control unit 435 causes the display unit 45 to display a 3D image of the 3D model. At this time, the display control unit 435 displays, on the 3D image, a region including the point having the 3D coordinates calculated in step S108. Thereby, the display control unit 435 displays, on the 3D image, a region corresponding to the region of interest of the 3D model (step S110). The user can confirm the position on the 3D model corresponding to the reference frame that is the region of interest. When step S110 is executed, Figure 2 the processing shown ends.

[0189] Alternatively, a device different from the PC 41 may execute steps S101 to S103, and the PC 41 may acquire the 3D model from this device. Therefore, steps S101 to S103 are not necessary.

[0190] Alternatively, the estimation unit 434 may not execute steps S108 and S109, and the display control unit 435 may display the camera coordinates of the camera that has acquired the reference frame on the 3D image of the 3D model. Since the camera coordinates are displayed on the 3D image, the user can confirm the position of the viewpoint of the reference frame. Therefore, steps S108 and S109 are not necessary.

[0191] The display control unit 435 may also display the 3D model and the region corresponding to the region of interest simultaneously in step S110. Alternatively, the display control unit 435 may display the 3D image of the 3D model before executing step S109, and display the region corresponding to the region of interest on the 3D image in step S110.

[0192] As described above, the 3D model includes the camera coordinates and pose information of the camera that has acquired two or more 2D images used to generate the 3D model. Therefore, the 3D model includes the camera coordinates of the first camera that has acquired the selected frame and the pose information of the first camera. The estimation unit 434 may also obtain, in step S107d, the camera coordinates of the first camera from the 3D model as an estimation result of the camera coordinates of the second camera that has acquired the reference frame. The estimation unit 434 may also obtain, in step S107d, the pose information of the first camera from the 3D model as an estimation result of the pose of the second camera. The composition of the selected frame is similar to the composition of the reference frame. Therefore, the estimation unit 434 may also use the camera coordinates of the first camera as approximate camera coordinates of the second camera. The estimation unit 434 may also use the pose information of the first camera as approximate pose information of the second camera.

[0193] In the above example, the image selection unit 433 selects a 2D image that is not included in two or more key frames used to generate the 3D model as the reference frame. The image selection unit 433 may also select one of the two or more key frames as the reference frame. In this case, the 3D model includes the camera coordinates of the second camera that has acquired the reference frame and the pose information of the second camera. The estimation unit 434 may also obtain the camera coordinates and the pose information from the 3D model.

[0194] An example of the image displayed on the display unit 45 will be described. Figures 8 to 13 An example of the display screen of the display unit 45 is shown. Hereinafter, for convenience, the following example is shown: after the 3D image of the 3D model is displayed on the display unit 45, the PC 41 selects one reference frame as the region of interest. There is no limitation on the timing when the PC 41 displays the 3D image and the timing when the PC 41 selects the reference frame. Therefore, either the display of the 3D image or the selection of the reference frame may be performed first.

[0195] Figures 8 to 13 The example shown shows the minimum structure in each embodiment of the present invention. Components other than Figures 8 to 13 the components of the user interface shown (buttons, check boxes, radio buttons, etc.) may also be arranged on the display screen. The same reference numerals are assigned to the common parts in Figures 8 to 13 .

[0196] When the CPU 43 starts the Figure 2 processing shown, the display control unit 435 causes the display unit 45 to display Figure 8 the dialog box A10 shown. In addition, the display control unit 435 displays a button B10 and a region R10 on the dialog box A10.

[0197] Button B10 is a button for inputting an instruction for generating a 3D model. The user can press the button on the display screen through the operation unit 44. When the display unit 45 is a touch panel, the user can press the button on the display screen by touching the display screen.

[0198] Region R10 is a region for displaying a 3D image of the 3D model. When the CPU 43 starts Figure 2 the processing shown, the 3D model has not been generated yet. At this time, no 3D image is displayed in region R10.

[0199] The user inputs an instruction for generating a 3D model by pressing button B10. At this time, the 3D model generation unit 432 starts the processing in step S103. The display control unit 435 can also cause the display unit 45 to display a button for displaying a 3D image of the generated 3D model.

[0200] After the 3D model is generated, the display control unit 435 causes a 3D image MI10 of the 3D model to be displayed in region R10 in step S110 ( Figure 9 ). At this time, the display control unit 435 displays a 3D image MI10 of the 3D model observed from a preset viewpoint in a preset direction. In addition, the display control unit 435 displays button B11 on dialog box A10. Button B11 is a button for inputting an instruction for reading in a reference frame as a region of interest.

[0201] After the 3D model is displayed, the user presses button B11. At this time, the display control unit 435 causes the display unit 45 to display a dialog box for selecting a still image recorded during the inspection. The user selects the still image for which they want to know the recorded position.

[0202] An example of selecting three still images (reference frames) will be described below. For example, identification information (file name) such as img01, img02, and img03 is assigned to the three still images. In the first embodiment, the region of interest is the entire region of the still image obtained during the inspection. Reading in the region of interest is not limited to reading in a still image. In the following example, the ranges captured in the three still images do not overlap with each other.

[0203] The image selection unit 433 reads out three still images from the memory 47 and executes step S106. The estimation unit 434 executes steps S107 to S109.

[0204] The display control unit 435 displays information indicating the positions on the 3D model corresponding to the three still images on the 3D image MI10 in step S110 ( Figure 10)。Specifically, the display control unit 435 displays the string CH11, the string CH12, and the string CH13 on the 3D image MI10, and also displays the line L11, the line L12, and the line L13 on the 3D image MI10.

[0205] Each string represents the file name of each still image among the three still images. The string CH11 represents the file name img01. The string CH12 represents the file name img02. The string CH13 represents the file name img03. Each line represents the position corresponding to each still image. The line L11 represents the position corresponding to the still image with the file name img01. The line L12 represents the position corresponding to the still image with the file name img02. The line L13 represents the position corresponding to the still image with the file name img03. Each position corresponds to the 3D coordinates calculated based on the camera coordinates and pose information of the camera that acquired the reference frame in step S108. By displaying the positions corresponding to the respective still images on the 3D model on the 3D image MI10, the user can confirm the positions corresponding to the regions of interest on the 3D model.

[0206] In addition, in step S110, the display control unit 435 causes the display unit 45 to display thumbnail images of each of the three still images. In Figure 10 the example shown, the display control unit 435 causes the region R20 within the dialog box A10 to display the thumbnail image IM11, the thumbnail image IM12, and the thumbnail image IM13.

[0207] Each thumbnail image corresponds to the still image (reference frame) that is the region of interest. The thumbnail image IM11 corresponds to the still image with the file name img01. The thumbnail image IM12 corresponds to the still image with the file name img02. The thumbnail image IM13 corresponds to the still image with the file name img03. The display control unit 435 does not need to cause the display unit 45 to display the images of the regions of interest (the thumbnail image IM11, the thumbnail image IM12, and the thumbnail image IM13).

[0208] In Figure 10 the example shown, the positions on the 3D image MI10 are associated with the file names of the still images. It is not necessary to display the file names. The positions on the 3D image MI10 can also be associated with identification information different from the file names. The positions on the 3D image MI10 can also be associated with text, colors, etc. different from the file names. For example, the display control unit 435 can also display the periphery of the thumbnail image IM11 corresponding to the file name img01 in a specified color, and display a marker in that color at the position on the 3D image MI10.

[0209] Figure 11 Shows another example of the display screen of the display unit 45. Omitting the content related toFigure 10 Description of the partially identical parts shown. The display control unit 435 displays the thumbnail images IM11, IM12, and IM13 together with the file names on the 3D image MI10 of the 3D model in step S110.

[0210] The display control unit 435 may also have a function of switching between displaying and not displaying the thumbnail images and file names. The display control unit 435 does not need to display the recorded still image itself as a thumbnail image. The display control unit 435 may also display a still image after artificial processing. For example, the display control unit 435 may perform a process of unifying the brightness of each image to the average brightness of a plurality of images among the images to be displayed. The display control unit 435 may also transform the still image into an image (free viewpoint image) obtained from a virtual viewpoint under the conditions of a specific object distance or a specific viewing angle. The display control unit 435 may perform this transformation within a range where the position of the recorded still image does not change significantly. Each embodiment of the present invention is not limited to the processing performed on the image when displaying the still image.

[0211] Figure 12 Another example of the display screen of the display unit 45 is shown. The description of the parts identical to those Figure 10 shown is omitted. The display control unit 435 displays, in step S110, information indicating the positions of the regions corresponding to the respective still images of the 3D model on the 3D image MI10. For example, the display control unit 435 displays the regions MR11, MR12, and MR13 on the 3D image MI10 of the 3D model. The region MR11 is the region of the 3D model corresponding to the still image having the file name img01. The region MR12 is the region of the 3D model corresponding to the still image having the file name img02. The region MR13 is the region of the 3D model corresponding to the still image having the file name img03. The regions MR11, MR12, and MR13 do not need to accurately represent the range of the subject captured in the still image.

[0212] The display control unit 435 may also perform a process (emphasis process) for improving visual recognition on the images of the respective regions in the regions MR11, MR12, and MR13. For example, the display control unit 435 may display the outlines of the respective regions with lines. The display control unit 435 may also display the respective regions with a specified color or a specified pattern. As long as the user can distinguish each region from other regions of the 3D model, the method of displaying each region is not limited to the above method.

[0213] The display control unit 435 displays the camera coordinates of the camera that captured each still image on the 3D image MI10 in step S110. For example, the display control unit 435 displaysFigure 12 The markers C11, C12, and C13 shown. Marker C11 represents the camera coordinates when a still image with the file name img01 is acquired. Marker C12 represents the camera coordinates when a still image with the file name img02 is acquired. Marker C13 represents the camera coordinates when a still image with the file name img03 is acquired.

[0214] The display control unit 435 may also perform an emphasis process on the images of the respective markers among the markers C11, C12, and C13. For example, the display control unit 435 may also display the outlines of the respective markers with lines. The display control unit 435 may also display the respective markers with a prescribed color or a prescribed pattern. As long as the user can recognize the respective markers, the method of displaying the respective markers is not limited to the above methods.

[0215] In step S110, the display control unit 435 displays the viewing angle information indicating the viewing angles of the respective still images on the 3D image MI10. For example, the display control unit 435 displays Figure 12 The lines L21, L22, and L23 shown. Line L21 represents the viewing angle of the still image with the file name img01. Line L22 represents the viewing angle of the still image with the file name img02. Line L23 represents the viewing angle of the still image with the file name img03. The viewing angle information represents the boundary between the area that enters the imaging field of view of the camera that acquired the still image and the area that does not enter the imaging field of view. The display control unit 435 may also calculate four lines representing the viewing angle based on four points corresponding to the four corners of the range captured in the still image on the 3D model and the camera coordinates of the camera that acquired the still image.

[0216] The display control unit 435 does not need to display all of the area of the 3D model corresponding to the still image, the camera coordinates, and the viewing angle information. The display control unit 435 may also display only one of the area of the 3D model and the camera coordinates without displaying the viewing angle information. The display control unit 435 may also display the area of the 3D model and the viewing angle information without displaying the camera coordinates. The display control unit 435 may also display the camera coordinates and the viewing angle information without displaying the area of the 3D model. The display control unit 435 may also display different types of information for each still image.

[0217] After displaying the 3D image MI10, the display control unit 435 may also change the viewpoint of the 3D image MI10. For example, the user operates the operation unit 44 to specify any one of the thumbnail images IM11, IM12, and IM13. Alternatively, the user operates the operation unit 44 to specify any one of the markers C11, C12, and C13. At this time, the display control unit 435 determines the still image corresponding to the thumbnail image or marker specified by the user.

[0218] The user can specify each thumbnail image or each marker through the operation unit 44. When the display unit 45 is a touch panel, the user can specify each thumbnail image or each marker by touching the display screen of the display unit 45. The display control unit 435 changes the viewpoint of the 3D image MI10 based on the camera coordinates of the camera that acquired the determined still image. In addition, the display control unit 435 changes the direction of the line of sight to a specified direction.

[0219] The display control unit 435 may also change the viewpoint of the 3D image MI10 to the point having the above-mentioned camera coordinates. The display control unit 435 may also change the viewpoint of the 3D image MI10 to the point having 3D coordinates close to the above-mentioned camera coordinates. The display control unit 435 may, after changing the viewpoint of the 3D image MI10, enlarge, reduce, or rotate the region of the 3D model corresponding to the still image. It is only necessary to change the viewpoint of the 3D image MI10 to the camera coordinates or 3D coordinates in the vicinity thereof, and the method of changing the viewpoint is not limited to the above method.

[0220] Refer to Figure 13 to illustrate an example of changing the viewpoint of the 3D image MI10. The display control unit 435 causes the thumbnail images IM11, IM12, and IM13 to be displayed in the region R20. For example, the user specifies the thumbnail image IM12. At this time, the display control unit 435 determines the still image corresponding to the thumbnail image IM12 and changes Figure 12 the viewpoint of the 3D image MI10 as shown. Thereby, the display control unit 435 displays Figure 13 the 3D image MI20 as shown. For example, the display control unit 435 changes the viewpoint in such a way that the point having the camera coordinates and at least a part of the region entering the imaging field of view of the camera can be displayed. The display control unit 435 may also perform processes such as enlargement, reduction, or rotation to make the region entering the imaging field of view of the camera easier to observe.

[0221] The display control unit 435 displays Figure 13The marker C21, the region MR21, and the line L31 shown. The marker C21 represents the camera coordinates when a still image corresponding to the thumbnail image IM12 is acquired. The region MR21 represents the region of the 3D model corresponding to the still image. The line L31 represents the viewing angle of the still image.

[0222] Refer to Figure 14 and Figure 15 to describe the structure of the endoscope device for acquiring a group of still images. Figure 14 Shows the appearance of the endoscope device 1. Figure 15 Shows the internal structure of the endoscope device 1. The endoscope device 1 photographs a subject to generate an image. The subject is an industrial product. The examiner can replace the optical adapter mounted at the front end of the insertion section 2, select the built-in image processing program, and add an image processing program to observe various subjects.

[0223] Figure 14 The endoscope device 1 shown has an insertion section 2, a main body section 3, an operation section 4, and a display section 5.

[0224] The insertion section 2 can be inserted into the interior of the subject. The insertion section 2 is an elongated tube that can be bent from the front end 20 to the base end. The insertion section 2 photographs the subject and outputs a video signal to the main body section 3. An optical adapter is mounted at the front end 20 of the insertion section 2. For example, a monocular optical adapter is mounted at the front end 20 of the insertion section 2. The main body section 3 is a control device having a storage section for storing the insertion section 2. The operation section 4 accepts operations on the endoscope device 1 by the user. The display section 5 has a display screen and displays an image of the subject acquired by the insertion section 2, an operation menu, etc. on the display screen.

[0225] The operation section 4 is a user interface. The display section 5 is a monitor (display) such as an LCD. The display section 5 can also be a touch panel. In this case, the operation section 4 and the display section 5 are integrated.

[0226] Figure 15The main body 3 shown has an endoscope unit 8, a CCU (Camera Control Unit) 9, and a control device 10. The endoscope unit 8 has a light source device (not shown) and a bending device. The light source device supplies illumination light required for observation. The bending device bends a bending mechanism built in the insertion portion 2. A imaging element 28 is built in the front end 20 of the insertion portion 2. The imaging element 28 is an image sensor. The imaging element 28 performs photoelectric conversion on the optical image of the subject formed by the optical adapter, and generates a imaging signal. The CCU 9 drives the imaging element 28. The imaging signal output from the imaging element 28 is input to the CCU 9. The CCU 9 performs preprocessing on the imaging signal obtained by the imaging element 28, including amplification and noise reduction. The CCU 9 converts the preprocessed imaging signal into a video signal such as an NTSC signal.

[0227] The control device 10 has an image signal processing circuit 12, a ROM (Read Only Memory) 13, a RAM (Random Access Memory) 14, a card interface 15, an external device interface 16, a control interface 17, and a CPU (Central Processing Unit) 18.

[0228] The image signal processing circuit 12 performs prescribed image processing on the video signal output from the CCU 9. For example, the image signal processing circuit 12 performs image processing related to improving visual recognition. For example, this image processing is color restoration, gray scale correction, noise suppression, and contour enhancement, etc. For example, the image signal processing circuit 12 synthesizes the video signal output from the CCU 9 with a graphic image signal generated by the CPU 18. The graphic image signal includes an image of an operation screen, etc. The image signal processing circuit 12 outputs the synthesized video signal to the display unit 5.

[0229] The ROM 13 is a non-volatile storage medium that records a program for controlling the operation of the endoscope device 1 by the CPU 18. The RAM 14 is a volatile storage medium that temporarily stores information used by the CPU 18 to control the endoscope device 1. The CPU 18 controls the operation of the endoscope device 1 based on the program recorded in the ROM 13.

[0230] A memory card 42, which is a removable storage medium, is connected to the card interface 15. The card interface 15 inputs control processing information, image information, etc. stored in the memory card 42 into the control device 10. In addition, the card interface 15 records control processing information, image information, etc. generated by the endoscope device 1 into the memory card 42.

[0231] An external device such as a USB device is connected to the external device interface 16. For example, Figure 1 the illustrated PC 41 is connected to the external device interface 16. The external device interface 16 sends information to the PC 41 and receives information from the PC 41. Thereby, the display unit 45 of the PC 41 can display information. In addition, the user can perform an operation related to the control of the endoscope device 1 by inputting an instruction to the PC 41.

[0232] The control interface 17 communicates with the operation unit 4, the endoscope unit 8, and the CCU 9 for operation control. The control interface 17 notifies the CPU 18 of the instruction input by the user to the operation unit 4. The control interface 17 outputs a control signal for controlling the light source device and the bending device to the endoscope unit 8. The control interface 17 outputs a control signal for controlling the imaging element 28 to the CCU 9.

[0233] The imaging element 28 is a camera that acquires a group of still images. The group of still images includes two or more 2D images. Each 2D image among the two or more 2D images is temporally associated with other 2D images included in the two or more 2D images. For example, when the group of still images is a moving image, each frame among two or more frames included in the moving image is associated with each other by a time stamp (time code).

[0234] For example, the imaging element 28 is a monocular camera having one viewpoint. In this case, each 2D image among the two or more 2D images is an image acquired by the monocular camera.

[0235] The imaging element 28 may also have two or more viewpoints. For example, the imaging element 28 may acquire a stereoscopic image including an image of a subject observed from a first viewpoint and an image of the subject observed from a second viewpoint different from the first viewpoint. Alternatively, the endoscope device 1 projects pattern light having a bright part and a dark part onto the subject, and the imaging element 28 acquires two or more images. The PC 41 may also generate a 3D model based on the stereoscopic image or the image of the subject onto which the pattern light is projected.

[0236] The image display method according to each aspect of the present invention includes a storage step, a first selection step, a second selection step, an estimation step, and a display step. In the storage step (step S104), the storage control unit 436 stores a 3D model (3D data) generated based on two or more 2D images of a subject in the memory 47 (storage medium). The 3D model includes 3D coordinates of two or more points of the subject and first camera coordinates. The first camera coordinates are 3D coordinates of the first camera that has acquired each of the two or more 2D images, and are associated with each of the two or more 2D images. In the first selection step (step S105), the image selection unit 433 selects a reference frame (reference image) as a 2D image of the subject. In the second selection step (step S106), the image selection unit 433 selects at least one 2D image from the two or more 2D images as a selection frame (selected image) based on the reference frame. In the estimation step (step S107), the estimation unit 434 estimates the 3D coordinates of the second camera, i.e., the second camera coordinates, which has acquired the reference frame, based on the first camera coordinates associated with the selection frame. In the display step (step S110), the display control unit 435 causes the display unit 45 (display) to display a 3D image MI10 in which at least one of the second camera coordinates and the 3D coordinates of one or more points of the subject calculated based on the second camera coordinates is visualized, and the 3D coordinates of one or more points of two or more points of the subject are visualized.

[0237] For example, the display control unit 435 displays a position corresponding to at least one of the second camera coordinates and the 3D coordinates calculated based on the second camera coordinates on the 3D image MI10. In Figure 12 the example shown, the display control unit 435 displays a marker C11 or the like indicating the second camera coordinates. In Figure 10 and Figure 12 the example shown, the display control unit 435 displays a string CH11, a line L11, or the like indicating a position on the 3D model corresponding to the still image on the 3D image MI10. The string CH11, the line L11, or the like shows the 3D coordinates calculated based on the second camera coordinates. In Figure 12 the example shown, the display control unit 435 displays a region MR11 or the like on the 3D image MI10 corresponding to the still image of the 3D model. The region MR11 or the like shows the 3D coordinates calculated based on the second camera coordinates.

[0238] Each aspect of the present invention may also include the following modification examples. The 3D model also includes first pose information representing the pose of the first camera. The first pose information is associated with each of two or more 2D images. In the estimation step (step S107), the estimation unit 434 also estimates the pose of the second camera based on the first pose information associated with the selected frame, and generates second pose information representing the estimated pose. In the estimation step (step S108), the estimation unit 434 also estimates the 3D coordinates of one or more points in the region of the subject captured in the reference frame based on the second camera coordinates and the second pose information. The 3D coordinates of one or more points in this region are visualized in the 3D image MI10.

[0239] For example, the estimation unit 434 calculates the 3D coordinates of the entire region captured in the reference frame. The estimation unit 434 may also calculate the 3D coordinates of a part of the region captured in the reference frame. When generating the 3D model, the 3D coordinates of two or more points constituting the 3D model are calculated. The estimation unit 434 calculates the 3D coordinates of points different from these two or more points.

[0240] For example, the display control unit 435 displays on the 3D image MI10 the positions corresponding to the 3D coordinates of one or more points in the region captured in the reference frame. In Figure 10 and Figure 12 In the example shown, the display control unit 435 displays on the 3D image MI10 a string CH11 or the like and a line L11 or the like indicating the position corresponding to the still image on the 3D model. In Figure 12 In the example shown, the display control unit 435 displays on the 3D image MI10 a region MR11 or the like corresponding to the still image of the 3D model. Figure 10 and Figure 12 In the 3D image MI10 shown, the 3D coordinates of one or more points in the region captured in the reference frame and the 3D coordinates of two or more points of the subject are visualized.

[0241] Each aspect of the present invention may also include the following modification examples. In the display step (step S110), the display control unit 435 also superimposes information indicating the position of the region captured in the reference frame on the 3D image MI10.

[0242] Each aspect of the present invention may also include the following modification examples. After the 3D image is displayed on the display unit 45, the display control unit 435 changes the viewpoint of the 3D image MI10 in the viewpoint change step (step S110).

[0243] Each aspect of the present invention may also include the following modification examples. In the display step (step S110), the display control unit 435 causes the display unit 45 to display viewpoint information indicating the viewpoint of the reference frame.

[0244] Each aspect of the present invention may also include the following modification examples. The 3D model also includes the 2D coordinates of the first points included in each of the two or more 2D images. In the estimation step (step S107), the estimation unit 434 estimates the second camera coordinates based on the first camera coordinates, the 2D coordinates of the first points ( Figure 6 such as point P11 KF etc.) included in one of the two or more 2D images, and the 2D coordinates of the second points ( Figure 6 such as point P11 d etc.) of the reference frame corresponding to the first points.

[0245] Each aspect of the present invention may also include the following modification examples. In the first selection step (step S105), the image selection unit 433 selects one of the one or more selected object images as the reference frame. The one or more selected object images are 2D images of the subject and are not included in the two or more 2D images.

[0246] Each aspect of the present invention may also include the following modification examples. In the second selection step (step S106), the image selection unit 433 uses the time information that correlates the two or more 2D images with each other in time to select at least one of the two or more 2D images as the selected frame.

[0247] In the first embodiment, the PC 41 displays a 3D image of the subject on the display unit 45. In this 3D image, at least one of the camera coordinates of the camera that acquired the reference frame and the 3D coordinates calculated based on the camera coordinates is visualized, and the 3D coordinates of one or more of the two or more points of the subject are visualized. Thus, the PC 41 can visualize the position having 3D coordinates associated with at least a part of the 2D image (reference frame) of the subject.

[0248] The user can confirm the position corresponding to the region of interest on the 3D model. Since this position is displayed on the 3D image, the user can confirm the relative position of the region of interest in the whole of the object to be inspected. Therefore, the PC 41 can provide useful information for repair or the next inspection to the user. The user does not need to copy the insertion length into a memo during the inspection.

[0249] The PC 41 performs highlighted display on the 3D image to improve the visual recognition of the region of interest. Alternatively, the PC 41 switches the viewpoint for displaying the 3D image. Thus, the user can confirm in detail the shape or inclination (direction), etc. of the region corresponding to the region of interest on the 3D model.

[0250] (First modification example of the first embodiment)

[0251] Describe a first modification of the first embodiment of the present invention. Figure 1 The PC 41 shown is changed to Figure 16 the PC 41a shown. Figure 16 Show the structure of PC 41a. PC 41a shows the position on the 3D model corresponding to the region of interest, and shows the three-dimensional distance (3D distance) between this position and the reference point on the 3D model. Omit the description of the structure that is the same as the Figure 1 structure shown.

[0252] Figure 1 The CPU 43 shown is changed to CPU 43a. The functions of CPU 43a include an image acquisition unit 430, a condition acceptance unit 431, a 3D model generation unit 432, an image selection unit 433, an estimation unit 434, a display control unit 435, a storage control unit 436, and a reference position acceptance unit 437. Figure 16 At least one module within the CPU 43a shown may also be constituted by a circuit different from the CPU 43a.

[0253] Each part within CPU 43a may also be constituted by at least one of a processor and a logic circuit. Each part within CPU 43a can include one or more processors. Each part within CPU 43a can include one or more logic circuits.

[0254] The user inputs the reference position on the 3D model into the PC 41a by operating the operation unit 44. The reference position is not limited to one point on the 3D model. The reference position may also be a region including two or more points on the 3D model. The reference position acceptance unit 437 accepts the reference position based on the information input by the user into the PC 41a. The storage control unit 436 stores the reference position accepted by the reference position acceptance unit 437 in the memory 47. The reference position acceptance unit 437 determines the reference point on the 3D model corresponding to the reference position. The 3D model includes the 3D coordinates of two or more points including the reference point. The reference position acceptance unit 437 calculates the 3D distance between the reference point and the region on the 3D model corresponding to the region of interest. The display control unit 435 displays the 3D distance calculated by the reference position acceptance unit 437 on the 3D image of the 3D model.

[0255] Figure 17 Show the process of the processing executed by the PC 41a. Omit the description of the processing that is the same as the Figure 2 processing shown.

[0256] The user inputs the reference position on the 3D model into the PC 41a by operating the operation unit 44. For example, the user designates the access port, which is the entrance for inserting the endoscope, as the reference position. Alternatively, the user designates a bent portion such as an elbow as the reference position. After step S110, the reference position reception unit 437 receives the reference position based on the information input by the user into the PC 41a (step S121).

[0257] After step S121, the reference position reception unit 437 determines the reference point corresponding to the reference position on the 3D model. For example, in the case where a specific area on the 3D model is designated as the reference position, the reference position reception unit 437 determines one point included in the area as the reference point. The reference position reception unit 437 calculates the 3D distance between the reference point and the area corresponding to the same area of concern of the 3D model (step S122).

[0258] Details of step S122 will be described. For example, the reference position reception unit 437 selects a representative point corresponding to the area of concern on the 3D model and calculates the 3D distance between the reference point and the representative point. In this case, the reference position reception unit 437 calculates the 3D distance between the reference point and one point among two or more points within the area of the 3D model. The reference position reception unit 437 may also calculate the 3D distance between the reference point and each of two or more points corresponding to the area of concern on the 3D model, and calculate the average of the 3D distances for all of the two or more points. In this case, the reference position reception unit 437 calculates the 3D distance between the reference point and an approximate point within the area of the 3D model. The approximate point coincides with any one of the two or more points on the 3D model or is located near any one of the two or more points.

[0259] After step S122, the display control unit 435 displays the 3D distance calculated in step S122 on the 3D image of the 3D model (step S123). When step S123 is executed, Figure 17 the processing shown ends.

[0260] In the above example, the user designates the reference position. The PC 41a may also automatically set the reference position in the 3D model. For example, the information of the reference position designated in advance by the user is stored in the memory 47. This information may also represent the reference position previously input by the user into the PC 41a. The PC 41a sets the reference position in the 3D model based on this information.

[0261] Figure 18 An example of the display screen of the display unit 45 is shown. The description of the part identical to the part Figure 10 shown is omitted.

[0262] In step S123, the display control unit 435 displays the reference position received by the reference position reception unit 437 and the 3D distance calculated by the reference position reception unit 437 on the 3D image MI10. In Figure 18 In the example shown, the display control unit 435 displays the reference position RF1 on the 3D image MI10, and also displays strings D11, D12, and D13 representing the 3D distances on the 3D image MI10. String D11 represents the 3D distance between the reference point corresponding to the reference position RF1 and the region corresponding to the still image (file name img01). String D12 represents the 3D distance between the reference point corresponding to the reference position RF1 and the region corresponding to the still image (file name img02). String D13 represents the 3D distance between the reference point corresponding to the reference position RF1 and the region corresponding to the still image (file name img03).

[0263] In Figure 18 In the example shown, the 3D image MI10 and the 3D distances are displayed in the region R10. The 3D distances may also be displayed in a region different from the region R10 where the 3D image MI10 is displayed.

[0264] In the above example, a reference length is required to convert the length on the 3D model into an actual length. The reference length is not essential. The reference position reception unit 437 may also calculate a 3D distance without an actual length scale. For example, the reference position reception unit 437 calculates a first 3D distance between a first reference point and a first point on the 3D model. In addition, the reference position reception unit 437 calculates a second 3D distance between a second reference point and a second point on the 3D model. The reference position reception unit 437 calculates the ratio of the first 3D distance to the second 3D distance. In this case, the 3D distance does not need to have a length scale. Therefore, the reference length is not required.

[0265] In the above example, the reference length is set based on an instruction from the user. It is not necessary for the user to specify the reference length. For example, in the case where the part where the reference position is set has a characteristic shape, pattern, or color, etc., the information of this part may be stored in the memory 47 in advance. The reference position reception unit 437 may also process the 2D image of the subject or the 3D model of the subject by using this information and detect the reference position. At this time, the reference position reception unit 437 can use known techniques.

[0266] Each aspect of the present invention may also include the following modification examples. In the display step (step S123), the display control unit 435 causes the display unit 45 to display the 3D distance between the reference point included in two or more points of the subject and the region of the subject captured in the reference frame (reference image).

[0267] In the first modification of the first embodiment, the PC 41a displays the 3D distance between the reference point and the area corresponding to the area of interest of the 3D model. The user can confirm how far the position on the 3D model is from the reference point. For example, the first modification of the first embodiment can be applied to pipe inspection.

[0268] (Second Modification of the First Embodiment)

[0269] The second modification of the first embodiment of the present invention will be described. In the second modification of the first embodiment, the PC 41 shown in Figure 1 is used. For example, the second modification of the first embodiment can be applied to gas turbine inspection. The gas turbine has two or more blades arranged periodically. The two or more blades are arranged along the circumferential direction of the gas turbine. The gas turbine rotates along the circumferential direction during inspection.

[0270] During inspection, a reference blade and a blade of interest are specified. The blade of interest is included in the still image specified by the user. The reference frame is the same as the still image specified by the user. Therefore, the blade of interest is included in the reference frame. The PC 41, or Figure 14 and Figure 15 the endoscope device 1 shown in detects the number of blades from the reference blade to the blade of interest. The PC 41 displays this number.

[0271] Figure 19 An example of a 2D image of the gas turbine as the subject in the second modification of the first embodiment is shown. In the Figure 19 image GT10 shown, two or more blades BD10 are captured. For example, the two or more blades BD10 are manufactured so as to have the same shape and the same size. The two or more blades BD10 are arranged at equal intervals.

[0272] Next, two methods for detecting the number of blades will be described. First, the first method in which the endoscope device 1 detects the number of blades will be described. For example, the endoscope device 1 has a projection optical system that projects laser light and a detection optical system that detects the laser light. For example, the projection optical system includes a laser pointer. The detection optical system has a laser detector. The laser detector may be the imaging element 28.

[0273] The projection optical system projects laser light onto the gas turbine. When a blade enters the imaging field of view of the imaging element 28, the laser light is reflected by the blade and enters the detection optical system. Therefore, the laser detector detects the laser light. When a blade does not enter the imaging field of view of the imaging element 28, the laser light is not reflected by the blade and thus does not enter the detection optical system. Therefore, the laser detector does not detect the laser light. The laser detector outputs a signal representing the detection result of the laser light to the CPU 18.

[0274] While the gas turbine is rotating, the CPU 18 detects a first period during which the laser detector detects the laser and a second period during which the laser detector does not detect the laser. The first period and the second period appear alternately. When the CPU 18 detects a group of one first period and one second period, the CPU 18 determines that one blade has been detected. The CPU 18 detects the number of blades by counting the number of groups of the two types of periods.

[0275] For example, the user designates a blade with characteristic damage or the like as a reference blade. The CPU 18 can also select a reference blade that meets a specified standard. After the imaging element 28 starts acquiring a group of still images (moving images), the CPU 18 determines whether the reference blade is detected in the images acquired by the imaging element 28. When the reference blade is detected, the CPU 18 starts detecting the number of blades. Each time one blade is detected, the CPU 18 increments the number of blades by 1. When the reference blade is detected next time, the CPU 18 ends the counting of the blades and starts detecting the number of blades again. The CPU 18 associates the information indicating the number of blades with each image acquired by the imaging element 28. For example, the CPU 18 records this information in the header of the moving image file.

[0276] The display control unit 435 obtains, in Figure 2 or Figure 17 the step S110 shown, the information associated with the reference frame selected by the image selection unit 433. This information indicates the number of blades from the reference blade to the target blade captured in this reference frame.

[0277] Next, a second method for the PC 41 to detect the number of blades will be described. When a reference blade is specified in the 3D model of the gas turbine, the display control unit 435 detects each blade arranged along the outer circumference of the gas turbine in a specified direction. The specified direction is the same as the rotation direction of the gas turbine or the direction opposite to this rotation direction. The display control unit 435 can also detect the blades in the 3D model by combining the 3D-CAD (Computer Aided Design) data of the gas turbine with the 3D model. The display control unit 435 detects, in Figure 2 or Figure 17 the step S110 shown, the number of blades from the reference blade to the target blade.

[0278] The display control unit 435 can also detect a first number of blades from the reference blade in a specified direction and a second number of blades from the reference blade in the direction opposite to this specified direction. The display control unit 435 can also select the smaller of the first number and the second number as the final number of blades.

[0279] The display control unit 435 causes the display unit 45 to display the number of blades detected by using the first method or the second method. For example, in step S110 shown in Figure 2 or Figure 17 the number is displayed on the 3D image of the 3D model.

[0280] Each aspect of the present invention may also include the following modification examples. The subject includes two or more local regions arranged periodically. In the display step (step S110), the display control unit 435 causes the display unit 45 to display the number of at least a part of the local regions from the reference part to the part of interest. The reference part is one of the two or more local regions. The part of interest is a local region different from the reference part and corresponding to the region of the subject captured in the reference frame (reference image).

[0281] In the above example, the subject is a gas turbine and includes two or more blades. The reference part is one of the two or more blades. In the above example, the reference part is the reference blade. In the above example, the part of interest is the blade of interest captured in the reference frame.

[0282] An example of the range of the blades for detecting the number is described. For example, the range includes the reference blade and the blade of interest, and includes all of one or more blades arranged between the reference blade and the blade of interest. The range may also include either the reference blade or the blade of interest, and includes all of one or more blades arranged between the reference blade and the blade of interest. Therefore, the range does not need to include both the reference blade and the blade of interest. The range may also not include the reference blade and the blade of interest, and includes all of one or more blades arranged between the reference blade and the blade of interest. When the blade of interest is the same as the reference blade, the range only includes the reference blade.

[0283] In the second modification example of the first embodiment, the PC 41 displays the number of blades from the reference blade to the blade of interest. The blade of interest is included in the region of interest. The user can confirm how far the blade of interest is from the reference blade.

[0284] (Third Modification Example of the First Embodiment)

[0285] The third modification example of the first embodiment of the present invention is described. In the third modification example of the first embodiment, Figure 1The PC 41 shown above. In the aforementioned first embodiment, devices such as the endoscope device 1 acquire a group of still images for generating a 3D model of the subject and a still image capturing the region of interest during one examination. Hereinafter, the still image capturing the region of interest will be referred to as the image of interest. In the third modification of the first embodiment, devices such as the endoscope device 1 acquire the group of still images and the image of interest during different examinations.

[0286] Refer to Figure 2 to describe the processing performed by the PC 41. The description of the processing identical to that in the first embodiment will be omitted. Figure 2 The processing in step S105 shown above is changed.

[0287] In the first embodiment, the time when the image of interest is recorded is embedded in the header of the moving image file. Therefore, in the first embodiment, the image selection unit 433 can determine the time when the image of interest is recorded by reading the header of the moving image file.

[0288] In the third modification of the first embodiment, the time of the examination for acquiring the group of still images (moving image file) is different from the time of the examination for acquiring the image of interest. Therefore, the time when the image of interest is recorded is not embedded in the header of the moving image file. The image selection unit 433 selects, in step S105, an image acquired with a composition similar to that of the image of interest from the moving image file. Thus, the image selection unit 433 selects a reference frame similar to the image of interest.

[0289] The image selection unit 433 may also use a well-known technique called "similar image search". Thus, the image selection unit 433 can search for an image (frame) acquired with a composition similar to that of a specific still image in the moving image file. As a representative example of similar image search, there are techniques such as Bag of Visual Words.

[0290] Even when the group of still images and the image of interest are acquired during different examinations, the PC 41 can select a reference frame corresponding to the image of interest. Therefore, the PC 41 can visualize the position having 3D coordinates associated with at least a part of the 2D image (reference frame) of the subject.

[0291] (Fourth modification of the first embodiment)

[0292] Describe the fourth modification of the first embodiment of the present invention. In the fourth modification of the first embodiment, the Figure 1 PC 41 shown above is used.

[0293] After the 3D model is displayed, the PC 41 accepts a specific area of the still image from the user as a new area of interest. The PC 41 displays, on the 3D image of the 3D model, the area of the 3D model corresponding to the area of interest. The area of interest is a part of the range captured in the reference frame.

[0294] Figure 20 Shows the process of the processing performed by the PC 41. Explanation of the same processing as Figure 2 the processing shown is omitted.

[0295] After the 3D image of the 3D model is displayed on the display unit 45 in step S110, the user operates the operation unit 44 to input information indicating the position of the area of interest in a specific still image to the PC 41. This still image is the reference frame selected in step S105. When two or more reference frames are selected in step S105, the user inputs information indicating the position of the area of interest in one of the reference frames to the PC 41. The estimation unit 434 accepts the area of interest based on the information input by the user to the PC 41. The estimation unit 434 may also accept two or more areas of interest (step S131).

[0296] After step S131, the estimation unit 434 calculates the 3D coordinates of the area of the 3D model corresponding to the area of interest. Thereby, the estimation unit 434 estimates the position on the 3D model corresponding to the area of interest (step S132).

[0297] Details of step S132 are described. The estimation unit 434 can execute step S132 by using a known technique. For example, the estimation unit 434 calculates the 3D coordinates of the area of the 3D model corresponding to the area of interest by performing the same processing as step S103. Since the area of interest is an area of the reference frame, the estimation unit 434 may also calculate the 3D coordinates of the area of the 3D model corresponding to the area of interest by using the result of step S108.

[0298] The area of interest may also be an area of a key frame used to generate the 3D model. When step S103 is executed, the 2D coordinates of each point of the key frame are associated with the 3D coordinates of each point of the 3D model. Therefore, the estimation unit 434 may also calculate the 3D coordinates of the area of the 3D model corresponding to the area of interest based on the relationship between the 2D coordinates and the 3D coordinates.

[0299] After step S132, the display control unit 435 causes the display unit 45 to redisplay the 3D image of the 3D model including the position estimated in step S132. At this time, the display control unit 435 may also change the viewpoint of the 3D image so as to facilitate the observation of the position estimated in step S132. The display control unit 435 may also enlarge, reduce, or rotate the region including the position estimated in step S132 (step S133). The user can confirm the position on the 3D model corresponding to the region of interest. When step S133 is executed, Figure 20 The processing shown ends.

[0300] In addition to Figure 20 the processing shown, Figure 17 steps S121 to S123 shown may also be executed. For example, steps S121 to S123 may be executed after step S133. Steps S121 to S123 may also be executed before executing step S131.

[0301] The display control unit 435 may also display the number of blades of the gas turbine by using the method shown in the second modification of the first embodiment in step S110 or step S133.

[0302] Figure 21 An example of the display screen of the display unit 45 is shown. The description of the part identical to the Figure 13 part shown is omitted.

[0303] The display control unit 435 causes the thumbnail image IM11, the thumbnail image IM12, and the thumbnail image IM13 to be displayed in the region R20. For example, the user designates a part of the thumbnail image IM12. At this time, the estimation unit 434 accepts a part of the thumbnail image IM12 as the region of interest MR31 in step S131. The display control unit 435 may also display the region of interest MR31 on the thumbnail image IM12. The estimation unit 434 estimates the position on the 3D model corresponding to the region of interest MR31 in step S132. The display control unit 435 displays, on the 3D image MI20 of the 3D model in step S133, information indicating the position of the region on the 3D model corresponding to the region of interest MR31. For example, the display control unit 435 displays the region MR32 on the 3D image MI20.

[0304] The display control unit 435 may also perform a process (emphasis process) for improving visual recognition on the image of the region MR32. For example, the display control unit 435 may display the contour of the region MR32 with a line. The display control unit 435 may also display the region MR32 with a prescribed color or a prescribed pattern. As long as the user can distinguish the region MR32 from other regions of the 3D model, the method of displaying the region MR32 is not limited to the above method.

[0305] In the fourth modification of the first embodiment, the PC 41 displays, on the 3D image of the 3D model, the area of the 3D model corresponding to the specific area of the still image. The user can confirm the position on the 3D model corresponding to this specific area.

[0306] The user can specify, as the area of interest, an abnormal part or the like captured in the still image. The user can confirm in detail the position and inclination (direction) of the area of the 3D model corresponding to the area of interest.

[0307] (Fifth modification of the first embodiment)

[0308] The fifth modification of the first embodiment of the present invention will be described. In the fifth modification of the first embodiment, the PC 41 shown in Figure 1 is used. Two or more reference frames are used. The ranges captured in the two or more reference frames overlap each other. The PC 41 displays the area that overlaps between the two or more reference frames.

[0309] The image selection unit 433 selects two or more reference frames from the group of still images. The image selection unit 433 selects selection frames from the group of still images based on each reference frame. Therefore, the image selection unit 433 selects two or more selection frames.

[0310] The estimation unit 434 estimates the camera coordinates of the camera that captured each reference frame. In addition, the estimation unit 434 estimates the pose of the camera that captured each reference frame, and generates pose information representing the estimated pose. Further, the estimation unit 434 estimates the 3D coordinates of the area of the subject captured in each reference frame.

[0311] The display control unit 435 causes the display unit 45 to display the 3D image of the 3D model. The display control unit 435 displays, on the 3D image, the position corresponding to at least one of the camera coordinates of the camera that captured each reference frame and the area of the 3D model corresponding to the area of interest. The display control unit 435 causes the display unit 45 to display information indicating the position of the area common (repeated area) between the two or more reference frames. The display control unit 435 causes the display unit 45 to display the area where the shape of the subject in the repeated area changes significantly.

[0312] Below, an example is described in which the user designates two still images (a first still image and a second still image) as regions of interest. The ranges captured in the two still images overlap each other. The first still image is a still image obtained in a past examination (for example, an examination conducted one year ago) and is not included in the group of still images to be processed. The second still image is a still image obtained in the latest examination and is included in the group of still images to be processed. Even when three or more still images become regions of interest, the PC 41 can use the same method as the method described below.

[0313] Figure 22 Shows the process of the processing executed by the PC 41. Explanation of the same processing as Figure 2 The processing shown is omitted.

[0314] Figure 2 The step S105 shown is changed to step S105a. The user inputs information indicating two still images to the PC 41 by operating the operation unit 44. The image selection unit 433 accepts two still images based on the information input by the user to the PC 41. The image selection unit 433 selects two reference frames (a first reference frame and a second reference frame) from the group of still images based on the two still images. The image selection unit 433 reads out two reference frames from the memory 47 (step S105a).

[0315] Details of step S105a are described. The image selection unit 433 selects a first reference frame based on the first still image. The method of selecting the first reference frame is the same as the method in the third modification of the first embodiment. The image selection unit 433 selects a still image obtained with a composition similar to that of the first still image from the group of still images, thereby selecting a first reference frame similar to the first still image.

[0316] The time when the first still image is obtained is very different from the time when each still image included in the group of still images is obtained. An abnormality may occur between the past examination and the latest examination. Therefore, sometimes an abnormality is not captured in the first still image but is captured in the first reference frame. In order to accurately select a first reference frame similar to the first still image, it is also possible to remove the abnormal region from each still image in the group of still images.

[0317] Alternatively, the user inputs information indicating the abnormal region to the PC 41 by operating the operation unit 44, and the image selection unit 433 accepts the region. The PC 41 can detect the abnormal region from the still image by using machine learning or the like. The image selection unit 433 can also select a first reference frame by removing the abnormal region from the ranges captured in each still image in the group of still images. The method of removing the abnormal region is not limited to the above method.

[0318] The image selection unit 433 selects a second reference frame based on the second still image. The method for selecting the second reference frame is the same as the method in the first embodiment. For example, the time when the second still image is recorded is embedded in the header of the moving image file. The image selection unit 433 determines the time when the second still image is recorded by reading the header of the moving image file. The image selection unit 433 reads out the still image recorded at the determined time from the memory 47 as the second reference frame. The second reference frame is the same as the second still image.

[0319] When the ranges captured in the two still images specified by the user overlap with each other, the image selection unit 433 may also select two reference frames by using the following method. First, the image selection unit 433 selects a second reference frame by using the above method. The image selection unit 433 selects one or more still images in the still image group that are temporally close to the second reference frame as processing targets. The image selection unit 433 selects a still image similar to the first still image from the processing targets. The selected still image is the first reference frame. Thereby, the frequency of selecting an incorrect frame as the first reference frame is reduced, and the processing time for selecting the first reference frame is shortened.

[0320] The image selection unit 433 selects two selected frames (a first selected frame and a second selected frame) based on the two reference frames in step S106. The method for selecting each selected frame is the same as the method in the first embodiment.

[0321] The estimation unit 434 estimates the position and orientation of the camera that acquired the first reference frame based on the first reference frame, the first selected frame, and the 3D model in step S107. In addition, the estimation unit 434 estimates the position and orientation of the camera that acquired the second reference frame based on the second reference frame, the second selected frame, and the 3D model in step S107. The method for estimating the position and orientation of the camera is the same as the method in the first embodiment.

[0322] The estimation unit 434 calculates the 3D coordinates of the region of the 3D model corresponding to the first reference frame in step S108. In addition, the estimation unit 434 calculates the 3D coordinates of the region of the 3D model corresponding to the second reference frame in step S108. Thereby, the estimation unit 434 estimates the positions on the 3D model corresponding to each of the first still image and the second still image. The method for estimating the positions on the 3D model is the same as the method in the first embodiment.

[0323] After step S109, the display control unit 435 detects a region common (repeated region) between the first reference frame and the second reference frame. This region is captured in the first reference frame and the second reference frame. The display control unit 435 may also detect the repeated region based only on the two-dimensional information of the two reference frames. Since the regions of the 3D model corresponding to the respective reference frames are determined in step S108, the display control unit 435 may also detect the repeated region by using the three-dimensional information of this region (step S141).

[0324] After step S141, the display control unit 435 calculates the amount of change in the shape of the subject in the repeated region and detects a region where this amount of change is equal to or greater than a specified amount. Thereby, the display control unit 435 detects a region where the shape of the subject changes significantly. The display control unit 435 may also detect this region based only on the two-dimensional information of the two reference frames. For example, the display control unit 435 may calculate the difference in the values of two adjacent pixels and detect a region where this difference exceeds a specified value. Alternatively, the display control unit 435 may detect a region where the color of the image is significantly different from the surrounding region. The display control unit 435 may also detect a region where the shape of the subject changes significantly by using the three-dimensional information of the repeated region. For example, the display control unit 435 may detect a region where the relative height or relative depth of the 3D shape of the subject with respect to the surrounding region exceeds a specified value (step S142).

[0325] Figure 2 The shown step S110 is changed to step S110a. After step S142, the display control unit 435 causes the display unit 45 to display the 3D image of the 3D model. At this time, the display control unit 435 displays a region including the point having the 3D coordinates calculated in step S108 on the 3D image. Thereby, the display control unit 435 displays the regions of the 3D model corresponding to the respective still images in the first still image and the second still image on the 3D image. In addition, the display control unit 435 causes the display unit 45 to display the repeated region detected in step S141 and the region detected in step S142 (step S110a). When step S110a is executed, Figure 22 the shown process ends.

[0326] The order of the processes executed by the PC 41 is not limited to Figure 22The order shown. For example, the display control unit 435 may also detect a duplicate region at any time between the moment of selecting the reference frame and the moment of displaying the 3D image. Therefore, step S141 may also be executed at any time between step S105a and step S110a. The display control unit 435 may also detect a region where the shape of the subject changes significantly at any time between the moment of selecting the reference frame and the moment of displaying the 3D image. Therefore, step S142 may also be executed at any time between step S105a and step S110a.

[0327] The display control unit 435 may also not execute step S141, and may also not display the duplicate region in step S110a. Therefore, step S141 is not necessary.

[0328] The display control unit 435 may also not execute step S142, and may also not display the region where the shape of the subject changes significantly in step S110a. Therefore, step S142 is not necessary.

[0329] In addition to Figure 22 the processing shown, steps S121 to S123 shown may also be executed. For example, steps S121 to S123 may also be executed after step S110a. Steps S110 shown and steps S121 to S123 may also be executed before executing step S141. Figure 17 Figure 17 the steps shown.

[0330] In addition to Figure 22 the processing shown, steps S131 to S133 shown may also be executed. For example, steps S131 to S133 may also be executed after step S110a. Steps S110 shown and steps S131 to S133 may also be executed before executing step S141. Figure 20 Figure 20 the steps shown.

[0331] The display control unit 435 may also display the number of blades of the gas turbine in step S110a by using the method shown in the second modification of the first embodiment.

[0332] Figure 23 An example of the display screen of the display unit 45 is shown. The description of the parts identical to the parts shown is omitted. Figure 13

[0333] The display control unit 435 causes the display unit 45 to display thumbnail images of each of the first still image and the second still image in step S110a. In Figure 23 ​​​In the example shown, the display control unit 435 causes the thumbnail images IM14 and IM15 to be displayed in the region R20. The thumbnail image IM14 is a thumbnail image of a still image with the file name img04. The thumbnail image IM15 is a thumbnail image of a still image with the file name img05.

[0334] In step S110a, the display control unit 435 displays the region of interest MR31 and the region MR32 of the 3D model on the 3D image MI20 of the 3D model. The region of interest MR31 is the region of the 3D model corresponding to the still image with the file name img04. The region MR32 is the region of the 3D model corresponding to the still image with the file name img05.

[0335] In step S110a, the display control unit 435 displays the marker C31 and the marker C32 on the 3D image MI20. The marker C31 represents the camera coordinates when the still image with the file name img04 is acquired. The marker C32 represents the camera coordinates when the still image with the file name img05 is acquired.

[0336] In step S110a, the display control unit 435 causes the lines L41 and L42 to be displayed. The line L41 represents the viewing angle of the still image with the file name img04. The line L42 represents the viewing angle of the still image with the file name img05.

[0337] In step S141, the display control unit 435 detects the region common to two reference frames (repeated region). In step S110a, the display control unit 435 causes the display unit 45 to display the information indicating the position of the repeated region. For example, the display control unit 435 displays the region MR41 on the thumbnail images IM14 and IM15, and displays the region MR42 on the thumbnail image IM15. The region MR41 is the repeated region. The region MR42 is the region not common to the two reference frames. In Figure 23 In the example shown, the imaging field of view of the camera that acquired the still image (file name img05) includes the imaging field of view of the camera that acquired the still image (file name img04). The thumbnail image IM14 is an image of only the region MR41. The thumbnail image IM15 is an image of the region MR41 and the region MR42.

[0338] The still image specified by the user is the same as or similar to the reference frame. Therefore, it is highly likely that almost the entire region common to the two reference frames is included in the two still images. In Figure 23 In the example shown, the display control unit 435 detects the region in the still image specified by the user that is the same as the repeated region of the reference frame. The display control unit 435 displays the detected region on the thumbnail image.

[0339] When the still image specified by the user is different from the reference frame, the display control unit 435 may also cause the display unit 45 to display the reference frame. The display control unit 435 may also display the repeated area on the reference frame.

[0340] The display control unit 435 detects the area where the shape of the subject changes significantly in step S142. The display control unit 435 displays the area MR43 on the thumbnail image IM14 and the thumbnail image IM15 in step S110a. In the area MR43, the shape of the subject changes significantly. The area MR43 is included in the area MR41. When the still image specified by the user is different from the reference frame, the display control unit 435 may also display the area where the shape of the subject changes significantly on the reference frame.

[0341] In Figure 23 In the example shown, the display control unit 435 displays the repeated area on the thumbnail image corresponding to each still image. The display control unit 435 may also display the repeated area on the 3D image MI20.

[0342] In Figure 23 In the example shown, the display control unit 435 displays the area where the shape of the subject changes significantly on the thumbnail image corresponding to each still image. The display control unit 435 may also display this area on the 3D image MI20.

[0343] The display control unit 435 performs processing (emphasis processing) for improving visual recognition on the images of each area of the area MR41, the area MR42, and the area MR43. For example, the display control unit 435 may also display the outline of each area with a line. The display control unit 435 may also display each area with a specified color or a specified pattern. As long as the user can distinguish each area from other areas of the still image or areas of the 3D model, the method of displaying each area is not limited to the above method.

[0344] Each aspect of the present invention may also include the following modification examples. In the first selection step (step S105a), the image selection unit 433 selects two or more reference frames (reference images). In the display step (step S110a), the display control unit 435 causes the display unit 45 to display the information indicating the position of the area common to two or more reference frames.

[0345] In the fifth modification example of the first embodiment, the PC 41 displays the information indicating the position of the area common to two or more reference frames (repeated area) in the display unit 45. The user can analyze and manage the relationship between two or more still images acquired at significantly different times.

[0346] The PC 41 displays, on the display unit 45, the area where the shape of the subject in the overlapping area changes significantly. The user can analyze and manage the changes in the shape of the subject over time.

[0347] (Sixth Variation of the First Embodiment)

[0348] Describe the sixth variation of the first embodiment of the present invention. Figure 1 The PC 41 shown is changed to Figure 24 the PC 41b shown. Figure 24 Show the structure of the PC 41b. The PC 41b displays a 3D image of design data (reference data) such as 3D-CAD instead of the 3D model. The description of the structure identical to that Figure 1 shown is omitted.

[0349] Figure 1 The CPU 43 shown is changed to the CPU 43b. The functions of the CPU 43b include an image acquisition unit 430, a condition reception unit 431, a 3D model generation unit 432, an image selection unit 433, an estimation unit 434, a display control unit 435, a storage control unit 436, a data reception unit 438, and a data processing unit 439. Figure 24 At least one module within the CPU 43b may also be constituted by a circuit different from the CPU 43b.

[0350] Each part within the CPU 43b may also be constituted by at least one of a processor and a logic circuit. Each part within the CPU 43b can include one or more processors. Each part within the CPU 43b can include one or more logic circuits.

[0351] The memory 47 stores reference data generated by the PC 41b or an external device. The reference data represents the 3D shape of the subject. The reference data is data independent of the 3D model. The data format of the 3D model and the data format of the reference data may also be different. The user inputs information for specifying the reference data to the PC 41b by operating the operation unit 44. The data reception unit 438 receives this information from the operation unit 44.

[0352] The 3D model includes the 3D coordinates of two or more points of the subject. The data processing unit 439 associates the 3D coordinates of these two or more points with the 3D shape shown by the reference data. Thereby, the data processing unit 439 correlates the position of the 3D model with the position of the reference data.

[0353] Figure 25 Show the process of the processing executed by the PC 41b. The description of the same processing as that Figure 2 shown is omitted.

[0354] After step S102, the user operates the operation unit 44 to input information for specifying reference data to the PC 41b. The data reception unit 438 receives this information from the operation unit 44 (step S151). After step S151, step S103 is executed.

[0355] After step S108, the data processing unit 439 reads out the 3D model generated in step S103 and the reference data indicated by the information received in step S151 from the memory 47. The data processing unit 439 correlates the position of the 3D model with the position of the reference data (step S152).

[0356] Details of step S152 will be described. When the data format of the 3D model is different from the data format of the reference data, the data processing unit 439 performs a process of unifying the data format of the 3D model and the data format of the reference data. The 3D model is 3D point cloud data. For example, when the reference data is solid data or surface data, the data processing unit 439 transforms the reference data into 3D point cloud data. The data processing unit 439 may also transform the 3D model into solid data or surface data.

[0357] After that, the data processing unit 439 correlates the position of the 3D model with the position of the reference data by using a known 3D data position alignment technique. The data processing unit 439 can easily perform position alignment by transforming the data format of the 3D model or the reference data. The user can also operate the operation unit 44 to align the position of the 3D model with the position of the reference data.

[0358] After step S152, the estimation unit 434 correlates the 3D coordinates calculated in step S108 with the reference data (step S153). Step S153 is the same as Figure 2 step S109 shown.

[0359] After step S153, the display control unit 435 causes the display unit 45 to display a 3D image of the reference data. At this time, the display control unit 435 displays a region including the point having the 3D coordinates calculated in step S108 on the 3D image. Thus, the display control unit 435 displays a region corresponding to the 3D model's region of interest on the 3D image (step S154). The user can confirm the position on the 3D model corresponding to the reference frame as the region of interest by confirming the 3D image of the reference data. When step S154 is executed, Figure 25 the processing shown ends.

[0360] The order of the processing executed by the PC 41b is not limited to Figure 25In the order shown. For example, the data reception unit 438 may also receive information for specifying reference data at any time before the data processing unit 439 processes the reference data. Therefore, step S151 may also be executed at any time before step S152 is executed.

[0361] In the case where usage regulations for the reference data are set in advance for the PC 41b, the data reception unit 438 does not need to receive information for specifying the reference data. Therefore, the CPU 43b does not need to have the data reception unit 438, and step S151 is not necessary.

[0362] In addition to Figure 25 the processing shown, Figure 17 steps S121 to S123 shown may also be executed. For example, steps S121 to S123 may also be executed after step S154. The display control unit 435 may also display the 3D distance on the 3D image of the reference data in step S123.

[0363] In addition to Figure 25 the processing shown, Figure 20 steps S131 to S133 shown may also be executed. For example, steps S131 to S133 may also be executed after step S154. The display control unit 435 may also cause the display unit 45 to display a 3D image of the reference data including the position estimated as the position of the new region of interest in step S133.

[0364] In addition to Figure 25 the processing shown, Figure 22 steps S141 and S142 shown may also be executed. For example, steps S141 and S142 may also be executed after step S154. After step S142, the display control unit 435 may also display the duplicate region detected in step S141 and the region detected in step S142 on the 3D image of the reference data. Steps S141 and S142 may also be executed before step S152 is executed. When step S154 is executed, the display control unit 435 may also display the duplicate region detected in step S141 and the region detected in step S142 on the 3D image of the reference data.

[0365] The display control unit 435 may also display the number of blades of the gas turbine in step S154 by using the method shown in the second modification of the first embodiment.

[0366] Each aspect of the present invention may also include the following modification examples. A 3D image is an image of shape data representing the 3D shape of a subject. In the association step (step S153), the data processing unit 439 associates the 3D coordinates of two or more points of the subject with the 3D shape shown by the shape data.

[0367] In the sixth modification example of the first embodiment, the PC 41b displays a 3D image of reference data such as 3D-CAD. Since the surface of the subject can be displayed with high quality, the user can confirm the surface of the subject in detail.

[0368] (Second Embodiment)

[0369] The second embodiment of the present invention will be described. Figure 1 The shown PC 41 is changed to Figure 26 the shown PC41c. Figure 26 The structure of the PC 41c is shown. The PC 41c has a function of reproducing a moving image and displays the moving image together with the 3D image of the 3D model. The description of the structure identical to the Figure 1 shown structure is omitted.

[0370] Figure 1 The shown CPU 43 is changed to the CPU 43c. The functions of the CPU 43c include an image acquisition unit 430, a condition acceptance unit 431, a 3D model generation unit 432, an image selection unit 433, an estimation unit 434, a display control unit 435, a storage control unit 436, and a moving image control unit 440. Figure 26 At least one module in the shown CPU 43c may also be constituted by a circuit different from the CPU 43c.

[0371] Each part in the CPU 43c may be constituted by at least one of a processor and a logic circuit. Each part in the CPU 43c can include one or more processors. Each part in the CPU 43c can include one or more logic circuits.

[0372] In the second embodiment, the moving image is used as a group of still images. Devices such as the endoscope device 1 acquire a moving image during an examination. The user can assign an index to a specific frame of the moving image by designating a frame of interest during the shooting of the moving image. The index has the same function as a chapter. The moving image control unit 440 controls the reproduction of the moving image.

[0373] Figure 27 and Figure 28 The process of the processing executed by the PC 41c is shown. The description of the processing identical to the Figure 2 shown processing is omitted.

[0374] Figure 2The step S105 shown is changed to step S105b. The image selection unit 433 reads the index embedded in the moving image and determines the frame corresponding to the index. The image selection unit 433 reads out this frame from the memory 47 as the reference frame (step S105b).

[0375] After step S110, the moving image control unit 440 reads the moving image from the memory 47 and reproduces the moving image. The moving image control unit 440 causes the display unit 45 to display the reproduced moving image. The moving image control unit 440 may also read the index embedded in the moving image and determine the frame corresponding to the index. The moving image control unit 440 may also display the information indicating the frame to which the index is assigned together with the moving image (step S161).

[0376] The user inputs information indicating a specific frame of the moving image to the PC 41c by operating the operation unit 44 during the reproduction of the moving image. Thereby, the user inputs information indicating a new region of interest to the PC 41c. The image selection unit 433 reads out the frame indicated by the input information from the memory 47 as the new reference frame (step S162).

[0377] After step S162, steps S163 to S166 are executed. Step S163 is the same as Figure 2 the step S106 shown. Step S164 is the same as Figure 2 the step S107 shown. Step S165 is the same as Figure 2 the step S108 shown. Step S166 is the same as Figure 2 the step S109 shown.

[0378] After step S166, the display control unit 435 displays a region including the point having the 3D coordinates calculated in step S165 on the 3D image of the 3D model. Thereby, the display control unit 435 displays the region corresponding to the new region of interest of the 3D model on the 3D image (step S167). When step S167 is executed, Figure 27 and Figure 28 the processing shown ends.

[0379] It is also possible to assign an index to the moving image after recording the moving image. For example, it may be that after the moving image is displayed on the display unit 45, the user operates the operation unit 44 to assign an index to a desired position in the moving image.

[0380] The order of the processing executed by the PC 41c is not limited to Figure 27 and Figure 28The order shown. For example, a moving image may also be displayed before step S105b is executed, and the user designates a specific frame of the moving image as the region of interest. The image selection unit 433 may also select this frame as the reference frame in step S105b. Thus, step S161 may also be executed before step S104 is executed.

[0381] The moving image control unit 440 may also temporarily stop the reproduction of the moving image. The image selection unit 433 may also select the frame being displayed when the reproduction of the moving image is stopped as the reference frame. It may also be that the user inputs information indicating the region of interest in the reference frame to the PC 41c by operating the operation unit 44. After that, the same processing as that of Figure 20 steps S131 to S133 shown may also be executed.

[0382] In addition to Figure 27 and Figure 28 the processing shown, steps S121 to S123 shown in Figure 17 may also be executed. For example, steps S121 to S123 may also be executed after step S167. Steps S121 to S123 may also be executed before step S161 is executed.

[0383] In addition to Figure 27 and Figure 28 the processing shown, steps S141 and S142 shown in Figure 22 may also be executed. For example, steps S141 and S142 may also be executed after step S166. Steps S141 and S142 may also be executed before step S161 is executed. When step S167 is executed, the display control unit 435 may also display the duplicate region detected in step S141 and the region detected in step S142 on the 3D image.

[0384] In addition to Figure 27 and Figure 28 the processing shown, steps S151 to S154 shown in Figure 25 may also be executed. For example, step S151 may also be executed between steps S102 and S103. Steps S152 to S154 may also be executed in place of steps S109 and S110. When step S167 is executed, the display control unit 435 may also display the region corresponding to the new region of interest of the 3D model on the 3D image of the reference data.

[0385] The display control unit 435 may also display the number of blades of the gas turbine by using the method shown in the second modification of the first embodiment in step S110 or step S167.

[0386] Figure 29 An example of the display screen of the display unit 45 is shown. Similar to the first embodiment, the following example is shown: after a 3D image of the 3D model is displayed on the display unit 45, the PC 41c reads in one reference frame as the region of interest. An index is assigned to the reference frame.

[0387] The display control unit 435 causes the display unit 45 to display Figure 29 the dialog box A20 shown. In addition, the display control unit 435 displays the region R10 on the dialog box A20. The region R10 is a region for displaying the 3D image of the 3D model.

[0388] After the 3D model is generated, the display control unit 435 causes the 3D image MI10 of the 3D model to be displayed in the region R10. At this time, the display control unit 435 displays the 3D image MI10 of the 3D model observed from a preset viewpoint.

[0389] The display control unit 435 displays the moving image IM21 on the dialog box A20 in step S161. At this time, the display control unit 435 displays the moving image IM21 in a region different from the region R10 where the 3D image MI10 is displayed. The 3D image MI10 and the moving image IM21 are arranged side by side in the horizontal direction.

[0390] The display control unit 435 displays the scroll bar SB10 on the dialog box A20 in step S161. The scroll bar SB10 shows the reproduction position or reproduction time of the moving image IM21. The display control unit 435 displays the index position IN11 and the index position IN12 on the scroll bar SB10 in step S161. The index position IN11 and the index position IN12 indicate the positions of the frames to which the index was assigned by the user during the shooting of the moving image IM21.

[0391] The display control unit 435 displays information indicating the positions on the 3D model corresponding to the respective index positions of the index position IN11 and the index position IN12 on the 3D image MI10 in step S161. Specifically, the display control unit 435 displays the string CH21 and the string CH22 on the 3D image MI10, and displays the line L51 and the line L52 on the 3D image MI10.

[0392] Each string represents an index name. The string CH21 represents the index name index1 corresponding to the index position IN11. The string CH22 represents the index name index2 corresponding to the index position IN12. Each line represents the position corresponding to the frame indicated by each index name. The line L51 represents the position corresponding to the still image indicated by the index name index1. The line L52 represents the position corresponding to the still image indicated by the index name index2. Each position corresponds to the 3D coordinates calculated based on the camera coordinates and pose information of the camera that acquired the reference frame in step S108.

[0393] The user can confirm the position of the frame to which the index is assigned. In addition, the user can confirm the position on the 3D model corresponding to the index assigned to the moving image IM21. Since the 3D image MI10 and the moving image IM21 are displayed, the user can correlate the position of the frame to which the index is assigned with the position on the 3D model corresponding to the same index.

[0394] After the 3D image MI10 is displayed, the display control unit 435 may also change the viewpoint of the 3D image MI10. For example, the user may also specify an index by operating the operation unit 44. At this time, the display control unit 435 may also determine the frame to which the index specified by the user is assigned. The display control unit 435 may change the viewpoint of the 3D image MI10 based on the camera coordinates of the camera that acquired the determined frame. The display control unit 435 may also enlarge, reduce, or rotate the area of the 3D model corresponding to the determined frame. The method of changing the viewpoint of the 3D image MI10 is the same as the method in the first embodiment. The method of enlarging, reducing, or rotating the area of the 3D model is the same as the method in the first embodiment.

[0395] After the moving image IM21 is displayed, the user operates the operation unit 44 to specify a specific position SP10 on the scroll bar SB10. Thereby, the user designates a specific frame of the moving image as the region of interest. The image selection unit 433 selects the frame specified by the user as the new reference frame in step S162. After steps S163 to S166 are executed, the display control unit 435 displays the position P31 of the region corresponding to the new region of interest on the 3D image MI10 in step S167.

[0396] The user can also temporarily stop the reproduction of the moving image. The user can also specify a part of the range captured in the frame being displayed. At this time, the estimation unit 434 can also accept a part of the frame as the region of interest. The estimation unit 434 can also estimate the position on the 3D model corresponding to the region of interest. The display control unit 435 can also display the region corresponding to the region of interest on the 3D image MI10. The method of accepting a part of the still image as the region of interest and visualizing the region on the 3D model corresponding to the region of interest is the same as the method in the fourth modification of the first embodiment.

[0397] In the second embodiment, the PC 41c displays a 3D image of the 3D model and also displays a moving image including two or more still images used when generating the 3D model. The PC 41c displays the position on the 3D image corresponding to the whole or a part of a specific frame on the 3D model. After the inspection is completed, the user can confirm in detail the position and posture of the region on the 3D model corresponding to the region of interest. Therefore, the user can efficiently report (comment on) the inspection results and can efficiently review the abnormality by using the inspection moving image.

[0398] (Third Embodiment)

[0399] The third embodiment of the present invention will be described. Figure 1 The PC 41 shown is changed to Figure 30 the PC41d shown. Figure 30 The structure of the PC 41d is shown. The PC 41d automatically sets the region of interest regardless of the operation performed by the user. The description of the structure identical to the Figure 1 structure shown is omitted.

[0400] Figure 1 The CPU 43 shown is changed to the CPU 43d. The functions of the CPU 43d include an image acquisition unit 430, a condition acceptance unit 431, a 3D model generation unit 432, an image selection unit 433, an estimation unit 434, a display control unit 435, a storage control unit 436, and a region detection unit 441. Figure 30 At least one module in the CPU 43d can also be constituted by a circuit different from the CPU 43d.

[0401] Each unit in the CPU 43d can also be constituted by at least one of a processor and a logic circuit. Each unit in the CPU 43d can include one or more processors. Each unit in the CPU 43d can include one or more logic circuits.

[0402] The region detection unit 441 detects a part of the region of the subject captured in the image as the region of interest. For example, the region of interest is a region estimated to contain an abnormality (abnormal region). The region of interest is not limited to the abnormal region. As long as the region of interest is set for the purpose of visualizing the position on the 3D model, the region of interest can also be a non-abnormal region. For example, the region of interest can also be the welding mark of the subject in pipe inspection, or a branch point such as an elbow of the subject. The region of interest can also be the reference blade in gas turbine inspection. The region of interest can also be a region serving as a landmark in the object to be inspected.

[0403] Figure 31 Shows the process of the processing executed by the PC 41d. The description of the same processing as Figure 2 the processing shown is omitted.

[0404] After step S104, the region detection unit 441 processes the image and detects the region of interest in the image (step S171). After step S171, step S105 is executed.

[0405] Details of step S171 will be described. Below, an example of detecting an abnormality in the object to be inspected by using machine learning will be described. The abnormal region is distinguished from the non-abnormal region in terms of shape, color, pattern, etc. The user pre-classifies the images (still images or moving images) managed in the past inspections into two types of images. The two types of images are known images that have captured abnormalities and known images that have not captured abnormalities. The region detection unit 441 processes these images as training data (training data with correct answers) for machine learning.

[0406] The region detection unit 441 generates learning data by performing learning with the training data as input. The learning data represents the characteristics of the abnormal region. The region detection unit 441 determines whether an abnormal region is included in an arbitrary image based on the learning data. When the region detection unit 441 determines that an abnormal region is included in the image, the region detection unit 441 detects the abnormal region as the region of interest. As long as the PC 41d can automatically detect the region of interest, the method for detecting the region of interest is not limited to the above method.

[0407] For example, the region detection unit 441 determines whether the region of interest is included in the still images included in the still image group. Alternatively, the region detection unit 441 detects the region of interest in the still image acquired at a time different from the time when the still image group is acquired. In this case, the region detection unit 441 determines whether the region of interest is included in a still image different from two or more still images included in the still image group.

[0408] The still image group includes two or more key frames used to generate a 3D model. The region detection unit 441 determines whether the key frames include a region of interest. Alternatively, the region detection unit 441 determines whether a region of interest is included in a still image that is not included in the two or more key frames. For example, in the case where the still image group includes a still image not used to generate a 3D model, the region detection unit 441 determines whether the region of interest is included in the still image. A still image not included in the still image group is not used to generate a 3D model. The region detection unit 441 may also determine whether the region of interest is included in the still image.

[0409] In step S105, the image selection unit 433 selects one still image from the two or more still images included in the still image group as the reference frame. When the region detection unit 441 detects a region of interest in a still image included in the still image group, the image selection unit 433 selects the still image as the reference frame. When the region detection unit 441 detects a region of interest in a still image different from all of the two or more still images included in the still image group, the image selection unit 433, in the same manner as the third modification of the first embodiment, selects an image obtained with a composition similar to that of the still image from the still image group. Thus, the image selection unit 433 selects the reference frame. The reference frame includes the detected region of interest in the still image.

[0410] When the region detection unit 441 detects a region of interest in a key frame used to generate a 3D model, the image selection unit 433 selects the key frame as the reference frame. When the region detection unit 441 detects a region of interest in a still image not included in the two or more key frames and the still image is included in the still image group, the image selection unit 433 selects the still image as the reference frame. When the region detection unit 441 detects a region of interest in a still image not included in the two or more key frames and the still image is not included in the still image group, the image selection unit 433, in the same manner as the third modification of the first embodiment, selects an image obtained with a composition similar to that of the still image from the still image group. Thus, the image selection unit 433 selects the reference frame. The reference frame includes the detected region of interest in the still image.

[0411] The conditions of the reference frame are pre-stored in the memory 47. In the above example, the conditions are shown by the learning data. The image selection unit 433 selects one still image from two or more still images included in the still image group in step S105. The selected still image includes a region of interest that satisfies the condition. In the above example, the image selection unit 433 selects the reference frame including the region of interest detected by the region detection unit 441. Alternatively, the image selection unit 433 selects the reference frame corresponding to the still image including the region of interest. Thus, the image selection unit 433 selects the still image that satisfies the condition shown by the learning data as the reference frame.

[0412] The estimation unit 434 detects the region of interest in the reference frame in step S108. When the image selection unit 433 selects the same still image as the still image including the region of interest detected by the region detection unit 441 as the reference frame, the reference frame includes the region of interest. The estimation unit 434 detects the region of interest. When the image selection unit 433 selects a still image different from the still image including the region of interest detected by the region detection unit 441 as the reference frame, the estimation unit 434 extracts a region similar to the region of interest in the reference frame. The estimation unit 434 processes the extracted region as the region of interest.

[0413] The estimation unit 434 calculates the 3D coordinates of the region corresponding to the region of interest of the 3D model in step S108. Thus, the estimation unit 434 estimates the position corresponding to the region of interest on the 3D model. The method of estimating the position on the 3D model is the same as the method in the first embodiment.

[0414] The display control unit 435 causes the display unit 45 to display the 3D image of the 3D model in step S110. At this time, the display control unit 435 causes the display unit 45 to display the region of interest.

[0415] The order of the processing executed by the PC 41d is not limited to Figure 31 the order shown. For example, the region of interest may be detected by the region detection unit 441 before the 3D model generation unit 432 generates the 3D model. Therefore, step S171 may be executed at any time between step S101 and step S104.

[0416] In addition to Figure 31 the processing shown, Figure 17 steps S121 to S123 shown may also be executed. For example, steps S121 to S123 may be executed after step S110.

[0417] In addition to Figure 31 the processing shown, Figure 20Steps S131 to S133 shown. For example, steps S131 to S133 may also be executed after step S110.

[0418] In addition to Figure 31 the processing shown, Figure 22 steps S141 and S142 shown may also be executed. For example, steps S141 and S142 may also be executed after step S109. Steps S142 may also be executed Figure 22 step S110a shown to replace step S110.

[0419] In addition to Figure 31 the processing shown, Figure 25 steps S151 to S154 shown may also be executed. For example, step S151 may also be executed between steps S102 and S103. Steps S152 to S154 may also be executed to replace steps S109 and S110.

[0420] In addition to Figure 31 the processing shown, Figure 28 steps S161 to S167 shown may also be executed. For example, steps S161 to S167 may also be executed after step S110.

[0421] The display control unit 435 may also display the number of blades of the gas turbine in step S110 by using the method shown in the second modification of the first embodiment.

[0422] Figure 32 An example of the display screen of the display unit 45 is shown. The description of the part identical to Figure 10 the part shown is omitted.

[0423] The display control unit 435 displays information indicating the position of the area representing the 3D model on the 3D image MI10 of the 3D model in step S110. For example, the display control unit 435 displays area MR51, area MR52, and area MR53 on the 3D image MI10. Area MR51 is the area of the 3D model corresponding to the still image with the file name img01. Area MR52 is the area of the 3D model corresponding to the still image with the file name img02. Area MR53 is the area of the 3D model corresponding to the still image with the file name img03.

[0424] In step S110, the display control unit 435 displays information indicating the positions of the regions of interest detected by the region detection unit 441 on the 3D image MI10. For example, the display control unit 435 displays the region of interest MR61, the region of interest MR62, and the region of interest MR63. The region of interest MR61 is displayed on the thumbnail image IM11, the region of interest MR62 is displayed on the thumbnail image IM12, and the region of interest MR63 is displayed on the thumbnail image IM13. The region MR51 corresponds to the region of interest MR61, the region MR52 corresponds to the region of interest MR62, and the region MR53 corresponds to the region of interest MR63.

[0425] The display control unit 435 may also perform a process (emphasis process) for improving visual recognition on the images of the regions MR51, MR52, MR53, the region of interest MR61, the region of interest MR62, and the region of interest MR63. For example, the display control unit 435 may display the contours of the respective regions with lines. The display control unit 435 may also display the respective regions with a specified color or a specified pattern. As long as the user can distinguish each region from other regions of the still image or regions of the 3D model, the method of displaying each region is not limited to the above method.

[0426] In Figure 32 In the example shown, the display control unit 435 displays the regions of interest in each of the three still images and also displays the regions of the 3D model corresponding to the respective regions of interest. Since images acquired from various viewpoints are displayed, the user can easily confirm whether the abnormal region detected by the region detection unit 441 is indeed abnormal.

[0427] The region detection unit 441 may detect abnormal regions in many still images. When the display control unit 435 displays all the still images in which abnormal regions are detected, it may be difficult for the user to confirm the abnormal regions. In this case, the display control unit 435 may also display only the representative still images in which abnormalities are detected.

[0428] After displaying the 3D image MI10, the display control unit 435 may also change the viewpoint of the 3D image MI10. For example, the user operates the operation unit 44 to specify any one of the regions of interest MR61, MR62, and MR63. Alternatively, the user operates the operation unit 44 to specify any one of the regions MR51, MR52, and MR53. At this time, the display control unit 435 determines a still image corresponding to the region specified by the user. The display control unit 435 changes the viewpoint of the 3D image MI10 based on the camera coordinates of the camera that has acquired the determined still image. The display control unit 435 may also enlarge, reduce, or rotate the region of the 3D model corresponding to the determined still image. The method of changing the viewpoint of the 3D image MI10 is the same as the method in the first embodiment. The method of enlarging, reducing, or rotating the region of the 3D model is the same as the method in the first embodiment. The user can easily confirm the abnormal region by changing the viewpoint of the 3D image MI10.

[0429] Each mode of the present invention may also include the following modification examples. In the first selection step (step S105), the image selection unit 433 selects a reference frame (reference image) including a region of interest (reference region) that satisfies a preset condition. In the estimation step (step S108), the estimation unit 434 estimates the 3D coordinates of one or more points of the region of interest.

[0430] In the third embodiment, the PC 41d automatically sets the region of interest regardless of the operation performed by the user. Since the amount of operation by the user is reduced, the user can concentrate on a prescribed task. For example, the user can confirm the relative position of the region of interest in the whole of the inspection object. When the abnormal region is detected as the region of interest, the user can confirm whether the abnormal region is indeed abnormal.

[0431] (Fourth Embodiment)

[0432] The fourth embodiment of the present invention will be described. In the fourth embodiment, the PC 41 shown Figure 1 is used.

[0433] Sometimes the user observes the 3D model to confirm in detail the state of the unevenness or the abnormal size, etc. of the region of interest. In the first to third embodiments, the 3D model shows the sparse 3D shape of the subject. Since the 3D model is not highly detailed, there is a possibility that the user cannot sufficiently confirm the state of the region of the 3D model corresponding to the region of interest. In order to obtain a highly detailed 3D model of a relatively wide range like the whole of the inspection object, a long processing time is required.

[0434] The user can confirm the overall image of the inspection object by observing the 3D model composed of sparse points. However, the accuracy for the user to confirm the fine structures such as concavities and convexities at specific positions of the subject, or the accuracy for measuring the dimensions of such structures may be insufficient.

[0435] In the fourth embodiment, the PC 41 generates a high-precision 3D model of the region corresponding to the region of interest of the 3D model and displays the 3D image of the 3D model. Thereby, the PC 41 solves the above problems.

[0436] Figure 33 Shows the process of the processing executed by the PC 41. The description of the same processing as that Figure 2 shown is omitted.

[0437] In step S103, the step S103d Figure 4 shown is not executed. Therefore, the 3D model generated in step S103 does not include the 3D coordinates of the points on the subject other than the feature points.

[0438] After the 3D image of the 3D model is displayed on the display unit 45 in step S110, the user operates the operation unit 44 to input information representing a specific still image to the PC 41. This still image is the reference frame selected in step S105. When two or more reference frames are selected in step S105, the user inputs information representing one reference frame to the PC 41. The estimation unit 434 accepts the reference frame as the region of interest based on the information input by the user to the PC 41. The region of interest is the whole of the range captured in the reference frame. The estimation unit 434 may also accept two or more reference frames (step S181).

[0439] After step S181, the estimation unit 434 calculates the 3D coordinates of the region of the 3D model corresponding to the reference frame accepted in step S181 and generates a 3D model including the 3D coordinates. The storage control unit 436 stores the 3D model generated by the estimation unit 434 in the memory 47 (step S182).

[0440] Details of step S182 are described. The estimation unit 434 calculates the 3D coordinates of two or more points of the reference frame based on the camera coordinates of the camera that acquired the reference frame, the pose information of the camera, and the selected frame. When the Figure 5 step S107d shown is executed, the estimation unit 434 acquires the camera coordinates and pose information of the camera that acquired the reference frame. The estimation unit 434 uses the camera coordinates and pose information in step S182.

[0441] Two or more points of the reference frame include those having Figure 5The points of the 2D coordinates determined in step S107c shown above. That is, two or more points of the reference frame include one or more feature points. Alternatively, two or more points of the reference frame include one or more points having 2D coordinates other than the 2D coordinates determined in step S107c. That is, two or more points of the reference frame include one or more points other than feature points. In the case where the 3D model includes the 3D coordinates of the feature points, the estimation unit 434 may also obtain the 3D coordinates of the feature points from the 3D model.

[0442] For example, the estimation unit 434 calculates the 3D coordinates by performing the same processing as that of Figure 4 step S103d shown above. At this time, for example, the estimation unit 434 uses the selection frame as Figure 3 image I1 shown above, and uses the reference frame as Figure 3 image I2 shown above. The estimation unit 434 calculates the 3D coordinates of the subject that has entered the imaging field of view of the camera that has acquired the reference frame by performing the above processing. In addition, the estimation unit 434 performs the same processing as that of Figure 4 step S103e shown above.

[0443] The estimation unit 434 may also calculate the 3D coordinates of the periphery of the range in addition to calculating the 3D coordinates of the range captured in the reference frame. For example, the estimation unit 434 may detect a common area (repeated area) between the reference frame and the selection frame from the selection frame. The estimation unit 434 may calculate the 3D coordinates of the area outside the repeated area in the selection frame.

[0444] The estimation unit 434 generates a high-precision 3D model corresponding to the reference frame by executing step S182. The density of the points of the 3D model generated in step S182 is higher than the density of the points included in the area corresponding to the reference frame among the two or more points of the 3D model generated in step S103. The point density represents the number of points included in a unit volume in the coordinate system of the 3D model.

[0445] After step S182, the display control unit 435 causes the display unit 45 to display a 3D image of the 3D model generated in step S182. Thereby, the display control unit 435 displays the area of the high-precision 3D model corresponding to the area of interest (step S183). The user can recognize the high-precision 3D model corresponding to the reference frame as the area of interest. When step S183 is executed, Figure 33 the processing shown above ends.

[0446] When the estimation unit 434 receives two or more reference frames in step S181, the estimation unit 434 may also generate a high-precision 3D model corresponding to each of the two or more reference frames in step S182. The display control unit 435 may also display a 3D image of the high-precision 3D model corresponding to each of the two or more reference frames on the display unit 45 in step S183.

[0447] The order of the processes executed by the PC 41 is not limited to Figure 33 the order shown. For example, it may be that after the image selection unit 433 selects a reference frame and before the display control unit 435 displays a 3D image of the sparse 3D model, the estimation unit 434 receives the reference frame. Thus, step S181 may be executed at any time between step S105 and step S110.

[0448] It may also be that after the estimation unit 434 obtains the camera coordinates and pose information of the camera that acquired the reference frame and before the display control unit 435 displays a 3D image of the sparse 3D model, the estimation unit 434 generates a high-precision 3D model. Thus, step S182 may be executed at any time between step S107 and step S110.

[0449] It may also be that after the estimation unit 434 obtains the camera coordinates and pose information of the camera that acquired the reference frame and before the display control unit 435 displays a 3D image of the sparse 3D model, the display control unit 435 displays a 3D image of the high-precision 3D model. Thus, step S183 may be executed at any time between step S107 and step S110.

[0450] The estimation unit 434 may also generate a high-precision 3D model corresponding to the reference frame selected in step S105. Therefore, the estimation unit 434 does not need to execute step S181.

[0451] It may also be that in addition to Figure 33 the processes shown, Figure 17 steps S121 to S123 shown are also executed. For example, steps S121 to S123 may be executed after step S183. Steps S121 to S123 may also be executed before executing step S181.

[0452] It may also be that in addition to Figure 33 the processes shown, Figure 20 steps S131 to S133 shown are also executed. For example, steps S131 to S133 may be executed after step S183. Steps S131 to S133 may also be executed before executing step S181.

[0453] In addition to Figure 33 the processing shown, the steps S141, S142, and S110a shown in Figure 22 may also be executed. For example, steps S141, S142, and S110a may be executed after step S183. Steps S141, S142, and S110a may also be executed before executing step S181.

[0454] In addition to Figure 33 the processing shown, the steps S151 to S154 shown in Figure 25 may also be executed. For example, step S151 may be executed between steps S102 and S103. Steps S152 to S154 may also be executed in place of steps S109 and S110.

[0455] In addition to Figure 33 the processing shown, the steps S161 to S167 shown in Figure 28 may also be executed. For example, steps S161 to S167 may be executed after step S183. Steps S161 to S167 may also be executed before executing step S181.

[0456] In addition to Figure 33 the processing shown, the step S171 shown in Figure 31 may also be executed. For example, step S171 may be executed between steps S104 and S105. The region detection unit 441 detects a region of interest in the image in step S171. The estimation unit 434 does not need to execute step S181. The estimation unit 434 may also calculate the 3D coordinates of the region of the 3D model corresponding to the region of interest in step S182 and generate a 3D model including the 3D coordinates.

[0457] The display control unit 435 may also display the number of blades of the gas turbine in step S110 by using the method shown in the second modification of the first embodiment.

[0458] Figure 34 An example of the display screen of the display unit 45 is shown. The description of the part identical to the part shown in Figure 10 is omitted.

[0459] The display control unit 435 and Figure 10Similarly, the thumbnail images IM11, IM12, and IM13 are displayed in the region R20. For example, the user designates the thumbnail image IM13. At this time, the estimation unit 434 accepts the reference frame corresponding to the thumbnail image IM13 as the region of interest in step S181. The estimation unit 434 generates a high-precision 3D model corresponding to the reference frame in step S182. The display control unit 435 displays a 3D image MI30 of the high-precision 3D model in the region R30 on the dialog box A10 in step S183.

[0460] The display control unit 435 displays the 3D image MI30 of the 3D model observed from a preset viewpoint in a preset direction. The display control unit 435 may also display the 3D image MI30 of the 3D model observed from the camera coordinates of the camera that acquired the reference frame in the direction indicated by the camera pose information. For example, the display control unit 435 displays the 3D image MI30 between the region R10 where the 3D image MI10 of the sparse 3D model is displayed and the region R20 where each thumbnail image is displayed.

[0461] In Figure 34 In the example shown, the region R10 where the 3D image MI10 is displayed and the region R30 where the 3D image MI30 is displayed are different from each other. The display control unit 435 may also display the 3D image MI10 and the 3D image MI30 in one region.

[0462] The display control unit 435 may also switch between a first display state and a second display state. The display control unit 435 does not display the 3D image MI30 but displays the 3D image MI10 in the first display state. The display control unit 435 does not display the 3D image MI10 but displays the 3D image MI30 in the second display state. The user may also input information indicating one of the first display state and the second display state to the PC 41 by operating the operation unit 44. The display control unit 435 may also implement the display state indicated by this information.

[0463] In Figure 34 In the example shown, the display control unit 435 causes the thumbnail images IM11, IM12, and IM13 to be displayed in the region R20. The display control unit 435 may also cause high-precision 3D models corresponding to the respective reference frames to be displayed in the region R20 instead of the respective thumbnail images.

[0464] The user may also input two or more measurement points of the 3D image MI30 to the PC 41 by operating the operation unit 44. The CPU 43 may also accept these two or more measurement points and measure the dimensions of the 3D model based on these two or more measurement points.

[0465] Each aspect of the present invention may also include the following modification examples. In the estimation step (step S182), the estimation unit 434 estimates the 3D coordinates of one or more points in the region of the subject captured in the reference frame based on the second camera coordinates of the second camera that has acquired the reference frame (reference image), the second pose information of the second camera, and the selected frame (selected image). The one or more points in this region are different from the points having 3D coordinates included in the 3D model (3D data) stored in the memory 47 in step S104. The 3D coordinates of the one or more points in this region are visualized in the 3D image MI30. In addition, in the 3D image MI30, the 3D coordinates of the one or more points included in the 3D coordinates included in the 3D model and included in this region are visualized. The density of the points having 3D coordinates visualized by the 3D image MI30 is higher than the density of the points included in the two or more points of the subject and included in this region.

[0466] Each aspect of the present invention may also include the following modification examples. In the first selection step (step S105), the image selection unit 433 selects a reference frame that includes a region of interest (reference region) that satisfies a preset condition. In the estimation step (step S182), the estimation unit 434 estimates the 3D coordinates of one or more points in the region of interest based on the second camera coordinates, the second pose information, and the selected frame (selected image). The one or more points in the region of interest are different from the points having 3D coordinates included in the 3D model stored in the memory 47 in step S104. The 3D coordinates of the one or more points in the region of interest are visualized in the 3D image. In addition, in the 3D image, the 3D coordinates of the one or more points included in the 3D coordinates included in the 3D model and included in the region of interest are visualized. The density of the points having 3D coordinates visualized by the 3D image is higher than the density of the points included in the two or more points of the subject and included in the region of interest.

[0467] In the fourth embodiment, the PC 41 generates a high-precision 3D model of the region corresponding to the region of interest of the 3D model. Compared with the case where the PC 41 generates a high-precision 3D model of the entire inspection object, an increase in processing time is suppressed. Since the PC 41 displays a 3D image of the high-precision 3D model, the user can confirm the position corresponding to the region of interest on the 3D model. In addition, the user can accurately confirm the abnormal state (concave-convex state or abnormal size, etc.) of the locality of the region of interest.

[0468] (Modification example of the fourth embodiment)

[0469] A modification example of the fourth embodiment of the present invention will be described. In the modification example of the fourth embodiment, Figure 1The PC 41 shown. The PC 41 does not need to display the 3D image of the sparse 3D model generated by the 3D model generation unit 432.

[0470] Figure 35 Shows the process of the processing executed by the PC 41. Explanation of the same processing as Figure 33 The processing shown is omitted.

[0471] After step S107, the estimation unit 434 calculates the 3D coordinates of the region of the 3D model corresponding to the reference frame in step S182, and generates a 3D model including the 3D coordinates. At this time, the estimation unit 434 uses the reference frame selected in step S105.

[0472] The 3D model generated by the 3D model generation unit 432 includes the camera coordinates and pose information of the camera that acquired two or more 2D images used for generating the 3D model. In the case where the 3D model generation unit 432 generates a 3D model without using a reference frame, the 3D model does not include the camera coordinates and pose information of the camera that acquired the reference frame. Therefore, the estimation unit 434 estimates the position and pose of the camera that acquired the reference frame in step S107.

[0473] In the above example, the image selection unit 433 selects a still image that is not included in two or more key frames used for generating the 3D model as the reference frame. The image selection unit 433 may also select one of the two or more key frames as the reference frame. In this case, the 3D model includes the camera coordinates of the second camera that acquired the reference frame and the pose information of the second camera. The estimation unit 434 may also acquire the camera coordinates and the pose information from the 3D model in step S107.

[0474] Figure 36 Shows an example of the display screen of the display unit 45. Explanation of the same part as Figure 34 The part shown is omitted.

[0475] The display control unit 435 does not display the region R10 for displaying the 3D image of the 3D model. The display control unit 435 Figure 34 Similarly, thumbnail images IM11, IM12, and IM13 are displayed in the region R20. For example, the user designates the thumbnail image IM13. At this time, the image selection unit 433 selects the reference frame corresponding to the thumbnail image IM13 in step S105. The estimation unit 434 generates a high-precision 3D model corresponding to the reference frame in step S182. The display control unit 435 displays the 3D image MI30 of the high-precision 3D model in the region R30 on the dialog box A10 in step S183.

[0476] The image display method according to each aspect of the present invention includes a storage step, a first selection step, a second selection step, an estimation step, and a display step. In the storage step (step S104), the storage control unit 436 stores a 3D model (3D data) generated based on two or more 2D images of a subject in a memory 47 (storage medium). The 3D model includes 3D coordinates of two or more points of the subject and first camera coordinates. The first camera coordinates are 3D coordinates of a first camera that has acquired each of the two or more 2D images, and are associated with each of the two or more 2D images. In the first selection step (step S105), the image selection unit 433 selects a reference frame (reference image) as a 2D image of the subject. In the second selection step (step S106), the image selection unit 433 selects at least one 2D image from the two or more 2D images as a selection frame (selected image) based on the reference frame. In the estimation step (step S182), the estimation unit 434 estimates 3D coordinates of one or more points in a region of the subject captured in the reference frame based on second camera coordinates of a second camera that has acquired the reference frame, second pose information of the second camera, and the selection frame. The one or more points in this region are different from the points having 3D coordinates included in the 3D model. In the display step (step S183), the display control unit 435 causes the display unit 45 (display) to display a 3D image MI30 in which the 3D coordinates of the one or more points in this region are visualized and the 3D coordinates of the one or more points included in the 3D coordinates included in the 3D model and included in this region are visualized. The density of the points having 3D coordinates visualized by the 3D image MI30 is higher than the density of the points included in the two or more points of the subject and included in this region.

[0477] In a modification of the fourth embodiment, instead of displaying a sparse 3D model of the entire inspection object, the PC 41 displays a high-precision 3D model of a region corresponding to the region of interest of the 3D model. Therefore, compared with the case of simultaneously displaying a sparse 3D model and a high-precision 3D model, the PC 41 can efficiently use the display screen of the display unit 45 to display the high-precision 3D model.

[0478] (Fifth Embodiment)

[0479] The fifth embodiment of the present invention will be described. In the fifth embodiment, the PC 41 shown in Figure 1 is used.

[0480] In the fifth embodiment, the PC 41 generates a 3D model by using a reference frame including a region of interest as a key frame. Before generating the 3D model, the image selection unit 433 selects a reference frame. When the 3D model generation unit 432 generates a 3D model, the 3D model generation unit 432 estimates the camera coordinates and pose of the camera that acquired the reference frame.

[0481] Figure 37 Shows the process of the processing performed by the PC 41. The description of the processing same as that Figure 2 shown is omitted.

[0482] After step S102, the image selection unit 433 selects one still image from two or more still images included in the still image group as a reference frame in step S105. For example, the image selection unit 433 determines the time when the still image was recorded by reading the header of the moving image file in the same manner as in the first embodiment. The image selection unit 433 selects the still image recorded at this time as the reference frame. In the case where the PC 41 ​ has the same function of reproducing a moving image as the PC 41c shown, the user can also specify a specific frame of the moving image during the reproduction of the moving image. The image selection unit 433 may also select the frame specified by the user as the reference frame.

[0483] After step S105, the 3D model generation unit 432 generates a 3D model of the subject based on two or more still images included in the still image group and the conditions for generating the 3D model (step S191). The two or more still images used for generating the 3D model include the reference frame.

[0484] Explains the process of the specific processing performed by the 3D model generation unit 432. ​ Shows the process of the processing performed by the 3D model generation unit 432.

[0485] The 3D model generation unit 432 reads out one still image included in the still image group from the memory 47 (step S191a). Step S191a is executed the same number of times as the number of still images included in the still image group. The 3D model generation unit 432 reads out a still image different from the already read still image from the memory 47 in step S191a. The two or more still images included in the still image group may also be arranged in order in advance according to a time stamp or the like. The 3D model generation unit 432 may also read out the still images from the memory 47 in this order in step S191a.

[0486] After step S191a, the 3D model generation unit 432 detects the feature points of the still image read out in step S191a (step S191b).

[0487] After step S191b, the 3D model generation unit 432 correlates the feature points of each of the read still images with the feature points detected in step S191b (step S191c). When step S191a is only executed once, step S191c is not executed.

[0488] After step S191c, the 3D model generation unit 432 determines whether all the images included in the still image group have been read out (step S191d). When the 3D model generation unit 432 determines in step S191d that some of the images have not been read out, step S191a is executed. Steps S191a to S191d are repeatedly executed until all the images included in the still image group have been processed.

[0489] When the 3D model generation unit 432 determines in step S191d that all the images included in the still image group have been read out, the 3D model generation unit 432 selects one or more key frames for generating a 3D model from the still image group (step S191e).

[0490] Details of step S191e will be described. The 3D model generation unit 432 selects key frames that satisfy a specified criterion. For example, the 3D model generation unit 432 selects an image without blur or a bright image as a key frame. In addition, the 3D model generation unit 432 confirms that the amount of activity between two key frames is equal to or more than a specified amount. Alternatively, when the reference frame selected in step S105 satisfies the conditions for a key frame, the 3D model generation unit 432 selects two or more key frames including the reference frame in step S191e.

[0491] After step S191e, the 3D model generation unit 432 adds the reference frame selected in step S105 as a key frame (step S191f). When two or more key frames including the reference frame are selected in step S191e, step S191f is not executed.

[0492] After step S191f, the 3D model generation unit 432 estimates the camera coordinates of the camera that acquired each key frame and the pose of the camera based on the 2D coordinates of the feature points of two or more key frames (step S191g).

[0493] After step S191g, the 3D model generation unit 432 calculates the 3D coordinates of each feature point (step S191h). Steps S191g and S191h are the same as step S103c shown in ​ When step S191h is executed, the processing shown in ​ ends.

[0494] Refer again to ​To illustrate the process of the processing performed after generating the 3D model. After step S191, step S104 is executed. After step S104, step S109 is executed.

[0495] The 3D model generation unit 432 calculates the 3D coordinates of the region of the 3D model corresponding to the reference frame in ​ the step S191h shown. Thus, the 3D model generation unit 432 estimates the position on the 3D model corresponding to the region of interest. The estimation unit 434 associates the 3D coordinates of this region with the 3D model in step S109.

[0496] In addition to ​ the processing shown, ​ the steps S121 to S123 shown may also be executed. For example, the steps S121 to S123 may be executed after step S110.

[0497] In addition to ​ the processing shown, ​ the steps S131 to S133 shown may also be executed. For example, the steps S131 to S133 may be executed after step S110.

[0498] In addition to ​ the processing shown, ​ the steps S141 and S142 shown may also be executed. For example, the steps S141 and S142 may be executed after step S109. The steps S142 may also be executed ​ the step S110a shown to replace step S110.

[0499] In addition to ​ the processing shown, ​ the steps S151 to S154 shown may also be executed. For example, the step S151 may be executed between step S102 and step S103. The steps S152 to S154 may also be executed to replace steps S109 and S110.

[0500] In addition to ​ the processing shown, ​ the steps S161 to S167 shown may also be executed. For example, the steps S161 to S167 may be executed after step S110.

[0501] In addition to ​ the processing shown, ​Step S171 shown above. For example, step S171 may also be executed between step S101 and step S102. Alternatively, step S171 may also be executed between step S102 and step S105. The region detection unit 441 detects a region of interest in the image in step S171. The image selection unit 433 may also select a reference frame including the region of interest detected by the region detection unit 441 in step S105.

[0502] In addition to ​ the processing shown above, ​ steps S181 to S183 shown above may also be executed. For example, steps S181 to S183 may also be executed after step S110.

[0503] The display control unit 435 may also display the number of blades of the gas turbine in step S110 by using the method shown in the second modification of the first embodiment.

[0504] The image display method according to each aspect of the present invention includes a selection step, an estimation step, and a display step. In the selection step (step S105), the image selection unit 433 selects a reference frame (reference image) that is a 2D image of the subject. In the estimation step (step S191), the 3D model generation unit 432 estimates the 3D coordinates and camera coordinates of two or more points of the subject based on two or more 2D images of the subject including the reference frame. The camera coordinates are the 3D coordinates of the camera that has acquired each 2D image among the two or more 2D images and are associated with each 2D image among the two or more 2D images. In the display step (step S110), the display control unit 435 causes the display unit 45 (display) to display a 3D image in which at least one of the camera coordinates of the camera that has acquired the reference frame and the 3D coordinates of one or more points of the subject calculated based on the camera coordinates is visualized and the 3D coordinates of one or more points among the two or more points of the subject are visualized.

[0505] In the fifth embodiment, before generating the 3D model, the PC 41 adds the reference frame including the region of interest as a key frame. The PC 41 generates the 3D model by using two or more key frames including the reference frame. When the PC 41 finishes generating the 3D model, the PC 41 can obtain the 3D coordinates of the region of the 3D model corresponding to the reference frame.

[0506] (Sixth Embodiment)

[0507] Describe the sixth embodiment of the present invention. In the sixth embodiment, two devices cooperate. The first device receives an instruction from the user to execute a process and notifies the user of the process result. The first device executes a process related to the display of an image or the like. The second device executes a process related to the generation of a 3D model or the like. For example, the first device has excellent portability, but the computing resources of the first device are small. The second device has poor portability, but the second device has rich computing resources. In the sixth embodiment, the advantages of the two devices can be exerted. There is the following situation: In a site where it is difficult to carry a large device or the like, a process using an image is required. In this situation, the sixth embodiment is effective.

[0508] ​ Show the structure of the image display system 6 based on the sixth embodiment. ​ The shown image display system 6 has a PC 41e and a 3D model generation device 7.

[0509] ​ The shown PC 41 is changed to ​ The shown PC 41e. ​ Show the structure of the PC 41e. The description of the structure identical to that ​ shown is omitted.

[0510] The communication unit 46 communicates with the 3D model generation device 7. For example, the communication unit 46 is connected to the 3D model generation device 7 through a cable or wirelessly. The communication between the communication unit 46 and the 3D model generation device 7 may also be executed via a LAN (Local Area Network) or the Internet.

[0511] ​ The shown CPU 43 is changed to CPU 43e. The functions of the CPU 43e include an image acquisition unit 430, a condition reception unit 431, an image selection unit 433, an estimation unit 434, a display control unit 435, a storage control unit 436, and a 3D model acquisition unit 442. ​ At least one module in the shown CPU 43e may also be constituted by a circuit different from the CPU 43e.

[0512] Each unit in the CPU 43e may also be constituted by at least one of a processor and a logic circuit. Each unit in the CPU 43e can include one or more processors. Each unit in the CPU 43e can include one or more logic circuits.

[0513] ​The 3D model generation unit 432 shown is changed to a 3D model acquisition unit 442. The 3D model acquisition unit 442 controls the communication unit 46 to communicate with the 3D model generation device 7. The 3D model acquisition unit 442 sends a group of still images for generating a 3D model and the conditions for generating a 3D model to the 3D model generation device 7. The 3D model acquisition unit 442 acquires a 3D model by receiving the 3D model from the 3D model generation device 7.

[0514] The 3D model generation device 7 receives a group of still images for generating a 3D model and the conditions for generating a 3D model from the PC 41e. The 3D model generation device 7 generates a 3D model of the subject based on two or more still images included in the group of still images and the conditions for generating a 3D model. The 3D model generation device 7 sends the generated 3D model to the PC 41e.

[0515] ​ Shows the process of the processing executed by the PC 41e. The description of the same processing as ​ the processing shown is omitted.

[0516] After step S102, the 3D model acquisition unit 442 controls the communication unit 46 to send the group of still images acquired in step S101 and the conditions received in step S102 to the 3D model generation device 7 (step S201).

[0517] After step S201, the 3D model acquisition unit 442 controls the communication unit 46 to receive the 3D model from the 3D model generation device 7 (step S202). After step S202, step S104 is executed.

[0518] In the above example, multiple processes are distributed between two devices. The number of devices is not limited, and the processes executed by each device are not limited. For example, it may be that multiple 3D model generation devices 7 execute time-consuming processes.

[0519] For example, after step S201, the PC 41e may also execute step S105 instead of steps S202 and S104. After step S105, the PC 41e may also send the reference frame selected in step S105 and the selection frame selected in step S106 to the 3D model generation device 7. The 3D model generation device 7 may also receive the reference frame and the selection frame from the PC 41e and perform the same processing as steps S107, S108, and S109. Alternatively, the 3D model generation device 7 may send the 3D model to the PC 41e, and the PC 41e may receive the 3D model from the 3D model generation device 7. The PC 41e may also display a 3D image of the 3D model on the display unit 45 in step S110 and display a region including points having 3D coordinates calculated by the 3D model generation device 7 on the 3D image.

[0520] In the sixth embodiment, multiple devices cooperate to execute processing. At least one of the multiple devices has the characteristics of excellent portability but small computing resources. At least one of the multiple devices has the characteristics of poor portability but rich computing resources. The image display system 6 can display a 3D model by leveraging the advantages of multiple devices.

[0521] (Seventh Embodiment)

[0522] The seventh embodiment of the present invention will be described. Sometimes, the images used to generate 3D models contain noise. Due to the influence of this noise, it is possible to generate a 3D model that does not accurately match the original shape of the subject. This noise is caused by image blur, inappropriate image brightness, or motion blur, etc. In the case where the image is very bright or very dark, the image brightness is inappropriate. Therefore, compared with the case of directly displaying the 3D model, in the case of displaying an image of reference data generated using CAD or the like, the user can more easily confirm the shape of the subject.

[0523] The reference data is not associated with the range of the subject captured in the still image recorded during the inspection. Therefore, in the case of only displaying the reference data, the user cannot confirm the range of the subject captured in the still image recorded during the inspection. The seventh embodiment solves this problem. The example of using 3D design data such as 3D-CAD as reference data corresponds to the sixth modification of the first embodiment described above. Below, an example of using 2D design data such as 2D-CAD as reference data will be described.

[0524] ​ The shown PC 41 is changed to ​ the shown PC 41f. ​ The structure of the PC 41f is shown. The PC 41f displays a 2D image of the reference data instead of the 3D image of the 3D model. The description related to​ Description of the same structure shown.

[0525] ​ The CPU 43 shown is changed to CPU 43f. The functions of CPU 43f include an image acquisition unit 430, a condition acceptance unit 431, a 3D model generation unit 432, an image selection unit 433, an estimation unit 434, a display control unit 435, a storage control unit 436, a data acceptance unit 438, a data processing unit 439, and a 2D data generation unit 443. ​ At least one module within the shown CPU 43f may also be constituted by a circuit different from CPU 43f.

[0526] Each unit within CPU 43f may also be constituted by at least one of a processor and a logic circuit. Each unit within CPU 43f can include one or more processors. Each unit within CPU 43f can include one or more logic circuits.

[0527] The memory 47 stores reference data generated by the PC 41f or an external device. The reference data shows the two-dimensional shape (2D shape) of the subject. The user inputs information for specifying the reference data to the PC 41f by operating the operation unit 44. The data acceptance unit 438 accepts this information from the operation unit 44.

[0528] The 2D data generation unit 443 transforms the 3D model into 2D data. The 2D data includes 2D coordinates corresponding to the 3D coordinates of two or more points of the subject. In addition, the 2D data includes 2D coordinates corresponding to the camera coordinates (first camera coordinates). The camera coordinates are the 3D coordinates of the camera (first camera) that acquires each of the two or more 2D images used to generate the 3D model and are associated with each of the two or more 2D images. The 2D data generation unit 443 calculates the 2D coordinates of the points having the 3D coordinates included in the 3D model. The 2D data generation unit 443 generates 2D data (2D image) including the calculated 2D coordinates.

[0529] The data processing unit 439 associates the 2D coordinates in the 2D data with the 2D shape shown by the reference data. Thereby, the data processing unit 439 correlates the position of the 2D data with the position of the reference data.

[0530] ​ Shows the process of the processing executed by the PC 41f. The description of the same processing as that ​ shown is omitted.

[0531] After step S109, the user operates the operation unit 44 to input information for specifying reference data to the PC 41f. The data reception unit 438 receives this information from the operation unit 44 (step S211).

[0532] After step S211, the 2D data generation unit 443 transforms the 3D model into 2D data of the subject observed from a specific viewpoint (step S212). At this time, a well-known technique for transforming a 3D model into 2D data can be applied. This technique is perspective projection or parallel projection, etc. The technique for transforming a 3D model into 2D data is not limited to the above techniques. The CPU 43f can also select the above-mentioned viewpoint according to the setting information stored in advance in the memory 47. Alternatively, the user can select the above-mentioned viewpoint by operating the operation unit 44.

[0533] The 3D coordinates of each point among two or more points of the subject are transformed into 2D coordinates in step S212. In addition, the camera coordinates are transformed into 2D coordinates in step S212. The 2D data generation unit 443 associates the 3D coordinates of each point among two or more points of the subject and the camera coordinates with the 2D coordinates in the 2D data. The storage control unit 436 stores the 2D data generated by the 2D data generation unit 443 in the memory 47.

[0534] The 3D coordinates calculated in step S108 are associated with the 3D model in step S109. When step S212 is executed, the 3D coordinates calculated in step S108 are associated with the 2D data. That is, the 3D coordinates calculated in step S108 are associated with the 2D coordinates in the 2D data.

[0535] After step S212, the data processing unit 439 reads out the 2D data generated in step S212 and the reference data indicated by the information received in step S211 from the memory 47. The data processing unit 439 correlates the position of the 2D data with the position of the reference data (step S213).

[0536] Details of step S213 will be described. When the data format of the 2D data is different from the data format of the reference data, the data processing unit 439 performs a process of unifying the data format of the 2D data and the data format of the reference data.

[0537] After that, the data processing unit 439 correlates the position of the 2D data with the position of the reference data by using a well-known position alignment technique. The data processing unit 439 can also correlate the position of the 2D data with the position of the reference data by detecting the position of a unique shape in the 2D shape of the subject.

[0538] After step S213, the estimation unit 434 associates the 3D coordinates calculated in step S108 with the reference data (step S214). The 3D coordinates calculated in step S108 are associated with the 2D coordinates in the 2D data. The positions of the 2D data and the reference data are associated with each other. The estimation unit 434 uses this relationship in step S214 to associate the 3D coordinates with the reference data.

[0539] After step S214, the display control unit 435 causes the display unit 45 to display a 2D image of the reference data. At this time, the display control unit 435 displays, on the 2D image, a region including the point having the 3D coordinates calculated in step S108. Thereby, the display control unit 435 displays, on the 2D image, a region corresponding to the region of interest on the 3D model (step S215). The user can confirm the position corresponding to the reference frame as the region of interest on the 3D model by confirming the 2D image of the reference data. The display control unit 435 may also display, on the 2D image, a region corresponding to the region of interest on the 3D model and display the 2D coordinates corresponding to the camera coordinates on the 2D image. Alternatively, the display control unit 435 may not display, on the 2D image, a region corresponding to the region of interest on the 3D model but display the 2D coordinates corresponding to the camera coordinates on the 2D image. When step S215 is executed, ​ the processing shown ends.

[0540] ​ An example of the display screen of the display unit 45 is shown. The description of the parts identical to the ​ parts shown is omitted.

[0541] The 2D data generation unit 443 generates, in step S212, 2D data of the 3D shape observed from a first viewpoint in the coordinate system of the 3D model in a preset direction. The display control unit 435 causes a 2D image DI10 of the 2D data to be displayed in the region R10 in step S215.

[0542] The display control unit 435 displays, in step S215, information indicating the position corresponding to the still image on the 3D model on the 2D image DI10. Specifically, the display control unit 435 displays the string CH11, the string CH12, and the string CH13 on the 2D image DI10, and displays the line L11, the line L12, and the line L13 on the 2D image DI10. In addition, the display control unit 435 displays the thumbnail image IM11, the thumbnail image IM12, and the thumbnail image IM13 on the 2D image DI10.

[0543] ​ Another example of the display screen of the display unit 45 is shown. The description of the parts identical to the ​ parts shown is omitted.

[0544] The 2D data generation unit 443 generates 2D data of the 3D shape observed from the second viewpoint in the coordinate system of the 3D model in a preset direction in step S212. The second viewpoint is different from the above-mentioned first viewpoint. The display control unit 435 causes the 2D image DI11 of the 2D data to be displayed in the region R10 in step S215. The display control unit 435 displays information indicating the position on the 3D model corresponding to the still image on the 2D image DI11 in step S215. This information includes the string CH11, the string CH12, the string CH13, the line L11, the line L12, the line L13, the thumbnail image IM11, the thumbnail image IM12, and the thumbnail image IM13.

[0545] The 2D data generation unit 443 may also generate 2D data of the 3D shape observed from the first viewpoint and 2D data of the 3D shape observed from the second viewpoint in step S212. The display control unit 435 may also cause the display unit 45 to display the 2D image DI10 and the 2D image DI11 in step S215. The display control unit 435 may also display information indicating the position on the 3D model corresponding to the still image on the 2D image DI10 and the 2D image DI11 in step S215.

[0546] Even when the first position and the second position that are different from each other in the 3D shape observed from the first viewpoint overlap each other, the first position and the second position in the 3D shape observed from the second viewpoint do not overlap each other. The user can easily confirm the position on the 3D model corresponding to the reference frame.

[0547] An example of using the 3D model obtained in the gas turbine inspection is described. The gas turbine has two or more blades and two or more fuel nozzles. The two or more blades and the two or more fuel nozzles are arranged in a circular ring shape. The 2D data generation unit 443 generates 2D data of the 3D shape of the gas turbine observed from a specific viewpoint in a direction parallel to the rotation axis of the gas turbine in step S212. The 2D data generation unit 443 may also generate 2D data of the cross section by using the data of the 3D model at the cross section perpendicular to the rotation axis of the gas turbine.

[0548] In the following example, a 3D model having a structure with two or more blades is used. The following method can also be applied to a 3D model having a structure with two or more fuel nozzles.

[0549] ​ An example of the display screen of the display unit 45 is shown. The description of the part identical to the ​ part shown is omitted.

[0550] The display control unit 435 causes a 2D image DI12 of 2D data to be displayed in the region R10 in step S215. Twelve blades BL11 are arranged on the disk DS10. In fact, dozens of blades or more than 100 blades are arranged on one disk. The center position CP10 represents the center of the disk DS10 on a plane perpendicular to the rotation axis.

[0551] The display control unit 435 displays information indicating positions on the 3D model corresponding to the still images on the 2D image DI12 in step S215. Specifically, the display control unit 435 displays the string CH16 and the string CH17 on the 2D image DI12, and displays the line L16 and the line L17 on the 2D image DI12. In addition, the display control unit 435 displays the thumbnail image IM16 and the thumbnail image IM17 on the 2D image DI12.

[0552] Each string shows the file name of each still image. The string CH16 shows the file name img06. The string CH17 shows the file name img07. Each line represents the position corresponding to each still image. The line L16 shows the position corresponding to the still image with the file name img06. The line L17 shows the position corresponding to the still image with the file name img07. Each position corresponds to the 3D coordinates calculated based on the camera coordinates and pose information of the camera that acquired the reference frame in step S108. The thumbnail image IM16 is a thumbnail image of the still image with the file name img06. The thumbnail image IM17 is a thumbnail image of the still image with the file name img07.

[0553] The 2D data generation unit 443 may also generate 2D data of the 3D shape of the gas turbine observed from a viewpoint different from the above-mentioned viewpoint in step S212. The display control unit 435 may also cause the display unit 45 to display two 2D images of two 2D data in step S215. The display control unit 435 may also display information indicating positions on the 3D model corresponding to the still images on these two 2D images in step S215.

[0554] The order of the processes executed by the PC 41f is not limited to ​ the order shown. For example, the data reception unit 438 may receive information for specifying reference data at any time before the data processing unit 439 processes the reference data. Therefore, step S211 may be executed at any time before executing step S213.

[0555] In the case where conditions for using prescribed reference data are set in advance for the PC 41f, the data reception unit 438 does not need to receive information for specifying reference data. Therefore, the CPU 43f does not need to have the data reception unit 438, and step S211 is not necessary.

[0556] Alternatively, the estimation unit 434 may not execute step S108 and step S109, the data processing unit 439 may not execute step S214, and the display control unit 435 may display 2D coordinates corresponding to the camera coordinates on the 2D image of the reference data. Since the camera coordinates are displayed on the 2D image, the user can confirm the position of the viewpoint of the reference frame. Therefore, step S108, step S109, and step S214 are not necessary.

[0557] In the first to sixth embodiments including various modification examples, the display control unit 435 may also cause the display unit 45 to display a 2D image of the reference data instead of the 3D image of the 3D model.

[0558] Each mode of the present invention may also include the following modification example. In the display step (step S215), the display control unit 435 causes the display unit 45 to display a 2D image in which at least one of the 2D coordinates corresponding to the second camera coordinates and the 3D coordinates of one or more points of the subject calculated based on the second camera coordinates is visualized, and the 2D coordinates corresponding to the 3D coordinates of one or more points among two or more points of the subject are visualized.

[0559] Each mode of the present invention may also include the following modification example. The 3D data further includes first pose information indicating the pose of the first camera. The first pose information is associated with each 2D image among two or more 2D images. In the estimation step (step S107), the estimation unit 434 also estimates the pose of the second camera based on the first pose information associated with the selected frame, and generates second pose information indicating the estimated pose. In the estimation step (step S108), the estimation unit 434 also estimates the 3D coordinates of one or more points in the region of the subject captured in the reference frame based on the second camera coordinates and the second pose information. The 2D coordinates corresponding to the 3D coordinates of one or more points in the region of the subject captured in the reference frame are visualized in the 2D image.

[0560] Each mode of the present invention may also include the following modification example. In the display step (step S215), the display control unit 435 also superimposes information indicating the position of the region captured in the reference frame on the 2D image.

[0561] Each mode of the present invention may also include the following modification example. In the generation step (step S212), the 2D data generation unit 443 generates a 2D image by using the 3D model.

[0562] Each aspect of the present invention may also include the following modification examples. In the generation step (step S212), the 2D data generation unit 443 generates a 2D image of the subject observed from the first viewpoint, i.e., the first image, and generates a 2D image of the subject observed from a second viewpoint different from the first viewpoint, i.e., the second image. In the display step (step S215), the display control unit 435 causes the display unit 45 to display the first image and the second image.

[0563] Each aspect of the present invention may also include the following modification examples. The 2D image is an image of shape data representing the 2D shape of the subject. In the association step (step S214), the data processing unit 439 associates the 2D coordinates corresponding to the 3D coordinates of two or more points of the subject with the 2D shape shown by the shape data.

[0564] In the seventh embodiment, the PC 41f displays a 2D image of the subject on the display unit 45. In this 2D image, the 2D coordinates corresponding to at least one of the camera coordinates of the camera that acquired the reference frame and the 3D coordinates calculated based on the camera coordinates are visualized, and the 2D coordinates corresponding to the 3D coordinates of one or more of two or more points of the subject are visualized. Thus, the PC 41f can visualize the positions having 3D coordinates associated with at least a part of the 2D image (reference frame) of the subject.

[0565] The user can confirm the position corresponding to the region of interest on the 3D model. Since the 2D image is an image of reference data such as 2D-CAD, the user can confirm the original shape of the subject.

[0566] (Modification example of the seventh embodiment)

[0567] A modification example of the seventh embodiment of the present invention will be described. In the modification example of the seventh embodiment, the PC 41f shown ​ is used. Instead of using reference data, the PC 41f transforms the 3D model into 2D data and displays an image of the 2D data. In the modification example of the seventh embodiment, the PC 41f does not need to have ​ the data reception unit 438 shown.

[0568] In a modification of the seventh embodiment, an example of using an inspection report is described. After the inspection is completed, the user creates an inspection report. For example, information indicating the overall positional relationship of the object to be inspected is attached to the inspection report. Since the amount of data of the 3D model is large and the method of processing the 3D model is complex, the 3D model does not need to be attached to the inspection report. The amount of data of the 2D data is smaller than that of the 3D model, and the method of processing the 2D data is simple. Therefore, the 2D data is suitable for the information to be attached to the inspection report. In addition, sometimes reference data cannot be obtained. In this case, the modification of the seventh embodiment is effective.

[0569] In the following example, the region of interest is the entire region of the reference frame. The region of interest may also be the region of a specific subject captured in a specific image.

[0570] ​ Shows the process of the processing executed by the PC 41f. The description of the same processing as that ​ shown is omitted.

[0571] After step S109, the 2D data generation unit 443 transforms the 3D model into 2D data of the object observed from a specific viewpoint in step S212.

[0572] After step S212, the data processing unit 439 attaches the 2D data to the inspection report (step S221). The storage control unit 436 stores the inspection report with the 2D data attached in the memory 47.

[0573] After step S221, the data processing unit 439 attaches the information indicating the 3D coordinates calculated in step S108 to the inspection report. That is, the data processing unit 439 attaches the information of the position corresponding to the region of interest on the 3D model to the inspection report (step S222).

[0574] Step S221 and step S222 may also be executed in any order. Therefore, step S222 may be executed, and then step S221 may be executed.

[0575] After step S222, the display control unit 435 causes the display unit 45 to display a 2D image of the 2D data. At this time, the display control unit 435 displays, on the 2D image, a region including the point having the 3D coordinates calculated in step S108. Thereby, the display control unit 435 displays, on the 2D image, the region corresponding to the region of interest on the 3D model (step S223). The user can confirm the position corresponding to the reference frame, which is the region of interest, on the 3D model by confirming the 2D image of the 2D data. The display control unit 435 displays, on the 2D image, the region corresponding to the region of interest on the 3D model and also displays the 2D coordinates corresponding to the camera coordinates on the 2D image. Alternatively, the display control unit 435 may not display, on the 2D image, the region corresponding to the region of interest on the 3D model but may display the 2D coordinates corresponding to the camera coordinates on the 2D image. When step S223 is executed, ​ The processing shown ends.

[0576] ​ An example of an inspection report is shown. ​ The inspection report IR10 shown includes 2D data DT10. The 2D data DT10 includes the same information as the information shown in the ​ region R10 shown. The 2D data DT10 includes a thumbnail image IM11, a thumbnail image IM12, and a thumbnail image IM13.

[0577] ​ An example of the information of the region of interest attached to the inspection report is shown. ​ The information INF10 shown includes information on three regions of interest. The information on each region of interest includes a thumbnail image of the region of interest and also includes abnormality information related to the abnormality in the region of interest. The abnormality information represents the inspection result and represents whether there is an abnormality in the region of interest. In the case where there is an abnormality in the region of interest, the abnormality information represents the severity of the abnormality.

[0578] In ​ the example shown, the thumbnail image IM11a is associated with the abnormality information AI11, the thumbnail image IM12a is associated with the abnormality information AI12, and the thumbnail image IM13a is associated with the abnormality information AI13. The thumbnail image IM11a is the same as the thumbnail image IM11. The thumbnail image IM12a is the same as the thumbnail image IM12. The thumbnail image IM13a is the same as the thumbnail image IM13. The scale of the thumbnail images included in the 2D data DT10 and the scale of the thumbnail images included in the information on the region of interest may also be different from each other. The aspect ratio of the thumbnail images included in the 2D data DT10 and the aspect ratio of the thumbnail images included in the information on the region of interest may also be different from each other.

[0579] Each still image (reference frame) is pre-associated with each piece of abnormality information. In step S222, the data processing unit 439 attaches the thumbnail image IM11a, the thumbnail image IM12a, and the thumbnail image IM12a to the inspection report. At this time, the data processing unit 439 associates the thumbnail image IM11a with the abnormality information AI11, associates the thumbnail image IM12a with the abnormality information AI12, and associates the thumbnail image IM13a with the abnormality information AI13. In addition, the data processing unit 439 associates the thumbnail image IM11a with the thumbnail image IM11, associates the thumbnail image IM12a with the thumbnail image IM12, and associates the thumbnail image IM13a with the thumbnail image IM13.

[0580] For example, in step S223, the display control unit 435 causes the display unit 45 to display the inspection report IR10. Thereby, the display control unit 435 causes the display unit 45 to display a 2D image of the 2D data DT10. For example, the user operates the operation unit 44 to confirm whether an abnormality is captured in the still image or to confirm the severity of the abnormality captured in the still image. Thereby, the user inputs an instruction for selecting the thumbnail image IM11 to the PC 41f. The CPU 43f receives this instruction from the operation unit 44. Each thumbnail image corresponds to a still image (reference frame) that is the region of interest. The above instruction indicates the reference frame corresponding to the thumbnail image IM11.

[0581] Based on the above instruction, the display control unit 435 causes the display unit 45 to display the information INF10. At this time, the display control unit 435 encloses the thumbnail image IM11a and the abnormality information AI11 with the line L60. Thereby, the display control unit 435 emphasizes the information associated with the reference frame indicated by the instruction input to the PC41f. The user can confirm whether an abnormality is captured in the still image or the severity of the abnormality. As long as the user can easily confirm the information corresponding to the selected thumbnail image, any method can be used to emphasize this information.

[0582] When the information INF10 is displayed, the display control unit 435 may also make the inspection report IR10 non-displayed. The display control unit 435 may also cause the display unit 45 to display the inspection report IR10 and the information INF10. When an instruction for selecting the thumbnail image IM11 is input to the PC 41f, the display control unit 435 may also extract the thumbnail image IM11a and the abnormality information AI11 from the information INF10 and cause the display unit 45 to display the thumbnail image IM11a and the abnormality information AI11.

[0583] When the information INF10 is displayed, the user can also select a thumbnail image. For example, the user operates the operation unit 44 to input an instruction for selecting the thumbnail image IM11a to the PC 41f. The CPU 43f accepts this instruction from the operation unit 44. The display control unit 435 causes the display unit 45 to display the inspection report IR10 based on this instruction, and emphasizes the thumbnail image IM11.

[0584] The information selected by the user is not limited to the thumbnail image. For example, the user can also select the file name of a still image (such as img01, etc.).

[0585] Alternatively, the estimation unit 434 may not execute step S108 and step S109, the data processing unit 439 may not execute step S222, and the display control unit 435 may display the 2D coordinates corresponding to the camera coordinates on the 2D image of the 2D data. Since the camera coordinates are displayed on the 2D image, the user can confirm the position of the viewpoint of the reference frame. Therefore, step S108, step S109, and step S222 are not necessary.

[0586] The 2D data does not need to be attached to the inspection report. Therefore, step S221 and step S222 are not necessary.

[0587] In the first to sixth embodiments including various modification examples, the display control unit 435 may also cause the display unit 45 to display the 2D image of the 2D data instead of the 3D image of the 3D model.

[0588] Each aspect of the present invention may also include the following modification example. The reference frame is associated in advance with the information of the subject (abnormal information AI11). When the display step (step S223) is executed and an instruction for selecting the reference frame is input, the display control unit 435 causes the display unit 45 to display the information (abnormal information AI11) associated with the reference frame indicated by this instruction in the information display step (step S223).

[0589] In the modification example of the seventh embodiment, the PC 41f can transform the 3D model into 2D data having a form suitable for attachment to the inspection report. Therefore, the PC 41f can reduce the workload of the operations required to produce the inspection report.

[0590] ​ The information contained in the shown 2D data DT10 is correlated with ​ the shown information INF10. The user and the approver of the inspection report can confirm the position on the 3D model corresponding to the region of interest and the information of the subject at that position. Therefore, the user and the approver of the inspection report can easily confirm the status of the inspection.

[0591] (Supplementary Note)

[0592] In each aspect of the present invention, the display control unit 435 causes the display unit 45 (display) to display an image of the subject visualizing a position indicated by at least one of the second camera coordinates and the 3D coordinates of one or more points of the subject calculated based on the second camera coordinates. The image of the subject is a three-dimensional image or a two-dimensional image.

[0593] The preferred embodiments of the present invention have been described above, but the present invention is not limited to these embodiments and their modified examples. Additions, omissions, substitutions, and other changes can be made without departing from the gist of the present invention. In addition, the present invention is not limited by the foregoing description, but only by the scope of the appended claims.

[0594] Description of Reference Numerals

[0595] 1: Endoscope device; 2: Insertion portion; 3: Main body portion; 4, 44: Operation portion; 5, 45: Display portion; 6: Image display system; 7: 3D model generation device; 8: Endoscope unit; 9: CCU; 10: Control device; 12: Video signal processing circuit; 13: ROM; 14: RAM; 15: Card interface; 16: External device interface; 17: Control interface; 18, 43, 43a, 43b, 43c, 43d, 43e, 43f: CPU; 20: Front end; 28: Image pickup element; 41, 41a, 41b, 41c, 41d, 41e: PC; 42: Memory card; 46: Communication portion; 47: Memory; 430: Image acquisition portion; 431: Condition acceptance portion; 432: 3D model generation portion; 433: Image selection portion; 434: Estimation portion; 435: Display control portion; 436: Storage control portion; 437: Reference position acceptance portion; 438: Data acceptance portion; 439: Data processing portion; 440: Moving image control portion; 441: Region detection portion; 442: 3D model acquisition portion; 443: 2D data generation portion.

Claims

1. An image display method, comprising the following steps: A storage step, in which a storage control unit stores three-dimensional data generated based on two or more two-dimensional images of a subject into a storage medium. The three-dimensional data includes three-dimensional coordinates of two or more points of the subject and first camera coordinates. The first camera coordinates are the three-dimensional coordinates of a first camera that acquires each of the two or more two-dimensional images, and are associated with each of the two or more two-dimensional images; A first selection step, in which a selection unit selects a reference image as the two-dimensional image of the subject; A second selection step, in which the selection unit selects at least one two-dimensional image from the two or more two-dimensional images as a selected image based on the reference image; An estimation step, in which an estimation unit estimates the three-dimensional coordinates of a second camera, i.e., second camera coordinates, that acquires the reference image based on the first camera coordinates associated with the selected image; And A display step, in which a display control unit causes a display to display an image of the subject in which at least one of the second camera coordinates and the three-dimensional coordinates of one or more points of the subject calculated by projecting one or more points of the reference image into the three-dimensional data based on the second camera coordinates is visualized.

2. The image display method according to claim 1, wherein The image of the subject is a three-dimensional image, In the three-dimensional image, at least one of the second camera coordinates and the three-dimensional coordinates of one or more points of the subject calculated based on the second camera coordinates is visualized, and the three-dimensional coordinates of one or more of the two or more points are visualized.

3. The image display method according to claim 2, wherein The three-dimensional data further includes first pose information indicating the pose of the first camera, The first pose information is associated with each of the two or more two-dimensional images, In the estimation step, the estimation unit further estimates the pose of the second camera based on the first pose information associated with the selected image, and generates second pose information indicating the estimated pose of the second camera, In the estimation step, the estimation unit further estimates the three-dimensional coordinates of one or more points of the region of the subject captured in the reference image based on the second camera coordinates and the second pose information, In the three-dimensional image, the three-dimensional coordinates of the one or more points of the region are visualized.

4. The image display method according to claim 3, wherein In the display step, the display control unit further causes information indicating the position of the region to be superimposed on the three-dimensional image.

5. The image display method according to claim 3, wherein It further includes a viewpoint change step, in which after the three-dimensional image is displayed on the display, the display control unit changes the viewpoint of the three-dimensional image.

6. The image display method according to claim 3, wherein In the first selection step, the selection unit selects two or more of the reference images, In the display step, the display control unit also causes the display to display information indicating the position of the common area between the two or more reference images.

7. The image display method according to claim 3, wherein the three-dimensional image is an image of shape data representing the three-dimensional shape of the subject, the image display method further includes an association step in which the data processing unit associates the three-dimensional coordinates of two or more points of the subject with the three-dimensional shape shown by the shape data.

8. The image display method according to claim 3, wherein in the display step, the display control unit also causes the display to display viewing angle information indicating the viewing angle of the reference image.

9. The image display method according to claim 3, wherein in the display step, the display control unit also causes the display to display the three-dimensional distance between a reference point included in two or more points of the subject and the area, the reference point being a point corresponding to an input reference position.

10. The image display method according to claim 3, wherein the subject includes two or more local regions arranged periodically, in the display step, the display control unit also causes the display to display the number of at least a part of the local regions from a reference part to a part of interest, the reference part being one of the two or more local regions, and the part of interest being a local region different from the reference part and corresponding to the area of the subject captured in the reference image.

11. The image display method according to claim 3, wherein the area is the whole of the range captured in the reference image.

12. The image display method according to claim 3, wherein the area is a part of the range captured in the reference image.

13. The image display method according to claim 3, wherein in the estimation step, the estimation unit estimates the three-dimensional coordinates of one or more points of the area of the subject captured in the reference image based on the second camera coordinates, the second pose information, and the selected image, the one or more points of the area are different from the points having the three-dimensional coordinates included in the three-dimensional data, the three-dimensional coordinates of the one or more points of the area are visualized in the three-dimensional image, and the three-dimensional coordinates of one or more points included in the area among the three-dimensional coordinates included in the three-dimensional data are visualized, the density of the points having the three-dimensional coordinates visualized by the three-dimensional image is higher than the density of the points included in the two or more points of the subject and included in the area.

14. The image display method according to claim 3, wherein in the first selection step, the selection unit selects the reference image including a reference area that satisfies a preset condition, in the estimation step, the estimation unit estimates the three-dimensional coordinates of one or more points of the reference area.

15. The image display method according to claim 14, wherein, in the estimation step, the estimation unit estimates the three-dimensional coordinates of the one or more points of the reference region based on the second camera coordinates, the second pose information, and the selected image, the one or more points of the reference region are different from the points having the three-dimensional coordinates included in the three-dimensional data, the three-dimensional coordinates of the one or more points of the reference region are visualized in the three-dimensional image, and the three-dimensional coordinates of the one or more points included in the reference region among the three-dimensional coordinates included in the three-dimensional data are visualized, the density of the points having the three-dimensional coordinates visualized by the three-dimensional image is higher than the density of the points included in the reference region among the two or more points of the subject.

16. The image display method according to claim 3, wherein, the three-dimensional data further includes the two-dimensional coordinates of the first points included in each of the two or more two-dimensional images, in the estimation step, the estimation unit estimates the second camera coordinates based on the first camera coordinates, the two-dimensional coordinates of the first points included in one of the two or more two-dimensional images, and the two-dimensional coordinates of the second points corresponding to the first points in the reference image.

17. The image display method according to claim 3, wherein, in the first selection step, the selection unit selects one of the one or more selectable images as the reference image, the one or more selectable images are two-dimensional images of the subject and are not included in the two or more two-dimensional images.

18. The image display method according to claim 3, wherein, each of the two or more two-dimensional images is temporally associated with the other two-dimensional images included in the two or more two-dimensional images.

19. The image display method according to claim 18, wherein, each of the two or more two-dimensional images is an image acquired by a monocular camera.

20. The image display method according to claim 18, wherein, in the second selection step, the selection unit selects at least one of the two or more two-dimensional images as the selected image by using the time information that temporally associates the two or more two-dimensional images with each other.

21. The image display method according to claim 1, wherein, the image of the subject is a two-dimensional image, in the two-dimensional image, the two-dimensional coordinates corresponding to at least one of the second camera coordinates and the three-dimensional coordinates of one or more points of the subject calculated based on the second camera coordinates are visualized, and the two-dimensional coordinates corresponding to the three-dimensional coordinates of one or more points among the two or more points are visualized.

22. The image display method according to claim 21, wherein, the three-dimensional data further includes first pose information indicating the pose of the first camera, The first pose information is associated with each of the two-dimensional images among the two or more two-dimensional images. In the estimating step, the estimating unit further estimates the pose of the second camera based on the first pose information associated with the selected image, and generates second pose information representing the estimated pose. In the estimating step, the estimating unit further estimates the three-dimensional coordinates of one or more points in the region of the subject captured in the reference image based on the second camera coordinates and the second pose information. In the two-dimensional image, the two-dimensional coordinates corresponding to the three-dimensional coordinates of the one or more points in the region are visualized.

23. The image display method according to claim 22, wherein In the displaying step, the display control unit further superimposes information indicating the position of the region on the two-dimensional image.

24. The image display method according to claim 21, wherein It further includes a generating step in which a generating unit generates the two-dimensional image by using the three-dimensional data.

25. The image display method according to claim 24, wherein In the generating step, the generating unit generates a first image, which is a two-dimensional image of the subject observed from a first viewpoint, and generates a second image, which is a two-dimensional image of the subject observed from a second viewpoint different from the first viewpoint. In the displaying step, the display control unit causes the display to display the first image and the second image.

26. The image display method according to claim 21, wherein The reference image is pre-associated with information of the subject. The image display method further includes an information display step in which the displaying step is performed, and when an instruction for selecting the reference image is input, the display control unit causes the display to display the information associated with the reference image indicated by the instruction.

27. The image display method according to claim 21, wherein The two-dimensional image is an image of shape data representing the two-dimensional shape of the subject. The image display method further includes an associating step in which a data processing unit associates the two-dimensional coordinates corresponding to the three-dimensional coordinates of the two or more points of the subject with the two-dimensional shape shown by the shape data.

28. A display control device having: A storage control unit that stores three-dimensional data generated based on two or more two-dimensional images of a subject in a storage medium. The three-dimensional data includes the three-dimensional coordinates of two or more points of the subject and first camera coordinates, which are the three-dimensional coordinates of a first camera that has acquired each of the two or more two-dimensional images and are associated with each of the two or more two-dimensional images. A selection unit that selects a reference image as a two-dimensional image of the subject and selects at least one two-dimensional image among the two or more two-dimensional images as a selected image based on the reference image. An estimation unit that estimates three-dimensional coordinates of a second camera that has acquired the reference image, i.e., second camera coordinates, based on the first camera coordinates associated with the selected image; and A display control unit that causes a display to display an image of the subject in which at least one of the positions indicated by the second camera coordinates and the three-dimensional coordinates of one or more points of the subject calculated by projecting one or more points of the reference image into the three-dimensional data based on the second camera coordinates is visualized.

29. A computer-readable storage medium storing a program for causing a computer to execute the following steps: A storage step of storing three-dimensional data generated based on two or more two-dimensional images of a subject in a storage medium, the three-dimensional data including three-dimensional coordinates of two or more points of the subject and first camera coordinates, the first camera coordinates being three-dimensional coordinates of a first camera that has acquired each of the two or more two-dimensional images and being associated with each of the two or more two-dimensional images; A first selection step of selecting a reference image as a two-dimensional image of the subject; A second selection step of selecting at least one of the two or more two-dimensional images as a selected image based on the reference image; An estimation step of estimating three-dimensional coordinates of a second camera that has acquired the reference image, i.e., second camera coordinates, based on the first camera coordinates associated with the selected image; and A display step of displaying, on a display, an image of the subject in which at least one of the positions indicated by the second camera coordinates and the three-dimensional coordinates of one or more points of the subject calculated by projecting one or more points of the reference image into the three-dimensional data based on the second camera coordinates is visualized.

30. A computer program product comprising a program for causing a computer to execute the method according to any one of claims 1 to 27.

Images