Three-dimensional image display method, three-dimensional image display device, and program

Abstract

To provide a three-dimensional image display method, a three-dimensional image display device, and a program that allow a user to recognize a region that has been inspected and a region that has not been inspected even when a subject does not have a unique shape.SOLUTION: In a three-dimensional image display method, a processor acquires three-dimensional data of a subject including one or more structures. The processor acquires reference data including information indicating the one or more structures. The processor associates, on the basis of information on a common structure, the information included in the three-dimensional data with the information included in the reference data. The common structure indicates a structure of the subject common to at least part of the information included in the three-dimensional data and at least part of the information included in the reference data. The processor displays, on a display, at least part of the three-dimensional data and at least part of the reference data.SELECTED DRAWING: Figure 49

Description

[Technical field]

[0001] The present invention relates to a three-dimensional image display method, a three-dimensional image display device, and a program. [Background technology]

[0002] Industrial endoscope devices are used to inspect the inside of boilers, gas turbines, automobile engines, pipes, etc. for abnormalities (such as scratches and corrosion). In an endoscopic inspection, the inspector records still images during the inspection to leave evidence of the presence or absence of an abnormality and its severity. After the endoscopic inspection is completed, an inspection report is generated. Generally, text indicating the condition of the abnormality shown in the recorded still image is added to the inspection report along with the still image.

[0003] Location information is included in the anomaly attachment. The location information indicates the position on the inspection object where the recorded still image was taken. The location information of the discovered anomaly is important when replacing or repairing the anomalous part, or when the next inspection is performed. One method of recording and managing the location information of anomalies is to associate a still image recorded during an inspection with a specific position in three-dimensional data (3D data) that shows the three-dimensional shape (3D shape) of the inspection object, and to visualize the position where the still image was taken. Using such a method, the position of the inspection object where the still image was taken becomes clear.

[0004] One method for acquiring 3D data of an object to be inspected is to reconstruct the 3D shape of the object by using videos recorded during the inspection. This method uses videos recorded during the inspection. Therefore, there is no need to bring special inspection equipment to the inspection site, and no need for blueprints of the object to be inspected.

[0005] In a method for reconstructing the 3D shape of an object to be inspected, it is necessary to correlate multiple images acquired from multiple viewpoints. When it is difficult to correlate multiple images, there is a problem that the 3D data is divided into multiple data. For example, the tip of the endoscope may suddenly move during video recording, causing a significant change in the composition of the image. Alternatively, halation or the like may occur in the image during video recording, causing a significant change in the state of the image. Alternatively, video recording may be interrupted and then resumed, and the object to be inspected that appears in the image after the video recording is resumed may not match the object to be inspected that appears in the image before the video recording was interrupted. In these cases, the 3D data is divided into multiple data.

[0006] It is preferable for users that 3D data covering as wide an area as possible of the inspection target be configured as a single piece of data without being divided. When multiple pieces of 3D data for the inspection target are generated, it is important to connect the multiple pieces of 3D data to generate 3D data for the wide area of ​​the inspection target. The following technology has been disclosed as a method for generating 3D data for a wide area by connecting multiple pieces of 3D data for partial areas.

[0007] The technology disclosed in Patent Document 1 provides a method for connecting multiple 3D data by using a relationship between the 2D image group used to generate the 3D data and the 3D data. The first 3D data is associated with the first 2D image group, and the second 3D data is associated with the second 2D image group. A first image and a second image having overlapping feature points are selected. The first image is included in the first 2D image group, and the second image is included in the second 2D image group. Translation and rotation of the 3D data are performed so that the 3D coordinates of the feature points in the first image and the 3D coordinates of the feature points in the second image match. Then, an error in the 3D coordinates is minimized.

[0008] The techniques disclosed in Patent Document 2 and Patent Document 3 each provide a method for connecting multiple 3D data based on the 3D shape indicated by the 3D data. In the technique disclosed in Patent Document 2, a ground feature, a plane feature, or a cylinder feature is assigned to multiple 3D data of a partial area. The multiple 3D data are connected so that the features match. In the technique disclosed in Patent Document 3, three planes are detected in each of the multiple 3D data. The normals of the three planes are mutually orthogonal. The multiple 3D data are connected so that the three planes match. [Prior art documents] [Patent documents]

[0009] [Patent Document 1] Patent No. 6040882 [Patent Document 2] Patent No. 6811763 [Patent Document 3] JP 2010-66595 A Summary of the Invention [Problem to be solved by the invention]

[0010] The present invention aims to provide a three-dimensional image display method, a three-dimensional image display device, and a program that enable a user to understand inspected and uninspected areas even when the subject does not have a unique shape. [Means for solving the problem]

[0011] The present invention is a three-dimensional image display method comprising: a first acquisition step in which a processor acquires three-dimensional data of a subject including one or more structures, the three-dimensional data including information indicating the three-dimensional coordinates of at least a portion of the one or more structures; a second acquisition step in which the processor acquires reference data including information indicating the one or more structures; an association step in which the processor associates the information included in the three-dimensional data with the information included in the reference data based on information of a common structure, the common structure indicating a structure of the subject that is common to at least a portion of the information included in the three-dimensional data and at least a portion of the information included in the reference data; and a display step in which the processor displays at least a portion of the three-dimensional data and at least a portion of the reference data on a display.

[0012] In the three-dimensional image display method of the present invention, the information included in the reference data indicates an area of ​​the one or more structures.

[0013] In the three-dimensional image display method of the present invention, the reference data is a known three-dimensional model.

[0014] In the three-dimensional image display method of the present invention, the reference data is data representing a two-dimensional shape on a plane.

[0015] In the three-dimensional image display method of the present invention, the reference data is received from an external device.

[0016] In the three-dimensional image display method of the present invention, the subject has two or more structures including the one or more structures.

[0017] In the three-dimensional image display method of the present invention, the reference data includes identification information corresponding to each of the two or more structures.

[0018] In the three-dimensional image display method of the present invention, the information of the common structure indicates the identification information of the structure that is common to at least a portion of the information contained in the three-dimensional data and at least a portion of the information contained in the reference data.

[0019] In the three-dimensional image display method of the present invention, the object is a turbine, and the two or more structures are blades.

[0020] In the three-dimensional image display method of the present invention, the subject has one structure, and the structure has a continuous same shape.

[0021] In the three-dimensional image display method of the present invention, the three-dimensional coordinates are defined in a first three-dimensional coordinate system, and the reference data includes two-dimensional coordinates or three-dimensional coordinates defined in a second coordinate system different from the first three-dimensional coordinate system.

[0022] In the three-dimensional image display method of the present invention, the processor converts the first three-dimensional coordinate system and the second coordinate system into a common coordinate system in the associating step.

[0023] In the three-dimensional image display method of the present invention, in the displaying step, the processor displays the three-dimensional data and the reference data in the common coordinate system on the display.

[0024] The three-dimensional image display method of the present invention further includes, after the second acquisition step, an input step in which the processor accepts an input of the identification information corresponding to each of the two or more structures.

[0025] In the three-dimensional image display method of the present invention, the reference data includes identification information corresponding to each of the one or more structures, and the association step includes an identification step in which the processor identifies an area of ​​the one or more structures indicated by the information included in the three-dimensional data, and an assignment step in which the processor assigns identification information to the area identified in the identification step, and in the association step, the processor identifies the common structure corresponding to the identification information of the area and the identification information included in the reference data, and the identification information of the area and the identification information included in the reference data match each other.

[0026] In the three-dimensional image display method of the present invention, in the associating step, the processor corrects at least one of the position, orientation, and scale of the three-dimensional data based on the information on the common structure.

[0027] In the three-dimensional image display method of the present invention, in the display step, the processor displays the information contained in the three-dimensional data on the display, and also displays information of structures that correspond to the information contained in the reference data and do not correspond to the information contained in the three-dimensional data on the display.

[0028] In the three-dimensional image display method of the present invention, in the display step, the processor displays an image of the reference data on the display.

[0029] In the three-dimensional image display method of the present invention, in the displaying step, the processor displays the identification information corresponding to each of the two or more structures on the display.

[0030] The present invention is a three-dimensional image display device having a processor, which acquires three-dimensional data of a subject including one or more structures, the three-dimensional data including information indicating three-dimensional coordinates of at least a portion of the one or more structures, acquires reference data including information indicating the one or more structures, and associates the information included in the three-dimensional data with the information included in the reference data based on information of a common structure, the common structure indicating a structure of the subject that is common to at least a portion of the information included in the three-dimensional data and at least a portion of the information included in the reference data, and displays at least a portion of the three-dimensional data and at least a portion of the reference data on a display.

[0031] The present invention is a program for causing a computer to execute the following steps: a first acquisition step of acquiring three-dimensional data of a subject including one or more structures, the three-dimensional data including information indicating three-dimensional coordinates of at least a portion of the one or more structures; a second acquisition step of acquiring reference data including information indicating the one or more structures; an association step of associating the information included in the three-dimensional data with the information included in the reference data based on information of a common structure, the common structure indicating a structure of the subject that is common to at least a portion of the information included in the three-dimensional data and at least a portion of the information included in the reference data; and a display step of displaying at least a portion of the three-dimensional data and at least a portion of the reference data on a display. Effect of the Invention

[0032] According to the present invention, the three-dimensional image display method, the three-dimensional image display device, and the program enable the user to understand the areas that have been inspected and the areas that have not been inspected, even if the subject does not have a unique shape. [Brief description of the drawings]

[0033] [Figure 1] 1 is a block diagram showing a configuration of an image display device according to a first embodiment of the present invention. [Diagram 2] 5 is a flowchart showing a procedure of processing executed by the image display device according to the first embodiment of the present invention. [Diagram 3] FIG. 2 is a diagram showing an example of an image displayed on a display unit according to the first embodiment of the present invention. [Figure 4] FIG. 4 is a diagram showing an example of a dialog box displayed on a display unit according to the first embodiment of the present invention. [Diagram 5] FIG. 4 is a diagram illustrating an example of a reference table in the first embodiment of the present invention. [Figure 6] 5 is a flowchart showing a procedure of processing executed by the image display device according to the first embodiment of the present invention. [Figure 7] FIG. 2 is a diagram showing an example of an image displayed on a display unit according to the first embodiment of the present invention. [Figure 8] FIG. 2 is a diagram showing an example of an image displayed on a display unit according to the first embodiment of the present invention. [Figure 9] FIG. 1 is a block diagram showing a configuration of an image display device according to a first modified example of the first embodiment of the present invention. [Figure 10] 10 is a flowchart showing a procedure of processing executed by an image display device according to a first modified example of the first embodiment of the present invention. [Figure 11] FIG. 11 is a diagram showing an example of an image displayed on a display unit according to a first modified example of the first embodiment of the present invention. [Figure 12] FIG. 11 is a diagram showing an example of an image displayed on a display unit according to a second modified example of the first embodiment of the present invention. [Figure 13] FIG. 11 is a diagram showing an example of an image displayed on a display unit according to a second modified example of the first embodiment of the present invention. [Figure 14] FIG. 13 is a diagram showing an example of a dialog box displayed on a display unit according to a third modified example of the first embodiment of the present invention. [Figure 15] FIG. 13 is a diagram showing an example of an image displayed on a display unit according to a third modified example of the first embodiment of the present invention. [Figure 16] FIG. 13 is a diagram showing an example of a U-shaped tube in a fourth modified example of the first embodiment of the present invention. [Figure 17] FIG. 13 is a diagram showing an example of an image displayed on a display unit according to a fourth modified example of the first embodiment of the present invention. [Figure 18] FIG. 13 is a diagram showing a configuration of a blade in a fifth modified example of the first embodiment of the present invention. [Figure 19] 13 is a flowchart showing a procedure of processing executed by an image display device according to a fifth modified example of the first embodiment of the present invention. [Figure 20] FIG. 13 is a diagram showing an example of an image displayed on a display unit according to a fifth modified example of the first embodiment of the present invention. [Figure 21] FIG. 13 is a diagram showing an example of an image displayed on a display unit according to a fifth modified example of the first embodiment of the present invention. [Figure 22] FIG. 11 is a perspective view showing the overall configuration of an endoscope apparatus according to a second embodiment of the present invention. [Figure 23] FIG. 11 is a block diagram showing the internal configuration of an endoscope apparatus according to a second embodiment of the present invention. [Figure 24] 13 is a block diagram showing the functional configuration of a CPU included in an endoscope apparatus according to a second embodiment of the present invention. FIG. [Diagram 25] 10 is a block diagram showing the functional configuration of a data generating unit included in an endoscope apparatus according to a second embodiment of the present invention. FIG. [Figure 26] FIG. 11 is a schematic diagram showing a situation in which an image is acquired in a second embodiment of the present invention. [Figure 27] 10 is a flowchart showing a procedure of a process for generating 3D data according to a second embodiment of the present invention. [Figure 28] FIG. 11 is a block diagram showing a configuration of an image display device according to a third embodiment of the present invention. [Figure 29] 10 is a flowchart showing a procedure of processing executed by an image display device according to a third embodiment of the present invention. [Diagram 30] 13 is a flowchart showing a procedure of processing executed by an image display device according to a fourth embodiment of the present invention. [Diagram 31] FIG. 13 is a diagram showing an example of an image displayed on a display unit according to a fourth embodiment of the present invention. [Diagram 32] FIG. 13 is a diagram showing an example of an image displayed on a display unit according to a fourth embodiment of the present invention. [Diagram 33] FIG. 13 is a diagram showing reference data in a first modified example of the fourth embodiment of the present invention. [Diagram 34] 13 is a flowchart showing a procedure of processing executed by an image display device according to a first modified example of the fourth embodiment of the present invention. [Diagram 35] FIG. 13 is a diagram showing an example of an image displayed on a display unit according to a first modified example of the fourth embodiment of the present invention. [Diagram 36] FIG. 13 is a diagram showing a structure of a combustion chamber in a second modified example of the fourth embodiment of the present invention. [Figure 37] FIG. 13 is a diagram showing an example of an image displayed on a display unit according to a second modified example of the fourth embodiment of the present invention. [Figure 38] FIG. 13 is a block diagram showing a configuration of an image display device according to a fifth embodiment of the present invention. [Figure 39] 13 is a flowchart showing a procedure of processing executed by an image display device according to a fifth embodiment of the present invention. [Diagram 40] FIG. 13 is a diagram showing an example of an image displayed on a display unit according to a fifth embodiment of the present invention. [Diagram 41] FIG. 13 is a diagram showing an example of an image displayed on a display unit according to a fifth embodiment of the present invention. [Diagram 42] FIG. 13 is a diagram showing an example of an image displayed on a display unit according to a fifth embodiment of the present invention. [Diagram 43] FIG. 13 is a diagram showing an example of an image displayed on a display unit according to a fifth embodiment of the present invention. [Diagram 44] FIG. 13 is a diagram illustrating a method for calculating a missing section in the fifth embodiment of the present invention. [Diagram 45] FIG. 13 is a diagram showing the relationship between camera trajectory data and 3D data in the fifth embodiment of the present invention. [Diagram 46] FIG. 13 is a diagram showing a moving image file in the fifth embodiment of the present invention. [Figure 47]13 is a flowchart showing a procedure of processing executed by an image display device according to a modified example of the fifth embodiment of the present invention. [Figure 48] FIG. 13 is a diagram showing an example of an image displayed on a display unit according to a modified example of the fifth embodiment of the present invention. [Figure 49] 13 is a flowchart showing the procedure of a process executed by an image display device according to a sixth embodiment of the present invention. [Figure 50] 13 is a flowchart showing the procedure of a process executed by an image display device according to a sixth embodiment of the present invention. [Figure 51] FIG. 13 is a diagram showing an example of an image displayed on a display unit according to a sixth embodiment of the present invention. [Figure 52] FIG. 13 is a diagram showing an example of an image displayed on a display unit according to a sixth embodiment of the present invention. [Figure 53] FIG. 13 is a diagram showing a structure of a heat exchanger according to a sixth embodiment of the present invention. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0034] Hereinafter, an embodiment of the present invention will be described with reference to the drawings.

[0035] In the techniques disclosed in Patent Documents 1, 2, and 3, the data of a common area must be included in the multiple 3D data. If the data of a common area is not included in the multiple 3D data, the multiple 3D data cannot be connected.

[0036] In the technology disclosed in Patent Document 1, if halation or the like occurs in at least one of the first image and the second image, and the first image and the second image appear differently from each other, multiple 3D data cannot be connected.

[0037] The first to fifth embodiments of the present invention and their variations aim to provide a three-dimensional image display method, a three-dimensional image display device, and a program that are capable of displaying three-dimensional shapes represented by two or more three-dimensional data even if the two or more three-dimensional data do not contain data of a common area.

[0038] (First embodiment) Fig. 1 shows a configuration of an image display device 50 according to a first embodiment of the present invention. The image display device 50 shown in Fig. 1 has a control unit 51, a data acquisition unit 52, a parameter generation unit 53, a conversion unit 54, a data generation unit 55, a display control unit 56, an information reception unit 57, and a structure estimation unit 58. An operation unit 70, a display unit 71, a communication unit 72, and a storage unit 73 shown in Fig. 1 are connected to the image display device 50. The image display device 50 may have at least one of the operation unit 70, the display unit 71, the communication unit 72, and the storage unit 73.

[0039] For example, the image display device 50 is a PC (Personal Computer). The image display device 50 may be any of a desktop PC, a laptop PC, and a tablet terminal. The image display device 50 may be a computer system that operates on the cloud.

[0040] The operation unit 70 is a user interface. For example, the operation unit 70 is at least one of a button, a switch, a key, a mouse, a joystick, a touchpad, a trackball, and a touch panel. The operation unit 70 accepts operations by a user. The user can input various information to the image display device 50 by operating the operation unit 70. The operation unit 70 accepts information input by the user and outputs the information to the image display device 50.

[0041] The display unit 71 has a display screen and displays an image of 3D data or the like on the display screen. The display unit 71 is a monitor (display) such as an LCD (Liquid Crystal Display). The display unit 71 may be a touch panel. In that case, the operation unit 70 and the display unit 71 are integrated.

[0042] The communication unit 72 communicates with an external device such as an endoscope device. For example, the communication unit 72 is connected to the external device via a cable or wirelessly. The communication between the communication unit 72 and the external device may be performed via a LAN (Local Area Network) or the Internet.

[0043] The storage unit 73 is a non-volatile recording medium. For example, the storage unit 73 is at least one of a static random access memory (SRAM), a read only memory (ROM), an erasable programmable read only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), and a flash memory.

[0044] The storage unit 73 stores two or more pieces of 3D data including first 3D data and second 3D data. The storage unit 73 may have a first storage unit that stores the first 3D data and a second storage unit that stores the second 3D data.

[0045] Each of the first 3D data and the second 3D data includes three-dimensional coordinates (3D coordinates) of two or more points of the object. Each of the first 3D data and the second 3D data indicates a 3D shape of the object. The first 3D data is generated by using two or more first images acquired at two or more viewpoints different from each other. The second 3D data is generated by using two or more second images acquired at two or more viewpoints different from each other. The two or more first images and the two or more second images are not completely the same. At least one of the two or more first images and at least one of the two or more second images are different from each other. All of the two or more first images may be different from any of the two or more second images. All of the two or more second images may be different from any of the two or more first images.

[0046] The first 3D data and the second 3D data are different from each other. At least a portion of the 3D coordinates included in the first 3D data are different from the 3D coordinates included in the second 3D data. One or more points included in the first 3D data are different from one or more points included in the second 3D data. At least a portion of the 3D coordinates included in the second 3D data are different from the 3D coordinates included in the second 3D data. One or more points included in the second 3D data are different from one or more points included in the first 3D data.

[0047] All of the 3D coordinates included in the first 3D data may be different from the 3D coordinates included in the second 3D data. All of the two or more points included in the first 3D data may be different from the two or more points included in the second 3D data. All of the 3D coordinates included in the second 3D data may be different from the 3D coordinates included in the first 3D data. All of the two or more points included in the second 3D data may be different from the two or more points included in the first 3D data.

[0048] That is, a subject region corresponding to at least a portion of the first 3D data does not exist in a subject region corresponding to the entirety of the second 3D data, and a subject region corresponding to at least a portion of the second 3D data does not exist in a subject region corresponding to the entirety of the first 3D data.

[0049] The three-dimensional coordinate system of the first 3D data (first 3D coordinate system) and the three-dimensional coordinate system of the second 3D data (second 3D coordinate system) are different from each other. One of the first 3D coordinate system and the second 3D coordinate system can be transformed into the other by using parameters indicating each of the position, the orientation, and the scale. At least one of the position, the orientation, and the scale is different between the first 3D coordinate system and the second 3D coordinate system. The orientation is defined by using three parameters. At least one parameter indicating the orientation may be different between the first 3D coordinate system and the second 3D coordinate system.

[0050] At least a portion of the two or more viewpoints at which the two or more first images are acquired may be different from the two or more viewpoints at which the two or more second images are acquired. All of the two or more viewpoints at which the two or more first images are acquired may be different from the two or more viewpoints at which the two or more second images are acquired. At least a portion of the two or more viewpoints at which the two or more second images are acquired may be different from the two or more viewpoints at which the two or more first images are acquired. All of the two or more viewpoints at which the two or more second images are acquired may be different from the two or more viewpoints at which the two or more first images are acquired.

[0051] The control unit 51 controls each unit of the image display device 50 .

[0052] The data acquisition unit 52 is connected to the storage unit 73 and acquires the first 3D data and the second 3D data from the storage unit 73 .

[0053] The parameter generating unit 53 generates conversion parameters for converting the first 3D coordinate system and the second 3D coordinate system into a common coordinate system. The common coordinate system is a 3D coordinate system common to the first 3D data and the second 3D data. By converting the first 3D coordinate system and the second 3D coordinate system into the common coordinate system, each of the first 3D data and the second 3D data is converted into 3D data in the common coordinate system. The conversion parameters include a first conversion parameter for converting the first 3D coordinate system into the common coordinate system and a second conversion parameter for converting the second 3D coordinate system into the common coordinate system. Each of the first conversion parameter and the second conversion parameter includes a position and orientation conversion parameter for converting the position and orientation of each 3D coordinate system and a scale conversion parameter for converting the scale of each 3D coordinate system.

[0054] The conversion unit 54 converts the first 3D coordinate system and the second 3D coordinate system into a common coordinate system by using the conversion parameters generated by the parameter generation unit 53. In this way, the conversion unit 54 converts each of the first 3D data and the second 3D data into 3D data in the common coordinate system.

[0055] The data generation unit 55 connects together the first 3D data and the second 3D data that have been converted into 3D data in a common coordinate system by the conversion unit 54. In this way, the data generation unit 55 generates 3D data for a wide range of the subject.

[0056] The display control unit 56 outputs an image of the 3D data to the display unit 71, and displays the image on the display unit 71. The display control unit 56 displays each image of the first 3D data and the second 3D data on the display unit 71. That is, the display control unit 56 displays each image of the 3D shape indicated by the first 3D data and the 3D shape indicated by the second 3D data on the display unit 71. In addition, the display control unit 56 displays an image of the 3D data generated by the data generation unit 55 on the display unit 71. That is, the display control unit 56 displays an image of the 3D shape indicated by the 3D data of a wide range of the subject on the display unit 71.

[0057] The information receiving unit 57 receives information output from the operation unit 70. Alternatively, the information receiving unit 57 receives information received by the communication unit 72. The information receiving unit 57 may receive information corresponding to a voice input to a microphone not shown in FIG. 1. The information receiving unit 57 may receive structural information related to a geometric structure of the subject. The structural information indicates the structure of the subject in a region where the 3D shape of the first 3D data and the 3D shape of the second 3D data are connected to each other. Hereinafter, the region is referred to as a connection region. The structural information is generated without using either the first 3D data or the second 3D data. The information receiving unit 57 may receive information that does not include structural information.

[0058] When the information receiving unit 57 receives information that does not include structure information, the structure estimation unit 58 estimates the structure of the subject in the connected region based on the information and generates structure information. When the information receiving unit 57 receives information that includes structure information, the structure estimation unit 58 is not used.

[0059] Each unit of the image display device 50 may be composed of at least one of a processor and a logic circuit. For example, the processor is at least one of a CPU (Central Processing Unit), a DSP (Digital Signal Processor), and a GPU (Graphics Processing Unit). For example, the logic circuit is at least one of an ASIC (Application Specific Integrated Circuit) and an FPGA (Field-Programmable Gate Array). Each unit of the image display device 50 may include one or more processors. Each unit of the image display device 50 may include one or more logic circuits.

[0060] The computer of the image display device 50 may load a program and execute the loaded program. The program includes instructions that define the operation of each part of the image display device 50. In other words, the function of each part of the image display device 50 may be realized by software.

[0061] The above program may be provided by a "computer-readable recording medium" such as a flash memory. The program may be transmitted from a computer that holds the program to the image display device 50 via a transmission medium or by a transmission wave in the transmission medium. A "transmission medium" that transmits the program is a medium that has a function of transmitting information. Media that have a function of transmitting information include networks (communication networks) such as the Internet and communication lines (communication lines) such as telephone lines. The above program may realize some of the functions described above. Furthermore, the above program may be a difference file (difference program). The functions described above may be realized by a combination of a program already recorded in the computer and a difference program.

[0062] The characteristic processing of the first embodiment will be described below. In the following description, it is assumed that the 3D data is generated based on a group of still images acquired by an endoscope device. The two or more first images used to generate the first 3D data and the two or more second images used to generate the second 3D data are generated by an endoscope device. In addition, the two or more first images and the two or more second images are generated based on an optical image of a subject acquired through a monocular optical system. The inspection device that acquires the group of still images is not limited to an endoscope device. As long as the inspection device has a camera, the inspection device may be any device.

[0063] For example, each of the two or more first images is a still image, and each of the two or more second images is a still image. The two or more first images may be all or a part of two or more images included in a video. The two or more second images may be all or a part of two or more images included in a video. The two or more first images may be included in a first video file, and the two or more second images may be included in a second video file different from the first video file. The two or more first images and the two or more second images do not need to be split into two video files. The two or more first images and the two or more second images may be included in one video file.

[0064] The process executed by the image display device 50 will be described with reference to Fig. 2. Fig. 2 shows the procedure of the process executed by the image display device 50.

[0065] The data acquisition unit 52 connects to the storage unit 73 and acquires the first 3D data from the storage unit 73 (step S100). After step S100, the display control unit 56 displays an image of the first 3D data on the display unit 71 (step S101).

[0066] After step S101, the data acquisition unit 52 connects to the storage unit 73 and acquires the second 3D data from the storage unit 73 (step S102). After step S102, the display control unit 56 displays an image of the second 3D data on the display unit 71 (step S103).

[0067] The order of steps S100 to S103 is not limited to the order shown in Fig. 2. For example, steps S102 and S103 may be executed first, and then steps S100 and S101 may be executed. Alternatively, steps S100 and S102 may be executed first, and then steps S101 and S103 may be executed. Steps S101 and S103 may be omitted.

[0068] 3 shows an example of an image displayed on the display unit 71. The display control unit 56 displays an image IMG11 on the display unit 71. The image IMG11 includes a region R11 and a region R12. An image of a first 3D shape 3D11 is displayed in the region R11. The first 3D shape 3D11 is represented by the first 3D data. An image of a second 3D shape 3D12 is displayed in the region R12. The second 3D shape 3D12 is represented by the second 3D data.

[0069] After step S103, the information receiving unit 57 receives information on the inspection target (step S104). In the following, an example in which the information receiving unit 57 receives information that does not include structure information from the operation unit 70 will be described.

[0070] The display control unit 56 displays a dialog box on the display unit 71. Fig. 4 shows an example of a dialog box displayed on the display unit 71. The display control unit 56 displays a dialog box DL10 on the display unit 71. The user can select information in the dialog box DL10 by operating the operation unit 70.

[0071] A cursor CS10 and a cursor CS11 are displayed in the dialog box DL10. Each of the cursor CS10 and the cursor CS11 indicates a currently selected item. Information INF10 is displayed in the dialog box DL10. The information INF10 includes a name of the examination object, and includes names of typical examination sites or structures present in the examination object.

[0072] The user selects a subject in the inspection target. When the region where the first 3D shape 3D11 and the second 3D shape 3D12 shown in FIG. 3 are connected to each other is a "straight pipe", the user selects "pipe" shown in FIG. 4, and then selects "straight pipe". When the user inputs information indicating that the selection has been confirmed to the image display device 50, the information receiving unit 57 receives the information selected by the user. At this time, the information receiving unit 57 receives the character string "straight pipe".

[0073] After step S104, the structure estimation unit 58 estimates the structure of the subject in the connected region based on the information received in step S104, and generates structure information (step S105).

[0074] An example of a method in which the structure estimation unit 58 generates the structure information will be described. For example, the structure estimation unit 58 uses a reference table TB11 shown in Fig. 5. The reference table TB11 includes information (input information) and structure information received by the information receiving unit 57. The input information and the structure information are associated with each other.

[0075] The structure estimation unit 58 converts the character string received by the information receiving unit 57 into structure information. For example, when the information receiving unit 57 receives the character string "straight pipe", the structure estimation unit 58 refers to the column CL11 in the reference table TB11 and acquires the structure information in the column CL11. For example, the structure information indicates that the 3D shapes of the first 3D data and the second 3D data are approximately cylindrical. Since the 3D data may contain foreign matter, the structure information indicates a shape close to a cylindrical shape rather than a perfect cylindrical shape. The structure information indicates that the inner diameter of the 3D shape of the first 3D data and the inner diameter of the 3D shape of the second 3D data are equal. The structure information indicates that the central axis of the cylinder of the first 3D data and the central axis of the cylinder of the second 3D data coincide with each other.

[0076] The method of generating the structure information is not limited to the method of using the reference table TB11. For example, the information receiving unit 57 may receive information similar to the structure information included in the reference table TB11 from the operation unit 70. In other words, the information receiving unit 57 may receive information including the structure information from the operation unit 70. In this case, the structure estimation unit 58 may be omitted.

[0077] The display control unit 56 may display an input box on the display unit 71. The user may input a word or keyword into the input box by using the operation unit 70 or a microphone. The structure estimation unit 58 may estimate the structure of the subject based on the input word or keyword, and generate structure information.

[0078] The method of acquiring the structure information of the connected regions is not limited to the above-mentioned method. For example, the structure estimation unit 58 may estimate the structure of the subject by using a technique such as AI (Artificial Intelligence).

[0079] After step S105, the parameter generating unit 53 generates conversion parameters for converting the first 3D coordinate system and the second 3D coordinate system into a common coordinate system (step S106).

[0080] The following describes the 3D coordinate system. The first 3D coordinate system in the first 3D data and the second 3D coordinate system in the second 3D data do not match each other. In order to synthesize the first 3D data and the second 3D data into a wide range of 3D data, it is necessary to convert the first 3D coordinate system and the second 3D coordinate system into an arbitrary common coordinate system. The common coordinate system may be a 3D coordinate system different from both the first 3D coordinate system and the second 3D coordinate system. The common coordinate system may be the same as the first 3D coordinate system or the second 3D coordinate system.

[0081] When the common coordinate system is different from both the first 3D coordinate system and the second 3D coordinate system, the parameter generation unit 53 generates a first conversion parameter for converting the first 3D coordinate system into the third 3D coordinate system, and generates a second conversion parameter for converting the second 3D coordinate system into the third 3D coordinate system.

[0082] On the other hand, if the common coordinate system is the same as the first 3D coordinate system, the parameter generating unit 53 generates only the second conversion parameters for converting the second 3D coordinate system into the first 3D coordinate system. Similarly, if the common coordinate system is the same as the second 3D coordinate system, the parameter generating unit 53 generates only the first conversion parameters for converting the first 3D coordinate system into the second 3D coordinate system.

[0083] The scale of the 3D coordinate system will be described below. The scale of the first 3D data and the scale of the second 3D data do not necessarily match each other. In order to synthesize the first 3D data and the second 3D data into a wide range of 3D data, it is necessary to make the scale of the first 3D data and the scale of the second 3D data match each other. The scale of the synthesized 3D data may be different from both the scale of the first 3D data and the scale of the second 3D data. The scale of the synthesized 3D data may be the same as the scale of the first 3D data or the scale of the second 3D data.

[0084] In the following, an example will be described in which the first 3D coordinate system is the reference of the coordinate system and is used as the common coordinate system. In the following example, the parameter generating unit 53 generates a position and orientation conversion parameter for matching the position and orientation of the second 3D coordinate system with the position and orientation of the first 3D coordinate system. In addition, the parameter generating unit 53 generates a scale conversion parameter for matching the scale of the second 3D coordinate system with the scale of the first 3D coordinate system. However, the method of generating the conversion parameter is not limited to the following example. The second 3D coordinate system may be used as the common coordinate system. A 3D coordinate system different from both the first 3D coordinate system and the second 3D coordinate system may be used as the common coordinate system.

[0085] In step S106, the parameter generating unit 53 executes the process shown in Fig. 6. Fig. 6 shows the procedure of the process executed by the parameter generating unit 53.

[0086] The structural information indicates that the 3D shapes of the first 3D data and the second 3D data are each substantially cylindrical. Therefore, the parameter generating unit 53 calculates the cylinder axis of the first 3D data (first cylinder axis) (step S106a). The cylinder axis of the first 3D data indicates the central axis of the cylinder of the first 3D data.

[0087] Fig. 7 shows an image similar to that of Fig. 3. Explanation of the same parts as those shown in Fig. 3 will be omitted. The parameter generating unit 53 calculates the first cylinder axis AX11.

[0088] After step S106a, the parameter generating unit 53 calculates the cylinder axis of the second 3D data (second cylinder axis) (step S106b). The cylinder axis of the second 3D data indicates the central axis of the cylinder of the second 3D data. The parameter generating unit 53 calculates the second cylinder axis AX12 shown in FIG. 7.

[0089] The order of steps S106a and S106b is not limited to the order shown in Fig. 6. Step S106a may be executed after step S106b is executed.

[0090] The structural information indicates that the inner diameter of the 3D shape of the first 3D data and the inner diameter of the 3D shape of the second 3D data are equal. Therefore, after step S106b, the parameter generating unit 53 generates a scale conversion parameter for correcting the scale of the second 3D data so that the inner diameter of the cylinder of the first 3D data and the inner diameter of the cylinder of the second 3D data match each other (step S106c). Specifically, the parameter generating unit 53 generates a scale conversion parameter for matching the diameter DM12 of the cylinder of the second 3D shape 3D12 shown in FIG. 7 with the diameter DM11 of the cylinder of the first 3D shape 3D11.

[0091] The structural information indicates that the central axis of the cylinder of the first 3D data and the central axis of the cylinder of the second 3D data coincide with each other. Therefore, after step S106c, the parameter generating unit 53 generates a position and orientation conversion parameter for correcting the position and orientation of the second 3D data so that the cylinder axis of the first 3D data and the cylinder axis of the second 3D data coincide with each other (step S106d). When step S106d is executed, the process shown in FIG. 6 ends.

[0092] When each 3D data includes a time stamp, the parameter generating unit 53 can identify the connection region by using the time stamp (time information). The time stamp indicates the time when each of the two or more images used to generate each of the first 3D data and the second 3D data was generated. For example, the first 3D data includes a time stamp from time t1 to time t2 shown in FIG. 7, and the second 3D data includes a time stamp from time t3 to time t4 shown in FIG. 7.

[0093] The parameter generating unit 53 identifies a region of the first 3D data associated with time t2 and a region of the second 3D data associated with time t3 as connection regions. Specifically, the parameter generating unit 53 identifies a first connection region CR11 of the first 3D shape 3D11 and a second connection region CR12 of the second 3D shape 3D12.

[0094] The first connection region CR11 includes the end of the first 3D shape 3D11. The second connection region CR12 includes the start of the second 3D shape 3D12. The parameter generation unit 53 generates position and orientation transformation parameters for matching the second cylinder axis AX12 in the second connection region CR12 with the first cylinder axis AX11 in the first connection region CR11.

[0095] The parameter generating unit 53 can convert the first 3D coordinate system and the second 3D coordinate system into a common coordinate system by executing steps S106a to S106d. When the information receiving unit 57 receives the character string "straight pipe", the parameter generating unit 53 generates position and orientation parameters and scale parameters for converting the first 3D coordinate system and the second 3D coordinate system into the common coordinate system.

[0096] After step S106, the conversion unit 54 converts the first 3D coordinate system and the second 3D coordinate system into a common coordinate system by using the conversion parameters generated in step S106. That is, the conversion unit 54 converts each of the first 3D data and the second 3D data into 3D data in the common coordinate system (step S107). When the first 3D coordinate system is used as the common coordinate system, the position, orientation, and scale of the first 3D coordinate system are not changed, and the position, orientation, and scale of the second 3D coordinate system are changed.

[0097] After step S107, the data generation unit 55 connects the first 3D data and the second 3D data in the common coordinate system to each other (step S108), thereby allowing the data generation unit 55 to generate 3D data of a wide range of the subject.

[0098] After the first 3D coordinate system and the second 3D coordinate system are converted into a common coordinate system, the first 3D data and the second 3D data do not need to be synthesized into one 3D data. Therefore, the data generator 55 may be omitted.

[0099] After step S108, the display control unit 56 displays an image of the 3D shape indicated by the 3D data generated in step S108 on the display unit 71 (step S109). When step S109 is executed, the process shown in FIG. 2 ends.

[0100] Fig. 8 shows an example of an image displayed on the display unit 71. Description of the same parts as those shown in Fig. 7 will be omitted. The display control unit 56 displays an image IMG12 on the display unit 71. The image IMG12 includes an area R13. Images of a first 3D shape 3D11 and a second 3D shape 3D12 are displayed in the area R13.

[0101] The first 3D shape 3D11 and the second 3D shape 3D12 are arranged such that the first connection region and the second connection region are connected to each other. In the example shown in Fig. 8, there is a gap between the first 3D shape 3D11 and the second 3D shape 3D12 to make it easier to understand the connection of the first connection region and the second connection region.

[0102] The display control unit 56 displays the region R14 and the region R15 in the region R13. The region R14 corresponds to the first connection region, and the region R15 corresponds to the second connection region. In the example shown in FIG. 8, the region R14 and the region R15 are each displayed as a line. Since the region R14 and the region R15 are each displayed as a line, the user can easily confirm the position where the first 3D shape 3D11 and the second 3D shape 3D12 are connected to each other.

[0103] The display control unit 56 may display the regions R14 and R15 in a first color, and may display the regions other than the regions R14 and R15 in a second color different from the first color. The positions of the regions R14 and R15 may be notified to the user by voice. As long as the user can distinguish the connection region from the region other than the connection region, the method of notifying the user of the region of the 3D shape is not limited to the above method.

[0104] The display control unit 56 may display the region R14 and the region R15 in different colors. When the user looks over a wide range of 3D shapes, the user can easily confirm how far the first 3D shape 3D11 and the second 3D shape 3D12 are from each other. In addition, the user can easily confirm where the first 3D shape 3D11 and the second 3D shape 3D12 are connected to each other.

[0105] The images of the first 3D shape 3D11 and the second 3D shape 3D12 may be added to the inspection report. The display control unit 56 may display the images of the first 3D shape 3D11 and the second 3D shape 3D12 on the display unit 71 by displaying the inspection report on the display unit 71.

[0106] In step S106, the parameter generating unit 53 may calculate the reliability of the connection between the first connection region and the second connection region. The reliability indicates the accuracy of the connection between the first connection region and the second connection region. For example, the parameter generating unit 53 performs cylinder fitting using the first 3D data. As a result, the parameter generating unit 53 calculates a cylinder that approximates the first 3D data, and calculates the central axis of the cylinder (first cylinder axis). Also, the parameter generating unit 53 performs cylinder fitting using the second 3D data. As a result, the parameter generating unit 53 calculates a cylinder that approximates the second 3D data, and calculates the central axis of the cylinder (second cylinder axis).

[0107] An error occurs between the 3D coordinates included in the 3D data and the calculated 3D coordinates of the cylinder. The parameter generating unit 53 calculates the error in the distance between each point of the 3D data and each point on the cylinder for each point of the 3D data, and calculates the average or standard deviation of the error as the reliability of the 3D data. The parameter generating unit 53 calculates statistical values ​​of the reliability of the first 3D data and the reliability of the second 3D data as the reliability of the connection in the first connection region and the second connection region. The statistical values ​​are the minimum, maximum, average, etc. of the reliability of the first 3D data and the reliability of the second 3D data.

[0108] In step S109, the display control unit 56 may display the reliability of the first 3D data and the reliability of the second 3D data on the display unit 71. For example, each reliability is displayed as a percentage. Alternatively, each reliability is displayed on a scale from 1 to 5. The method of displaying the reliability is not limited to the above example.

[0109] When the 3D shape shown by the 3D data is a perfect cylinder, the parameter generating unit 53 can accurately calculate the cylinder axis. Therefore, the reliability is high. On the other hand, when the 3D data includes an error, the central axis of the cylinder that approximates the 3D data is shifted from the original central axis. Therefore, the reliability is low. The reliability is displayed so that the user can check the accuracy of the connection between the first connection region and the second connection region.

[0110] The three-dimensional image display method of each aspect of the present invention includes a first acquisition step, a second acquisition step, a conversion step, and a display step. In the first acquisition step (step S100), the data acquisition unit 52 connects to a storage unit 73 (recording medium) that stores first 3D data of a subject and second 3D data of the subject, and acquires the first 3D data from the storage unit 73. The first 3D data includes 3D coordinates defined in a first 3D coordinate system. The second 3D data includes 3D coordinates defined in a second 3D coordinate system different from the first 3D coordinate system. At least a part of the area of ​​the subject corresponding to the first 3D data is different from at least a part of the area of ​​the subject corresponding to the second 3D data. In the second acquisition step (step S102), the data acquisition unit 52 connects to the storage unit 73, and acquires the second 3D data from the storage unit 73. In a conversion step (step S107), the conversion unit 54 converts the first 3D coordinate system and the second 3D coordinate system into a three-dimensional common coordinate system based on structural information related to the geometric structure of the subject. The structural information is generated without using either the first 3D data or the second 3D data. In a display step (step S109), the display control unit 56 displays an image of the first 3D data in the common coordinate system and an image of the second 3D data in the common coordinate system on the display unit 71 (display).

[0111] The three-dimensional image display device according to each aspect of the present invention includes a data acquisition unit 52, a conversion unit 54, and a display control unit 56. The data acquisition unit 52 is connected to a storage unit 73 (recording medium) that stores first 3D data of a subject and second 3D data of the subject, and acquires the first 3D data and the second 3D data from the storage unit 73. The conversion unit 54 converts the first 3D coordinate system and the second 3D coordinate system into a three-dimensional common coordinate system based on structural information related to the geometric structure of the subject. The display control unit 56 displays an image of the first 3D data in the common coordinate system and an image of the second 3D data in the common coordinate system on a display unit 71 (display).

[0112] Each aspect of the present invention may include the following modifications: Each of the two or more first images and each of the two or more second images include time information (time stamp). In a conversion step (step S107), the conversion unit 54 converts the first 3D coordinate system and the second 3D coordinate system into a common coordinate system based on the time information.

[0113] Each aspect of the present invention may include the following modified examples. In a conversion step (step S107), the conversion unit 54 converts the first 3D coordinate system and the second 3D coordinate system into a common coordinate system so that a first region (first connection region CR11) of the subject and a second region (second connection region CR12) of the subject are connected to each other. The first region corresponds to the 3D coordinates included in the first 3D data. The second region corresponds to the 3D coordinates included in the second 3D data. In a display step (step S109), the display control unit 56 displays information indicating the positions of the first region and the second region on the display unit 71 (display).

[0114] Each aspect of the present invention may include the following modified examples: In a display step (step S109), the display control unit 56 displays information indicating the accuracy of the connection between the first region (first connection region CR11) and the second region (second connection region CR12) on the display unit 71 (display).

[0115] Each aspect of the present invention may include the following modifications. In a conversion step (step S107), the conversion unit 54 generates position conversion parameters and orientation conversion parameters for converting the first 3D coordinate system and the second 3D coordinate system into a common coordinate system based on the structural information. In the conversion step (step S107), the conversion unit 54 converts the first 3D coordinate system and the second 3D coordinate system into the common coordinate system by using the position conversion parameters and the orientation conversion parameters.

[0116] Each aspect of the present invention may include the following modified examples. In a generating step (step S106), the parameter generating unit 53 generates a scale conversion parameter for correcting at least one of the scale of the 3D shape represented by the first 3D data and the scale of the 3D shape represented by the second 3D data. In a converting step (step S107), the converting unit 54 converts the first 3D coordinate system and the second 3D coordinate system into a common coordinate system by using the scale conversion parameter.

[0117] Each aspect of the invention may include the following variations: the first 3D data is generated by using two or more first images acquired at two or more different viewpoints; the second 3D data is generated by using two or more second images acquired at two or more different viewpoints; at least one of the two or more first images and at least one of the two or more second images are different from each other.

[0118] Each aspect of the present invention may include the following variations: The two or more first images are at least a portion of two or more images included in a first video. The two or more second images are at least a portion of two or more images included in a second video that is the same as or different from the first video.

[0119] Each aspect of the invention may include the following variations: The two or more first images and the two or more second images are generated by an endoscope.

[0120] Each aspect of the present invention may include the following variations: The two or more first images and the two or more second images are generated based on an optical image of a subject acquired through a monocular optical system.

[0121] In the first embodiment, there is a case where the first 3D data and the second 3D data do not include 3D data of a common area. Even in that case, the image display device 50 can display an image of a wide range of 3D shapes represented by two or more 3D data. Therefore, the user can capture the entire structure of the inspection target in the field of view and can confirm the structure.

[0122] There may be a case where 3D data corresponding to a region between a first connection region in a 3D shape represented by the first 3D data and a second connection region in a 3D shape represented by the second 3D data is missing. Even in such a case, the image display device 50 can display an image of a wide range of 3D shapes represented by two or more 3D data.

[0123] (First Modification of the First Embodiment) A first modified example of the first embodiment of the present invention will be described. The image display device 50 shown in Fig. 1 is changed to an image display device 50a shown in Fig. 9. Fig. 9 shows the configuration of the image display device 50a. The description of the same configuration as that shown in Fig. 1 will be omitted.

[0124] An image display device 50a shown in FIG. 9 includes a control unit 51, a data acquisition unit 52, a parameter generation unit 53, a conversion unit 54, a data generation unit 55, a display control unit 56, an information reception unit 57, a structure estimation unit 58, and an adjustment unit 59.

[0125] Each unit of the image display device 50a may be composed of at least one of a processor and a logic circuit. Each unit of the image display device 50a may include one or more processors. Each unit of the image display device 50a may include one or more logic circuits.

[0126] The adjustment unit 59 adjusts at least one of the position and orientation of at least one of the first 3D data and the second 3D data in the common coordinate system. For example, the adjustment unit 59 executes an adjustment process of the first 3D data and the second 3D data. Alternatively, the adjustment unit 59 executes an adjustment process of only the first 3D data or only the second 3D data. The adjustment unit 59 adjusts the position and orientation of the 3D data in the adjustment process. Alternatively, the adjustment unit 59 adjusts only the position of the 3D data or only the orientation of the 3D data in the adjustment process.

[0127] After adjustment unit 59 executes the adjustment process, display control unit 56 redisplays the images of the first 3D data and the second 3D data on display unit 71. This changes at least one of the position and the orientation of at least one of the images of the first 3D data and the second 3D data.

[0128] The process executed by the image display device 50a will be described with reference to Fig. 10. Fig. 10 shows the procedure of the process executed by the image display device 50a. The description of the same process as that shown in Fig. 2 will be omitted.

[0129] For example, in step S108, an image IMG12 shown in Fig. 8 is displayed on the display unit 71. In the example shown in Fig. 8, the end of the first 3D shape 3D11 coincides with the start of the second 3D shape 3D12. There are cases where 3D data is missing between the end of the first 3D shape 3D11 and the start of the second 3D shape 3D12.

[0130] In a first variation of the first embodiment, the user estimates the length of a section between the end of the first 3D shape 3D11 and the beginning of the second 3D shape 3D12. 3D data in that section is missing. The user can change the arrangement of the first 3D shape 3D11 and the second 3D shape 3D12 so that the end of the first 3D shape 3D11 and the beginning of the second 3D shape 3D12 are separated by the length of the section.

[0131] The user operates the operation unit 70 to input an instruction to move at least one of the first 3D shape 3D11 and the second 3D shape 3D12 to the image display device 50a. The information receiving unit 57 receives the instruction. An example in which the second 3D shape 3D12 moves will be described below. The adjustment unit 59 adjusts at least one of the position and the orientation of the second 3D shape 3D12 based on the instruction received by the information receiving unit 57 (step S110).

[0132] The 3D data generated in step S108 includes 3D data corresponding to the first 3D data and 3D data corresponding to the second 3D data. The adjustment unit 59 changes the 3D coordinates of the 3D data corresponding to the second 3D data. In this way, the adjustment unit 59 adjusts at least one of the position and the orientation of the 3D shape indicated by the 3D data.

[0133] After step S110, the display control unit 56 displays an image of the 3D shape indicated by the 3D data processed in step S110 on the display unit 71 (step S111). When step S111 is executed, the process shown in FIG.

[0134] Fig. 11 shows an example of an image displayed on the display unit 71. Description of the same parts as those shown in Fig. 8 will be omitted. The display control unit 56 displays the image IMG13 on the display unit 71.

[0135] The user inputs, to the image display device 50a, an instruction to move the second 3D shape 3D12 closer to or farther from the first 3D shape 3D11. Alternatively, the user inputs, to the image display device 50a, an instruction to rotate the second 3D shape 3D12.

[0136] The operations performed by the user to freely change the position and orientation of the second 3D shape 3D12 are complicated. Therefore, the movement of the second 3D shape 3D12 may be restricted. For example, when "straight pipe" is selected, the user may move the second 3D shape 3D12 within a range in which the direction of the cylinder axis does not change. In this case, the user can move the second 3D shape 3D12 in a direction parallel to the cylinder axis and can rotate the second 3D shape 3D12 around the cylinder axis.

[0137] Even if the subject is a straight pipe, the conditions for restricting the movement of the second 3D shape 3D12 are not limited to the above example.

[0138] 11, the user inputs an instruction to move the second 3D shape 3D12 to the right into the image display device 50a. The adjustment unit 59 changes the 3D coordinates of the 3D data so that the second 3D shape 3D12 moves to the right.

[0139] Each aspect of the present invention may include the following modified examples: In an adjustment step (step S110), the adjustment unit 59 adjusts at least one of the position and the orientation of at least one of the first 3D data and the second 3D data in the common coordinate system.

[0140] In a first modification of the first embodiment, at least one of the positions and orientations of the first 3D shape and the second 3D shape in the wide range of 3D shapes is adjusted, and through this adjustment, the 3D shape represented by the wide range of 3D data becomes closer to the structure of the actual inspection target, thereby improving the quality of the 3D data.

[0141] (Second Modification of the First Embodiment) A second modified example of the first embodiment of the present invention will now be described. In the second modified example of the first embodiment, an image display device 50 shown in FIG.

[0142] The object in the first embodiment described above is a straight pipe, whereas in a second modified example of the first embodiment, an example in which the object is a 90-degree joint will be described.

[0143] For example, the user selects "pipe" shown in Fig. 4, and then selects "90 degree joint." When the user inputs information indicating that the selection has been confirmed to the image display device 50, the information accepting unit 57 accepts the information selected by the user. At this time, the information accepting unit 57 accepts the character string "90 degree joint."

[0144] For example, the structural information of a 90-degree joint indicates that the 3D shapes of the first 3D data and the second 3D data are approximately cylindrical. The structural information indicates that the inner diameter of the 3D shape of the first 3D data and the inner diameter of the 3D shape of the second 3D data are equal. The structural information indicates that the central axis of the cylinder of the first 3D data and the central axis of the cylinder of the second 3D data are perpendicular to each other. In step S106, the parameter generating unit 53 generates conversion parameters based on the structural information.

[0145] 12 shows an example of an image displayed on the display unit 71. The display control unit 56 displays an image IMG21 on the display unit 71. The image IMG21 includes an area R21. Images of a first 3D shape 3D21 and a second 3D shape 3D22 are displayed in the area R21. The first 3D shape 3D21 is represented by the first 3D data. The second 3D shape 3D22 is represented by the second 3D data.

[0146] In the example shown in Fig. 12, the parameter generating unit 53 generates a position and orientation conversion parameter for matching the position and orientation of the second 3D coordinate system with the position and orientation of the first 3D coordinate system. The parameter generating unit 53 also generates a scale conversion parameter for matching the scale of the second 3D coordinate system with the scale of the first 3D coordinate system. The parameter generating unit 53 calculates a first cylinder axis AX21 of the first 3D shape 3D21 and a second cylinder axis AX22 of the second 3D shape 3D22. The parameter generating unit 53 generates a scale conversion parameter for matching a diameter DM22 of the cylinder of the second 3D shape 3D22 with a diameter DM21 of the cylinder of the first 3D shape 3D21.

[0147] The first 3D data includes time stamps from time t1 to time t2 shown in Fig. 12, and the second 3D data includes time stamps from time t3 to time t4 shown in Fig. 12. The parameter generating unit 53 identifies a first connection region of the first 3D shape 3D21 associated with time t2, and identifies a second connection region of the second 3D shape 3D22 associated with time t3.

[0148] The first connection region includes the end of the first 3D shape 3D21. The second connection region includes the start of the second 3D shape 3D22. The parameter generation unit 53 generates position and orientation transformation parameters for making the second cylinder axis AX22 in the second connection region perpendicular to the first cylinder axis AX21 in the first connection region.

[0149] The second connection region is disposed in a position close to the first connection region. The parameter generating unit 53 generates position and posture transformation parameters so that the region between the first connection region and the second connection region is small. 3D data of the region between the first connection region and the second connection region (the region of the 90-degree joint) is missing. The parameter generating unit 53 generates position and posture transformation parameters so that the first connection region and the second connection region do not overlap each other.

[0150] In addition to the 3D data of the curved area, the 3D data of the straight pipe section may be missing. Therefore, the length of the section between the first connection area and the second connection area is actually unknown. After the image IMG21 shown in FIG. 12 is displayed, the position or orientation of the 3D shape may be adjusted as in the first modification of the first embodiment.

[0151] The display control unit 56 displays the region R22 and the region R23 in the region R21. The region R22 corresponds to the first connection region, and the region R23 corresponds to the second connection region.

[0152] The purpose of each embodiment of the present invention is to display an image of a wide range of 3D shapes even if the first 3D data and the second 3D data do not include 3D data of a common area. Therefore, the position and orientation of the first 3D data and the second 3D data do not need to be precisely adjusted. The user may adjust the position and orientation. The first 3D data and the second 3D data may be roughly connected to each other.

[0153] Fig. 13 shows an example of an image displayed on the display unit 71. Description of the same parts as those shown in Fig. 12 will be omitted. The display control unit 56 displays an image IMG22 on the display unit 71.

[0154] A point on the first cylinder axis AX21 at the end of the first 3D shape 3D21 coincides with a point on the second cylinder axis AX22 at the beginning of the second 3D shape 3D22. A part of the first connection area and a part of the second connection area overlap each other. The structure of the subject to be inspected is actually different from the structure shown in FIG. 13. When the first 3D data and the second 3D data do not need to be connected to each other accurately, the display control unit 56 can display the image IMG22 on the display unit 71 as shown in FIG. 13. In order to improve the appearance of a wide range of 3D shapes, the user may adjust the position and orientation of the 3D data in the same manner as in the first modification of the first embodiment.

[0155] In the second modification of the first embodiment, when the object is a 90-degree joint, the image display device 50 executes a process similar to the process of selecting a straight pipe as the object. This allows the image display device 50 to connect the first 3D data and the second 3D data to each other, and to display images of a wide range of 3D shapes.

[0156] (Third Modification of the First Embodiment) A third modified example of the first embodiment of the present invention will now be described. In the third modified example of the first embodiment, an image display device 50 shown in FIG.

[0157] The subject in the second modified example of the first embodiment described above is a 90-degree joint of the same diameter. On the other hand, in the third modified example of the first embodiment, an example in which the subject is a 90-degree joint of different diameters will be described. There are cases in which joints of different diameters are used in pipes. The third modified example of the first embodiment is applied to the inspection of such pipes.

[0158] After the dialog box DL10 shown in Fig. 4 is displayed on the display unit 71, the display control unit 56 displays a dialog box DL20 shown in Fig. 14 on the display unit 71. The user operates the operation unit 70 to input the inner diameter of the first 3D data into an input box IB21 and to input the inner diameter of the second 3D data into an input box IB22.

[0159] The information accepting unit 57 accepts the inner diameter inputted in the input box IB21 and the inner diameter inputted in the input box IB22. In step S106, the parameter generating unit 53 generates a scale conversion parameter based on the inner diameter of the cylinder of the first 3D data and the inner diameter of the cylinder of the second 3D data.

[0160] An example will be described in which the subject is a 90-degree joint, and the inner diameter of the cylinder of the second 3D data is 1.5 times the inner diameter of the cylinder of the first 3D data. FIG. 15 shows an example of an image displayed on the display unit 71. The display control unit 56 displays an image IMG31 on the display unit 71. The image IMG31 includes an area R31. Images of a first 3D shape 3D31 and a second 3D shape 3D32 are displayed in the area R31. The first 3D shape 3D31 is shown by the first 3D data. The second 3D shape 3D32 is shown by the second 3D data.

[0161] In the example shown in Fig. 15, the parameter generating unit 53 generates a position and orientation conversion parameter for matching the position and orientation of the second 3D coordinate system with the position and orientation of the first 3D coordinate system. The parameter generating unit 53 also generates a scale conversion parameter for matching the scale of the second 3D coordinate system with the scale of the first 3D coordinate system. The parameter generating unit 53 calculates a first cylinder axis AX31 of the first 3D shape 3D31 and a second cylinder axis AX32 of the second 3D shape 3D32. The parameter generating unit 53 generates a scale conversion parameter for making the diameter DM32 of the cylinder of the second 3D shape 3D32 1.5 times the diameter DM31 of the cylinder of the first 3D shape 3D31.

[0162] The first 3D data includes time stamps from time t1 to time t2 shown in Fig. 15, and the second 3D data includes time stamps from time t3 to time t4 shown in Fig. 15. The parameter generating unit 53 identifies a first connection region of the first 3D shape 3D31 associated with time t2, and identifies a second connection region of the second 3D shape 3D32 associated with time t3.

[0163] The first connection region includes the end of the first 3D shape 3D31. The second connection region includes the start of the second 3D shape 3D32. The parameter generation unit 53 generates position and orientation transformation parameters for making the second cylinder axis AX32 in the second connection region perpendicular to the first cylinder axis AX31 in the first connection region.

[0164] The second connection region is disposed in a position close to the first connection region. The parameter generating unit 53 generates position and orientation transformation parameters so that the region between the first connection region and the second connection region is small. 3D data of the region between the first connection region and the second connection region is missing. The parameter generating unit 53 generates position and orientation transformation parameters so that the first connection region and the second connection region do not overlap with each other. After an image of the 3D data of a wide range is displayed, the user may adjust the position and orientation of the 3D data in the same manner as in the first modification of the first embodiment.

[0165] The display control unit 56 displays the region R32 and the region R33 in the region R31. The region R32 corresponds to the first connection region, and the region R33 corresponds to the second connection region.

[0166] In the third modified example of the first embodiment, when the subject is a 90-degree joint of different diameters, the image display device 50 executes a process similar to the process of selecting a straight pipe as the subject. This enables the image display device 50 to connect the first 3D data and the second 3D data to each other, and to display images of a wide range of 3D shapes.

[0167] (Fourth Modification of the First Embodiment) A fourth modified example of the first embodiment of the present invention will now be described. In the fourth modified example of the first embodiment, an image display device 50 shown in FIG.

[0168] The object in the first embodiment described above is a straight pipe. The object in the second and third modified examples of the first embodiment described above is a 90-degree joint. On the other hand, in the fourth modified example of the first embodiment, an example in which the object is a U-shaped pipe will be described. The heat exchange tube or the like has a U-shaped pipe.

[0169] FIG. 16 shows an example of a U-shaped pipe. The U-shaped pipe has a straight pipe section SP41 and an arc section CP41. In the following, an example in which a part of the 3D data is missing in the arc section CP41 will be described. The first 3D data and the second 3D data are connected to each other in the arc section CP41. When the first 3D data and the second 3D data are connected to each other in the straight pipe section SP41, the image display device 50 executes a process similar to the process in the first embodiment in which the subject is a straight pipe, and connects the first 3D data and the second 3D data to each other.

[0170] For example, the structural information of a U-shaped pipe indicates that the 3D shapes of the first 3D data and the second 3D data are approximately cylindrical. The structural information indicates that the arc portion is approximately donut shaped. The range of the arc portion for which 3D data exists is a range of 180 degrees, i.e., only a semicircle. The structural information indicates that the inner diameter of the 3D shape of the first 3D data and the inner diameter of the 3D shape of the second 3D data are equal. The structural information indicates that the two central axes of the two cylinders in the straight pipe portion are parallel to each other, and that the arc portion has an arbitrary radius of curvature. In step S106, the parameter generating unit 53 generates conversion parameters based on the structural information.

[0171] 17 shows an example of an image displayed on the display unit 71. The display control unit 56 displays an image IMG41 on the display unit 71. The image IMG41 includes an area R41. Images of a first 3D shape 3D41 and a second 3D shape 3D42 are displayed in the area R41. The first 3D shape 3D41 is represented by the first 3D data. The second 3D shape 3D42 is represented by the second 3D data.

[0172] In the example shown in Fig. 17, the parameter generating unit 53 generates a position and orientation conversion parameter for matching the position and orientation of the second 3D coordinate system with the position and orientation of the first 3D coordinate system. The parameter generating unit 53 also generates a scale conversion parameter for matching the scale of the second 3D coordinate system with the scale of the first 3D coordinate system. The parameter generating unit 53 calculates a first cylinder axis AX41 of the first 3D shape 3D41 and a second cylinder axis AX42 of the second 3D shape 3D42. The first cylinder axis AX41 and the second cylinder axis AX42 are linear in the straight pipe portion and arcuate in the arcuate portion.

[0173] The parameter generating unit 53 generates a scale conversion parameter for matching the diameter DM42 of the cylinder of the second 3D shape 3D42 with the diameter DM41 of the cylinder of the first 3D shape 3D41. The two inner diameters of the two cylinders in the straight pipe section of the U-shaped pipe are often the same. Therefore, the image display device 50 can generate the scale conversion parameter by using a method similar to the method for generating the scale conversion parameter in the first embodiment.

[0174] The first 3D data includes time stamps from time t1 to time t2 shown in Fig. 17, and the second 3D data includes time stamps from time t3 to time t4 shown in Fig. 17. The parameter generating unit 53 identifies a first connection region of the first 3D shape 3D41 associated with time t2, and identifies a second connection region of the second 3D shape 3D42 associated with time t3.

[0175] The first connection region includes the end of the first 3D shape 3D 41. The second connection region includes the start of the second 3D shape 3D 42. The parameter generation unit 53 generates position and orientation transformation parameters for connecting the first connection region and the second connection region to each other. The first connection region and the second connection region are connected to each other under the condition that the arc portion has a predetermined radius of curvature.

[0176] When a part of the 3D data in the arc section is missing, information indicating the position of the connection region and the angle of the corresponding arc section is not available. When the second 3D data includes 3D data of a straight pipe section, the parameter generating unit 53 can estimate the section where the 3D data is missing by using the condition that the two central axes of the two cylinders in the straight pipe section are parallel to each other.

[0177] The display control unit 56 displays the region R42 and the region R43 in the region R41. The region R42 corresponds to the first connection region, and the region R43 corresponds to the second connection region.

[0178] There is a case where the second 3D data includes only 3D data in the arc portion. In that case, it is difficult for the parameter generating unit 53 to generate position and orientation conversion parameters for accurately connecting the first 3D data and the second 3D data to each other. The parameter generating unit 53 may generate temporary position and orientation conversion parameters so that there is no area between the first 3D data and the second 3D data. After an image of the 3D data of a wide range is displayed, the user may adjust the position and orientation of the 3D data in the same manner as in the first modified example of the first embodiment. At this time, the user may adjust the position of the 3D data along the arc.

[0179] (Fifth Modification of the First Embodiment) A fifth modified example of the first embodiment of the present invention will now be described. In the fifth modified example of the first embodiment, an image display device 50 shown in FIG.

[0180] The subject in the first embodiment and the second to fourth modified examples of the first embodiment described above is an inspection portion of a pipe. Meanwhile, in the fifth modified example of the first embodiment, an example in which the subject is an inspection portion of a gas turbine will be described. Gas turbines are the main inspection objects in aircraft engines and generators, etc. Below, an example in blade inspection will be described. Blades are a typical inspection portion of a gas turbine of an aircraft engine.

[0181] The size of the turbine blade of the generator is different from that of the gas turbine blade of the aircraft engine. However, the structure of the turbine blade is similar to that of the gas turbine blade. Therefore, the following process may be applied to the turbine blade inspection.

[0182] A gas turbine has two or more blades fixed in an annular and radial pattern to a circular part called a disk.

[0183] FIG. 18 shows a schematic diagram of the blade configuration. In FIG. 18, 12 blades BL50 are arranged on a disk DS50. In practice, several tens of blades or even more than 100 blades are arranged on one disk. A center position CP50 indicates the center of the disk DS50 in a plane perpendicular to the rotation axis of the engine. Multiple stages of disks are arranged in each of the compressor section and the turbine section of the gas turbine. Each stage of disks has a number of blades that function as either moving blades or stationary blades. In the inspection manual, an ID is often assigned to each blade.

[0184] When inspecting the rotor blades, a scope is inserted into the engine through an access port on the engine. The rotor blades are rotated manually or automatically, and the tip of the scope is fixed. An inspector performs the inspection in this state and checks whether there are any abnormalities in each blade.

[0185] On the other hand, when the stator vanes (nozzle guide vanes) located immediately aft of the combustion chamber are inspected, the scope is inserted from the access port of the combustion chamber toward the rear of the combustion chamber. Hereinafter, the nozzle guide vanes are referred to as stator vanes. When inspecting stator vanes, the blades cannot be rotated. The inspector moves the tip of the scope around the circumference of the stator vane disk to check whether there are any abnormalities with each blade.

[0186] Information INF10 in the dialog box DL10 shown in Fig. 4 includes information about the gas turbine to be inspected and information about the blades in the gas turbine. For example, the user selects "gas turbine" and then selects "blade." When the user inputs information indicating that the selection has been confirmed into the image display device 50, the information accepting unit 57 accepts the information selected by the user. At this time, the information accepting unit 57 accepts the character string "blade."

[0187] For example, the structural information of the blades indicates that the blades in each of the first 3D data and the second 3D data are arranged at equal intervals on a circumference having an arbitrary radius of curvature. The structural information indicates that the blades have the same length. In step S106, the parameter generating unit 53 generates the transformation parameters based on the structural information.

[0188] When the engine model, section, and number of stages are specified, the total number of blades, the length of each blade, and the ID of each blade are known. The engine model indicates the model of the engine in which the blade is arranged. The section indicates the compressor section or the turbine section. The number of stages indicates the position (stage number) of a disk in a stage including multiple disks. The structural information may include the total number of blades, the length of each blade, and the ID of each blade. In this case, the parameter generating unit 53 can generate position and posture conversion parameters for accurately connecting the first 3D data and the second 3D data to each other.

[0189] The image display device 50 executes the process shown in Fig. 2. The specific process in step S106 differs from the process when the inspection object is a pipe.

[0190] In step S106, the parameter generating unit 53 executes the process shown in Fig. 19. Fig. 19 shows the procedure of the process executed by the parameter generating unit 53. In the following example, the parameter generating unit 53 generates a position and orientation conversion parameter for matching the position and orientation of the second 3D coordinate system with the position and orientation of the first 3D coordinate system. The parameter generating unit 53 also generates a scale conversion parameter for matching the scale of the second 3D coordinate system with the scale of the first 3D coordinate system.

[0191] The structural information indicates that the blades have the same length. Therefore, the parameter generating unit 53 generates a scale conversion parameter for correcting the scale of the second 3D data so that the blade length of the first 3D data and the blade length of the second 3D data match each other (step S106e). The structural information may include information indicating the overall length of the blade. The parameter generating unit 53 may determine, based on the information, that the blade length of the first 3D data and the blade length of the second 3D data are the same.

[0192] After step S106e, the parameter generating unit 53 calculates the center position of the disc in the first 3D data (first center position) (step S106f). In the example shown in Fig. 18, the parameter generating unit 53 calculates the center position CP50.

[0193] The parameter generating unit 53 may use any method to calculate the center position of the disk. For example, the parameter generating unit 53 may calculate the position where straight lines extending in the length direction of each blade intersect as the center position. The parameter generating unit 53 may detect the blades in the 3D data by performing pattern matching. The parameter generating unit 53 may calculate a circumscribing circle or an inscribing circle based on the positional relationship of the blades, and calculate the center position of the circle.

[0194] After step S106f, the parameter generating unit 53 executes the same process as step S106f to calculate the center position of the disc in the second 3D data (second center position) (step S106g). The method of calculating the center position of the disc in step S106g may be the same as or different from the method of calculating the center position of the disc in step S106f.

[0195] After step S106g, the parameter generating unit 53 generates position and orientation transformation parameters for correcting the positions of the first 3D data and the second 3D data so that the first center position and the second center position coincide with each other (step S106h).

[0196] After step S106h, the parameter generating unit 53 calculates a position and posture conversion parameter for arranging the start blade in the second 3D data as the blade next to the end blade in the first 3D data (step S106i). When step S106i is executed, the process shown in FIG. 19 ends.

[0197] 20 shows an example of an image displayed on the display unit 71. The display control unit 56 displays an image IMG51 on the display unit 71. The image IMG51 includes an area R51. Images of a first 3D shape 3D51 and a second 3D shape 3D52 are displayed in the area R51. The first 3D shape 3D51 is represented by the first 3D data. The second 3D shape 3D52 is represented by the second 3D data.

[0198] The first 3D shape 3D51 includes four blades. The first 3D data includes time stamps from time t1 to time t2 shown in FIG. 20. The time stamp (t1) of blade BL51 is the earliest, and the time stamp (t2) of blade BL52 is the latest. The second 3D shape 3D52 includes eight blades. The second 3D data includes time stamps from time t3 to time t4 shown in FIG. 20. The time stamp (t3) of blade BL53 is the earliest, and the time stamp (t4) of blade BL54 is the latest.

[0199] The parameter generation unit 53 identifies a first connection region of a first 3D shape 3D 51 associated with time t2, and identifies a second connection region of a second 3D shape 3D 52 associated with time t3. The first connection region includes a terminal blade BL52 in the first 3D shape 3D 51. The second connection region includes a starting blade BL53 in the second 3D shape 3D 52. The parameter generation unit 53 generates position and posture transformation parameters for connecting the second connection region to the first connection region.

[0200] The parameter generation unit 53 identifies a first connection region of a first 3D shape 3D51 associated with time t1, and identifies a second connection region of a second 3D shape 3D52 associated with time t4. The first connection region includes a starting blade BL51 in the first 3D shape 3D51. The second connection region includes a terminal blade BL54 in the second 3D shape 3D52. The parameter generation unit 53 generates position and posture transformation parameters for connecting the second connection region to the first connection region.

[0201] The display control unit 56 displays the region R52 and the region R53 in the region R51. Each of the region R52 and the region R53 includes a first connection region and a second connection region.

[0202] To connect a first connection region including the blade BL51 and a second connection region including the blade BL54 to each other, a positional relationship between the blade BL51 and the blade BL54 may be determined based on features of two or more images. The two or more images are used to generate the first 3D data and the second 3D data. The first connection region and the second connection region may be connected to each other based on the features. This method is called loop closing. Known techniques can be applied to the loop closing.

[0203] When the total number of blades that actually exist is the same as the total number of blades in the 3D data, the parameter generating unit 53 can generate position and orientation conversion parameters for accurately connecting the first 3D data and the second 3D data to each other. The total number of blades in the 3D data is the total number of blades in the first 3D data and the second 3D data.

[0204] Fig. 21 shows an example of an image displayed on the display unit 71 when the total number of blades in the 3D data is one less than the total number of blades that actually exist. Explanation of the same parts as those shown in Fig. 20 will be omitted.

[0205] The second 3D shape 3D52 shown in Fig. 21 does not include the blade BL54 shown in Fig. 20. The second 3D shape 3D52 shown in Fig. 21 may not include the blade BL53 shown in Fig. 20. Therefore, in reality, the second 3D shape 3D52 may not include either the blade BL53 or the blade BL54.

[0206] After an image of the wide range of 3D data is displayed, the user may adjust the position and orientation of the 3D data in the same manner as in the first modification of the first embodiment. At this time, the user may adjust the position of the second 3D data by rotating the second 3D data. The user may rotate the first 3D data and the second 3D data. The position of the 3D data may be adjusted only on the circumference of a circle having an arbitrary diameter.

[0207] Second embodiment A second embodiment of the present invention will be described. In the second embodiment, a method for generating first 3D data and second 3D data will be described. In the following, an example in which the image display device is an endoscope device will be described.

[0208] The configuration of an endoscope device 1 in the second embodiment will be described with reference to Fig. 22 and Fig. 23. Fig. 22 shows the external appearance of the endoscope device 1. Fig. 23 shows the internal configuration of the endoscope device 1.

[0209] The endoscope device 1 shown in Fig. 22 has an insertion section 2, a main body section 3, an operation section 4, and a display section 5. The endoscope device 1 captures an image of a subject and generates an image. The subject is an industrial product. In order to observe various subjects, a user can replace the optical adapter attached to the tip 20 of the insertion section 2, select a built-in image processing program, and add an image processing program.

[0210] The insertion section 2 is inserted into the inside of the subject. The insertion section 2 is a long, thin tube that is bendable from the tip 20 to the base end. The insertion section 2 captures an image of the subject and outputs an image signal to the main body section 3. An optical adapter is attached to the tip 20 of the insertion section 2. For example, a monocular optical adapter is attached to the tip 20. The main body section 3 is a control device that includes a storage section that stores the insertion section 2. The operation section 4 accepts user operations on the endoscope device 1. The display section 5 has a display screen, and displays an image of the subject acquired by the insertion section 2, an operation menu, etc. on the display screen.

[0211] The operation unit 4 is a user interface. The display unit 5 is a monitor (display) such as an LCD (Liquid Crystal Display). The display unit 5 may be a touch panel. In that case, the operation unit 4 and the display unit 5 are integrated.

[0212] The main body 3 shown in FIG. 23 includes an endoscope unit 8, a CCU (Camera Control Unit) 9, and a control device 10.

[0213] The endoscope unit 8 has a light source device and a bending device, not shown. The light source device supplies illumination light required for observation to the tip 20. The bending device bends the bending mechanism built into the insertion portion 2.

[0214] The imaging element 28 is built into the tip 20 of the insertion portion 2. The imaging element 28 is an image sensor. The imaging element 28 photoelectrically converts an optical image of a subject formed by the optical adapter, and generates an imaging signal.

[0215] The CCU 9 drives the imaging element 28. An imaging signal output from the imaging element 28 is input to the CCU 9. The CCU 9 performs pre-processing, including amplification and noise removal, on the imaging signal acquired by the imaging element 28. The CCU 9 converts the pre-processed imaging signal into a video signal such as an NTSC signal.

[0216] The control device 10 includes a video signal processing circuit 12, a ROM (Read Only Memory) 13, a RAM (Random Access Memory) 14, a card interface 15, an external device interface 16, a control interface 17, and a CPU (Central Processing Unit) 18.

[0217] The video signal processing circuit 12 performs a predetermined video processing on the video signal output from the CCU 9. For example, the video signal processing circuit 12 performs video processing related to improving visibility. For example, the video processing includes color reproduction, gradation correction, noise suppression, and edge enhancement. For example, the video signal processing circuit 12 synthesizes the video signal output from the CCU 9 with a graphic image signal generated by the CPU 18. The graphic image signal includes an image of an operation screen, etc. The video signal processing circuit 12 outputs the synthesized video signal to the display unit 5.

[0218] The ROM 13 is a non-volatile recording medium in which a program is recorded for the CPU 18 to control the operation of the endoscope apparatus 1. The RAM 14 is a volatile recording medium in which information used by the CPU 18 to control the endoscope apparatus 1 is temporarily stored. The CPU 18 controls the operation of the endoscope apparatus 1 based on the program recorded in the ROM 13.

[0219] A memory card 42 is connected to the card interface 15. The memory card 42 is a recording medium that is detachable from the endoscope device 1. The card interface 15 imports the control processing information, image information, etc. stored in the memory card 42 into the control device 10. In addition, the card interface 15 records the control processing information, image information, etc. generated by the endoscope device 1 in the memory card 42.

[0220] An external device such as a USB device is connected to the external device interface 16. For example, a personal computer (PC) 41 is connected to the external device interface 16. The external device interface 16 transmits information to the PC 41 and receives information from the PC 41. This allows the PC 41 to display information. Also, a user can perform operations related to the control of the endoscope device 1 by inputting instructions to the PC 41.

[0221] The control interface 17 communicates with the operation section 4, the endoscope unit 8, and the CCU 9 for operational control. The control interface 17 notifies the CPU 18 of information input to the operation section 4 by a user. The control interface 17 outputs control signals for controlling the light source device and the bending device to the endoscope unit 8. The control interface 17 outputs control signals for controlling the image sensor 28 to the CCU 9.

[0222] The program executed by the CPU 18 may be recorded in a computer-readable recording medium. The program recorded in this recording medium may be read and executed by a computer other than the endoscope device 1. For example, the PC 41 may read and execute the program. The PC 41 may control the endoscope device 1 by transmitting control information for controlling the endoscope device 1 to the endoscope device 1 according to the program. Alternatively, the PC 41 may obtain a video signal from the endoscope device 1 and process the obtained video signal.

[0223] The image sensor 28 is a camera that captures a set of still images. The set of still images includes two or more images. Each of the two or more images is temporally associated with other images included in the set of two or more images. For example, each of the two or more images is a still image. A video may be used instead of the set of still images. Each of the two or more frames included in the video is associated with one another by a timestamp (time code).

[0224] For example, the image sensor 28 is a monocular camera having one viewpoint. In this case, each of the two or more still images is an image captured by the monocular camera. The camera in the second embodiment includes the image sensor 28 and an observation optical system.

[0225] As described above, the endoscope device 1 has the imaging element 28 and the CPU 18. The imaging element 28 captures an image of a subject and generates an imaging signal. The imaging signal includes an image of the subject. Thus, the imaging element 28 captures an image of the subject and obtains the image of the subject. The image is a two-dimensional image (2D image). The image obtained by the imaging element 28 is input to the CPU 18 via the video signal processing circuit 12.

[0226] Fig. 24 shows a functional configuration of CPU 18. The functions of CPU 18 include a control unit 180, a data acquisition unit 181, a parameter generation unit 182, a conversion unit 183, a data generation unit 184, a display control unit 185, an information reception unit 186, a structure estimation unit 187, and a data generation unit 188. At least one of the blocks shown in Fig. 24 may be configured by a circuit different from CPU 18.

[0227] Each unit shown in Fig. 24 may be composed of at least one of a processor and a logic circuit. Each unit shown in Fig. 24 may include one or more processors. Each unit shown in Fig. 24 may include one or more logic circuits.

[0228] The control unit 180 controls the processes executed by each unit shown in FIG.

[0229] The data acquisition unit 181 has the same function as the data acquisition unit 52 shown in Fig. 1. The data acquisition unit 181 is connected to the RAM 14, and acquires the first 3D data and the second 3D data from the RAM 14. The data acquisition unit 181 may acquire the first 3D data and the second 3D data from a recording medium in the PC 41 or the memory card 42.

[0230] The parameter generating unit 182 has the same function as the parameter generating unit 53 shown in Fig. 1. The parameter generating unit 182 generates conversion parameters for converting the first 3D coordinate system and the second 3D coordinate system into a common coordinate system.

[0231] The conversion unit 183 has the same function as the conversion unit 54 shown in Fig. 1. The conversion unit 183 converts the first 3D coordinate system and the second 3D coordinate system into a common coordinate system by using the conversion parameters generated by the parameter generation unit 182. In this way, the conversion unit 183 converts each of the first 3D data and the second 3D data into 3D data in the common coordinate system.

[0232] The data generating unit 184 has the same function as the data generating unit 55 shown in Fig. 1. The data generating unit 184 connects the first 3D data and the second 3D data converted into 3D data in a common coordinate system by the conversion unit 183. In this way, the data generating unit 184 generates 3D data of a wide range of the subject.

[0233] The display control unit 185 controls the processing executed by the video signal processing circuit 12. The CCU 9 outputs a video signal. The video signal includes color data of each pixel of the image acquired by the imaging element 28. The display control unit 185 causes the video signal processing circuit 12 to output the video signal output from the CCU 9 to the display unit 5. The video signal processing circuit 12 outputs the video signal to the display unit 5. The display unit 5 displays an image based on the video signal output from the video signal processing circuit 12. As a result, the display control unit 185 displays the image acquired by the imaging element 28 on the display unit 5.

[0234] The display control unit 185 displays various information on the display unit 5. That is, the display control unit 185 displays various information on an image. The various information may include a cursor. The cursor is a mark that allows the user to specify a specific point on the image.

[0235] For example, the display control unit 185 generates a graphic image signal of various information. The display control unit 185 outputs the generated graphic image signal to the video signal processing circuit 12. The video signal processing circuit 12 combines the video signal output from the CCU 9 and the graphic image signal output from the CPU 18. This causes the various information to be superimposed on the image. The video signal processing circuit 12 outputs the combined video signal to the display unit 5. The display unit 5 displays the image on which the various information is superimposed.

[0236] 1. The display control unit 185 generates a graphic image signal of 3D data. The display control unit 185 outputs the graphic image signal to the video signal processing circuit 12. The same processing as that described above is executed, and the display unit 5 displays an image of the 3D data. As a result, the display control unit 185 displays the image of the 3D data on the display unit 5.

[0237] A user inputs various information to the endoscope device 1 by operating the operation unit 4. The operation unit 4 outputs the information input by the user. The information is input to a control interface 17, which is an input unit. The information is output from the control interface 17 to the CPU 18. An information receiving unit 186 receives the information input to the endoscope device 1 via the operation unit 4.

[0238] For example, the user operates the operation unit 4 to input cursor position information to the endoscope device 1. If the display unit 5 is configured as a touch panel, the user touches the screen of the display unit 5 to input position information indicating a position on an image to the endoscope device 1. The information receiving unit 186 receives the position information input to the endoscope device 1. The information receiving unit 186 calculates a position on the image based on the position information. The display control unit 185 displays the cursor at the position calculated by the information receiving unit 186.

[0239] Moreover, the information receiving unit 186 has the same function as the information receiving unit 57 shown in Fig. 1. The information receiving unit 186 receives structure information.

[0240] The structure estimation unit 187 has the same function as the structure estimation unit 58 shown in Fig. 1. When the information receiving unit 186 receives information that does not include structure information, the structure estimation unit 187 estimates the structure of the subject in the connection region based on the information, and generates structure information.

[0241] The data generating unit 188 has the functions shown in Fig. 25. Fig. 25 shows the functional configuration of the data generating unit 188. The functions of the data generating unit 188 include an image acquiring unit 1880, a condition accepting unit 1881, and a data calculating unit 1882.

[0242] The image acquisition unit 1880 acquires the still image group from the RAM 14. The image acquisition unit 1880 may acquire the still image group from a recording medium in the PC 41 or the memory card 42.

[0243] A user operates the operation unit 4 to input information indicating conditions for generating 3D data to the endoscope device 1. The condition receiving unit 1881 receives the conditions for generating 3D data based on the information input by the user. Specifically, the conditions include internal parameters of the camera, distortion correction parameters of the camera, setting values, and a reference length. The setting values ​​are used in various processes for generating 3D data. The reference length is used to match the 3D data with the scale of the actual subject.

[0244] The data calculation unit 1882 generates (reconstructs) 3D data of the subject based on two or more images included in the still image group. The data calculation unit 1882 does not need to use all images included in the still image group. When the still image group includes three or more images, the data calculation unit 1882 generates 3D data based on all or a part of the still image group. The 3D data includes 3D coordinates of two or more points (three-dimensional point cloud) of the subject, camera coordinates, and posture information.

[0245] The camera coordinates indicate the 3D coordinates of a camera that captured each of the two or more images, and are associated with each of the two or more images. The camera coordinates indicate the 3D coordinates of a viewpoint when an image was captured. For example, the camera coordinates indicate the 3D coordinates of an observation optical system that the camera has. The attitude information indicates the attitude of a camera that captured each of the two or more images, and is associated with each of the two or more images. For example, the attitude information indicates the attitude of an observation optical system that the camera has.

[0246] A specific procedure of the process executed by the data calculation unit 1882 will be described. The data calculation unit 1882 uses the still image group acquired by the image acquisition unit 1880 and the conditions accepted by the condition acceptance unit 1881. An example in which the data calculation unit 1882 uses two images (still images) included in the still image group will be described below. When the two images are generated, the two viewpoints of the camera are different from each other. Even when three or more images are used, the basic principle is the same as when two images are used. The method described below can also be applied to the case where three or more images are used.

[0247] In the method described below, feature points are detected in each of two images captured from two different viewpoints, and the feature points are associated with each other. In addition, in the method described below, the camera position, the camera attitude, and the 3D coordinates of the feature points are estimated based on the feature points. The method using the feature point information is called the Indirect Method. The method applied to each embodiment of the present invention is not limited to this method.

[0248] For example, there is a method of directly using pixel values ​​of two images captured at two different viewpoints. By using this method, the camera position, camera pose, and 3D coordinates corresponding to each pixel are estimated. This method is called the Direct Method. This method may be used in each embodiment of the present invention. Any method may be used as long as it is possible to estimate the camera position, camera pose, and 3D coordinates of an object by using two or more images captured at two or more different viewpoints.

[0249] Fig. 26 shows a schematic diagram of an image acquisition situation in which two images of a subject are acquired. In the following description, the term "camera" is used in a broad sense. In the case where an endoscope acquires images, the camera in the following description specifically refers to the observation optical system at the tip of the endoscope.

[0250] As shown in Fig. 26, first, an image I1 is acquired in an imaging state c1 of the camera. Next, an image I2 is acquired in an imaging state c2 of the camera. At least one of the imaging position and the imaging attitude is different between the imaging states c1 and c2. In Fig. 26, both the imaging position and the imaging attitude are different between the imaging states c1 and c2.

[0251] In each embodiment of the present invention, it is assumed that the images I1 and I2 are acquired by the same endoscope. In addition, in each embodiment of the present invention, it is assumed that the parameters of the objective optical system of the endoscope do not change. The parameters of the objective optical system include focal length, distortion aberration, and pixel size of the image sensor. Hereinafter, the parameters of the objective optical system are abbreviated to internal parameters for convenience. Under such conditions, the internal parameters describing the characteristics of the optical system of the endoscope can be commonly used regardless of the position and attitude of the camera (observation optical system). In each embodiment of the present invention, it is assumed that the internal parameters are acquired at the time of shipment from the factory. In addition, in each embodiment of the present invention, it is assumed that the internal parameters are known when the image is acquired.

[0252] In each embodiment of the present invention, it is assumed that two or more images are extracted from a group of still images, and that the group of still images is acquired by one endoscope. However, the present invention is not limited to this. For example, the present invention can be applied to a case where a 3D model is restored using a group of still images acquired by a plurality of endoscopes. In this case, it is only necessary that the image I1 and the image I2 are acquired using different endoscope devices, and different internal parameters are held for each endoscope. Even if the internal parameters are unknown, it is possible to perform calculations using the internal parameters as variables. Therefore, the subsequent procedures do not change significantly depending on whether the internal parameters are known or not.

[0253] The process of calculating the 3D coordinates of an object based on two images and generating 3D data will be described with reference to Fig. 27. Fig. 27 shows the procedure of the process of generating 3D data.

[0254] First, the data calculation unit 1882 executes a feature point detection process (step S200). In the feature point detection process, the data calculation unit 1882 detects feature points of each of the two images. A feature point refers to a corner, an edge, or the like having a large image brightness gradient among the subject information captured in the image. Methods for detecting this feature point include SIFT (Scale-invariant Feature Transform) and FAST (Features from Accelerated Segment Test). The data calculation unit 1882 can detect feature points in the images by using such methods.

[0255] Figure 26 shows the feature point P 11 is detected, and feature point P 12In Figure 26, only one feature point is shown for each image, but in reality, multiple feature points are detected in each image. The number of feature points detected may vary between images. Each feature point detected from each image is converted into data called a feature amount. A feature amount is data that represents the characteristics of a feature point.

[0256] After step S200, the data calculation unit 1882 executes a feature point association process (step S201). In the feature point association process, the data calculation unit 1882 compares the correlation of feature amounts between images for each feature point detected by the feature point detection process (step S200). When the correlation of feature amounts is compared and feature points with similar feature amounts are found in each image, the data calculation unit 1882 holds the information in the RAM 14. In this way, the data calculation unit 1882 associates the feature points of each image with each other. On the other hand, when feature points with similar feature amounts are not found, the data calculation unit 1882 discards the information of the feature points.

[0257] After step S201, the data calculation unit 1882 reads out the coordinates of feature points (feature point pairs) of two images associated with each other from the RAM 14. The data calculation unit 1882 executes a position and orientation calculation process based on the read out coordinates (step S202). In the position and orientation calculation process, the data calculation unit 1882 calculates the relative position and orientation between the imaging state c1 of the camera that acquired the image I1 and the imaging state c2 of the camera that acquired the image I2. More specifically, the data calculation unit 1882 calculates a matrix E by solving the following equation (1) that utilizes the epipolar constraint.

[0258]

number

[0259] The matrix E is called a fundamental matrix. The fundamental matrix E is a matrix that holds the relative position and orientation between the imaging state c1 of the camera that acquired the image I1 and the imaging state c2 of the camera that acquired the image I2. In equation (1), the matrix p1 is a matrix that includes the coordinates of feature points detected from the image I1. The matrix p2 is a matrix that includes the coordinates of feature points detected from the image I2. The fundamental matrix E includes information about the relative position and orientation of the camera, and therefore corresponds to the external parameters of the camera. The data calculation unit 1882 can solve the fundamental matrix E by using a known algorithm.

[0260] As shown in FIG. 26, when the amount of change in camera position (relative position) is t and the amount of change in camera attitude (relative attitude) is R, formulas (2) and (3) hold true.

[0261]

number

[0262] In equation (2), the amount of movement in the x-axis direction is t x and the amount of movement in the y-axis direction is t y and the amount of movement in the z-axis direction is t z In equation (3), the amount of rotation around the x-axis, α, is R x (α), and the amount of rotation around the y-axis is R y (β), and the amount of rotation around the z-axis γ is R z After the fundamental matrix E is calculated, an optimization process called bundle adjustment may be performed to improve the accuracy of the reconstruction of the 3D coordinates.

[0263] The data calculation unit 1882 calculates 3D coordinates (camera coordinates) in the coordinate system of the 3D model by using the calculated amount of change in position of the camera. For example, the data calculation unit 1882 defines the 3D coordinates of the camera that acquired the image I1. The data calculation unit 1882 calculates the 3D coordinates of the camera that acquired the image I2 based on the 3D coordinates of the camera that acquired the image I1 and the amount of change in position of the camera that acquired the image I2.

[0264] The data calculation unit 1882 calculates posture information in the coordinate system of the 3D model by using the calculated posture change amount of the camera. For example, the data calculation unit 1882 defines posture information of the camera that acquired the image I1. The data calculation unit 1882 generates posture information of the camera that acquired the image I2 based on the posture information of the camera that acquired the image I1 and the posture change amount of the camera that acquired the image I2.

[0265] The data calculation unit 1882 executes a position and orientation calculation process (step S202) to generate three-dimensional shape (3D shape) data (3D shape data) including 3D coordinates (camera coordinates) at the camera position and orientation information indicating the camera orientation. When a method such as Structure from Motion or visual-SLAM is applied to the position and orientation calculation process (step S202), the data calculation unit 1882 further calculates the 3D coordinates of each feature point in step S202. The 3D shape data generated in step S202 does not include 3D coordinates of points on the subject other than the feature points. Therefore, the 3D shape data indicates a sparse 3D shape of the subject.

[0266] The 3D shape data includes the 3D coordinates of each feature point, the above camera coordinates, and the above posture information. The 3D coordinates of each feature point are defined in the coordinate system of the 3D data. The 3D coordinates of each feature point are associated with the two-dimensional coordinates (2D coordinates) of each feature point. The 2D coordinates of each feature point are defined in the coordinate system of the image in which each feature point is included. The 2D and 3D coordinates of each feature point are associated with the image in which each feature point is included.

[0267] After step S202, the data calculation unit 1882 executes a three-dimensional shape reconstruction process based on the relative position and orientation (position change amount t and orientation change amount R) of the camera calculated in step S202 (step S203). The data calculation unit 1882 generates 3D data of the subject in the three-dimensional shape reconstruction process. Methods for reconstructing the three-dimensional shape of the subject include patch-based multi-view stereo (PMVS) and matching processing using parallelized stereo. However, the means are not particularly limited.

[0268] In step S203, the data calculation unit 1882 calculates the 3D coordinates of points on the subject other than the feature points. The 3D coordinates of each point other than the feature points are defined in the coordinate system of the 3D data. The 3D coordinates of each point are associated with the 2D coordinates of each point. The 2D coordinates of each point are defined in the coordinate system of the 2D image in which each point is included. The 2D coordinates and 3D coordinates of each point are associated with the 2D image in which each point is included. The data calculation unit 1882 updates the 3D shape data. The updated 3D shape data includes the 3D coordinates of each feature point, the 3D coordinates of each point other than the feature points, the camera coordinates, and the posture information. The 3D shape data updated in step S203 includes the 3D coordinates of points on the subject other than the feature points in addition to the 3D coordinates of the feature points. Therefore, the 3D shape data indicates a dense 3D shape of the subject.

[0269] After step S203, the data calculation unit 1882 executes a three-dimensional coordinate conversion process (step S204) based on the 3D shape data processed in the three-dimensional shape restoration process (step S203) and the reference length accepted by the condition acceptance unit 1881. In the three-dimensional coordinate conversion process, the data calculation unit 1882 converts the 3D shape data of the subject into three-dimensional coordinate data (3D data) having a dimension of length. When step S204 is executed, the process shown in FIG. 27 ends.

[0270] To reduce processing time, step S203 may be omitted, in which case, after step S202 is performed, step S204 is performed without performing step S203.

[0271] Step S204 may be omitted. In this case, after step S203 is executed, step S204 is not executed and the process shown in FIG. 27 ends. In this case, the 3D data indicates the relative shape of the subject without the dimension of length. Even when the 3D data indicates the relative shape of the subject, the endoscope device 1 can identify the area of ​​the 3D data corresponding to the camera coordinates.

[0272] In order for 3D data to be generated according to the principle shown in Fig. 26, at least a part of the area of ​​each image must be common to at least a part of the area of ​​at least one other image. That is, the area of ​​the first image and the area of ​​the second image different from the first image include a common area. The area other than the common area in the first image and the area other than the common area in the second image are different from each other.

[0273] After the 3D data is generated, the endoscope device 1 executes a process similar to that shown in Fig. 2. The endoscope device 1 can connect the first 3D data and the second 3D data to each other, and can display a wide range of 3D shape images. The image display device 50 shown in Fig. 1 may have a data generation unit 188.

[0274] (Third embodiment) A third embodiment of the present invention will now be described, in which data generated by a sensor is used to estimate the structure in connected regions of an object.

[0275] The image display device 50 shown in Fig. 1 is changed to an image display device 50b shown in Fig. 28. Fig. 28 shows the configuration of the image display device 50b. Description of the same configuration as that shown in Fig. 1 will be omitted.

[0276] The image display device 50b shown in Fig. 28 has a control unit 51, a data acquisition unit 52, a parameter generation unit 53, a conversion unit 54, a data generation unit 55, a display control unit 56, and a structure estimation unit 58. The image display device 50b does not have the information reception unit 57 shown in Fig. 1. The operation unit 70, the display unit 71, the communication unit 72, the storage unit 73, and the sensor 74 shown in Fig. 28 are connected to the image display device 50b. The image display device 50b may have at least one of the operation unit 70, the display unit 71, the communication unit 72, the storage unit 73, and the sensor 74.

[0277] Each unit of the image display device 50b may be composed of at least one of a processor and a logic circuit. Each unit of the image display device 50b may include one or more processors. Each unit of the image display device 50b may include one or more logic circuits.

[0278] The sensor 74 outputs sensor data. The sensor data is influenced by the geometric structure of the object. The sensor 74 is an image sensor (image pickup element).

[0279] The sensor data may include, as sensor values, pixel values, acceleration measurements, or angular velocity measurements. An image sensor generates an image of an object, in which the structure of the object is visible.

[0280] The sensor data is stored in the storage unit 73. The structure estimation unit 58 uses the sensor data to estimate the structure of the subject in the connected region and generate structure information.

[0281] The process executed by the image display device 50b will be described with reference to Fig. 29. Fig. 29 shows the procedure of the process executed by the image display device 50b. The description of the same process as that shown in Fig. 2 will be omitted.

[0282] After step S103, the data acquisition unit 52 connects to the storage unit 73, and acquires sensor data in the connection area from the storage unit 73 (step S120).

[0283] The 3D data includes a time stamp. In the sensor data, the sensor value and the time stamp are associated with each other. Therefore, the data acquisition unit 52 can acquire the sensor data generated at the time indicated by the time stamp of the 3D data. For example, the data acquisition unit 52 acquires the sensor data in the section from the time of the end of the first 3D data to the time of the start of the second 3D data from the storage unit 73.

[0284] In the following, an example will be described in which the sensor 74 is an image sensor and the data acquisition unit 52 acquires two or more images from the storage unit 73. The following processing can also be applied to the case in which the data acquisition unit 52 acquires one image from the storage unit 73.

[0285] After step S120, the structure estimation unit 58 estimates the structure of the subject in the connected region by using the two or more images acquired in step S120, and generates structure information (step S121). After step S121, step S106 is executed.

[0286] The structure estimation unit 58 may use any method to estimate the structure of the object. For example, the structure estimation unit 58 may estimate the structure of the object by using AI. The structure estimation unit 58 may estimate the structure of the object by using image feature detection, feature amount calculation, and support vector machine (SVM).

[0287] When the structure estimation unit 58 determines that the subject is a straight pipe, the structure estimation unit 58 may obtain the structure information of the straight pipe from a reference table TB11 shown in FIG.

[0288] Each aspect of the invention may include the following variations: Structural information is generated based on data output from the sensor 74 .

[0289] In the third embodiment, the structure estimation unit 58 estimates the structure of the subject in the connected region by using sensor data. The image display device 50b does not need to accept information from the user.

[0290] (First modified example of the third embodiment) A first modified example of the third embodiment of the present invention will be described. In the first modified example of the third embodiment, an image display device 50b shown in Fig. 28 is used. In the first modified example of the third embodiment, an endoscope device 1 shown in Fig. 22 is used to generate 3D data.

[0291] In the third embodiment described above, the structure estimation unit 58 estimates the structure of the subject in the connected region by using an image generated by an image sensor. On the other hand, in a first modified example of the third embodiment, the structure estimation unit 58 uses sensor data output from an Inertial Measurement Unit (IMU) instead of an image.

[0292] An IMU is used as the sensor 74. The sensor 74 has an acceleration sensor and an angular velocity sensor, and detects acceleration and angular velocity. For example, the sensor 74 is disposed at the tip 20 of the insertion section 2. When the insertion section 2 is inserted into a pipe, the insertion section 2 moves according to the structure of the pipe. Therefore, the sensor 74 outputs sensor data according to the structure of the pipe. The structure estimation section 58 calculates the trajectory of the tip 20 by using the sensor data. The trajectory indicates two or more positions where the tip 20 is disposed.

[0293] Image display device 50b executes the process shown in Fig. 29. In step S121, structure estimation unit 58 uses sensor data to calculate the trajectory of tip 20. In step S121, structure estimation unit 58 estimates the structure of the subject in the connection region based on the trajectory, and generates structure information.

[0294] For example, the structure estimation unit 58 estimates the structure of the subject by performing the following process: Each of the first 3D data and the second 3D data includes 3D coordinates in a first interval, a second interval, and a third interval. The first interval includes a start end of the 3D shape. The second interval includes an end end of the 3D shape. The third interval is disposed between the first interval and the second interval. For example, the second interval in the first 3D data and the first interval in the second 3D data correspond to a connection region.

[0295] The structure estimation unit 58 analyzes the trajectory of the tip 20 in each of the second section in the first 3D data, the third section in the first 3D data close to the second section, the first section in the second 3D data, and the third section in the second 3D data close to the first section. If the structure estimation unit 58 determines that the tip 20 moves in a substantially straight line in all sections, the structure estimation unit 58 determines that the subject is a straight pipe. If the structure estimation unit 58 determines that the moving direction of the tip 20 turns approximately 90 degrees while the tip 20 passes through each of the above sections, the structure estimation unit 58 determines that the subject is a 90-degree joint. The structure estimation unit 58 generates structure information according to the structure of the subject.

[0296] (Second modified example of the third embodiment) A second modified example of the third embodiment of the present invention will be described. In the second modified example of the third embodiment, an image display device 50b shown in FIG. 28 is used. The image display device 50b does not need to have a sensor 74. In the second modified example of the third embodiment, an endoscope device 1 shown in FIG. 22 is used to generate 3D data.

[0297] In the above-described third embodiment, the structure estimation unit 58 estimates the structure of the subject in the connection region by using the image generated by the image sensor. On the other hand, in the second modified example of the third embodiment, the structure estimation unit 58 uses history information of operations related to bending of the insertion unit 2 instead of the image.

[0298] The user operates the operation section 4 to input information indicating the bending direction and bending amount to the endoscope device 1. The control section 180 of the CPU 18 generates a control signal based on the information input by the user. The control signal is output to the endoscope unit 8 via the control interface 17. The bending device of the endoscope unit 8 bends the bending mechanism based on the control signal.

[0299] The control unit 180 generates history information of operations related to bending of the insertion unit 2. The history information includes information indicating a bending direction and a bending amount. The communication unit 72 receives the history information from the endoscope device 1. The history information is stored in the storage unit 73.

[0300] The image display device 50b executes the process shown in Fig. 29. In step S20, the data acquisition unit 52 acquires history information instead of sensor data from the storage unit 73. In step S121, the structure estimation unit 58 uses the history information instead of the sensor data to estimate the structure of the subject in the connection region and generate structure information.

[0301] For example, the structure estimation unit 58 estimates the structure of the subject by performing the following process: Each of the first 3D data and the second 3D data includes 3D coordinates in a first interval, a second interval, and a third interval. The first interval includes a start end of the 3D shape. The second interval includes an end end of the 3D shape. The third interval is disposed between the first interval and the second interval. For example, the second interval in the first 3D data and the first interval in the second 3D data correspond to a connection region.

[0302] The structure estimation unit 58 analyzes history information for each of the second section in the first 3D data, the third section in the first 3D data close to the second section, the first section in the second 3D data, and the third section in the second 3D data close to the first section. If the structure estimation unit 58 determines that the insertion portion 2 is not curved in all sections, the structure estimation unit 58 determines that the subject is a straight pipe. If the structure estimation unit 58 determines that the insertion portion 2 is curved approximately 90 degrees while the tip 20 of the insertion portion 2 passes through each of the above sections, the structure estimation unit 58 determines that the subject is a 90-degree joint.

[0303] Each aspect of the present invention may include the following variations: The first 3D data is generated by using two or more first images. The second 3D data is generated by using two or more second images. The two or more first images and the two or more second images are generated based on an optical image of a subject acquired by the insertion portion 2. The insertion portion 2 is inserted into an object having a subject and is bendable. The structural information is generated based on information indicating a bending direction and a bending amount of the insertion portion 2.

[0304] (Fourth embodiment) A fourth embodiment of the present invention will be described. In the fourth embodiment, an image display device 50 shown in FIG. 1 is used. Reference data is used as structural information. For example, the reference data is design data including design values ​​of a geometric structure. For example, the design data is generated by using 3D-CAD (computer-aided design).

[0305] The process executed by the image display device 50 will be described with reference to Fig. 30. Fig. 30 shows the procedure of the process executed by the image display device 50. The description of the same process as that shown in Fig. 2 will be omitted.

[0306] After step S103, the information receiving unit 57 receives the reference data (step S130). For example, the communication unit 72 receives the reference data from an external device. The information receiving unit 57 receives the reference data received by the communication unit 72.

[0307] After step S130, the display control unit 56 displays the image of the reference data on the display unit 71 (step S131).

[0308] FIG. 31 shows an example of an image displayed on the display unit 71. The display control unit 56 displays an image IMG61 on the display unit 71. The image IMG61 includes a region R61, a region R62, and a region R63. An image of a first 3D shape 3D61 is displayed in the region R61. The first 3D shape 3D61 is shown by the first 3D data. An image of a second 3D shape 3D62 is displayed in the region R62. The second 3D shape 3D62 is shown by the second 3D data. An image of a third 3D shape 3D63 is displayed in the region R63. The third 3D shape 3D63 is shown by the reference data.

[0309] When information accepting unit 57 accepts reference data, three types of images shown in Fig. 31 are displayed. When information accepting unit 57 accepts reference data, it is not necessary for the image of the reference data to be displayed on display unit 71. Therefore, it is not necessary to execute step S131.

[0310] After step S131, the parameter generating unit 53 generates conversion parameters by using the reference data (step S132). After step S132, step S107 is executed.

[0311] The parameter generating unit 53 executes the following process in step S132. The parameter generating unit 53 executes shape matching using the first 3D data and the reference data. As a result, the parameter generating unit 53 detects the 3D shape of the reference data corresponding to the 3D shape of the first 3D data. The parameter generating unit 53 executes shape matching using the second 3D data and the reference data. As a result, the parameter generating unit 53 detects the 3D shape of the reference data corresponding to the 3D shape of the second 3D data.

[0312] A known method may be applied to 3D shape matching. The parameter generating unit 53 may perform a matching process that uses color information in addition to shape information.

[0313] The parameter generating unit 53 generates a position and orientation conversion parameter for matching the position and orientation of the first 3D coordinate system with the position and orientation of the common coordinate system. The parameter generating unit 53 generates a scale conversion parameter for matching the scale of the first 3D coordinate system with the scale of the common coordinate system. The common coordinate system is the 3D coordinate system of the reference data.

[0314] The parameter generating unit 53 generates a position and orientation transformation parameter for matching the position and orientation of the second 3D coordinate system with the position and orientation of the common coordinate system, and generates a scale transformation parameter for matching the scale of the second 3D coordinate system with the scale of the common coordinate system.

[0315] FIG. 32 shows an example of an image displayed on the display unit 71. The description of the same parts as those shown in FIG. 31 will be omitted. The display control unit 56 displays an image IMG62 on the display unit 71. The image IMG62 includes an area R64. Images of a first 3D shape 3D61, a second 3D shape 3D62, and a third 3D shape 3D63 are displayed in the area R64. The user can confirm that each of the first 3D data and the second 3D data is placed at a correct position in the reference data. This allows the user to confirm that the first 3D data and the second 3D data are accurately connected to each other.

[0316] The reference data used as the structural information is not limited to the design data, and may be data other than the design data.

[0317] For example, 3D data generated by using two or more images of the object of inspection generated in a previous inspection may be used as the reference data. The 3D data is generated in the process shown in Fig. 27. The 3D data is different from both the first 3D data and the second 3D data. The 3D data used as the reference data may be generated by an equipment different from the endoscopic equipment that generates the 3D data of the object inside the object of inspection.

[0318] As long as the parameter generating unit 53 can match the positions of the first 3D data and the second 3D data with the position of the reference data, the reference data does not need to be 3D data. The reference data may be two-dimensional data (2D data). The parameter generating unit 53 converts the 3D data into 2D information to match the positions of the 3D data and the 2D data. The 2D information indicates a two-dimensional shape on an arbitrary plane. The parameter generating unit 53 matches the position of the 2D information with the position of the 2D data.

[0319] Each aspect of the present invention may include the following variations: The structural information is design data including design values ​​of a geometric structure of the subject, or is 3D data different from both the first 3D data and the second 3D data.

[0320] In the fourth embodiment, the image display device 50 can connect the first 3D data and the second 3D data to each other, and can display images of a wide range of 3D shapes on the image of the reference data.

[0321] (First modified example of the fourth embodiment) A first modified example of the fourth embodiment of the present invention will be described. In the above-mentioned fourth embodiment, the ID of each blade is not used to connect the first 3D data and the second 3D data to each other. On the other hand, in the first modified example of the fourth embodiment, the first 3D data and the second 3D data are connected to each other so that the ID of each blade of the first 3D data and the second 3D data matches the ID of each blade of the reference data.

[0322] In a first modified example of the fourth embodiment, the image display device 50 shown in FIG. 1 is used. The storage unit 73 stores reference data of a gas turbine. FIG. 33 shows the reference data of a gas turbine GT70. In the example shown in FIG. 33, the gas turbine GT70 has 12 blades. An ID is assigned to each blade. The reference data shown in FIG. 33 may be a cross-sectional view of the gas turbine GT70 or may be 3D-CAD data.

[0323] Each ID corresponds to the position of the blade. For example, when consecutive IDs are assigned to the blades, two consecutive IDs are assigned to two adjacent blades. The structural information in the first modification of the fourth embodiment is the reference data and the ID assigned to each blade.

[0324] In step S106, the parameter generating unit 53 executes the process shown in Fig. 34. Fig. 34 shows the procedure of the process executed by the parameter generating unit 53.

[0325] The parameter generating unit 53 identifies blade regions in the first 3D data (step S106j). Each blade region includes one blade.

[0326] The parameter generating unit 53 may use any method to identify the blade region. For example, the parameter generating unit 53 extracts two or more shapes that are similar to each other in the first 3D data as the blade region. Alternatively, the parameter generating unit 53 performs shape matching using a preregistered blade shape and the first 3D data, and extracts a blade region that includes a shape that matches the blade shape.

[0327] After step S106j, the parameter generating unit 53 assigns an ID to the blade region identified in step S106j (step S106k).

[0328] The parameter generating unit 53 may use any method to assign IDs to blade regions. For example, an image of the first 3D data is displayed on the display unit 71. The user operates the operation unit 70 to input the ID of each blade to the image display device 50. The parameter generating unit 53 assigns the ID to the blade region. The parameter generating unit 53 may also detect a reference blade having a characteristic structure from the first 3D data, and assign IDs to each blade in order starting from the reference blade.

[0329] A turning tool may be used when the endoscope device generates an image used to generate 3D data. The turning tool automatically rotates the disk. The turning tool outputs information on the rotation angle of the disk to the endoscope device. Based on the information, the endoscope device determines the position of the blade shown in the image.

[0330] For example, before the turning tool rotates the disk, the endoscopic instrument sets the blade that appears in the image as the reference blade. The total number of blades is known, and the angle between two adjacent blades is known. While the disk is rotating, the endoscopic instrument determines which blade appears in the image based on the rotation angle of the disk. When the disk has rotated once, the endoscopic instrument can determine that the reference blade appears in the image again.

[0331] The endoscopic instrument assigns an ID to each blade and embeds the ID in the image. The endoscopic instrument may assign an ID to each blade in the image and embed the ID in the image without using a turning tool.

[0332] The 3D coordinates of each point included in the 3D data are associated with the image used to generate the 3D data. The parameter generating unit 53 obtains the ID of the image associated with the blade region identified in step S106j, and assigns the ID to the blade region.

[0333] After step S106k, the parameter generating unit 53 identifies the blade region in the second 3D data (step S106l). Step S106l is similar to step S106j.

[0334] After step S106l, the parameter generating unit 53 assigns an ID to the blade region identified in step S106l (step S106m). Step S106m is similar to step S106k.

[0335] After step S106m, the parameter generation unit 53 calculates transformation parameters for correcting the position, orientation, and scale of the first 3D data so that the ID of the blade in the reference data and the ID of the blade region in the first 3D data match each other (step S106n).

[0336] After step S106n, the parameter generating unit 53 calculates conversion parameters for correcting the position, orientation, and scale of the second 3D data so that the ID of the blade in the reference data and the ID of the blade region in the second 3D data match each other (step S106o). When step S106o is executed, the process shown in FIG. 34 ends.

[0337] FIG. 35 shows an example of an image displayed on the display unit 71. The display control unit 56 displays an image IMG71 on the display unit 71. The image IMG71 includes an area R71. Images of a first 3D shape 3D71, a second 3D shape 3D72, and a third 3D shape 3D73 are displayed in the area R71. The first 3D shape 3D71 is represented by the first 3D data. The second 3D shape 3D72 is represented by the second 3D data. The third 3D shape 3D73 is represented by reference data.

[0338] The first 3D shape 3D71 includes two blades. The IDs of the two blades are 2 and 3. The position of the first 3D data is set to a position overlapping with the two blades of the reference data assigned IDs 2 and 3.

[0339] The second 3D shape 3D72 includes five blades. The IDs of the five blades are 7 to 11. The position of the second 3D data is set to a position overlapping with the five blades of the reference data that are assigned IDs 7 to 11.

[0340] Blades assigned IDs 1, 4, 5, 6, or 12 do not exist in the first 3D data or the second 3D data. Blades in the reference data assigned these IDs are displayed.

[0341] The display control unit 56 may display each blade on the display unit 71 so that the user can distinguish between the blades included in the first 3D data or the second 3D data and the blades included in the reference data. For example, the color of the blades included in the first 3D data or the second 3D data may be different from the color of the blades included in the reference data. The user can check the blades included in the 3D data and the blades not included in the 3D data.

[0342] The parameter generating unit 53 may execute the process shown in Fig. 19 in addition to the process shown in Fig. 34. This enables the parameter generating unit 53 to connect the first 3D data and the second 3D data to each other more accurately.

[0343] Each aspect of the present invention may include the following modifications: The subject includes two or more objects, and the structural information indicates positions at which the two or more objects are arranged.

[0344] In the first modification of the fourth embodiment, the parameter generating unit 53 generates the conversion parameters by using the ID as the structural information, so that the image display device 50 can accurately connect the first 3D data and the second 3D data.

[0345] (Second modified example of the fourth embodiment) A second modified example of the fourth embodiment of the present invention will now be described. In the second modified example of the fourth embodiment, an image display device 50 shown in FIG.

[0346] The object in the first modified example of the fourth embodiment described above is a gas turbine blade. On the other hand, the object in the second modified example of the fourth embodiment is a combustion chamber of the gas turbine. The combustion chamber includes a fuel injection nozzle and a plate as typical inspection parts. The plate is disposed around the fuel injection nozzle.

[0347] Fuel injection nozzles, like blades, are often arranged in a ring. An ID is often assigned to each fuel injection nozzle. Each ID has a value according to the position where each fuel injection nozzle is arranged. Each fuel injection nozzle is arranged on an arbitrary circumference, and the center position of the circle is defined.

[0348] FIG. 36 shows the structure of a combustion chamber CC80. The combustion chamber CC80 has 12 fuel injection nozzles NZ80. The actual number of fuel injection nozzles varies depending on the engine model. The 12 fuel injection nozzles NZ80 are arranged in the front stage of the combustion chamber CC80. A center position CP80 indicates the center of a circle in which the 12 fuel injection nozzles NZ80 are arranged. In a second modification of the fourth embodiment, the first 3D data and the second 3D data are connected to each other by using an ID assigned to the fuel injection nozzle.

[0349] Information INF10 in the dialog box DL10 shown in Fig. 4 includes information on the gas turbine to be inspected and information on the combustion chamber in the gas turbine. For example, the user selects "gas turbine" and then selects "combustion chamber." When the user inputs information indicating that the selection has been confirmed into the image display device 50, the information receiving unit 57 receives the information selected by the user. At this time, the information receiving unit 57 receives the character string "combustion chamber."

[0350] For example, the structural information of the combustion chamber indicates that the fuel injection nozzles in each of the first 3D data and the second 3D data are arranged at equal intervals on a circumference having an arbitrary radius of curvature. The structural information indicates that the fuel injection nozzles have the same size. In step S106, the parameter generating unit 53 generates the conversion parameters based on the structural information.

[0351] When the engine model is specified, the total number of fuel injection nozzles, the size of each fuel injection nozzle, and the ID of each fuel injection nozzle are known. The structural information may include the total number of fuel injection nozzles, the size of each fuel injection nozzle, and the ID of each fuel injection nozzle. In this case, the parameter generating unit 53 can generate position and orientation conversion parameters for accurately connecting the first 3D data and the second 3D data to each other.

[0352] In step S106, the parameter generating unit 53 executes the process shown in Fig. 34. The process shown in Fig. 34 is also applied when the subject is a combustion chamber.

[0353] FIG. 37 shows an example of an image displayed on the display unit 71. The display control unit 56 displays an image IMG81 on the display unit 71. The image IMG81 includes an area R81. Images of a first 3D shape 3D81, a second 3D shape 3D82, and a third 3D shape 3D83 are displayed in the area R81. The first 3D shape 3D81 is represented by the first 3D data. The second 3D shape 3D82 is represented by the second 3D data. The third 3D shape 3D83 is represented by reference data.

[0354] The first 3D shape 3D81 includes four fuel injection nozzles. The IDs of the four fuel injection nozzles are 2 to 5. The position of the first 3D data is set to a position overlapping with the four fuel injection nozzles of the reference data assigned IDs 2 to 5.

[0355] The second 3D shape 3D82 includes four fuel injection nozzles. The IDs of the four fuel injection nozzles are 9 to 12. The position of the second 3D data is set to a position overlapping with the four fuel injection nozzles of the reference data assigned IDs 9 to 12.

[0356] Fuel injection nozzles assigned with IDs 1, 6, 7, or 8 do not exist in the first 3D data and the second 3D data. Fuel injection nozzles in the reference data assigned with these IDs are displayed.

[0357] The display control unit 56 may display each blade on the display unit 71 so that the user can distinguish between the fuel injection nozzles included in the first 3D data or the second 3D data and the fuel injection nozzles included in the reference data. For example, the color of the fuel injection nozzles included in the first 3D data or the second 3D data may be different from the color of the fuel injection nozzles included in the reference data. The user can check the fuel injection nozzles included in the 3D data and the fuel injection nozzles not included in the 3D data.

[0358] In the second modification of the fourth embodiment, the parameter generating unit 53 generates the conversion parameters by using the ID as the structural information, so that the image display device 50 can accurately connect the first 3D data and the second 3D data.

[0359] The first and second variants of the fourth embodiment are applied to the inspection of a gas turbine of an aircraft engine. These variants may be applied to the inspection of a tubular structure. For example, these variants may be applied to the inspection of a heat exchange tube. The heat exchange tube has dozens of thin tubes. An ID may be assigned to each tube. The parameter generating unit 53 can connect the first 3D data and the second 3D data to each other by using the ID.

[0360] Fifth embodiment A fifth embodiment of the present invention will be described. In the fifth embodiment, the structural information is camera trajectory data. The camera trajectory data indicates two or more positions at which a camera generating two or more images used to generate the 3D data is placed. The positions correspond to the camera coordinates described above. That is, the camera trajectory data indicates the trajectory of the camera. The camera trajectory data is associated with each of the first 3D data and the second 3D data. When the endoscope device 1 shown in FIG. 22 is used, the camera trajectory data indicates two or more positions at which the tip 20 of the insertion section 2 is placed. By using the camera trajectory data, the first 3D data and the second 3D data are connected to each other.

[0361] The image display device 50 shown in Fig. 1 is changed to an image display device 50c shown in Fig. 38. Fig. 38 shows the configuration of the image display device 50c. Description of the same configuration as that shown in Fig. 1 will be omitted.

[0362] 38 includes a control unit 51, a data acquisition unit 52, a parameter generation unit 53, a conversion unit 54, a data generation unit 55, a display control unit 56, and a missing section calculation unit 60. The image display device 50c does not include the information acceptance unit 57 and the structure estimation unit 58 shown in FIG.

[0363] Each unit of the image display device 50c may be composed of at least one of a processor and a logic circuit. Each unit of the image display device 50c may include one or more processors. Each unit of the image display device 50c may include one or more logic circuits.

[0364] The storage unit 73 stores the first camera trajectory data and the second camera trajectory data. The first camera trajectory data is associated with the first 3D data, and the second camera trajectory data is associated with the second 3D data. The data acquisition unit 52 is connected to the storage unit 73 and acquires the first camera trajectory data and the second camera trajectory data from the storage unit 73.

[0365] The missing section calculation unit 60 calculates the position of the missing section by using the first camera trajectory data and the second camera trajectory data. The missing section includes 3D data corresponding to a missing region of the subject. The missing region is not included in the first region and the second region of the subject. The first region corresponds to 3D coordinates included in the first 3D data. The second region corresponds to 3D coordinates included in the second 3D data. The missing section calculation unit 60 can calculate the length of the missing section based on the position of the missing section.

[0366] The process executed by the image display device 50c will be described with reference to Fig. 39. Fig. 39 shows the procedure of the process executed by the image display device 50c. The description of the same process as that shown in Fig. 2 will be omitted.

[0367] The data acquisition unit 52 connects to the storage unit 73 and acquires the first 3D data and the first camera trajectory data from the storage unit 73 (step S140). After step S140, the display control unit 56 displays images of the first 3D data and the first camera trajectory data on the display unit 71 (step S141).

[0368] After step S141, the data acquisition unit 52 connects to the storage unit 73 and acquires the second 3D data and the second camera trajectory data from the storage unit 73 (step S142). After step S142, the display control unit 56 displays images of the second 3D data and the second camera trajectory data on the display unit 71 (step S143).

[0369] The order of steps S140 to S143 is not limited to the order shown in Fig. 39. For example, steps S142 and S143 may be executed first, and then steps S140 and S141 may be executed. Alternatively, steps S140 and S142 may be executed first, and then steps S141 and S143 may be executed. Steps S141 and S143 may be omitted.

[0370] After step S143, the parameter generating unit 53 generates transformation parameters based on the relationship between the first camera trajectory data and the second camera trajectory data (step S144). The parameter generating unit 53 generates position and attitude transformation parameters for connecting the first 3D data and the second 3D data to each other by using the following method. The following method can be applied to pipe inspection and blade (moving blade) inspection.

[0371] When the subject is a pipe, the parameter generation unit 53 executes the following process. The parameter generation unit 53 calculates a first straight line that approximates the camera trajectory indicated by the first camera trajectory data. The parameter generation unit 53 also calculates a second straight line that approximates the camera trajectory indicated by the second camera trajectory data. The parameter generation unit 53 generates position and orientation conversion parameters for correcting the positions and orientations of the first 3D data and the second 3D data so that the first straight line and the second straight line match each other.

[0372] The parameter generating unit 53 may calculate the first straight line by using first camera trajectory data associated with the entire first 3D data. The parameter generating unit 53 may calculate the first straight line by using first camera trajectory data associated with a connection region identified based on a timestamp of the first 3D data.

[0373] The parameter generating unit 53 may calculate the second straight line by using second camera trajectory data associated with the entire second 3D data. The parameter generating unit 53 may calculate the second straight line by using second camera trajectory data associated with a connection region identified based on a timestamp of the second 3D data.

[0374] FIG. 40 shows an example of an image displayed on the display unit 71 before the first 3D data and the second 3D data are connected to each other. The display control unit 56 displays an image IMG91 on the display unit 71. The image IMG91 includes an area R91 and an area R92. An image of the first 3D shape 3D91 is displayed in the area R91. The first 3D shape 3D91 is shown by the first 3D data. An image of the second 3D shape 3D92 is displayed in the area R92. The second 3D shape 3D92 is shown by the second 3D data.

[0375] A first camera trajectory TR91 and a second camera trajectory TR92 are shown in Fig. 40. The first camera trajectory TR91 indicates two or more positions included in the first camera trajectory data. The second camera trajectory TR92 indicates two or more positions included in the second camera trajectory data. It is not necessary for the display control unit 56 to display the first camera trajectory TR91 and the second camera trajectory TR92 on the display unit 71.

[0376] FIG. 41 shows an example of an image displayed on the display unit 71 after the first 3D data and the second 3D data are connected to each other. In FIG. 41, step S145 described later is not taken into consideration. Description of the same parts as those shown in FIG. 40 will be omitted. The display control unit 56 displays an image IMG92 on the display unit 71. The image IMG92 includes an area R93. Images of the first 3D shape 3D91 and the second 3D shape 3D92 are displayed in the area R93.

[0377] The first 3D shape 3D91 and the second 3D shape 3D92 are arranged so that a first straight line approximating the first camera trajectory TR91 and a second straight line approximating the second camera trajectory TR92 coincide with each other. The straight line L91 is formed by the first straight line and the second straight line.

[0378] When the subject is a blade (moving surface), the parameter generating unit 53 executes the following process. The parameter generating unit 53 calculates a first curve that approximates the camera trajectory indicated by the first camera trajectory data. The parameter generating unit 53 also calculates a second curve that approximates the camera trajectory indicated by the second camera trajectory data. Each of the first curve and the second curve is an arc. The parameter generating unit 53 generates position and orientation conversion parameters for correcting the position and orientation of each of the first 3D data and the second 3D data such that the first curve and the second curve are connected to each other on the circumference of a circle having an arbitrary diameter.

[0379] FIG. 42 shows an example of an image displayed on the display unit 71 before the first 3D data and the second 3D data are connected to each other. The display control unit 56 displays an image IMG101 on the display unit 71. The image IMG101 includes an area R101 and an area R102. An image of a first 3D shape 3D101 is displayed in the area R101. The first 3D shape 3D101 is shown by the first 3D data. An image of a second 3D shape 3D102 is displayed in the area R102. The second 3D shape 3D102 is shown by the second 3D data.

[0380] A first camera trajectory TR101 and a second camera trajectory TR102 are shown in Fig. 42. The first camera trajectory TR101 indicates two or more positions included in the first camera trajectory data. The second camera trajectory TR102 indicates two or more positions included in the second camera trajectory data. It is not necessary for the display control unit 56 to display the first camera trajectory TR101 and the second camera trajectory TR102 on the display unit 71.

[0381] FIG. 43 shows an example of an image displayed on the display unit 71 after the first 3D data and the second 3D data are connected to each other. In FIG. 43, step S145 described later is not taken into consideration. Description of the same parts as those shown in FIG. 42 will be omitted. The display control unit 56 displays an image IMG102 on the display unit 71. The image IMG102 includes an area R103. Images of the first 3D shape 3D101 and the second 3D shape 3D102 are displayed in the area R103.

[0382] The first 3D shape 3D101 and the second 3D shape 3D102 are arranged such that a first curve approximating the first camera trajectory TR101 and a second curve approximating the second camera trajectory TR102 are arranged on one circumference of a circle. The circle CR101 is formed by the first curve and the second curve.

[0383] The parameter generating unit 53 can generate position and posture conversion parameters according to the object by using the above method. When the object is a pipe, the scale of each of the first 3D data and the second 3D data is not corrected. When the object is a blade (moving surface), the scale of each of the first 3D data and the second 3D data may be corrected.

[0384] After step S144, the missing section calculation unit 60 calculates the missing section in the connection region by using the first camera trajectory data and the second camera trajectory data (step S145). The missing section calculation unit 60 calculates the position of the missing section by using the following method. The following method can be applied to pipe inspection and blade (moving blade) inspection.

[0385] When the subject is a pipe, the missing section calculation section 60 executes the following process: Figure 44 shows a method for calculating the missing section.

[0386] Each of the two or more points P111, the two or more points P112, and the two or more points P113 indicates the position (3D coordinates) of the camera when the camera generated the image. The two or more points P111 constitute first camera trajectory data and are associated with the first 3D data 3D111. The two or more points P112 constitute second camera trajectory data and are associated with the second 3D data 3D112. The two or more points P113 indicate the position of the camera in an area where no 3D data exists.

[0387] Since there is no 3D data corresponding to two or more points P113, there is no camera trajectory data corresponding to two or more points P113. Two or more points P113 constitute a missing section.

[0388] The first 3D data 3D111 is generated by using two or more images FR111. The second 3D data 3D112 is generated by using two or more images FR112. The two or more images FR113 are not used to generate the first 3D data 3D111 and the second 3D data 3D112. The two or more images FR111, the two or more images FR112, and the two or more images FR113 are included in one video file. Each of the two or more images FR111, the two or more images FR112, and the two or more images FR113 has a frame number.

[0389] Each of the two or more points P111 is associated with the image FR111. Each of the two or more points P112 is associated with the image FR112. Each of the two or more points P113 is not associated with the image FR113.

[0390] Not all of the two or more images FR111 are necessarily used to generate the first 3D data 3D111. Two or more key frames included in the two or more images FR111 are used to generate the first 3D data 3D111. The number of key frames is equal to or less than the total number of the two or more images FR111. Similarly, two or more key frames included in the two or more images FR112 are used to generate the second 3D data 3D112.

[0391] There is a correlation between the number of points P111 and the number of images FR111 (key frames). There is a correlation between the number of points P112 and the number of images FR112 (key frames). The missing section calculation unit 60 calculates the number of points P111 and points P112 per image. The missing section calculation unit 60 calculates the number of points P113 by using the numbers of points P111 and points P112 per image and the number of images FR113.

[0392] The missing section calculation unit 60 calculates the interval (three-dimensional distance) between the points P111 or P112. The missing section calculation unit 60 calculates the length LT111 of the missing section by using the interval and the number of the points P113. If the video data used to generate the first 3D data is different from the video data used to generate the second 3D data, the above method is not used.

[0393] When the subject is a blade (moving surface), the missing section calculation unit 60 executes the following process: Figures 45 and 46 show a method for calculating the missing section.

[0394] FIG. 45 shows the relationship between the camera trajectory data and the 3D data. Two or more points P121, two or more points P122, two or more points P123, and two or more points P124 each indicate the position (3D coordinates) of the camera when the camera generated an image. Two or more points P121 constitute the first camera trajectory data and are associated with the first 3D data 3D121. Two or more points P122 constitute the second camera trajectory data and are associated with the second 3D data 3D122. Two or more points P123 and two or more points P124 indicate the position of the camera in an area where no 3D data exists.

[0395] Since there is no 3D data corresponding to two or more points P123 and no 3D data corresponding to two or more points P124, there is no camera trajectory data corresponding to two or more points P123 and no camera trajectory data corresponding to two or more points P124. Two or more points P123 constitute a first missing section. Two or more points P124 constitute a second missing section.

[0396] FIG. 46 shows a video file. The first 3D data 3D121 is generated by using two or more images FR121. The second 3D data 3D122 is generated by using two or more images FR122. The two or more images FR123 and the two or more images FR124 are not used to generate the first 3D data 3D121 and the second 3D data 3D122. The two or more images FR121, the two or more images FR122, the two or more images FR123, and the two or more images FR124 are included in one video file. Each of the two or more images FR121, the two or more images FR122, the two or more images FR123, and the two or more images FR124 has a frame number.

[0397] Each of the two or more points P121 is associated with the image FR121. Each of the two or more points P122 is associated with the image FR122. Each of the two or more points P123 is not associated with the image FR123. Each of the two or more points P124 is not associated with the image FR124.

[0398] Not all of the two or more images FR121 are necessarily used to generate the first 3D data 3D121. Two or more key frames included in the two or more images FR121 are used to generate the first 3D data 3D121. The number of key frames is equal to or less than the total number of the two or more images FR121. Similarly, two or more key frames included in the two or more images FR122 are used to generate the second 3D data 3D122.

[0399] There is a correlation between the number of points P121 and the number of images FR121 (key frames). There is a correlation between the number of points P122 and the number of images FR122 (key frames). The missing section calculation unit 60 calculates the number of points P121 and points P122 per image. The missing section calculation unit 60 calculates the number of points P123 by using the numbers of points P121 and points P122 per image and the number of images FR123.

[0400] The missing section calculation unit 60 calculates the interval (angle) of point P121 or point P122. The missing section calculation unit 60 calculates the angle AG121 of the first missing section by using that interval and the number of points P113. The missing section calculation unit 60 may calculate the interval (angle) of the blades by using the interval of point P121 and the number of blades in the 3D data 3D121. The missing section calculation unit 60 may calculate the number of blades in the first missing section by using that interval and the angle AG121 of the first missing section.

[0401] The missing section calculation unit 60 calculates the angle AG122 of the second missing section by subtracting the first angle, the second angle, and the angle AG121 from the angle of one revolution of the blade (360 degrees). The first angle corresponds to the range in which the blades of the 3D data 3D121 are arranged. The second angle corresponds to the range in which the blades of the 3D data 3D122 are arranged. The missing section calculation unit 60 may calculate the number of blades in the second missing section by subtracting the number of blades of the 3D data 3D121, the number of blades of the 3D data 3D122, and the number of blades in the first missing section from the total number of blades. When the video data used to generate the first 3D data is different from the video data used to generate the second 3D data, the above method is not used.

[0402] After step S145, in step S106, the parameter generating unit 53 generates transformation parameters based on the missing section. At this time, the parameter generating unit 53 generates transformation parameters for separating the first 3D data and the second 3D data from each other by the length or angle of the missing section.

[0403] Each aspect of the present invention may include the following modifications: The structural information indicates two or more positions at which the tip 20 of the insertion portion 2 to be inserted into the object having the subject is positioned.

[0404] Each aspect of the present invention may include the following modifications. The structural information includes first position information (first camera trajectory data) and second position information (second camera trajectory data). The first position information indicates two or more positions where the tip 20 is placed to acquire two or more first images. The second position information indicates two or more positions where the tip 20 is placed to acquire two or more second images. In a generating step (step S144), the parameter generating unit 53 generates position conversion parameters and orientation conversion parameters for converting the first 3D coordinate system and the second 3D coordinate system into a common coordinate system based on the first position information and the second position information. In a converting step (step S107), the converting unit 54 converts the first 3D coordinate system and the second 3D coordinate system into a common coordinate system by using the position conversion parameters and the orientation conversion parameters.

[0405] Each aspect of the present invention may include the following variations: the first 3D data is generated by using two or more first images; the second 3D data is generated by using two or more second images; and the two or more first images and the two or more second images form one video file.

[0406] Each aspect of the present invention may include the following modified examples. In a calculation step (step S145), the missing section calculation unit 60 calculates the position of the missing area based on the number of two or more first images, the number of two or more second images, and the number of third images. The third image is arranged temporally between the two or more first images and the two or more second images in the video file. The missing area is an area of ​​the subject that is different from both the first area of ​​the subject and the second area of ​​the subject. The first area corresponds to 3D coordinates included in the first 3D data. The second area corresponds to 3D coordinates included in the second 3D data. In a conversion step (step S107), the conversion unit 54 converts the first 3D coordinate system and the second 3D coordinate system into a common coordinate system based on the position of the missing area.

[0407] In the fifth embodiment, the image display device 50c can connect the first 3D data and the second 3D data to each other, and can display a wide range of 3D shape images. Moreover, the image display device 50c can display an image of a 3D shape close to the original shape of the subject by taking into account the missing section.

[0408] (Modification of the fifth embodiment) A modified example of the fifth embodiment of the present invention will be described. In the above-mentioned fifth embodiment, camera trajectory data generated through image processing is used. On the other hand, in the modified example of the fifth embodiment, the position of the camera is calculated by using sensor data, and data indicating the position is used instead of the camera trajectory data. In the modified example of the fifth embodiment, an image display device 50c shown in FIG. 38 is used.

[0409] For example, an IMU or an insertion length sensor is used. The IMU is disposed at the tip 20 of the insertion portion 2. The insertion length sensor is disposed at the base end of the insertion portion 2 shown in FIG. 22, or disposed on a drum that houses the insertion portion 2. The insertion length sensor detects the length (insertion length) of the portion of the insertion portion 2 that is inserted into the test subject.

[0410] The process executed by the image display device 50c will be described with reference to Fig. 47. Fig. 47 shows the procedure of the process executed by the image display device 50c. The description of the same process as that shown in Fig. 2 will be omitted.

[0411] After step S103, the data acquisition unit 52 connects to the storage unit 73 and acquires sensor data in the connection region from the storage unit 73 (step S120). For example, the data acquisition unit 52 acquires sensor data in a section from the time of the end of the first 3D data to the time of the start of the second 3D data from the storage unit 73. Step S120 is the same as step S120 shown in FIG.

[0412] After step S120, the parameter generating unit 53 generates conversion parameters based on the sensor data acquired in step S120 (step S150).

[0413] For example, the parameter generating unit 53 calculates the trajectory of the tip 20 of the insertion unit 2 by using sensor data of the IMU. The trajectory indicates two or more positions where the tip 20 is located. The parameter generating unit 53 determines the movement of the tip 20 based on the trajectory. Specifically, the parameter generating unit 53 determines whether the tip 20 moves straight or whether the tip 20 turns 90 degrees. In addition, the missing section calculating unit 60 calculates the length of the missing section by using the method in the fifth embodiment.

[0414] When the sensor data of the insertion length sensor is used, the parameter generating unit 53 cannot determine whether the tip 20 moves straight or not, or whether the tip 20 will bend 90 degrees. The missing section calculating unit 60 can calculate the length of the missing section by using the sensor data of the insertion length sensor. When the history information of the operation related to the bending of the insertion unit 2 in the second modified example of the third embodiment is used, the parameter generating unit 53 can determine whether the tip 20 moves straight or not, or whether the tip 20 will bend 90 degrees, based on the history information. The missing section calculating unit 60 can calculate the shape of the missing section based on the history information.

[0415] When a gas turbine vane is inspected, the tip of the scope moves on the circumference of the disk of the vane. Even in this case, the parameter generating unit 53 can calculate the length of the missing section by using the sensor data of the insertion length sensor.

[0416] Fig. 48 shows an example of an image displayed on the display unit 71. The display control unit 56 displays an image IMG131 on the display unit 71. The image IMG131 includes an area R131. Images of a first 3D shape 3D131 and a second 3D shape 3D132 are displayed in the area R131. The first 3D shape 3D131 is represented by the first 3D data. The second 3D shape 3D132 is represented by the second 3D data.

[0417] 48, the parameter generating unit 53 determines that the tip 20 is bent by 90 degrees. Therefore, the parameter generating unit 53 generates a position and orientation transformation parameter for making the central axis of the second 3D shape 3D 132 perpendicular to the central axis of the first 3D shape 3D 131. The first 3D shape 3D 131 and the second 3D shape 3D 132 are arranged so that the angle AG 131 between the first 3D shape 3D 131 and the second 3D shape 3D 132 is 90 degrees.

[0418] In the second modification of the third embodiment described above, the parameter generating unit 53 generates the transformation parameters by using the structural information. On the other hand, in the modification of the fifth embodiment, the parameter generating unit 53 generates the transformation parameters by using the sensor data without using the structural information.

[0419] Each aspect of the present invention may include the following modifications. In a calculation step (step S150), the missing section calculation unit 60 calculates the shape of the missing area. In a conversion step (step S107), the conversion unit 54 converts the first 3D coordinate system and the second 3D coordinate system into a common coordinate system based on the shape of the missing area.

[0420] In a modification of the fifth embodiment, the image display device 50c can connect the first 3D data and the second 3D data to each other, and can display a wide range of 3D shaped images. The image display device 50c can also display 3D shaped images by taking missing sections into account.

[0421] Sixth embodiment A sixth embodiment of the present invention will be described. Industrial endoscope devices are used to inspect the inside of boilers, gas turbines, automobile engines, pipes, etc. for abnormalities (such as scratches and corrosion). In order to enable verification of the contents of the inspection performed by the inspector, videos are generally recorded during the inspection.

[0422] There is a known technique for reconstructing the 3D shape of an object by using a moving image. The moving image is generated based on an optical image acquired through a monocular optical system. By using this technique, an endoscope device can acquire 3D data of a wide range of objects.

[0423] The 3D data of the inspection target may be used to allow the user to understand which areas have been inspected and which areas have not been inspected. 3D data is generated for the inspected areas, and 3D data is not generated for the areas not inspected. If the inspection target has a unique shape, the user can understand which areas of the 3D data correspond to which areas of the actual inspection target. The user can understand which areas of the inspection target have been inspected by checking the 3D shape shown by the 3D data.

[0424] However, there are few unique structures among typical inspection targets in endoscopic inspection, such as heat exchanger tubes or gas turbines. Therefore, it is difficult for users to grasp which region of the 3D data corresponds to which region of the inspection target. Even when users visually compare the 3D data with reference data of the inspection target, it is difficult for users to grasp which region of the 3D data corresponds to which region of the reference data.

[0425] The sixth embodiment provides a method for allowing a user to grasp inspected and uninspected areas even when the object does not have a unique shape. For example, the object has a structure (e.g., a pipe) with a continuous same shape. Or, the object has a plurality of structures (e.g., blades) with the same shape.

[0426] The sixth embodiment is almost the same as the fourth embodiment. However, in the sixth embodiment, the 3D data is not divided into two. In the following, an example in which the subject is a blade is described. The 3D shape shown by the 3D data is close to a ring.

[0427] In the sixth embodiment, the image display device 50 shown in Fig. 1 is used. Processing executed by the image display device 50 will be described with reference to Fig. 49. Fig. 49 shows the procedure of the processing executed by the image display device 50.

[0428] The data acquisition unit 52 is connected to the storage unit 73, and acquires 3D data from the storage unit 73 (step S200). After step S200, the display control unit 56 displays an image of the 3D data on the display unit 71 (step S201). In the first to fifth embodiments described above, two types of 3D data are acquired from the storage unit 73, and an image of each of the 3D data is displayed on the display unit 71. On the other hand, in the sixth embodiment, one type of 3D data is acquired from the storage unit 73, and an image of the 3D data is displayed on the display unit 71.

[0429] After step S201, the information receiving unit 57 receives reference data (step S202). For example, the communication unit 72 receives the reference data from an external device. The information receiving unit 57 receives the reference data received by the communication unit 72. The reference data includes structural information. An ID is assigned to each of two or more blades having the same shape. The reference data includes the ID of each blade.

[0430] The reference data may be data generated by using 3D-CAD, or data showing a two-dimensional shape in any plane, etc. As long as the reference data includes structural information and an ID is associated with the reference data, the format of the reference data is not limited to the above examples.

[0431] It is not necessary for an ID to be assigned to a blade before the information accepting unit 57 accepts the reference data. An ID may be assigned to a blade by a user after the information accepting unit 57 accepts the reference data. Alternatively, an ID may be assigned to a blade automatically.

[0432] After step S202, the display control unit 56 displays the image of the reference data accepted in step S201 on the display unit 71 (step S203).

[0433] The order of steps S200 to S203 is not limited to the order shown in Fig. 49. For example, steps S202 and S203 may be executed first, and then steps S200 and S201 may be executed. Alternatively, steps S200 and S202 may be executed first, and then steps S201 and S203 may be executed. Steps S201 and S203 may be omitted.

[0434] After step S203, the parameter generating unit 53 generates conversion parameters for converting the 3D coordinate system of the 3D data and the 3D coordinate system of the reference data into a common coordinate system (step S204).

[0435] An example will be described below in which the 3D coordinate system of the reference data is the reference for the coordinate system and is used as the common coordinate system. In the following example, the parameter generating unit 53 generates a position and orientation conversion parameter for matching the position and orientation of the 3D coordinate system of the 3D data with the position and orientation of the 3D coordinate system of the reference data. In addition, the parameter generating unit 53 generates a scale conversion parameter for matching the scale of the 3D coordinate system of the 3D data with the scale of the 3D coordinate system of the reference data. However, the method of generating the conversion parameter is not limited to the following example. The 3D coordinate system of the 3D data may be used as the common coordinate system. A 3D coordinate system different from both the 3D coordinate system of the 3D data and the 3D coordinate system of the reference data may be used as the common coordinate system.

[0436] In step S204, the parameter generating unit 53 executes the process shown in Fig. 50. Fig. 50 shows the procedure of the process executed by the parameter generating unit 53.

[0437] The parameter generating unit 53 identifies blade regions in the 3D data (step S204a). Each blade region includes one blade.

[0438] After step S204a, the parameter generating unit 53 assigns an ID to the blade region identified in step S204a (step S204b).

[0439] After step S204b, the parameter generating unit 53 calculates conversion parameters for correcting the position, orientation, and scale of the 3D data so that the ID of the blade in the reference data and the ID of the blade region in the 3D data match each other (step S204c). When step S204c is executed, the process shown in FIG. 50 ends.

[0440] FIG. 51 shows an example of an image displayed on the display unit 71. The display control unit 56 displays an image IMG141 on the display unit 71. The image IMG141 includes an area R141 and an area R142. An image of a first 3D shape 3D141 is displayed in the area R141. The first 3D shape 3D141 is shown by 3D data. An image of a second 3D shape 3D142 is displayed in the area R142. The second 3D shape 3D142 is shown by reference data.

[0441] The first 3D shape 3D141 includes 11 blades with IDs 1 through 4 and 6 through 12. The blade with ID 5 is missing and is not included in the 3D data.

[0442] The second 3D shape 3D142 includes 12 blades. The IDs of the 12 blades are 1 to 12.

[0443] After step S204, the conversion unit 54 converts the 3D coordinate system of the 3D data and the 3D coordinate system of the reference data into a common coordinate system by using the conversion parameters generated in step S204. That is, the conversion unit 54 converts each of the 3D data and the reference data into 3D data in the common coordinate system (step S205). When the 3D coordinate system of the reference data is used as the common coordinate system, the position, orientation, and scale of the 3D coordinate system of the reference data are not changed, and the position, orientation, and scale of the 3D coordinate system of the 3D data are changed.

[0444] After step S205, the display control unit 56 displays an image of the 3D shape of the 3D data and the reference data on the display unit 71. At this time, the display control unit 56 controls the state of the image so that the user can distinguish between an area corresponding to the 3D data and an area not corresponding to the 3D data (step S206). When step S206 is executed, the process shown in FIG. 49 ends.

[0445] Fig. 52 shows an example of an image displayed on the display unit 71. Description of the same parts as those shown in Fig. 51 will be omitted. The display control unit 56 displays an image IMG142 on the display unit 71. The image IMG142 includes an area R143. Images of a 3D shape 3D141 and a 3D shape 3D142 are displayed in the area R143.

[0446] For example, the display control unit 56 displays blades with IDs of 1 to 4 and 6 to 12 in a first color, and displays a blade with ID of 5 in a second color different from the first color. The blade with ID of 5 is not included in the 3D data. The user can confirm the blades included in the reference data and the blades not included in the reference data. That is, the user can confirm the blades in the reference data that correspond to the 3D data and the blades in the reference data that do not correspond to the 3D data.

[0447] The display control unit 56 may execute a process for emphasizing an area corresponding to the 3D data or an area not corresponding to the 3D data. The user may be notified of the positions of the area corresponding to the 3D data and the area not corresponding to the 3D data by voice. As long as the user can distinguish the area corresponding to the 3D data from the area not corresponding to the 3D data, the method of notifying the user of the area of ​​the 3D data is not limited to the above method.

[0448] The method in the sixth embodiment may be applied to the inspection of a combustion chamber of a gas turbine or the inspection of a heat exchanger of a gas turbine. Fig. 53 shows the structure of a heat exchanger HE150. The heat exchanger HE150 has two or more heat exchange tubes HT150. In general, an ID is assigned to each of the two or more heat exchange tubes in the reference data.

[0449] An ID may be assigned to the 3D data of each heat exchange tube. The image display device 50 may execute the same process as that shown in Fig. 49 and Fig. 50. In this way, the image display device 50 may match each of the position, posture, and scale of the 3D data of each heat exchange tube with each of the position, posture, and scale of the heat exchange tube in the reference data.

[0450] In the sixth embodiment, the image display device 50 allows the user to grasp the inspected area and the uninspected area by using the 3D data and the reference data. Compared with the case where only the 3D data is used, the user can easily distinguish the inspected area from the uninspected area. Therefore, the quality of the inspection report created by the inspector is improved.

[0451] The manager of the examination department or the member who requested the examination approves the examination. Generally, the person who approves the examination does not have much knowledge about endoscopic examination. By checking the data showing the areas that have been examined and the areas that have not been examined, the approver can objectively judge whether the examination has been performed correctly or not. Therefore, the reliability of the examination is guaranteed.

[0452] Although the preferred embodiments of the present invention have been described above, the present invention is not limited to these embodiments and their modifications. Addition, omission, substitution, and other modifications of the configuration are possible without departing from the spirit of the present invention. Furthermore, the present invention is not limited by the above description, but is limited only by the scope of the attached claims. [Explanation of symbols]

[0453] 1 Endoscopic device 2 Insertion section 3 Main body 4,70 Control unit 5,71 Display section 8 Endoscope Unit 9 CCU 10 Control device 12 Video signal processing circuit 13 ROM 14 RAM 15 Card Interface 16 External device interface 17 Control Interface 18 CPU 20 Tip 28 Image sensor 50, 50a, 50b, 50c Image display device 51,180 Control section 52,181 Data Acquisition Section 53,182 Parameter Generation Section 54,183 Converter 55,184,188 Data Generation Division 56,185 Display control unit 57,186 Information Reception Department 58,187 Structure estimation part 59 Adjustment part 60 Missing section calculation unit 72 Communications Department 73 Memory section 74 Sensors 1880 Image Acquisition Unit 1881 Conditions Reception Department 1882 Data Calculation Department

Claims

1. a first acquiring step in which a processor acquires three-dimensional data of a subject including one or more structures, the three-dimensional data including information indicating three-dimensional coordinates of at least a portion of the one or more structures; a second acquisition step in which the processor acquires reference data including information indicative of the one or more structures; an associating step in which the processor associates the information included in the three-dimensional data with the information included in the reference data based on information of a common structure, the common structure indicating a structure of the subject that is common to at least a portion of the information included in the three-dimensional data and at least a portion of the information included in the reference data; a display step of the processor displaying at least a portion of the three-dimensional data and at least a portion of the reference data on a display; A three-dimensional image display method comprising the steps of:

2. The information included in the reference data indicates an area of ​​the one or more structures. The three-dimensional image display method according to claim 1 .

3. The reference data is a known three-dimensional model. The three-dimensional image display method according to claim 2.

4. The reference data is data that indicates a two-dimensional shape on a plane. The three-dimensional image display method according to claim 2.

5. The reference data is received from an external device. The three-dimensional image display method according to claim 1 .

6. The subject has two or more structures including the one or more structures. The three-dimensional image display method according to claim 1 .

7. The reference data includes identification information corresponding to each of the two or more structures. The three-dimensional image display method according to claim 6.

8. The information of the common structure indicates identification information of the structure that is common to at least a portion of the information included in the three-dimensional data and at least a portion of the information included in the reference data. The three-dimensional image display method according to claim 7.

9. The subject is a turbine, and the two or more structures are blades. The three-dimensional image display method according to claim 8.

10. The subject has one structure, and the structure has a continuous shape. The three-dimensional image display method according to claim 1 .

11. The three-dimensional coordinates are defined in a first three-dimensional coordinate system; The reference data includes two-dimensional or three-dimensional coordinates defined in a second coordinate system different from the first three-dimensional coordinate system. The three-dimensional image display method according to claim 1 .

12. The processor converts the first three-dimensional coordinate system and the second coordinate system into a common coordinate system in the associating step. The three-dimensional image display method according to claim 11.

13. In the display step, the processor displays the three-dimensional data and the reference data in the common coordinate system on the display. The three-dimensional image display method according to claim 12.

14. The method further includes, after the second acquisition step, an input step in which the processor receives input of the identification information corresponding to each of the two or more structures. The three-dimensional image display method according to claim 7.

15. the reference data includes identification information corresponding to each of the one or more structures; The associating step includes: A step of identifying an area of ​​the one or more structures indicated by the information included in the three-dimensional data by the processor; an assigning step of the processor assigning identification information to the area identified in the identifying step; Including, In the associating step, the processor identifies the common structure corresponding to the identification information of the region and the identification information included in the reference data; The identification information of the region and the identification information included in the reference data match each other. The three-dimensional image display method according to claim 1 .

16. In the associating step, the processor corrects at least one of a position, an orientation, and a scale of the three-dimensional data based on the information of the common structure. The three-dimensional image display method according to claim 15.

17. The processor, in the display step, displays the information included in the three-dimensional data on the display, and, in the display step, displays information on a structure that corresponds to the information included in the reference data and does not correspond to the information included in the three-dimensional data on the display. The three-dimensional image display method according to claim 1 .

18. In the display step, the processor displays an image of the reference data on the display. The three-dimensional image display method according to claim 1 .

19. In the display step, the processor displays the identification information corresponding to each of the two or more structures on the display. The three-dimensional image display method according to claim 7.

20. a processor, the processor comprising: acquiring three-dimensional data of a subject including one or more structures, the three-dimensional data including information indicating three-dimensional coordinates of at least a portion of the one or more structures; obtaining reference data including information indicative of the one or more structures; the information included in the three-dimensional data and the information included in the reference data are associated with each other based on information of a common structure, the common structure indicating a structure of the subject that is common to at least a portion of the information included in the three-dimensional data and at least a portion of the information included in the reference data; At least a part of the three-dimensional data and at least a part of the reference data are displayed on a display. Three-dimensional image display device.

21. a first acquiring step of acquiring three-dimensional data of a subject including one or more structures, the three-dimensional data including information indicating three-dimensional coordinates of at least a portion of the one or more structures; a second acquisition step of acquiring reference data including information indicative of the one or more structures; an associating step of associating the information included in the three-dimensional data with the information included in the reference data based on information of a common structure, the common structure indicating a structure of the subject that is common to at least a portion of the information included in the three-dimensional data and at least a portion of the information included in the reference data; a display step of displaying at least a part of the three-dimensional data and at least a part of the reference data on a display; A program for causing a computer to execute the following.

Images