3D image display method, 3D image display device, and program

Abstract

To provide a three-dimensional image display method, a three-dimensional image display device, and a program that allow a user to recognize a region that has been inspected and a region that has not been inspected even when a subject does not have a unique shape.SOLUTION: In a three-dimensional image display method, a processor acquires three-dimensional data of a subject including one or more structures. The processor acquires reference data including information indicating the one or more structures. The processor associates, on the basis of information on a common structure, the information included in the three-dimensional data with the information included in the reference data. The common structure indicates a structure of the subject common to at least part of the information included in the three-dimensional data and at least part of the information included in the reference data. The processor displays, on a display, at least part of the three-dimensional data and at least part of the reference data.SELECTED DRAWING: Figure 49

Description

[Technical Field]

[0001] The present invention relates to a three-dimensional image display method, a three-dimensional image display device, and a program. [Background technology]

[0002] Industrial endoscope devices are used to inspect the interior of boilers, gas turbines, automobile engines, pipes, etc. for defects (such as scratches and corrosion). During an endoscopy inspection, the inspector records still images during the inspection to record the presence or absence of defects and their severity as evidence of the inspection. After the endoscopy inspection is completed, an inspection report is generated. Typically, text describing the condition of any defects captured in the recorded still images is added to the inspection report along with the still images.

[0003] Location information is included in the anomaly information. The location information indicates the location on the object being inspected where the recorded still image was acquired. The location information of a discovered anomaly is important when replacing or repairing the abnormal part, or when the next inspection is performed. One method for recording and managing the location information of an anomaly is to associate the still image recorded during the inspection with a specific location in three-dimensional data (3D data) that shows the three-dimensional shape (3D shape) of the object being inspected, and to visualize the location where the still image was acquired. Using this method, the location on the object being inspected where the still image was acquired becomes clear.

[0004] One method for obtaining 3D data of an object to be inspected is to reconstruct the 3D shape of the object by using video recorded during the inspection. This method uses video recorded during the inspection. Therefore, there is no need to bring special inspection equipment to the inspection site, and no need for blueprints of the object to be inspected.

[0005] In a method for reconstructing the 3D shape of an object to be inspected, it is necessary to correlate multiple images acquired from multiple viewpoints. When it is difficult to correlate multiple images, the 3D data may be split into multiple pieces of data. For example, the tip of the endoscope may suddenly move during video recording, significantly changing the image composition. Alternatively, halation or other issues may occur in the image during video recording, significantly changing the state of the image. Alternatively, video recording may be interrupted and then resumed, and the object of inspection shown in the image after the video recording resumes may not match the object of inspection shown in the image before the video recording was interrupted. In these cases, the 3D data may be split into multiple pieces of data.

[0006] It is preferable for users that 3D data covering as wide an area as possible of the inspection object be constructed as a single piece of data without being divided. When multiple pieces of 3D data for the inspection object are generated, it is important to connect the multiple pieces of 3D data to generate 3D data for the wide area of ​​the inspection object. The following technology has been disclosed as a method for generating 3D data for a wide area by connecting multiple pieces of 3D data for partial areas.

[0007] The technology disclosed in Patent Document 1 provides a method for connecting multiple 3D data by using the relationship between the 2D images used to generate the 3D data and the 3D data. First 3D data is associated with the first 2D image group, and second 3D data is associated with the second 2D image group. A first image and a second image having overlapping feature points are selected. The first image is included in the first 2D image group, and the second image is included in the second 2D image group. The 3D data are translated and rotated so that the three-dimensional coordinates of the feature points in the first image and the three-dimensional coordinates of the feature points in the second image match. Then, the error in the three-dimensional coordinates is minimized.

[0008] The techniques disclosed in Patent Documents 2 and 3 each provide a method for connecting multiple pieces of 3D data based on the 3D shape represented by the 3D data. In the technique disclosed in Patent Document 2, ground features, plane features, or cylinder features are assigned to multiple pieces of 3D data of a partial area. The multiple pieces of 3D data are connected so that the features match. In the technique disclosed in Patent Document 3, three planes are detected in each of the multiple pieces of 3D data. The normals of the three planes are orthogonal to each other. The multiple pieces of 3D data are connected so that the three planes match. [Prior art documents] [Patent documents]

[0009] [Patent Document 1] Patent No. 6040882 [Patent Document 2] Patent No. 6811763 [Patent Document 3] Japanese Patent Application Laid-Open No. 2010-66595 Summary of the Invention [Problem to be solved by the invention]

[0010] The present invention aims to provide a three-dimensional image display method, a three-dimensional image display device, and a program that enable a user to understand inspected and uninspected areas even when the subject does not have a unique shape. [Means for solving the problem]

[0011] The present invention is a three-dimensional image display method comprising: a first acquisition step in which a processor acquires three-dimensional data of a subject including one or more structures, the three-dimensional data including information indicating the three-dimensional coordinates of at least a portion of the one or more structures; a second acquisition step in which the processor acquires reference data including information indicating the one or more structures; an association step in which the processor associates the information included in the three-dimensional data with the information included in the reference data based on information of a common structure, the common structure indicating a structure of the subject that is common to at least a portion of the information included in the three-dimensional data and at least a portion of the information included in the reference data; and a display step in which the processor displays at least a portion of the three-dimensional data and at least a portion of the reference data on a display.

[0012] In the three-dimensional image display method of the present invention, the information included in the reference data indicates the area of ​​the one or more structures.

[0013] In the three-dimensional image display method of the present invention, the reference data is a known three-dimensional model.

[0014] In the three-dimensional image display method of the present invention, the reference data is data that indicates a two-dimensional shape on a plane.

[0015] In the three-dimensional image display method of the present invention, the reference data is received from an external device.

[0016] In the three-dimensional image display method of the present invention, the subject has two or more structures including the one or more structures.

[0017] In the three-dimensional image display method of the present invention, the reference data includes identification information corresponding to each of the two or more structures.

[0018] In the three-dimensional image display method of the present invention, the information of the common structure indicates the identification information of the structure that is common to at least a portion of the information contained in the three-dimensional data and at least a portion of the information contained in the reference data.

[0019] In the three-dimensional image display method of the present invention, the object is a turbine, and the two or more structures are blades.

[0020] In the three-dimensional image display method of the present invention, the subject has one structure, and the structure has a continuous same shape.

[0021] In the three-dimensional image display method of the present invention, the three-dimensional coordinates are defined in a first three-dimensional coordinate system, and the reference data includes two-dimensional coordinates or three-dimensional coordinates defined in a second coordinate system different from the first three-dimensional coordinate system.

[0022] In the three-dimensional image display method of the present invention, the processor converts the first three-dimensional coordinate system and the second coordinate system into a common coordinate system in the associating step.

[0023] In the three-dimensional image display method of the present invention, the processor displays the three-dimensional data and the reference data in the common coordinate system on the display in the display step.

[0024] The three-dimensional image display method of the present invention further includes, after the second obtaining step, an input step in which the processor receives input of the identification information corresponding to each of the two or more structures.

[0025] In the three-dimensional image display method of the present invention, the reference data includes identification information corresponding to each of the one or more structures, and the associating step includes an identifying step in which the processor identifies the area of ​​the one or more structures indicated by the information included in the three-dimensional data, and an assigning step in which the processor assigns identification information to the area identified in the identifying step, and in the associating step, the processor identifies the common structure corresponding to the identification information of the area and the identification information included in the reference data, and the identification information of the area and the identification information included in the reference data match each other.

[0026] In the three-dimensional image display method of the present invention, in the associating step, the processor corrects at least one of the position, orientation, and scale of the three-dimensional data based on the information about the common structure.

[0027] In the three-dimensional image display method of the present invention, the processor displays the information contained in the three-dimensional data on the display in the display step, and also displays information about structures that correspond to the information contained in the reference data but do not correspond to the information contained in the three-dimensional data on the display in the display step.

[0028] In the three-dimensional image display method of the present invention, the processor displays an image of the reference data on the display in the displaying step.

[0029] In the three-dimensional image display method of the present invention, the processor displays the identification information corresponding to each of the two or more structures on the display in the displaying step.

[0030] The present invention is a three-dimensional image display device having a processor, which acquires three-dimensional data of a subject including one or more structures, the three-dimensional data including information indicating the three-dimensional coordinates of at least a portion of the one or more structures, acquires reference data including information indicating the one or more structures, and associates the information included in the three-dimensional data with the information included in the reference data based on information of a common structure, the common structure indicating a structure of the subject that is common to at least a portion of the information included in the three-dimensional data and at least a portion of the information included in the reference data, and displays at least a portion of the three-dimensional data and at least a portion of the reference data on a display.

[0031] The present invention is a program for causing a computer to execute the following steps: a first acquisition step for acquiring three-dimensional data of a subject including one or more structures, the three-dimensional data including information indicating the three-dimensional coordinates of at least a portion of the one or more structures; a second acquisition step for acquiring reference data including information indicating the one or more structures; an association step for associating the information included in the three-dimensional data with the information included in the reference data based on information of a common structure, the common structure indicating a structure of the subject that is common to at least a portion of the information included in the three-dimensional data and at least a portion of the information included in the reference data; and a display step for displaying at least a portion of the three-dimensional data and at least a portion of the reference data on a display. [Effects of the Invention]

[0032] According to the present invention, the three-dimensional image display method, three-dimensional image display device, and program enable the user to understand the inspected and uninspected areas even if the subject does not have a unique shape. [Brief explanation of the drawings]

[0033] [Figure 1] 1 is a block diagram showing a configuration of an image display device according to a first embodiment of the present invention. [Figure 2] 5 is a flowchart showing the procedure of processing executed by the image display device according to the first embodiment of the present invention. [Figure 3] FIG. 3 is a diagram showing an example of an image displayed on a display unit according to the first embodiment of the present invention. [Figure 4] FIG. 3 is a diagram showing an example of a dialog box displayed on a display unit according to the first embodiment of the present invention. [Figure 5] FIG. 3 is a diagram illustrating an example of a reference table in the first embodiment of the present invention. [Figure 6] 5 is a flowchart showing the procedure of processing executed by the image display device according to the first embodiment of the present invention. [Figure 7] FIG. 3 is a diagram showing an example of an image displayed on a display unit according to the first embodiment of the present invention. [Figure 8] FIG. 3 is a diagram showing an example of an image displayed on a display unit according to the first embodiment of the present invention. [Figure 9] FIG. 1 is a block diagram showing a configuration of an image display device according to a first modified example of the first embodiment of the present invention. [Figure 10] 10 is a flowchart showing the procedure of processing executed by an image display device according to a first modified example of the first embodiment of the present invention. [Figure 11] FIG. 10 is a diagram showing an example of an image displayed on a display unit according to a first modified example of the first embodiment of the present invention. [Figure 12] FIG. 10 is a diagram showing an example of an image displayed on a display unit according to a second modified example of the first embodiment of the present invention. [Figure 13] FIG. 10 is a diagram showing an example of an image displayed on a display unit according to a second modified example of the first embodiment of the present invention. [Figure 14] FIG. 10 is a diagram showing an example of a dialog box displayed on a display unit according to a third modified example of the first embodiment of the present invention. [Figure 15] FIG. 10 is a diagram showing an example of an image displayed on a display unit according to a third modified example of the first embodiment of the present invention. [Figure 16] FIG. 10 is a diagram showing an example of a U-shaped pipe in a fourth modified example of the first embodiment of the present invention. [Figure 17] FIG. 10 is a diagram showing an example of an image displayed on a display unit according to a fourth modified example of the first embodiment of the present invention. [Figure 18] FIG. 10 is a diagram showing a configuration of a blade in a fifth modified example of the first embodiment of the present invention. [Figure 19] 10 is a flowchart showing a procedure of processing executed by an image display device according to a fifth modified example of the first embodiment of the present invention. [Figure 20] FIG. 10 is a diagram showing an example of an image displayed on a display unit according to a fifth modified example of the first embodiment of the present invention. [Figure 21] FIG. 10 is a diagram showing an example of an image displayed on a display unit according to a fifth modified example of the first embodiment of the present invention. [Figure 22] FIG. 10 is a perspective view showing the overall configuration of an endoscope apparatus according to a second embodiment of the present invention. [Figure 23] FIG. 4 is a block diagram showing the internal configuration of an endoscope apparatus according to a second embodiment of the present invention. [Figure 24] FIG. 10 is a block diagram showing the functional configuration of a CPU included in an endoscope apparatus according to a second embodiment of the present invention. [Figure 25] FIG. 10 is a block diagram showing the functional configuration of a data generating unit included in an endoscope apparatus according to a second embodiment of the present invention. [Figure 26] FIG. 10 is a schematic diagram showing an image acquisition situation in a second embodiment of the present invention. [Figure 27] 10 is a flowchart showing the procedure of a process for generating 3D data according to a second embodiment of the present invention. [Figure 28] FIG. 10 is a block diagram showing a configuration of an image display device according to a third embodiment of the present invention. [Figure 29] 10 is a flowchart showing the procedure of processing executed by an image display device according to a third embodiment of the present invention. [Figure 30] 10 is a flowchart showing the procedure of processing executed by an image display device according to a fourth embodiment of the present invention. [Figure 31] FIG. 10 is a diagram showing an example of an image displayed on a display unit according to a fourth embodiment of the present invention. [Figure 32] FIG. 10 is a diagram showing an example of an image displayed on a display unit according to a fourth embodiment of the present invention. [Figure 33] FIG. 13 is a diagram showing reference data in a first modified example of the fourth embodiment of the present invention. [Figure 34] 13 is a flowchart showing the procedure of processing executed by an image display device according to a first modified example of the fourth embodiment of the present invention. [Figure 35] FIG. 13 is a diagram showing an example of an image displayed on a display unit according to a first modified example of the fourth embodiment of the present invention. [Figure 36] FIG. 10 is a diagram showing the structure of a combustion chamber in a second modified example of the fourth embodiment of the present invention. [Figure 37] FIG. 13 is a diagram showing an example of an image displayed on a display unit according to a second modified example of the fourth embodiment of the present invention. [Figure 38] FIG. 10 is a block diagram showing the configuration of an image display device according to a fifth embodiment of the present invention. [Figure 39] 10 is a flowchart showing the procedure of processing executed by an image display device according to a fifth embodiment of the present invention. [Figure 40] FIG. 13 is a diagram showing an example of an image displayed on a display unit according to a fifth embodiment of the present invention. [Figure 41] FIG. 13 is a diagram showing an example of an image displayed on a display unit according to a fifth embodiment of the present invention. [Figure 42] FIG. 13 is a diagram showing an example of an image displayed on a display unit according to a fifth embodiment of the present invention. [Figure 43] FIG. 13 is a diagram showing an example of an image displayed on a display unit according to a fifth embodiment of the present invention. [Figure 44] FIG. 13 is a diagram illustrating a method for calculating missing segments in the fifth embodiment of the present invention. [Figure 45] FIG. 13 is a diagram showing the relationship between camera trajectory data and 3D data in the fifth embodiment of the present invention. [Figure 46] FIG. 13 is a diagram showing a moving image file in the fifth embodiment of the present invention. [Figure 47]13 is a flowchart showing the procedure of processing executed by an image display device according to a modified example of the fifth embodiment of the present invention. [Figure 48] FIG. 13 is a diagram showing an example of an image displayed on a display unit according to a modified example of the fifth embodiment of the present invention. [Figure 49] 13 is a flowchart showing the procedure of processing executed by an image display device according to a sixth embodiment of the present invention. [Figure 50] 13 is a flowchart showing the procedure of processing executed by an image display device according to a sixth embodiment of the present invention. [Figure 51] FIG. 13 is a diagram showing an example of an image displayed on a display unit according to a sixth embodiment of the present invention. [Figure 52] FIG. 13 is a diagram showing an example of an image displayed on a display unit according to a sixth embodiment of the present invention. [Figure 53] FIG. 10 is a diagram showing the structure of a heat exchanger according to a sixth embodiment of the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0034] Hereinafter, an embodiment of the present invention will be described with reference to the drawings.

[0035] The techniques disclosed in Patent Documents 1, 2, and 3 require that data of a common region be included in multiple 3D data. If data of a common region is not included in multiple 3D data, the multiple 3D data cannot be connected.

[0036] In the technology disclosed in Patent Document 1, if halation or the like occurs in at least one of the first image and the second image, and the appearance of the first image and the appearance of the second image differ from each other, it is not possible to connect multiple 3D data.

[0037] The first to fifth embodiments of the present invention and their variations aim to provide a three-dimensional image display method, a three-dimensional image display device, and a program that can display three-dimensional shapes represented by two or more three-dimensional data even if the two or more three-dimensional data do not contain data of a common area.

[0038] (First embodiment) Fig. 1 shows the configuration of an image display device 50 according to a first embodiment of the present invention. The image display device 50 shown in Fig. 1 includes a control unit 51, a data acquisition unit 52, a parameter generation unit 53, a conversion unit 54, a data generation unit 55, a display control unit 56, an information reception unit 57, and a structure estimation unit 58. An operation unit 70, a display unit 71, a communication unit 72, and a storage unit 73 shown in Fig. 1 are connected to the image display device 50. The image display device 50 may include at least one of the operation unit 70, the display unit 71, the communication unit 72, and the storage unit 73.

[0039] For example, the image display device 50 is a PC (Personal Computer). The image display device 50 may be any of a desktop PC, a laptop PC, and a tablet terminal. The image display device 50 may also be a computer system that operates on the cloud.

[0040] The operation unit 70 is a user interface. For example, the operation unit 70 is at least one of a button, a switch, a key, a mouse, a joystick, a touchpad, a trackball, and a touch panel. The operation unit 70 accepts operations by the user. The user can input various information to the image display device 50 by operating the operation unit 70. The operation unit 70 accepts information input by the user and outputs the information to the image display device 50.

[0041] The display unit 71 has a display screen and displays images of 3D data and the like on the display screen. The display unit 71 is a monitor (display) such as an LCD (Liquid Crystal Display). The display unit 71 may be a touch panel. In this case, the operation unit 70 and the display unit 71 are integrated.

[0042] The communication unit 72 communicates with an external device such as an endoscope device. For example, the communication unit 72 is connected to the external device via a cable or wirelessly. The communication between the communication unit 72 and the external device may be performed via a LAN (Local Area Network) or the Internet.

[0043] The storage unit 73 is a non-volatile recording medium, and is, for example, at least one of a static random access memory (SRAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), and a flash memory.

[0044] The storage unit 73 stores two or more pieces of 3D data including first 3D data and second 3D data. The storage unit 73 may have a first storage unit that stores the first 3D data and a second storage unit that stores the second 3D data.

[0045] Each of the first 3D data and the second 3D data includes three-dimensional coordinates (3D coordinates) of two or more points of the object. Each of the first 3D data and the second 3D data indicates a 3D shape of the object. The first 3D data is generated by using two or more first images acquired from two or more different viewpoints. The second 3D data is generated by using two or more second images acquired from two or more different viewpoints. The two or more first images and the two or more second images are not completely identical. At least one of the two or more first images and at least one of the two or more second images are different from each other. All of the two or more first images may be different from any of the two or more second images. All of the two or more second images may be different from any of the two or more first images.

[0046] The first 3D data and the second 3D data are different from each other. At least some of the 3D coordinates included in the first 3D data are different from 3D coordinates included in the second 3D data. One or more points included in the first 3D data are different from one or more points included in the second 3D data. At least some of the 3D coordinates included in the second 3D data are different from 3D coordinates included in the second 3D data. One or more points included in the second 3D data are different from one or more points included in the first 3D data.

[0047] All of the 3D coordinates included in the first 3D data may be different from the 3D coordinates included in the second 3D data. All of two or more points included in the first 3D data may be different from all of two or more points included in the second 3D data. All of the 3D coordinates included in the second 3D data may be different from the 3D coordinates included in the first 3D data. All of the two or more points included in the second 3D data may be different from all of the two or more points included in the first 3D data.

[0048] That is, the subject area corresponding to at least a portion of the first 3D data does not exist in the subject area corresponding to the entire second 3D data, and the subject area corresponding to at least a portion of the second 3D data does not exist in the subject area corresponding to the entire first 3D data.

[0049] The three-dimensional coordinate system of the first 3D data (first 3D coordinate system) and the three-dimensional coordinate system of the second 3D data (second 3D coordinate system) are different from each other. One of the first 3D coordinate system and the second 3D coordinate system can be transformed into the other by using parameters indicating each of the position, orientation, and scale. At least one of the position, orientation, and scale differs between the first 3D coordinate system and the second 3D coordinate system. The orientation is defined by using three parameters. At least one parameter indicating the orientation may differ between the first 3D coordinate system and the second 3D coordinate system.

[0050] At least some of the two or more viewpoints from which the two or more first images were acquired may be different from two or more viewpoints from which the two or more second images were acquired. All of the two or more viewpoints from which the two or more first images were acquired may be different from two or more viewpoints from which the two or more second images were acquired. At least some of the two or more viewpoints from which the two or more second images were acquired may be different from two or more viewpoints from which the two or more first images were acquired. All of the two or more viewpoints from which the two or more second images were acquired may be different from two or more viewpoints from which the two or more first images were acquired.

[0051] The control unit 51 controls each unit of the image display device 50 .

[0052] The data acquisition unit 52 is connected to the storage unit 73 and acquires the first 3D data and the second 3D data from the storage unit 73.

[0053] The parameter generation unit 53 generates transformation parameters for transforming the first 3D coordinate system and the second 3D coordinate system into a common coordinate system. The common coordinate system is a 3D coordinate system common to the first 3D data and the second 3D data. By transforming the first 3D coordinate system and the second 3D coordinate system into the common coordinate system, each of the first 3D data and the second 3D data is transformed into 3D data in the common coordinate system. The transformation parameters include a first transformation parameter for transforming the first 3D coordinate system into the common coordinate system and a second transformation parameter for transforming the second 3D coordinate system into the common coordinate system. Each of the first transformation parameter and the second transformation parameter includes a position / orientation transformation parameter for transforming the position and orientation of each 3D coordinate system and a scale transformation parameter for transforming the scale of each 3D coordinate system.

[0054] The conversion unit 54 converts the first 3D coordinate system and the second 3D coordinate system into the common coordinate system by using the conversion parameters generated by the parameter generation unit 53. In this way, the conversion unit 54 converts each of the first 3D data and the second 3D data into 3D data in the common coordinate system.

[0055] The data generation unit 55 connects together the first 3D data and the second 3D data converted into 3D data in the common coordinate system by the conversion unit 54. In this way, the data generation unit 55 generates 3D data of a wide range of the subject.

[0056] The display control unit 56 outputs an image of the 3D data to the display unit 71 and displays the image on the display unit 71. The display control unit 56 displays each image of the first 3D data and the second 3D data on the display unit 71. That is, the display control unit 56 displays each image of the 3D shape indicated by the first 3D data and the 3D shape indicated by the second 3D data on the display unit 71. The display control unit 56 also displays an image of the 3D data generated by the data generation unit 55 on the display unit 71. That is, the display control unit 56 displays an image of the 3D shape indicated by the 3D data of a wide range of the subject on the display unit 71.

[0057] The information receiving unit 57 receives information output from the operation unit 70. Alternatively, the information receiving unit 57 receives information received by the communication unit 72. The information receiving unit 57 may also receive information corresponding to voice input to a microphone not shown in FIG. 1. The information receiving unit 57 may also receive structural information related to the geometric structure of the subject. The structural information indicates the structure of the subject in a region where the 3D shape of the first 3D data and the 3D shape of the second 3D data are connected to each other. Hereinafter, this region will be referred to as a connected region. The structural information is generated without using either the first 3D data or the second 3D data. The information receiving unit 57 may also receive information that does not include structural information.

[0058] When the information receiving unit 57 receives information that does not include structural information, the structure estimation unit 58 estimates the structure of the subject in the connected region based on the information and generates structural information. When the information receiving unit 57 receives information that includes structural information, the structure estimation unit 58 is not used.

[0059] Each unit of the image display device 50 may be configured with at least one of a processor and a logic circuit. For example, the processor is at least one of a CPU (Central Processing Unit), a DSP (Digital Signal Processor), and a GPU (Graphics Processing Unit). For example, the logic circuit is at least one of an ASIC (Application Specific Integrated Circuit) and an FPGA (Field-Programmable Gate Array). Each unit of the image display device 50 may include one or more processors. Each unit of the image display device 50 may include one or more logic circuits.

[0060] The computer of the image display device 50 may load a program and execute the loaded program. The program includes instructions that define the operation of each unit of the image display device 50. In other words, the function of each unit of the image display device 50 may be realized by software.

[0061] The above program may be provided by a "computer-readable recording medium" such as a flash memory. The program may be transmitted from a computer storing the program to the image display device 50 via a transmission medium or by transmission waves in the transmission medium. A "transmission medium" that transmits the program is a medium that has the function of transmitting information. Media that have the function of transmitting information include networks (communication networks) such as the Internet and communication lines (communication lines) such as telephone lines. The above program may realize some of the above functions. Furthermore, the above program may be a difference file (difference program). The above functions may be realized by combining a program already recorded on the computer with the difference program.

[0062] The following describes characteristic processing of the first embodiment. In the following description, it is assumed that 3D data is generated based on a group of still images acquired by an endoscopic device. The two or more first images used to generate the first 3D data and the two or more second images used to generate the second 3D data are generated by the endoscopic device. Furthermore, the two or more first images and the two or more second images are generated based on an optical image of a subject acquired through a monocular optical system. The inspection device that acquires the group of still images is not limited to an endoscopic device. The inspection device may be any device as long as it has a camera.

[0063] For example, each of the two or more first images may be a still image, and each of the two or more second images may be a still image. The two or more first images may be all or a part of two or more images included in a video. The two or more second images may be all or a part of two or more images included in a video. The two or more first images may be included in a first video file, and the two or more second images may be included in a second video file different from the first video file. The two or more first images and the two or more second images do not need to be separated into two video files. The two or more first images and the two or more second images may be included in a single video file.

[0064] The processing executed by the image display device 50 will be described with reference to Fig. 2. Fig. 2 shows the procedure of the processing executed by the image display device 50.

[0065] The data acquisition unit 52 connects to the storage unit 73 and acquires the first 3D data from the storage unit 73 (step S100). After step S100, the display control unit 56 displays an image of the first 3D data on the display unit 71 (step S101).

[0066] After step S101, the data acquisition unit 52 connects to the storage unit 73 and acquires the second 3D data from the storage unit 73 (step S102). After step S102, the display control unit 56 displays an image of the second 3D data on the display unit 71 (step S103).

[0067] The order of steps S100 to S103 is not limited to the order shown in Fig. 2. For example, steps S102 and S103 may be performed first, followed by steps S100 and S101. Alternatively, steps S100 and S102 may be performed first, followed by steps S101 and S103. Steps S101 and S103 may be omitted.

[0068] FIG. 3 shows an example of an image displayed on the display unit 71. The display control unit 56 displays an image IMG11 on the display unit 71. The image IMG11 includes an area R11 and an area R12. An image of a first 3D shape 3D11 is displayed in the area R11. The first 3D shape 3D11 is represented by the first 3D data. An image of a second 3D shape 3D12 is displayed in the area R12. The second 3D shape 3D12 is represented by the second 3D data.

[0069] After step S103, the information receiving unit 57 receives information on the inspection target (step S104). In the following, an example will be described in which the information receiving unit 57 receives information that does not include structural information from the operation unit .

[0070] The display control unit 56 displays a dialog box on the display unit 71. Fig. 4 shows an example of a dialog box displayed on the display unit 71. The display control unit 56 displays a dialog box DL10 on the display unit 71. The user can select information in the dialog box DL10 by operating the operation unit 70.

[0071] A cursor CS10 and a cursor CS11 are displayed in the dialog box DL10. Each of the cursors CS10 and CS11 indicates a currently selected item. Information INF10 is displayed in the dialog box DL10. The information INF10 includes the name of the examination object and the names of typical examination regions or structures present in the examination object.

[0072] The user selects a subject to be inspected. If the region where the first 3D shape 3D11 and the second 3D shape 3D12 shown in FIG. 3 are connected to each other is a "straight pipe," the user selects "pipe" shown in FIG. 4, and then selects "straight pipe." When the user inputs information indicating that the selection has been finalized into the image display device 50, the information accepting unit 57 accepts the information selected by the user. At this time, the information accepting unit 57 accepts the character string "straight pipe."

[0073] After step S104, the structure estimation unit 58 estimates the structure of the subject in the connected region based on the information received in step S104, and generates structure information (step S105).

[0074] An example of a method in which the structure estimation unit 58 generates structural information will be described. For example, the structure estimation unit 58 uses a reference table TB11 shown in Fig. 5. The reference table TB11 includes information (input information) and structural information received by the information receiving unit 57. The input information and structural information are associated with each other.

[0075] The structure estimation unit 58 converts the character string received by the information receiving unit 57 into structural information. For example, if the information receiving unit 57 receives the character string "straight pipe," the structure estimation unit 58 references column CL11 in the reference table TB11 and acquires the structural information in column CL11. For example, the structural information indicates that the 3D shapes of the first 3D data and the second 3D data are approximately cylindrical. Because the 3D data may contain foreign matter, the structural information indicates a shape that is close to a cylindrical shape rather than a perfect cylindrical shape. The structural information indicates that the inner diameter of the 3D shape of the first 3D data and the inner diameter of the 3D shape of the second 3D data are equal. The structural information indicates that the central axis of the cylinder of the first 3D data and the central axis of the cylinder of the second 3D data are the same.

[0076] The method for generating the structural information is not limited to the method using the reference table TB11. For example, the information receiving unit 57 may receive information similar to the structural information included in the reference table TB11 from the operation unit 70. In other words, the information receiving unit 57 may receive information including the structural information from the operation unit 70. In this case, the structure estimation unit 58 may be omitted.

[0077] The display control unit 56 may display an input box on the display unit 71. The user may input a word or keyword into the input box by using the operation unit 70 or a microphone. The structure estimation unit 58 may estimate the structure of the subject based on the input word or keyword and generate structure information.

[0078] The method for acquiring the structural information of the connected regions is not limited to the above method. For example, the structure estimation unit 58 may estimate the structure of the subject by using a technique such as AI (Artificial Intelligence).

[0079] After step S105, the parameter generating unit 53 generates transformation parameters for transforming the first 3D coordinate system and the second 3D coordinate system into a common coordinate system (step S106).

[0080] The following describes 3D coordinate systems. The first 3D coordinate system in the first 3D data and the second 3D coordinate system in the second 3D data do not match each other. In order to combine the first 3D data and the second 3D data into a wide range of 3D data, the first 3D coordinate system and the second 3D coordinate system need to be converted into an arbitrary common coordinate system. The common coordinate system may be a 3D coordinate system different from both the first 3D coordinate system and the second 3D coordinate system. The common coordinate system may be the same as the first 3D coordinate system or the second 3D coordinate system.

[0081] If the common coordinate system is different from both the first 3D coordinate system and the second 3D coordinate system, the parameter generation unit 53 generates a first transformation parameter for transforming the first 3D coordinate system into the third 3D coordinate system, and generates a second transformation parameter for transforming the second 3D coordinate system into the third 3D coordinate system.

[0082] On the other hand, if the common coordinate system is the same as the first 3D coordinate system, the parameter generation unit 53 generates only the second transformation parameters for transforming the second 3D coordinate system into the first 3D coordinate system. Similarly, if the common coordinate system is the same as the second 3D coordinate system, the parameter generation unit 53 generates only the first transformation parameters for transforming the first 3D coordinate system into the second 3D coordinate system.

[0083] The following describes the scale of the 3D coordinate system. The scale of the first 3D data and the scale of the second 3D data do not necessarily match each other. In order to combine the first 3D data and the second 3D data into 3D data with a wide range, the scale of the first 3D data and the scale of the second 3D data must match each other. The scale of the combined 3D data may be different from both the scale of the first 3D data and the scale of the second 3D data. The scale of the combined 3D data may be the same as the scale of the first 3D data or the scale of the second 3D data.

[0084] The following describes an example in which the first 3D coordinate system is the reference coordinate system and is used as the common coordinate system. In the following example, the parameter generation unit 53 generates position and orientation transformation parameters for matching the position and orientation of the second 3D coordinate system with the position and orientation of the first 3D coordinate system. The parameter generation unit 53 also generates scale transformation parameters for matching the scale of the second 3D coordinate system with the scale of the first 3D coordinate system. However, the method of generating transformation parameters is not limited to the following example. The second 3D coordinate system may be used as the common coordinate system. A 3D coordinate system different from both the first 3D coordinate system and the second 3D coordinate system may be used as the common coordinate system.

[0085] In step S106, the parameter generating unit 53 executes the process shown in Fig. 6. Fig. 6 shows the procedure of the process executed by the parameter generating unit 53.

[0086] The structural information indicates that the 3D shapes of the first 3D data and the second 3D data are each approximately cylindrical. Therefore, the parameter generating unit 53 calculates the cylinder axis of the first 3D data (first cylinder axis) (step S106a). The cylinder axis of the first 3D data indicates the central axis of the cylinder of the first 3D data.

[0087] Fig. 7 shows an image similar to Fig. 3. Explanation of the same parts as those shown in Fig. 3 will be omitted. The parameter generating unit 53 calculates the first cylinder axis AX11.

[0088] After step S106a, the parameter generating unit 53 calculates the cylinder axis of the second 3D data (second cylinder axis) (step S106b). The cylinder axis of the second 3D data indicates the central axis of the cylinder of the second 3D data. The parameter generating unit 53 calculates the second cylinder axis AX12 shown in FIG. 7.

[0089] The order of steps S106a and S106b is not limited to the order shown in Fig. 6. Step S106b may be executed first, followed by step S106a.

[0090] The structural information indicates that the inner diameter of the 3D shape of the first 3D data and the inner diameter of the 3D shape of the second 3D data are equal. Therefore, after step S106b, the parameter generation unit 53 generates a scale conversion parameter for correcting the scale of the second 3D data so that the inner diameter of the cylinder of the first 3D data and the inner diameter of the cylinder of the second 3D data match each other (step S106c). Specifically, the parameter generation unit 53 generates a scale conversion parameter for matching the diameter DM12 of the cylinder of the second 3D shape 3D12 shown in FIG. 7 with the diameter DM11 of the cylinder of the first 3D shape 3D11.

[0091] The structural information indicates that the central axis of the cylinder of the first 3D data and the central axis of the cylinder of the second 3D data coincide. Therefore, after step S106c, the parameter generation unit 53 generates position and orientation transformation parameters for correcting the position and orientation of the second 3D data so that the cylinder axes of the first 3D data and the second 3D data coincide with each other (step S106d). When step S106d is executed, the process shown in FIG. 6 ends.

[0092] If each piece of 3D data includes a timestamp, the parameter generating unit 53 can identify the connected region by using the timestamp (time information). The timestamp indicates the time at which each of the two or more images used to generate each of the first and second 3D data was generated. For example, the first 3D data includes a timestamp from time t1 to time t2 shown in FIG. 7, and the second 3D data includes a timestamp from time t3 to time t4 shown in FIG. 7.

[0093] The parameter generation unit 53 identifies, as connection regions, a region of the first 3D data associated with time t2 and a region of the second 3D data associated with time t3. Specifically, the parameter generation unit 53 identifies a first connection region CR11 of the first 3D shape 3D11 and a second connection region CR12 of the second 3D shape 3D12.

[0094] The first connection region CR11 includes the end of the first 3D shape 3D11. The second connection region CR12 includes the start of the second 3D shape 3D12. The parameter generation unit 53 generates position and posture transformation parameters for aligning the second cylinder axis AX12 in the second connection region CR12 with the first cylinder axis AX11 in the first connection region CR11.

[0095] The parameter generating unit 53 can convert the first 3D coordinate system and the second 3D coordinate system into a common coordinate system by executing steps S106a to S106d. When the information receiving unit 57 receives the character string "straight pipe," the parameter generating unit 53 generates position and orientation parameters and scale parameters for converting the first 3D coordinate system and the second 3D coordinate system into the common coordinate system.

[0096] After step S106, the conversion unit 54 converts the first 3D coordinate system and the second 3D coordinate system into a common coordinate system by using the conversion parameters generated in step S106. That is, the conversion unit 54 converts each of the first 3D data and the second 3D data into 3D data in the common coordinate system (step S107). When the first 3D coordinate system is used as the common coordinate system, the position, orientation, and scale of the first 3D coordinate system are not changed, and the position, orientation, and scale of the second 3D coordinate system are changed.

[0097] After step S107, the data generation unit 55 connects the first 3D data and the second 3D data in the common coordinate system to each other (step S108), thereby generating 3D data of a wide range of the subject.

[0098] After the first 3D coordinate system and the second 3D coordinate system are converted into a common coordinate system, the first 3D data and the second 3D data do not need to be combined into one 3D data, and therefore the data generator 55 may be omitted.

[0099] After step S108, the display control unit 56 displays an image of the 3D shape indicated by the 3D data generated in step S108 on the display unit 71 (step S109). When step S109 is executed, the processing shown in FIG. 2 ends.

[0100] Fig. 8 shows an example of an image displayed on the display unit 71. Description of parts that are the same as those shown in Fig. 7 will be omitted. The display control unit 56 displays an image IMG12 on the display unit 71. The image IMG12 includes an area R13. Images of a first 3D shape 3D11 and a second 3D shape 3D12 are displayed in the area R13.

[0101] The first 3D shape 3D11 and the second 3D shape 3D12 are arranged so that the first connection region and the second connection region are connected to each other. In the example shown in Figure 8, there is a gap between the first 3D shape 3D11 and the second 3D shape 3D12 to make it easier to understand the connection between the first connection region and the second connection region.

[0102] The display control unit 56 displays regions R14 and R15 in region R13. Region R14 corresponds to the first connection region, and region R15 corresponds to the second connection region. In the example shown in FIG. 8, regions R14 and R15 are each displayed as a line. Because regions R14 and R15 are each displayed as a line, the user can easily confirm the positions where the first 3D shape 3D11 and the second 3D shape 3D12 are connected to each other.

[0103] The display control unit 56 may display regions R14 and R15 in a first color, and may display regions other than regions R14 and R15 in a second color different from the first color. The positions of regions R14 and R15 may be notified to the user by voice. As long as the user can distinguish between the connection region and the region other than the connection region, the method of notifying the user of the region of the 3D shape is not limited to the above method.

[0104] The display control unit 56 may display the region R14 and the region R15 in different colors. When the user looks over a wide range of 3D shapes, the user can easily see how far the first 3D shape 3D11 and the second 3D shape 3D12 are from each other. In addition, the user can easily see where the first 3D shape 3D11 and the second 3D shape 3D12 are connected to each other.

[0105] Images of the first 3D shape 3D11 and the second 3D shape 3D12 may be added to the examination report. The display control unit 56 may display the images of the first 3D shape 3D11 and the second 3D shape 3D12 on the display unit 71 by displaying the examination report on the display unit 71.

[0106] In step S106, the parameter generation unit 53 may calculate a reliability of the connection between the first connection region and the second connection region. The reliability indicates the accuracy of the connection between the first connection region and the second connection region. For example, the parameter generation unit 53 performs cylindrical fitting using the first 3D data. As a result, the parameter generation unit 53 calculates a cylinder that approximates the first 3D data and calculates the central axis of the cylinder (first cylinder axis). Furthermore, the parameter generation unit 53 performs cylindrical fitting using the second 3D data. As a result, the parameter generation unit 53 calculates a cylinder that approximates the second 3D data and calculates the central axis of the cylinder (second cylinder axis).

[0107] An error occurs between the 3D coordinates included in the 3D data and the calculated 3D coordinates of the cylinder. The parameter generation unit 53 calculates the error in the distance between each point of the 3D data and each point on the cylinder for each point of the 3D data, and calculates the average or standard deviation of the errors as the reliability of the 3D data. The parameter generation unit 53 calculates statistical values ​​of the reliability of the first 3D data and the reliability of the second 3D data as the reliability of the connection in the first connection region and the second connection region. The statistical values ​​are the minimum, maximum, average, etc. of the reliability of the first 3D data and the reliability of the second 3D data.

[0108] In step S109, the display control unit 56 may display the reliability of the first 3D data and the reliability of the second 3D data on the display unit 71. For example, each reliability may be displayed as a percentage. Alternatively, each reliability may be displayed on a scale of 1 to 5. The method of displaying the reliability is not limited to the above example.

[0109] If the 3D shape represented by the 3D data is a perfect cylinder, the parameter generation unit 53 can accurately calculate the cylinder axis. Therefore, the reliability is high. On the other hand, if the 3D data contains errors, the central axis of the cylinder approximating the 3D data will deviate from the original central axis. Therefore, the reliability is low. The reliability is displayed, allowing the user to check the accuracy of the connection between the first connection region and the second connection region.

[0110] Each aspect of the three-dimensional image display method of the present invention includes a first acquisition step, a second acquisition step, a conversion step, and a display step. In the first acquisition step (step S100), the data acquisition unit 52 connects to a storage unit 73 (recording medium) that stores first 3D data of a subject and second 3D data of the subject, and acquires the first 3D data from the storage unit 73. The first 3D data includes 3D coordinates defined in a first 3D coordinate system. The second 3D data includes 3D coordinates defined in a second 3D coordinate system different from the first 3D coordinate system. At least a portion of the area of ​​the subject corresponding to the first 3D data is different from at least a portion of the area of ​​the subject corresponding to the second 3D data. In the second acquisition step (step S102), the data acquisition unit 52 connects to the storage unit 73 and acquires the second 3D data from the storage unit 73. In a conversion step (step S107), the conversion unit 54 converts the first 3D coordinate system and the second 3D coordinate system into a three-dimensional common coordinate system based on structural information related to the geometric structure of the subject. The structural information is generated without using either the first 3D data or the second 3D data. In a display step (step S109), the display control unit 56 displays an image of the first 3D data in the common coordinate system and an image of the second 3D data in the common coordinate system on the display unit 71 (display).

[0111] The three-dimensional image display device according to each aspect of the present invention includes a data acquisition unit 52, a conversion unit 54, and a display control unit 56. The data acquisition unit 52 is connected to a storage unit 73 (recording medium) that stores first 3D data of a subject and second 3D data of the subject, and acquires the first 3D data and the second 3D data from the storage unit 73. The conversion unit 54 converts the first 3D coordinate system and the second 3D coordinate system into a three-dimensional common coordinate system based on structural information related to the geometric structure of the subject. The display control unit 56 displays an image of the first 3D data in the common coordinate system and an image of the second 3D data in the common coordinate system on a display unit 71 (display).

[0112] Each aspect of the present invention may include the following modification: Each of the two or more first images and each of the two or more second images includes time information (time stamp), and in a conversion step (step S107), the conversion unit 54 converts the first 3D coordinate system and the second 3D coordinate system into a common coordinate system based on the time information.

[0113] Each aspect of the present invention may include the following modifications. In a conversion step (step S107), the conversion unit 54 converts the first 3D coordinate system and the second 3D coordinate system into a common coordinate system so that a first region of the subject (first connected region CR11) and a second region of the subject (second connected region CR12) are connected to each other. The first region corresponds to the 3D coordinates included in the first 3D data. The second region corresponds to the 3D coordinates included in the second 3D data. In a display step (step S109), the display control unit 56 displays information indicating the positions of the first region and the second region on the display unit 71 (display).

[0114] Each aspect of the present invention may include the following modifications: In a display step (step S109), the display control unit 56 displays information indicating the accuracy of the connection between the first region (first connection region CR11) and the second region (second connection region CR12) on the display unit 71 (display).

[0115] Each aspect of the present invention may include the following modifications: In a conversion step (step S107), the conversion unit 54 generates position conversion parameters and orientation conversion parameters for converting the first 3D coordinate system and the second 3D coordinate system into a common coordinate system based on the structural information. In the conversion step (step S107), the conversion unit 54 converts the first 3D coordinate system and the second 3D coordinate system into the common coordinate system by using the position conversion parameters and orientation conversion parameters.

[0116] Each aspect of the present invention may include the following modifications: In a generating step (step S106), the parameter generating unit 53 generates a scale conversion parameter for correcting at least one of the scale of the 3D shape represented by the first 3D data and the scale of the 3D shape represented by the second 3D data. In a converting step (step S107), the converting unit 54 converts the first 3D coordinate system and the second 3D coordinate system into a common coordinate system by using the scale conversion parameter.

[0117] Each aspect of the present invention may include the following variations: the first 3D data is generated by using two or more first images acquired from two or more different viewpoints; the second 3D data is generated by using two or more second images acquired from two or more different viewpoints; and at least one of the two or more first images and at least one of the two or more second images are different from each other.

[0118] Each aspect of the present invention may include the following variations: The two or more first images are at least a portion of two or more images included in a first video, and the two or more second images are at least a portion of two or more images included in a second video that is the same as or different from the first video.

[0119] Each aspect of the present invention may include the following variations: The two or more first images and the two or more second images are generated by an endoscope.

[0120] Each aspect of the present invention may include the following modifications: The two or more first images and the two or more second images are generated based on an optical image of a subject acquired through a monocular optical system.

[0121] In the first embodiment, the first 3D data and the second 3D data may not include 3D data of a common region. Even in such a case, the image display device 50 can display an image of a wide range of 3D shapes represented by two or more 3D data. This allows the user to view the entire structure of the object of inspection and confirm that structure.

[0122] In some cases, 3D data corresponding to the region between the first connection region in the 3D shape represented by the first 3D data and the second connection region in the 3D shape represented by the second 3D data is missing. Even in such cases, the image display device 50 can display images of a wide range of 3D shapes represented by two or more 3D data.

[0123] (First Modification of the First Embodiment) A first modified example of the first embodiment of the present invention will be described. The image display device 50 shown in Fig. 1 is changed to an image display device 50a shown in Fig. 9. Fig. 9 shows the configuration of the image display device 50a. Description of the same configuration as that shown in Fig. 1 will be omitted.

[0124] An image display device 50a shown in FIG. 9 includes a control unit 51, a data acquisition unit 52, a parameter generation unit 53, a conversion unit 54, a data generation unit 55, a display control unit 56, an information reception unit 57, a structure estimation unit 58, and an adjustment unit 59.

[0125] Each unit of the image display device 50a may be configured with at least one of a processor and a logic circuit. Each unit of the image display device 50a may include one or more processors. Each unit of the image display device 50a may include one or more logic circuits.

[0126] The adjustment unit 59 adjusts at least one of the position and orientation of at least one of the first 3D data and the second 3D data in the common coordinate system. For example, the adjustment unit 59 executes an adjustment process for the first 3D data and the second 3D data. Alternatively, the adjustment unit 59 executes an adjustment process for only the first 3D data or only the second 3D data. The adjustment unit 59 adjusts the position and orientation of the 3D data in the adjustment process. Alternatively, the adjustment unit 59 adjusts only the position of the 3D data or only the orientation of the 3D data in the adjustment process.

[0127] After adjustment unit 59 performs the adjustment process, display control unit 56 displays the images of the first 3D data and the second 3D data again on display unit 71. This changes at least one of the position and orientation of at least one of the images of the first 3D data and the second 3D data.

[0128] The processing executed by the image display device 50a will be described using Fig. 10. Fig. 10 shows the procedure of the processing executed by the image display device 50a. The description of the same processing as that shown in Fig. 2 will be omitted.

[0129] For example, in step S108, the image IMG12 shown in Fig. 8 is displayed on the display unit 71. In the example shown in Fig. 8, the end of the first 3D shape 3D11 and the start of the second 3D shape 3D12 coincide with each other. There may be a case where 3D data is missing between the end of the first 3D shape 3D11 and the start of the second 3D shape 3D12.

[0130] In a first variation of the first embodiment, a user estimates the length of a section between the end of the first 3D shape 3D11 and the beginning of the second 3D shape 3D12. 3D data in that section is missing. The user can then reposition the first 3D shape 3D11 and the second 3D shape 3D12 so that the end of the first 3D shape 3D11 and the beginning of the second 3D shape 3D12 are separated by the length of that section.

[0131] The user operates the operation unit 70 to input an instruction to move at least one of the first 3D shape 3D11 and the second 3D shape 3D12 to the image display device 50a. The information receiving unit 57 receives the instruction. An example in which the second 3D shape 3D12 moves will be described below. The adjustment unit 59 adjusts at least one of the position and orientation of the second 3D shape 3D12 based on the instruction received by the information receiving unit 57 (step S110).

[0132] The 3D data generated in step S108 includes 3D data corresponding to the first 3D data and 3D data corresponding to the second 3D data. Adjustment unit 59 changes the 3D coordinates of the 3D data corresponding to the second 3D data. In this way, adjustment unit 59 adjusts at least one of the position and orientation of the 3D shape indicated by the 3D data.

[0133] After step S110, the display control unit 56 displays an image of the 3D shape indicated by the 3D data processed in step S110 on the display unit 71 (step S111). When step S111 is executed, the processing shown in FIG. 10 ends.

[0134] 11 shows an example of an image displayed on the display unit 71. Explanation of the same parts as those shown in FIG.

[0135] The user inputs, into the image display device 50a, an instruction to move the second 3D shape 3D12 closer to or farther away from the first 3D shape 3D11. Alternatively, the user inputs, into the image display device 50a, an instruction to rotate the second 3D shape 3D12.

[0136] The operations performed by the user to freely change the position and orientation of the second 3D shape 3D12 are complex. Therefore, the movement of the second 3D shape 3D12 may be restricted. For example, if a "straight pipe" is selected, the user may move the second 3D shape 3D12 within a range that does not change the direction of the cylindrical axis. In this case, the user can move the second 3D shape 3D12 in a direction parallel to the cylindrical axis and can rotate the second 3D shape 3D12 around the cylindrical axis.

[0137] Even when the subject is a straight pipe, the conditions for restricting the movement of the second 3D shape 3D12 are not limited to the above example.

[0138] 11, the user inputs an instruction to the image display device 50a to move the second 3D shape 3D12 to the right. The adjustment unit 59 changes the 3D coordinates of the 3D data so that the second 3D shape 3D12 moves to the right.

[0139] Each aspect of the present invention may include the following modifications: In an adjustment step (step S110), the adjustment unit 59 adjusts at least one of the position and orientation of at least one of the first 3D data and the second 3D data in the common coordinate system.

[0140] In a first modification of the first embodiment, at least one of the positions and orientations of the first 3D shape and the second 3D shape in the wide range of 3D shapes is adjusted. Through this adjustment, the 3D shape represented by the wide range of 3D data becomes closer to the actual structure of the inspection target, thereby improving the quality of the 3D data.

[0141] (Second Modification of the First Embodiment) A second modification of the first embodiment of the present invention will now be described. In the second modification of the first embodiment, an image display device 50 shown in FIG.

[0142] The object in the first embodiment described above is a straight pipe, whereas in a second modified example of the first embodiment, an example in which the object is a 90-degree joint will be described.

[0143] For example, the user selects "pipe" shown in Fig. 4, and then selects "90-degree joint." When the user inputs information indicating that the selection has been confirmed into the image display device 50, the information accepting unit 57 accepts the information selected by the user. At this time, the information accepting unit 57 accepts the character string "90-degree joint."

[0144] For example, the structural information of a 90-degree joint indicates that the 3D shapes of the first 3D data and the second 3D data are approximately cylindrical. The structural information indicates that the inner diameter of the 3D shape of the first 3D data and the inner diameter of the 3D shape of the second 3D data are equal. The structural information indicates that the central axis of the cylinder of the first 3D data and the central axis of the cylinder of the second 3D data are perpendicular to each other. In step S106, the parameter generation unit 53 generates conversion parameters based on the structural information.

[0145] 12 shows an example of an image displayed on the display unit 71. The display control unit 56 displays an image IMG21 on the display unit 71. The image IMG21 includes an area R21. Images of a first 3D shape 3D21 and a second 3D shape 3D22 are displayed in the area R21. The first 3D shape 3D21 is represented by the first 3D data. The second 3D shape 3D22 is represented by the second 3D data.

[0146] 12, the parameter generation unit 53 generates position and orientation transformation parameters for matching the position and orientation of the second 3D coordinate system with those of the first 3D coordinate system. The parameter generation unit 53 also generates scale transformation parameters for matching the scale of the second 3D coordinate system with that of the first 3D coordinate system. The parameter generation unit 53 calculates a first cylinder axis AX21 of the first 3D shape 3D21 and a second cylinder axis AX22 of the second 3D shape 3D22. The parameter generation unit 53 generates scale transformation parameters for matching the diameter DM22 of the cylinder of the second 3D shape 3D22 with the diameter DM21 of the cylinder of the first 3D shape 3D21.

[0147] The first 3D data includes time stamps from time t1 to time t2 shown in Fig. 12, and the second 3D data includes time stamps from time t3 to time t4 shown in Fig. 12. The parameter generating unit 53 identifies a first connection region of the first 3D shape 3D21 associated with time t2, and identifies a second connection region of the second 3D shape 3D22 associated with time t3.

[0148] The first connection region includes the end of the first 3D shape 3D21. The second connection region includes the start of the second 3D shape 3D22. The parameter generation unit 53 generates position and posture transformation parameters so that the second cylinder axis AX22 in the second connection region is perpendicular to the first cylinder axis AX21 in the first connection region.

[0149] The second connection region is positioned close to the first connection region. The parameter generation unit 53 generates position and posture transformation parameters so that the region between the first connection region and the second connection region is small. 3D data is missing from the region between the first connection region and the second connection region (the 90-degree joint region). The parameter generation unit 53 generates position and posture transformation parameters so that the first connection region and the second connection region do not overlap each other.

[0150] In addition to the 3D data of the curved region, 3D data of the straight pipe section may be missing. Therefore, the length of the section between the first connection region and the second connection region is actually unknown. After the image IMG21 shown in FIG. 12 is displayed, the position or orientation of the 3D shape may be adjusted, as in the first modification of the first embodiment.

[0151] The display control unit 56 displays the region R22 and the region R23 in the region R21. The region R22 corresponds to the first connection region, and the region R23 corresponds to the second connection region.

[0152] The purpose of each embodiment of the present invention is to display an image of a wide range of 3D shapes even if the first 3D data and the second 3D data do not contain 3D data of a common area. Therefore, the positions and orientations of the first 3D data and the second 3D data do not need to be precisely adjusted. The user may adjust the positions and orientations. The first 3D data and the second 3D data may be loosely connected to each other.

[0153] 13 shows an example of an image displayed on the display unit 71. Explanation of the same parts as those shown in FIG.

[0154] The point on the first cylindrical axis AX21 at the end of the first 3D shape 3D21 coincides with the point on the second cylindrical axis AX22 at the start of the second 3D shape 3D22. Part of the first connection region and part of the second connection region overlap each other. The structure of the subject to be inspected actually differs from the structure shown in FIG. 13. If the first 3D data and the second 3D data do not need to be accurately connected to each other, the display control unit 56 can display the image IMG22 on the display unit 71 as shown in FIG. 13. To improve the appearance of a wide range of 3D shapes, the user may adjust the position and orientation of the 3D data in the same manner as in the first modification of the first embodiment.

[0155] In the second modification of the first embodiment, when the object is a 90-degree joint, the image display device 50 executes a process similar to the process of selecting a straight pipe as the object. This enables the image display device 50 to connect the first 3D data and the second 3D data to each other and display images of a wide range of 3D shapes.

[0156] (Third Modification of the First Embodiment) A third modification of the first embodiment of the present invention will now be described. In the third modification of the first embodiment, an image display device 50 shown in FIG.

[0157] The object in the second modified example of the first embodiment described above is a 90-degree joint of the same diameter. On the other hand, in the third modified example of the first embodiment, an example will be described in which the object is a 90-degree joint of different diameters. There are cases in which joints of different diameters are used in pipes. The third modified example of the first embodiment is applied to the inspection of such pipes.

[0158] After the dialog box DL10 shown in Fig. 4 is displayed on the display unit 71, the display control unit 56 displays a dialog box DL20 shown in Fig. 14 on the display unit 71. The user operates the operation unit 70 to input the inner diameter of the first 3D data into an input box IB21 and input the inner diameter of the second 3D data into an input box IB22.

[0159] The information accepting unit 57 accepts the inner diameter input into the input box IB21 and the inner diameter input into the input box IB22. In step S106, the parameter generating unit 53 generates a scale conversion parameter based on the inner diameter of the cylinder of the first 3D data and the inner diameter of the cylinder of the second 3D data.

[0160] An example will be described in which the subject is a 90-degree joint and the inner diameter of the cylinder of the second 3D data is 1.5 times the inner diameter of the cylinder of the first 3D data. FIG. 15 shows an example of an image displayed on the display unit 71. The display control unit 56 displays an image IMG31 on the display unit 71. The image IMG31 includes an area R31. Images of a first 3D shape 3D31 and a second 3D shape 3D32 are displayed in the area R31. The first 3D shape 3D31 is represented by the first 3D data. The second 3D shape 3D32 is represented by the second 3D data.

[0161] 15, the parameter generation unit 53 generates position and orientation transformation parameters for matching the position and orientation of the second 3D coordinate system with those of the first 3D coordinate system. The parameter generation unit 53 also generates scale transformation parameters for matching the scale of the second 3D coordinate system with that of the first 3D coordinate system. The parameter generation unit 53 calculates a first cylinder axis AX31 of the first 3D shape 3D31 and a second cylinder axis AX32 of the second 3D shape 3D32. The parameter generation unit 53 generates scale transformation parameters for making the diameter DM32 of the cylinder of the second 3D shape 3D32 1.5 times the diameter DM31 of the cylinder of the first 3D shape 3D31.

[0162] The first 3D data includes time stamps from time t1 to time t2 shown in Fig. 15, and the second 3D data includes time stamps from time t3 to time t4 shown in Fig. 15. The parameter generation unit 53 identifies a first connection region of the first 3D shape 3D31 associated with time t2, and identifies a second connection region of the second 3D shape 3D32 associated with time t3.

[0163] The first connection region includes the end of the first 3D shape 3D31. The second connection region includes the start of the second 3D shape 3D32. The parameter generation unit 53 generates position and posture transformation parameters so that the second cylinder axis AX32 in the second connection region is perpendicular to the first cylinder axis AX31 in the first connection region.

[0164] The second connection region is positioned close to the first connection region. The parameter generation unit 53 generates position and orientation transformation parameters so that the region between the first connection region and the second connection region is small. 3D data in the region between the first connection region and the second connection region is missing. The parameter generation unit 53 generates position and orientation transformation parameters so that the first connection region and the second connection region do not overlap with each other. After an image of a wide range of 3D data is displayed, the user may adjust the position and orientation of the 3D data in the same manner as in the first modification of the first embodiment.

[0165] The display control unit 56 displays the region R32 and the region R33 in the region R31. The region R32 corresponds to the first connection region, and the region R33 corresponds to the second connection region.

[0166] In the third modification of the first embodiment, when the object is a 90-degree joint with different diameters, the image display device 50 executes a process similar to the process of selecting a straight pipe as the object. This enables the image display device 50 to connect the first 3D data and the second 3D data to each other and display images of a wide range of 3D shapes.

[0167] (Fourth Modification of the First Embodiment) A fourth modified example of the first embodiment of the present invention will now be described. In the fourth modified example of the first embodiment, an image display device 50 shown in FIG.

[0168] In the first embodiment described above, the object is a straight pipe. In the second and third modified examples of the first embodiment described above, the object is a 90-degree joint. On the other hand, in the fourth modified example of the first embodiment, an example in which the object is a U-shaped pipe will be described. The heat exchange tube or the like has a U-shaped pipe.

[0169] FIG. 16 shows an example of a U-shaped pipe. The U-shaped pipe has a straight pipe section SP41 and an arc section CP41. An example in which part of the 3D data is missing from the arc section CP41 will be described below. The first 3D data and the second 3D data are connected to each other at the arc section CP41. When the first 3D data and the second 3D data are connected to each other at the straight pipe section SP41, the image display device 50 performs processing similar to that in the first embodiment in which the subject is a straight pipe, and connects the first 3D data and the second 3D data to each other.

[0170] For example, the structural information of a U-shaped pipe indicates that the 3D shapes of the first 3D data and the second 3D data are approximately cylindrical. The structural information indicates that the arc portion is approximately donut-shaped. The range of the arc portion for which 3D data exists is a 180-degree range, i.e., a semicircle only. The structural information indicates that the inner diameter of the 3D shape of the first 3D data and the inner diameter of the 3D shape of the second 3D data are equal. The structural information indicates that the two central axes of the two cylinders in the straight pipe section are parallel to each other and that the arc portion has an arbitrary radius of curvature. In step S106, the parameter generation unit 53 generates conversion parameters based on the structural information.

[0171] FIG. 17 shows an example of an image displayed on the display unit 71. The display control unit 56 displays an image IMG41 on the display unit 71. The image IMG41 includes an area R41. Images of a first 3D shape 3D41 and a second 3D shape 3D42 are displayed in the area R41. The first 3D shape 3D41 is represented by the first 3D data. The second 3D shape 3D42 is represented by the second 3D data.

[0172] 17, the parameter generation unit 53 generates position and orientation transformation parameters for matching the position and orientation of the second 3D coordinate system with those of the first 3D coordinate system. The parameter generation unit 53 also generates scale transformation parameters for matching the scale of the second 3D coordinate system with that of the first 3D coordinate system. The parameter generation unit 53 calculates a first cylinder axis AX41 of the first 3D shape 3D41 and a second cylinder axis AX42 of the second 3D shape 3D42. The first cylinder axis AX41 and the second cylinder axis AX42 are linear in the straight pipe portion and arc-shaped in the arc portion.

[0173] The parameter generation unit 53 generates a scale conversion parameter for matching the diameter DM42 of the cylinder of the second 3D shape 3D42 with the diameter DM41 of the cylinder of the first 3D shape 3D41. The two inner diameters of the two cylinders in the straight pipe section of a U-shaped pipe are often the same. Therefore, the image display device 50 can generate the scale conversion parameter by using a method similar to the method for generating the scale conversion parameter in the first embodiment.

[0174] The first 3D data includes time stamps from time t1 to time t2 shown in Fig. 17, and the second 3D data includes time stamps from time t3 to time t4 shown in Fig. 17. The parameter generation unit 53 identifies a first connection region of the first 3D shape 3D41 associated with time t2, and identifies a second connection region of the second 3D shape 3D42 associated with time t3.

[0175] The first connection region includes the end of the first 3D shape 3D41. The second connection region includes the start of the second 3D shape 3D42. The parameter generation unit 53 generates position and orientation transformation parameters for connecting the first connection region and the second connection region to each other. The first connection region and the second connection region are connected to each other under the condition that the arc portions have a predetermined radius of curvature.

[0176] If a portion of the 3D data in the arc section is missing, information indicating the position of the connection region and the angle of the corresponding arc section is not available. If the second 3D data includes 3D data of a straight pipe section, the parameter generating unit 53 can estimate the section where 3D data is missing by using the condition that the two central axes of the two cylinders in the straight pipe section are parallel to each other.

[0177] The display control unit 56 displays the region R42 and the region R43 in the region R41. The region R42 corresponds to the first connection region, and the region R43 corresponds to the second connection region.

[0178] The second 3D data may include only 3D data in the arc portion. In this case, it is difficult for the parameter generation unit 53 to generate position and orientation transformation parameters for accurately connecting the first 3D data and the second 3D data to each other. The parameter generation unit 53 may generate temporary position and orientation transformation parameters so that there is no area between the first 3D data and the second 3D data. After a wide range of 3D data images is displayed, the user may adjust the position and orientation of the 3D data in the same manner as in the first modification of the first embodiment. In this case, the user may adjust the position of the 3D data along the arc.

[0179] (Fifth Modification of the First Embodiment) A fifth modification of the first embodiment of the present invention will now be described. In the fifth modification of the first embodiment, an image display device 50 shown in FIG.

[0180] In the first embodiment and the second to fourth modified examples of the first embodiment described above, the subject is an inspection portion of a pipe. On the other hand, in the fifth modified example of the first embodiment, an example will be described in which the subject is an inspection portion of a gas turbine. Gas turbines are the main inspection targets in aircraft engines, generators, etc. Below, an example of blade inspection will be described. Blades are a typical inspection portion of a gas turbine of an aircraft engine.

[0181] The size of the turbine blades of a generator is different from the size of the gas turbine blades of an aircraft engine. However, the structure of the turbine blades is similar to that of the gas turbine blades. Therefore, the following process may be applied to the turbine blade inspection.

[0182] A gas turbine has two or more blades fixed in an annular and radial pattern to a disk-shaped part called a disk.

[0183] FIG. 18 shows a schematic diagram of a blade configuration. In FIG. 18, 12 blades BL50 are arranged on a disk DS50. In practice, several tens of blades or even more than 100 blades are arranged on one disk. A center position CP50 indicates the center of the disk DS50 in a plane perpendicular to the rotation axis of the engine. Multiple stages of disks are arranged in each of the compressor section and turbine section of a gas turbine. Each stage of disk has a number of blades that function as either rotor blades or stator vanes. In inspection manuals, an ID is often assigned to each blade.

[0184] When inspecting the rotor blades, a scope is inserted into the engine through an access port. The rotor blades are rotated manually or automatically, and the tip of the scope is fixed. The inspector then performs the inspection in this state, checking for any abnormalities in each blade.

[0185] On the other hand, when inspecting the stator vanes (nozzle guide vanes) located immediately aft of the combustion chamber, a scope is inserted through the access port of the combustion chamber toward the rear of the chamber. Hereinafter, the nozzle guide vanes will be referred to as stator vanes. When inspecting stator vanes, the blades cannot be rotated. The inspector moves the tip of the scope around the circumference of the stator vane disk to check for any abnormalities in each blade.

[0186] Information INF10 in the dialog box DL10 shown in FIG. 4 includes information about the gas turbine to be inspected and information about the blades in the gas turbine. For example, the user selects "gas turbine" and then selects "blade." When the user inputs information indicating that the selection has been confirmed into the image display device 50, the information accepting unit 57 accepts the information selected by the user. At this time, the information accepting unit 57 accepts the character string "blade."

[0187] For example, the structural information of the blades indicates that the blades in each of the first 3D data and the second 3D data are arranged at equal intervals on a circumference with an arbitrary radius of curvature. The structural information indicates that the blades have the same length. In step S106, the parameter generation unit 53 generates transformation parameters based on the structural information.

[0188] When the engine model, section, and number of stages are specified, the total number of blades, the length of each blade, and the ID of each blade are known. The engine model indicates the model of the engine in which the blade is located. The section indicates the compressor section or the turbine section. The number of stages indicates the position (stage number) of a disk in a stage that includes multiple disk stages. The structural information may include the total number of blades, the length of each blade, and the ID of each blade. In this case, the parameter generation unit 53 can generate position and posture transformation parameters for accurately connecting the first 3D data and the second 3D data to each other.

[0189] The image display device 50 executes the process shown in Fig. 2. The specific process in step S106 differs from the process when the inspection object is a pipe.

[0190] In step S106, the parameter generation unit 53 executes the process shown in Fig. 19. Fig. 19 shows the procedure of the process executed by the parameter generation unit 53. In the following example, the parameter generation unit 53 generates position and orientation conversion parameters for matching the position and orientation of the second 3D coordinate system with the position and orientation of the first 3D coordinate system. The parameter generation unit 53 also generates scale conversion parameters for matching the scale of the second 3D coordinate system with the scale of the first 3D coordinate system.

[0191] The structural information indicates that the blades have the same length. Therefore, the parameter generating unit 53 generates a scale conversion parameter for correcting the scale of the second 3D data so that the blade lengths of the first 3D data and the second 3D data match each other (step S106e). The structural information may include information indicating the overall length of the blade. Based on this information, the parameter generating unit 53 may determine that the blade lengths of the first 3D data and the second 3D data are the same.

[0192] After step S106e, the parameter generating unit 53 calculates the center position of the disc in the first 3D data (first center position) (step S106f). In the example shown in Fig. 18, the parameter generating unit 53 calculates the center position CP50.

[0193] The parameter generating unit 53 may use any method to calculate the center position of the disc. For example, the parameter generating unit 53 may calculate the center position as the position where lines extending in the length direction of each blade intersect. The parameter generating unit 53 may detect the blades in the 3D data by performing pattern matching. The parameter generating unit 53 may calculate a circumscribing circle or an inscribing circle based on the positional relationship of the blades, and then calculate the center position of that circle.

[0194] After step S106f, the parameter generating unit 53 executes the same process as step S106f to calculate the center position of the disc in the second 3D data (second center position) (step S106g). The method for calculating the center position of the disc in step S106g may be the same as or different from the method for calculating the center position of the disc in step S106f.

[0195] After step S106g, the parameter generation unit 53 generates position and posture transformation parameters for correcting the positions of the first 3D data and the second 3D data so that the first center position and the second center position coincide with each other (step S106h).

[0196] After step S106h, the parameter generating unit 53 calculates position and posture transformation parameters for arranging the starting blade in the second 3D data as the blade next to the terminal blade in the first 3D data (step S106i). When step S106i is executed, the process shown in FIG. 19 ends.

[0197] FIG. 20 shows an example of an image displayed on the display unit 71. The display control unit 56 displays an image IMG51 on the display unit 71. The image IMG51 includes an area R51. Images of a first 3D shape 3D51 and a second 3D shape 3D52 are displayed in the area R51. The first 3D shape 3D51 is represented by first 3D data. The second 3D shape 3D52 is represented by second 3D data.

[0198] The first 3D shape 3D51 includes four blades. The first 3D data includes time stamps from time t1 to time t2 shown in FIG. 20. The time stamp (t1) of blade BL51 is the earliest, and the time stamp (t2) of blade BL52 is the latest. The second 3D shape 3D52 includes eight blades. The second 3D data includes time stamps from time t3 to time t4 shown in FIG. 20. The time stamp (t3) of blade BL53 is the earliest, and the time stamp (t4) of blade BL54 is the latest.

[0199] The parameter generation unit 53 identifies a first connection region of the first 3D shape 3D51 associated with time t2, and identifies a second connection region of the second 3D shape 3D52 associated with time t3. The first connection region includes a terminal blade BL52 in the first 3D shape 3D51. The second connection region includes a starting blade BL53 in the second 3D shape 3D52. The parameter generation unit 53 generates position and posture transformation parameters for connecting the second connection region with the first connection region.

[0200] The parameter generation unit 53 identifies a first connection region of a first 3D shape 3D51 associated with time t1, and identifies a second connection region of a second 3D shape 3D52 associated with time t4. The first connection region includes a starting blade BL51 in the first 3D shape 3D51. The second connection region includes a terminal blade BL54 in the second 3D shape 3D52. The parameter generation unit 53 generates position and posture transformation parameters for connecting the second connection region with the first connection region.

[0201] The display control unit 56 displays the region R52 and the region R53 in the region R51. Each of the region R52 and the region R53 includes a first connection region and a second connection region.

[0202] To connect a first connection region including blade BL51 and a second connection region including blade BL54 to each other, the positional relationship between blade BL51 and blade BL54 may be determined based on features of two or more images. The two or more images are used to generate first 3D data and second 3D data. The first connection region and the second connection region may be connected to each other based on the features. This method is called loop closing. Known techniques can be applied to loop closing.

[0203] When the total number of blades that actually exist is the same as the total number of blades in the 3D data, the parameter generator 53 can generate position and orientation transformation parameters for accurately connecting the first 3D data and the second 3D data. The total number of blades in the 3D data is the total number of blades in the first 3D data and the second 3D data.

[0204] 21 shows an example of an image displayed on the display unit 71 when the total number of blades in the 3D data is one less than the total number of blades that actually exist. Explanation of the same parts as those shown in FIG. 20 will be omitted.

[0205] The second 3D shape 3D52 shown in Fig. 21 does not include the blade BL54 shown in Fig. 20. The second 3D shape 3D52 shown in Fig. 21 may not include the blade BL53 shown in Fig. 20. Therefore, in reality, the second 3D shape 3D52 may not include either the blade BL53 or the blade BL54.

[0206] After a wide range of 3D data images are displayed, the user may adjust the position and orientation of the 3D data in the same manner as in the first modification of the first embodiment. At this time, the user may adjust the position of the second 3D data by rotating the second 3D data. The user may also rotate the first 3D data and the second 3D data. The position of the 3D data may be adjusted only on the circumference of a circle with any diameter.

[0207] (Second embodiment) A second embodiment of the present invention will be described. In the second embodiment, a method for generating first 3D data and second 3D data will be described. In the following, an example in which the image display device is an endoscope device will be described.

[0208] The configuration of the endoscope device 1 in the second embodiment will be described with reference to Fig. 22 and Fig. 23. Fig. 22 shows the external appearance of the endoscope device 1. Fig. 23 shows the internal configuration of the endoscope device 1.

[0209] The endoscope device 1 shown in FIG. 22 has an insertion section 2, a main body section 3, an operation section 4, and a display section 5. The endoscope device 1 captures an image of a subject and generates an image. The subject is an industrial product. To observe various subjects, the user can replace the optical adapter attached to the tip 20 of the insertion section 2, select a built-in image processing program, and add an image processing program.

[0210] The insertion section 2 is inserted into the interior of the subject. The insertion section 2 is a long, thin tube that is bendable from the tip 20 to the base end. The insertion section 2 captures an image of the subject and outputs an image signal to the main body 3. An optical adapter is attached to the tip 20 of the insertion section 2. For example, a monocular optical adapter is attached to the tip 20. The main body 3 is a control device that includes a storage section for storing the insertion section 2. The operation section 4 accepts user operations on the endoscope device 1. The display section 5 has a display screen, and displays images of the subject acquired by the insertion section 2, operation menus, etc. on the display screen.

[0211] The operation unit 4 is a user interface. The display unit 5 is a monitor (display) such as an LCD (Liquid Crystal Display). The display unit 5 may be a touch panel. In this case, the operation unit 4 and the display unit 5 are integrated.

[0212] The main body 3 shown in FIG. 23 includes an endoscope unit 8, a CCU (Camera Control Unit) 9, and a control device 10.

[0213] The endoscope unit 8 has a light source device and a bending device, not shown. The light source device supplies illumination light necessary for observation to the tip 20. The bending device bends the bending mechanism built into the insertion portion 2.

[0214] The imaging element 28 is built into the tip 20 of the insertion portion 2. The imaging element 28 is an image sensor. The imaging element 28 photoelectrically converts an optical image of a subject formed by the optical adapter, and generates an imaging signal.

[0215] The CCU 9 drives the imaging element 28. An imaging signal output from the imaging element 28 is input to the CCU 9. The CCU 9 performs preprocessing, including amplification and noise removal, on the imaging signal acquired by the imaging element 28. The CCU 9 converts the preprocessed imaging signal into a video signal such as an NTSC signal.

[0216] The control device 10 includes a video signal processing circuit 12, a ROM (Read Only Memory) 13, a RAM (Random Access Memory) 14, a card interface 15, an external device interface 16, a control interface 17, and a CPU (Central Processing Unit) 18.

[0217] The video signal processing circuit 12 performs predetermined video processing on the video signal output from the CCU 9. For example, the video signal processing circuit 12 performs video processing related to improving visibility. For example, the video processing includes color reproduction, gradation correction, noise suppression, and edge enhancement. For example, the video signal processing circuit 12 combines the video signal output from the CCU 9 with a graphic image signal generated by the CPU 18. The graphic image signal includes an image of an operation screen, etc. The video signal processing circuit 12 outputs the combined video signal to the display unit 5.

[0218] The ROM 13 is a non-volatile recording medium that stores a program for the CPU 18 to control the operation of the endoscope device 1. The RAM 14 is a volatile recording medium that temporarily stores information used by the CPU 18 to control the endoscope device 1. The CPU 18 controls the operation of the endoscope device 1 based on the program stored in the ROM 13.

[0219] A memory card 42 is connected to the card interface 15. The memory card 42 is a recording medium that is detachable from the endoscope device 1. The card interface 15 imports the control processing information, image information, etc. stored in the memory card 42 into the control device 10. The card interface 15 also records the control processing information, image information, etc. generated by the endoscope device 1 onto the memory card 42.

[0220] An external device such as a USB device is connected to the external device interface 16. For example, a personal computer (PC) 41 is connected to the external device interface 16. The external device interface 16 transmits information to the PC 41 and receives information from the PC 41. This allows the PC 41 to display information. Furthermore, a user can perform operations related to the control of the endoscope device 1 by inputting instructions to the PC 41.

[0221] The control interface 17 communicates with the operation unit 4, the endoscope unit 8, and the CCU 9 for operational control. The control interface 17 notifies the CPU 18 of information input by the user to the operation unit 4. The control interface 17 outputs control signals to the endoscope unit 8 for controlling the light source device and the bending device. The control interface 17 outputs control signals to the CCU 9 for controlling the image sensor 28.

[0222] The program executed by the CPU 18 may be recorded on a computer-readable recording medium. The program recorded on this recording medium may be read and executed by a computer other than the endoscope device 1. For example, the program may be read and executed by the PC 41. The PC 41 may control the endoscope device 1 by transmitting control information for controlling the endoscope device 1 to the endoscope device 1 in accordance with the program. Alternatively, the PC 41 may acquire a video signal from the endoscope device 1 and process the acquired video signal.

[0223] The image sensor 28 is a camera that captures a group of still images. The group of still images includes two or more images. Each of the two or more images is temporally associated with the other images included in the group of two or more images. For example, each of the two or more images is a still image. A video may be used instead of the group of still images. Each of the two or more frames included in the video is associated with each other by a timestamp (time code).

[0224] For example, the image sensor 28 is a monocular camera with one viewpoint. In this case, each of the two or more still images is an image captured by the monocular camera. The camera in the second embodiment includes the image sensor 28 and an observation optical system.

[0225] As described above, the endoscope device 1 has the imaging element 28 and the CPU 18. The imaging element 28 captures an image of a subject and generates an imaging signal. The imaging signal includes an image of the subject. Therefore, the imaging element 28 captures an image of the subject and generates the image. The image is a two-dimensional image (2D image). The image captured by the imaging element 28 is input to the CPU 18 via the video signal processing circuit 12.

[0226] 24 shows the functional configuration of CPU 18. The functions of CPU 18 include control unit 180, data acquisition unit 181, parameter generation unit 182, conversion unit 183, data generation unit 184, display control unit 185, information reception unit 186, structure estimation unit 187, and data generation unit 188. At least one of the blocks shown in FIG. 24 may be configured by a circuit different from CPU 18.

[0227] Each unit shown in Fig. 24 may be configured with at least one of a processor and a logic circuit. Each unit shown in Fig. 24 may include one or more processors. Each unit shown in Fig. 24 may include one or more logic circuits.

[0228] The control unit 180 controls the processing executed by each unit shown in FIG.

[0229] The data acquisition unit 181 has the same function as the data acquisition unit 52 shown in Fig. 1. The data acquisition unit 181 is connected to the RAM 14 and acquires the first 3D data and the second 3D data from the RAM 14. The data acquisition unit 181 may acquire the first 3D data and the second 3D data from a recording medium in the PC 41 or the memory card 42.

[0230] The parameter generation unit 182 has the same function as the parameter generation unit 53 shown in Fig. 1. The parameter generation unit 182 generates conversion parameters for converting the first 3D coordinate system and the second 3D coordinate system into a common coordinate system.

[0231] The conversion unit 183 has the same function as the conversion unit 54 shown in Fig. 1. The conversion unit 183 converts the first 3D coordinate system and the second 3D coordinate system into a common coordinate system by using the conversion parameters generated by the parameter generation unit 182. In this way, the conversion unit 183 converts each of the first 3D data and the second 3D data into 3D data in the common coordinate system.

[0232] The data generation unit 184 has the same function as the data generation unit 55 shown in Fig. 1. The data generation unit 184 connects the first 3D data and the second 3D data converted into 3D data in a common coordinate system by the conversion unit 183. In this way, the data generation unit 184 generates 3D data of a wide range of the subject.

[0233] The display control unit 185 controls the processing executed by the video signal processing circuit 12. The CCU 9 outputs a video signal. The video signal includes color data for each pixel of the image acquired by the imaging element 28. The display control unit 185 causes the video signal processing circuit 12 to output the video signal output from the CCU 9 to the display unit 5. The video signal processing circuit 12 outputs the video signal to the display unit 5. The display unit 5 displays an image based on the video signal output from the video signal processing circuit 12. As a result, the display control unit 185 displays the image acquired by the imaging element 28 on the display unit 5.

[0234] The display control unit 185 displays various types of information on the display unit 5. That is, the display control unit 185 displays various types of information on an image. The various types of information may include a cursor. The cursor is a mark that allows the user to designate a specific point on the image.

[0235] For example, the display control unit 185 generates a graphic image signal of various information. The display control unit 185 outputs the generated graphic image signal to the video signal processing circuit 12. The video signal processing circuit 12 combines the video signal output from the CCU 9 and the graphic image signal output from the CPU 18. As a result, the various information is superimposed on the image. The video signal processing circuit 12 outputs the combined video signal to the display unit 5. The display unit 5 displays the image on which the various information is superimposed.

[0236] 1. The display control unit 185 generates a graphic image signal of 3D data. The display control unit 185 outputs the graphic image signal to the video signal processing circuit 12. The same processing as that described above is executed, and the display unit 5 displays an image of the 3D data. As a result, the display control unit 185 displays the image of the 3D data on the display unit 5.

[0237] The user operates the operation unit 4 to input various information to the endoscope device 1. The operation unit 4 outputs the information input by the user. The information is input to the control interface 17, which is an input unit. The information is output from the control interface 17 to the CPU 18. The information receiving unit 186 receives the information input to the endoscope device 1 via the operation unit 4.

[0238] For example, the user operates the operation unit 4 to input cursor position information to the endoscope device 1. If the display unit 5 is configured as a touch panel, the user touches the screen of the display unit 5 to input position information indicating a position on the image to the endoscope device 1. The information receiving unit 186 receives the position information input to the endoscope device 1. The information receiving unit 186 calculates the position on the image based on the position information. The display control unit 185 displays the cursor at the position calculated by the information receiving unit 186.

[0239] Furthermore, the information receiving unit 186 has the same function as the information receiving unit 57 shown in Fig. 1. The information receiving unit 186 receives structural information.

[0240] The structure estimation unit 187 has the same function as the structure estimation unit 58 shown in Fig. 1. When the information receiving unit 186 receives information that does not include structural information, the structure estimation unit 187 estimates the structure of the subject in the connected region based on the information, and generates structural information.

[0241] The data generation unit 188 has the functions shown in Fig. 25. Fig. 25 shows the functional configuration of the data generation unit 188. The functions of the data generation unit 188 include an image acquisition unit 1880, a condition acceptance unit 1881, and a data calculation unit 1882.

[0242] The image acquisition unit 1880 acquires the still image group from the RAM 14. The image acquisition unit 1880 may acquire the still image group from a recording medium in the PC 41 or the memory card 42.

[0243] The user operates the operation unit 4 to input information indicating the conditions for generating 3D data to the endoscope device 1. The condition receiving unit 1881 receives the conditions for generating 3D data based on the information input by the user. Specifically, the conditions include the camera's internal parameters, the camera's distortion correction parameters, setting values, and a reference length. The setting values ​​are used in various processes for generating 3D data. The reference length is used to match the 3D data with the scale of the actual subject.

[0244] The data calculation unit 1882 generates (reconstructs) 3D data of the subject based on two or more images included in the still image group. The data calculation unit 1882 does not need to use all of the images included in the still image group. When the still image group includes three or more images, the data calculation unit 1882 generates 3D data based on all or part of the still image group. The 3D data includes 3D coordinates of two or more points (three-dimensional point cloud) of the subject, camera coordinates, and posture information.

[0245] The camera coordinates indicate the 3D coordinates of the camera that captured each of the two or more images, and are associated with each of the two or more images. The camera coordinates indicate the 3D coordinates of the viewpoint when the image was captured. For example, the camera coordinates indicate the 3D coordinates of the observation optical system of the camera. The attitude information indicates the attitude of the camera that captured each of the two or more images, and is associated with each of the two or more images. For example, the attitude information indicates the attitude of the observation optical system of the camera.

[0246] A specific processing procedure executed by the data calculation unit 1882 will be described. The data calculation unit 1882 uses the group of still images acquired by the image acquisition unit 1880 and the conditions accepted by the condition acceptance unit 1881. An example in which the data calculation unit 1882 uses two images (still images) included in the group of still images will be described below. When the two images are generated, the two viewpoints of the camera are different from each other. Even when three or more images are used, the basic principle remains the same as when two images are used. The method described below can also be applied when three or more images are used.

[0247] In the method described below, feature points are detected in each of two images captured from two different viewpoints, and the multiple feature points are associated with each other. Furthermore, in the method described below, the camera position, camera orientation, and 3D coordinates of the feature points are estimated based on the multiple feature points. The method using feature point information is called the indirect method. The method applicable to each embodiment of the present invention is not limited to this method.

[0248] For example, there is a method of directly using pixel values ​​of two images acquired from two different viewpoints. By using this method, the camera position, camera orientation, and 3D coordinates corresponding to each pixel are estimated. This method is called the Direct Method. This method may be used in each embodiment of the present invention. Any method may be used as long as it is possible to estimate the camera position, camera orientation, and 3D coordinates of an object by using two or more images acquired from two or more different viewpoints.

[0249] Figure 26 shows a schematic diagram of an image acquisition situation in which two images of a subject are acquired. In the following explanation, the term "camera" is used in a broad sense. When an endoscope acquires images, the term "camera" in the following explanation specifically refers to the observation optical system at the tip of the endoscope.

[0250] As shown in Fig. 26, first, an image I1 is acquired in an imaging state c1 of the camera. Next, an image I2 is acquired in an imaging state c2 of the camera. At least one of the imaging position and the imaging attitude is different between the imaging states c1 and c2. In Fig. 26, both the imaging position and the imaging attitude are different between the imaging states c1 and c2.

[0251] In each embodiment of the present invention, it is assumed that images I1 and I2 are acquired by the same endoscope. Furthermore, in each embodiment of the present invention, it is assumed that the parameters of the objective optical system of the endoscope do not change. The parameters of the objective optical system include focal length, distortion, and pixel size of the image sensor. Hereinafter, for convenience, the parameters of the objective optical system are abbreviated as intrinsic parameters. Assuming such conditions, the intrinsic parameters that describe the characteristics of the optical system of the endoscope can be commonly used regardless of the position and orientation of the camera (observation optical system). In each embodiment of the present invention, it is assumed that the intrinsic parameters are acquired at the time of shipment from the factory. Furthermore, in each embodiment of the present invention, it is assumed that the intrinsic parameters are known when the images are acquired.

[0252] In each embodiment of the present invention, it is assumed that two or more images are extracted from a group of still images, and that the group of still images is acquired by a single endoscope. However, the present invention is not limited to this. For example, the present invention can also be applied to cases where a 3D model is reconstructed using multiple groups of still images acquired by multiple endoscopes. In this case, it is sufficient that images I1 and I2 are acquired using different endoscopic devices, and different internal parameters are stored for each endoscope. Even if the internal parameters are unknown, calculations can be performed using the internal parameters as variables. Therefore, whether or not the internal parameters are known does not significantly affect the subsequent procedures.

[0253] The process of calculating the 3D coordinates of a subject based on two images and generating 3D data will be described using Fig. 27. Fig. 27 shows the procedure for the process of generating 3D data.

[0254] First, the data calculation unit 1882 executes a feature point detection process (step S200). In the feature point detection process, the data calculation unit 1882 detects feature points in each of the two images. A feature point refers to a corner, an edge, or the like with a large image brightness gradient among the subject information captured in the image. Methods for detecting these feature points include SIFT (Scale-Invariant Feature Transform) and FAST (Features from Accelerated Segment Test). The data calculation unit 1882 can detect feature points in the images by using such methods.

[0255] Figure 26 shows the feature point P 11 is detected and the feature point P 12This shows an example where only one feature point is detected for each image. In Figure 26, only one feature point is displayed for each image, but in reality, multiple feature points are detected for each image. The number of feature points detected may differ between images. Each feature point detected from each image is converted into data called a feature value. A feature value is data that represents the characteristics of a feature point.

[0256] After step S200, the data calculation unit 1882 executes a feature point association process (step S201). In the feature point association process, the data calculation unit 1882 compares the correlation of feature amounts between images for each feature point detected by the feature point detection process (step S200). When the correlation of feature amounts is compared and feature points with similar feature amounts are found in each image, the data calculation unit 1882 stores that information in the RAM 14. In this way, the data calculation unit 1882 associates the feature points of each image with each other. On the other hand, when feature points with similar feature amounts are not found, the data calculation unit 1882 discards the information of those feature points.

[0257] After step S201, the data calculation unit 1882 reads out the coordinates of feature points (feature point pairs) of the two images associated with each other from the RAM 14. The data calculation unit 1882 executes a position and orientation calculation process based on the read coordinates (step S202). In the position and orientation calculation process, the data calculation unit 1882 calculates the relative position and orientation between the imaging state c1 of the camera that captured image I1 and the imaging state c2 of the camera that captured image I2. More specifically, the data calculation unit 1882 calculates a matrix E by solving the following equation (1) that uses an epipolar constraint:

[0258]

number

[0259] The matrix E is called the fundamental matrix. The fundamental matrix E is a matrix that holds the relative position and orientation between the imaging state c1 of the camera that captured image I1 and the imaging state c2 of the camera that captured image I2. In equation (1), the matrix p1 is a matrix that contains the coordinates of feature points detected from image I1. The matrix p2 is a matrix that contains the coordinates of feature points detected from image I2. The fundamental matrix E contains information about the relative position and orientation of the cameras, and therefore corresponds to the extrinsic parameters of the cameras. The data calculation unit 1882 can solve the fundamental matrix E by using a known algorithm.

[0260] As shown in FIG. 26, when the amount of change in camera position (relative position) is t and the amount of change in camera attitude (relative attitude) is R, equations (2) and (3) hold true.

[0261]

number

[0262] In equation (2), the amount of movement in the x-axis direction is t x The amount of movement in the y-axis direction is expressed as t y and the amount of movement in the z-axis direction is t z In equation (3), the rotation amount α around the x-axis is R x (α), and the rotation amount around the y-axis is R y (β), and the rotation amount around the z-axis γ is R z After the fundamental matrix E is calculated, an optimization process called bundle adjustment may be performed to improve the accuracy of the reconstruction of the 3D coordinates.

[0263] The data calculation unit 1882 calculates 3D coordinates (camera coordinates) in the coordinate system of the 3D model by using the calculated amount of change in position of the camera. For example, the data calculation unit 1882 defines the 3D coordinates of the camera that acquired image I1. The data calculation unit 1882 calculates the 3D coordinates of the camera that acquired image I2 based on the 3D coordinates of the camera that acquired image I1 and the amount of change in position of the camera that acquired image I2.

[0264] The data calculation unit 1882 calculates posture information in the coordinate system of the 3D model by using the calculated posture change amount of the camera. For example, the data calculation unit 1882 defines posture information of the camera that acquired image I1. The data calculation unit 1882 generates posture information of the camera that acquired image I2 based on the posture information of the camera that acquired image I1 and the posture change amount of the camera that acquired image I2.

[0265] The data calculation unit 1882 executes a position and orientation calculation process (step S202) to generate three-dimensional shape (3D shape) data (3D shape data) including 3D coordinates (camera coordinates) at the camera position and orientation information indicating the camera orientation. Furthermore, when a method such as Structure from Motion or visual-SLAM is applied to the position and orientation calculation process (step S202), the data calculation unit 1882 further calculates the 3D coordinates of each feature point in step S202. The 3D shape data generated in step S202 does not include 3D coordinates of points on the object other than the feature points. Therefore, the 3D shape data indicates a sparse 3D shape of the object.

[0266] The 3D shape data includes the 3D coordinates of each feature point, the camera coordinates described above, and the posture information described above. The 3D coordinates of each feature point are defined in the coordinate system of the 3D data. The 3D coordinates of each feature point are associated with the two-dimensional coordinates (2D coordinates) of each feature point. The 2D coordinates of each feature point are defined in the coordinate system of the image in which each feature point is included. The 2D and 3D coordinates of each feature point are associated with the image in which each feature point is included.

[0267] After step S202, the data calculation unit 1882 executes a three-dimensional shape reconstruction process based on the relative position and orientation (position change amount t and orientation change amount R) of the camera calculated in step S202 (step S203). In the three-dimensional shape reconstruction process, the data calculation unit 1882 generates 3D data of the subject. Methods for reconstructing the three-dimensional shape of the subject include patch-based multi-view stereo (PMVS) and rectified stereo matching. However, the method is not particularly limited.

[0268] In step S203, the data calculation unit 1882 calculates the 3D coordinates of points on the object other than the feature points. The 3D coordinates of each point other than the feature points are defined in the coordinate system of the 3D data. The 3D coordinates of each point are associated with the 2D coordinates of each point. The 2D coordinates of each point are defined in the coordinate system of the 2D image in which each point is included. The 2D coordinates and 3D coordinates of each point are associated with the 2D image in which each point is included. The data calculation unit 1882 updates the 3D shape data. The updated 3D shape data includes the 3D coordinates of each feature point, the 3D coordinates of each point other than the feature points, camera coordinates, and posture information. The 3D shape data updated in step S203 includes the 3D coordinates of points on the object other than the feature points in addition to the 3D coordinates of the feature points. Therefore, the 3D shape data indicates a dense 3D shape of the object.

[0269] After step S203, the data calculation unit 1882 executes a three-dimensional coordinate conversion process (step S204) based on the 3D shape data processed in the three-dimensional shape reconstruction process (step S203) and the reference length accepted by the condition acceptance unit 1881. In the three-dimensional coordinate conversion process, the data calculation unit 1882 converts the 3D shape data of the subject into three-dimensional coordinate data (3D data) having a dimension of length. When step S204 is executed, the process shown in FIG. 27 ends.

[0270] To shorten the processing time, step S203 may be omitted, in which case, after step S202 is executed, step S204 is executed without executing step S203.

[0271] Step S204 may be omitted. In this case, after step S203 is executed, step S204 is not executed and the process shown in FIG. 27 ends. In this case, the 3D data indicates the relative shape of the subject without the dimension of length. Even when the 3D data indicates the relative shape of the subject, the endoscope device 1 can identify the area of ​​the 3D data corresponding to the camera coordinates.

[0272] In order for 3D data to be generated according to the principle shown in Fig. 26, at least a portion of the area of ​​each image must be common to at least a portion of each area of ​​at least one other image. That is, an area of ​​a first image and an area of ​​a second image different from the first image include a common area. The area other than the common area in the first image and the area other than the common area in the second image are different from each other.

[0273] After the 3D data is generated, the endoscope device 1 executes the same process as that shown in Fig. 2. The endoscope device 1 can connect the first 3D data and the second 3D data to each other and display a wide range of 3D shape images. The image display device 50 shown in Fig. 1 may include a data generation unit 188.

[0274] (Third embodiment) A third embodiment of the present invention will now be described, in which data generated by a sensor is used to estimate the structure in connected regions of an object.

[0275] The image display device 50 shown in Fig. 1 is changed to an image display device 50b shown in Fig. 28. Fig. 28 shows the configuration of the image display device 50b. Description of the same configuration as that shown in Fig. 1 will be omitted.

[0276] 28 includes a control unit 51, a data acquisition unit 52, a parameter generation unit 53, a conversion unit 54, a data generation unit 55, a display control unit 56, and a structure estimation unit 58. The image display device 50b does not include the information reception unit 57 shown in FIG. 1. An operation unit 70, a display unit 71, a communication unit 72, a storage unit 73, and a sensor 74 shown in FIG. 28 are connected to the image display device 50b. The image display device 50b may include at least one of the operation unit 70, the display unit 71, the communication unit 72, the storage unit 73, and the sensor 74.

[0277] Each unit of the image display device 50b may be configured with at least one of a processor and a logic circuit. Each unit of the image display device 50b may include one or more processors. Each unit of the image display device 50b may include one or more logic circuits.

[0278] The sensor 74 outputs sensor data, which is affected by the geometric structure of the object. The sensor 74 is an image sensor (image pickup element).

[0279] The sensor data may include, as sensor values, pixel values, acceleration measurements, or angular velocity measurements. An image sensor generates an image of an object, in which the structure of the object is visible.

[0280] The sensor data is stored in the storage unit 73. The structure estimation unit 58 uses the sensor data to estimate the structure of the subject in the connected region and generate structure information.

[0281] The processing executed by the image display device 50b will be described using Fig. 29. Fig. 29 shows the procedure of the processing executed by the image display device 50b. The description of the same processing as that shown in Fig. 2 will be omitted.

[0282] After step S103, the data acquisition unit 52 connects to the storage unit 73 and acquires sensor data in the connection area from the storage unit 73 (step S120).

[0283] The 3D data includes a timestamp. In the sensor data, the sensor value and the timestamp are associated with each other. Therefore, the data acquisition unit 52 can acquire the sensor data generated at the time indicated by the timestamp of the 3D data. For example, the data acquisition unit 52 acquires, from the storage unit 73, the sensor data for the section from the time of the end of the first 3D data to the time of the start of the second 3D data.

[0284] The following describes an example in which the sensor 74 is an image sensor and the data acquisition unit 52 acquires two or more images from the storage unit 73. The following processing can also be applied to the case in which the data acquisition unit 52 acquires one image from the storage unit 73.

[0285] After step S120, the structure estimation unit 58 estimates the structure of the subject in the connected region by using the two or more images acquired in step S120, and generates structure information (step S121). After step S121, step S106 is executed.

[0286] The structure estimation unit 58 may use any method to estimate the structure of the object. For example, the structure estimation unit 58 may estimate the structure of the object by using AI. The structure estimation unit 58 may estimate the structure of the object by detecting image features, calculating feature amounts, and using a support vector machine (SVM).

[0287] When the structure estimation unit 58 determines that the object is a straight pipe, the structure estimation unit 58 may acquire the structure information of the straight pipe from a reference table TB11 shown in FIG.

[0288] Each aspect of the present invention may include the following variations: The structural information is generated based on the data output from the sensor 74 .

[0289] In the third embodiment, the structure estimation unit 58 estimates the structure of the subject in the connected region by using sensor data. The image display device 50b does not need to receive information from the user.

[0290] (First modified example of the third embodiment) A first modified example of the third embodiment of the present invention will be described. In the first modified example of the third embodiment, an image display device 50b shown in Fig. 28 is used. In the first modified example of the third embodiment, an endoscope device 1 shown in Fig. 22 is used to generate 3D data.

[0291] In the third embodiment described above, the structure estimation unit 58 estimates the structure of the subject in the connected region by using an image generated by an image sensor. On the other hand, in the first modified example of the third embodiment, the structure estimation unit 58 uses sensor data output from an Inertial Measurement Unit (IMU) instead of an image.

[0292] An IMU is used as the sensor 74. The sensor 74 has an acceleration sensor and an angular velocity sensor, and detects acceleration and angular velocity. For example, the sensor 74 is disposed at the tip 20 of the insertion section 2. When the insertion section 2 is inserted into a pipe, the insertion section 2 moves according to the structure of the pipe. Therefore, the sensor 74 outputs sensor data according to the structure of the pipe. The structure estimation unit 58 calculates the trajectory of the tip 20 by using the sensor data. The trajectory indicates two or more positions where the tip 20 is positioned.

[0293] Image display device 50b executes the processing shown in Fig. 29. In step S121, structure estimation unit 58 uses sensor data to calculate the trajectory of tip 20. In step S121, structure estimation unit 58 estimates the structure of the subject in the connected region based on the trajectory, and generates structure information.

[0294] For example, the structure estimation unit 58 estimates the structure of the subject by performing the following process: Each of the first 3D data and the second 3D data includes 3D coordinates in a first section, a second section, and a third section. The first section includes the start of the 3D shape. The second section includes the end of the 3D shape. The third section is located between the first section and the second section. For example, the second section in the first 3D data and the first section in the second 3D data correspond to a connected region.

[0295] The structure estimation unit 58 analyzes the trajectory of the tip 20 in each of the second section in the first 3D data, the third section in the first 3D data that is close to the second section, the first section in the second 3D data, and the third section in the second 3D data that is close to the first section. If the structure estimation unit 58 determines that the tip 20 moves in a substantially straight line in all sections, the structure estimation unit 58 determines that the object is a straight pipe. If the structure estimation unit 58 determines that the direction of movement of the tip 20 turns approximately 90 degrees as it passes through each of the above sections, the structure estimation unit 58 determines that the object is a 90-degree joint. The structure estimation unit 58 generates structural information according to the structure of the object.

[0296] (Second modified example of the third embodiment) A second modified example of the third embodiment of the present invention will be described. In the second modified example of the third embodiment, an image display device 50b shown in FIG. 28 is used. The image display device 50b does not need to have a sensor 74. In the second modified example of the third embodiment, an endoscope device 1 shown in FIG. 22 is used to generate 3D data.

[0297] In the third embodiment described above, the structure estimation unit 58 estimates the structure of the subject in the connection region by using an image generated by an image sensor. On the other hand, in the second modified example of the third embodiment, the structure estimation unit 58 uses history information of operations related to the bending of the insertion unit 2 instead of an image.

[0298] The user operates the operation unit 4 to input information indicating the bending direction and bending amount to the endoscope device 1. The control unit 180 of the CPU 18 generates a control signal based on the information input by the user. The control signal is output to the endoscope unit 8 via the control interface 17. The bending device of the endoscope unit 8 bends the bending mechanism based on the control signal.

[0299] The control unit 180 generates history information of operations related to bending of the insertion unit 2. The history information includes information indicating the bending direction and bending amount. The communication unit 72 receives the history information from the endoscope device 1. The history information is stored in the storage unit 73.

[0300] Image display device 50b executes the process shown in Fig. 29. In step S20, data acquisition unit 52 acquires history information instead of sensor data from storage unit 73. In step S121, structure estimation unit 58 uses the history information instead of sensor data to estimate the structure of the subject in the connected region and generate structure information.

[0301] For example, the structure estimation unit 58 estimates the structure of the subject by performing the following process: Each of the first 3D data and the second 3D data includes 3D coordinates in a first section, a second section, and a third section. The first section includes the start of the 3D shape. The second section includes the end of the 3D shape. The third section is located between the first section and the second section. For example, the second section in the first 3D data and the first section in the second 3D data correspond to a connected region.

[0302] The structure estimation unit 58 analyzes history information for each of the second section in the first 3D data, the third section in the first 3D data that is close to the second section, the first section in the second 3D data, and the third section in the second 3D data that is close to the first section. If the structure estimation unit 58 determines that the insertion portion 2 is not curved in any of the sections, the structure estimation unit 58 determines that the object is a straight pipe. If the structure estimation unit 58 determines that the insertion portion 2 is curved approximately 90 degrees while the tip 20 of the insertion portion 2 passes through each of the above sections, the structure estimation unit 58 determines that the object is a 90-degree joint.

[0303] Each aspect of the present invention may include the following variations: The first 3D data is generated by using two or more first images; The second 3D data is generated by using two or more second images; The two or more first images and the two or more second images are generated based on an optical image of the subject acquired by the insertion unit 2; The insertion unit 2 is inserted into an object having the subject and is bendable; and the structural information is generated based on information indicating the bending direction and bending amount of the insertion unit 2.

[0304] (Fourth embodiment) A fourth embodiment of the present invention will be described. In the fourth embodiment, an image display device 50 shown in FIG. 1 is used. Reference data is used as structural information. For example, the reference data is design data including design values ​​of a geometric structure. For example, the design data is generated by using 3D-CAD (computer-aided design).

[0305] The processing executed by the image display device 50 will be described using Fig. 30. Fig. 30 shows the procedure of the processing executed by the image display device 50. The description of the same processing as that shown in Fig. 2 will be omitted.

[0306] After step S103, the information receiving unit 57 receives the reference data (step S130). For example, the communication unit 72 receives the reference data from an external device. The information receiving unit 57 receives the reference data received by the communication unit 72.

[0307] After step S130, the display control unit 56 displays the image of the reference data on the display unit 71 (step S131).

[0308] FIG. 31 shows an example of an image displayed on the display unit 71. The display control unit 56 displays an image IMG61 on the display unit 71. The image IMG61 includes an area R61, an area R62, and an area R63. An image of a first 3D shape 3D61 is displayed in the area R61. The first 3D shape 3D61 is represented by the first 3D data. An image of a second 3D shape 3D62 is displayed in the area R62. The second 3D shape 3D62 is represented by the second 3D data. An image of a third 3D shape 3D63 is displayed in the area R63. The third 3D shape 3D63 is represented by the reference data.

[0309] When information accepting unit 57 accepts reference data, three types of images shown in Fig. 31 are displayed. When information accepting unit 57 accepts reference data, it is not necessary for the image of the reference data to be displayed on display unit 71. Therefore, it is not necessary to execute step S131.

[0310] After step S131, the parameter generating unit 53 generates conversion parameters by using the reference data (step S132). After step S132, step S107 is executed.

[0311] The parameter generation unit 53 performs the following process in step S132. The parameter generation unit 53 performs shape matching using the first 3D data and the reference data. As a result, the parameter generation unit 53 detects the 3D shape of the reference data that corresponds to the 3D shape of the first 3D data. The parameter generation unit 53 performs shape matching using the second 3D data and the reference data. As a result, the parameter generation unit 53 detects the 3D shape of the reference data that corresponds to the 3D shape of the second 3D data.

[0312] Known methods may be applied to 3D shape matching. The parameter generating unit 53 may perform a matching process that uses color information in addition to shape information.

[0313] The parameter generation unit 53 generates position and orientation transformation parameters for matching the position and orientation of the first 3D coordinate system with those of the common coordinate system. The parameter generation unit 53 generates scale transformation parameters for matching the scale of the first 3D coordinate system with that of the common coordinate system. The common coordinate system is the 3D coordinate system of the reference data.

[0314] The parameter generation unit 53 generates a position and orientation transformation parameter for matching the position and orientation of the second 3D coordinate system with the position and orientation of the common coordinate system, and generates a scale transformation parameter for matching the scale of the second 3D coordinate system with the scale of the common coordinate system.

[0315] FIG. 32 shows an example of an image displayed on the display unit 71. Description of the same parts as those shown in FIG. 31 will be omitted. The display control unit 56 displays an image IMG62 on the display unit 71. The image IMG62 includes an area R64. Images of a first 3D shape 3D61, a second 3D shape 3D62, and a third 3D shape 3D63 are displayed in the area R64. The user can confirm that the first 3D data and the second 3D data are each positioned in the correct position in the reference data. This allows the user to confirm that the first 3D data and the second 3D data are accurately connected to each other.

[0316] The reference data used as structural information is not limited to design data, and may be data other than design data.

[0317] For example, 3D data generated by using two or more images of the object of inspection generated in a previous inspection may be used as the reference data. The 3D data is generated in the process shown in FIG. 27. The 3D data is different from both the first 3D data and the second 3D data. The 3D data used as the reference data may be generated by an endoscopic device other than the device that generates the 3D data of the object inside the object of inspection.

[0318] As long as the parameter generation unit 53 can match the positions of the first 3D data and the second 3D data with the position of the reference data, the reference data does not need to be 3D data. The reference data may be two-dimensional data (2D data). The parameter generation unit 53 converts the 3D data into 2D information to match the positions of the 3D data with the positions of the 2D data. The 2D information indicates a two-dimensional shape on an arbitrary plane. The parameter generation unit 53 matches the positions of the 2D information with the positions of the 2D data.

[0319] Each aspect of the present invention may include the following modifications: The structural information is design data including design values ​​of the geometric structure of the subject, or is 3D data different from both the first 3D data and the second 3D data.

[0320] In the fourth embodiment, the image display device 50 can connect the first 3D data and the second 3D data to each other, and can display images of a wide range of 3D shapes on the image of the reference data.

[0321] (First modified example of the fourth embodiment) A first modification of the fourth embodiment of the present invention will be described. In the fourth embodiment described above, the IDs of the blades are not used to connect the first 3D data and the second 3D data to each other. On the other hand, in the first modification of the fourth embodiment, the first 3D data and the second 3D data are connected to each other so that the IDs of the blades in the first 3D data and the second 3D data match the IDs of the blades in the reference data.

[0322] In a first modification of the fourth embodiment, the image display device 50 shown in FIG. 1 is used. A storage unit 73 stores reference data of a gas turbine. FIG. 33 shows the reference data of a gas turbine GT70. In the example shown in FIG. 33, the gas turbine GT70 has 12 blades. An ID is assigned to each blade. The reference data shown in FIG. 33 may be a cross-sectional view of the gas turbine GT70 or 3D-CAD data.

[0323] Each ID corresponds to the position of a blade. For example, when consecutive IDs are assigned to blades, two consecutive IDs are assigned to two adjacent blades. The structural information in the first variant of the fourth embodiment is the reference data and the ID assigned to each blade.

[0324] In step S106, the parameter generating unit 53 executes the process shown in Fig. 34. Fig. 34 shows the procedure of the process executed by the parameter generating unit 53.

[0325] The parameter generating unit 53 identifies blade regions in the first 3D data (step S106j). Each blade region includes one blade.

[0326] The parameter generation unit 53 may use any method to identify the blade region. For example, the parameter generation unit 53 may extract two or more shapes that are similar to each other in the first 3D data as the blade region. Alternatively, the parameter generation unit 53 may perform shape matching using a pre-registered blade shape and the first 3D data, and extract a blade region that includes a shape that matches the blade shape.

[0327] After step S106j, the parameter generating unit 53 assigns an ID to the blade region identified in step S106j (step S106k).

[0328] The parameter generation unit 53 may use any method to assign IDs to blade regions. For example, an image of the first 3D data is displayed on the display unit 71. The user operates the operation unit 70 to input the ID of each blade into the image display device 50. The parameter generation unit 53 assigns the ID to the blade region. The parameter generation unit 53 may also detect a reference blade with a characteristic structure from the first 3D data and assign IDs to each blade in order starting from the reference blade.

[0329] A turning tool may be used when the endoscopic device generates images used to generate 3D data. The turning tool automatically rotates a disk. The turning tool outputs information about the rotation angle of the disk to the endoscopic device. Based on the information, the endoscopic device determines the position of the blade shown in the image.

[0330] For example, before the turning tool rotates the disk, the endoscopic device sets the blade shown in the image as the reference blade. The total number of blades is known, and the angle between two adjacent blades is known. While the disk is rotating, the endoscopic device determines which blade is shown in the image based on the rotation angle of the disk. When the disk has rotated once, the endoscopic device can determine that the reference blade is again shown in the image.

[0331] The endoscopic instrument assigns an ID to each blade and embeds the ID in the image. The endoscopic instrument may assign an ID to each blade in the image and embed the ID in the image without using a turning tool.

[0332] The 3D coordinates of each point included in the 3D data are associated with the image used to generate the 3D data. The parameter generating unit 53 obtains the ID of the image associated with the blade region identified in step S106j and assigns the ID to the blade region.

[0333] After step S106k, the parameter generating unit 53 identifies the blade region in the second 3D data (step S106l). Step S106l is similar to step S106j.

[0334] After step S106l, the parameter generating unit 53 assigns an ID to the blade region identified in step S106l (step S106m). Step S106m is similar to step S106k.

[0335] After step S106m, the parameter generation unit 53 calculates transformation parameters for correcting the position, posture, and scale of the first 3D data so that the ID of the blade in the reference data and the ID of the blade region in the first 3D data match each other (step S106n).

[0336] After step S106n, the parameter generating unit 53 calculates transformation parameters for correcting the position, orientation, and scale of the second 3D data so that the ID of the blade in the reference data and the ID of the blade region in the second 3D data match each other (step S106o). When step S106o is executed, the process shown in Fig. 34 ends.

[0337] FIG. 35 shows an example of an image displayed on the display unit 71. The display control unit 56 displays an image IMG71 on the display unit 71. The image IMG71 includes an area R71. Images of a first 3D shape 3D71, a second 3D shape 3D72, and a third 3D shape 3D73 are displayed in the area R71. The first 3D shape 3D71 is represented by first 3D data. The second 3D shape 3D72 is represented by second 3D data. The third 3D shape 3D73 is represented by reference data.

[0338] The first 3D shape 3D71 includes two blades. The IDs of the two blades are 2 and 3. The position of the first 3D data is set to a position that overlaps with the two blades in the reference data that are assigned IDs 2 and 3.

[0339] The second 3D shape 3D72 includes five blades. The IDs of the five blades are 7 to 11. The position of the second 3D data is set to a position that overlaps with the five blades of the reference data that are assigned IDs 7 to 11.

[0340] Blades assigned IDs 1, 4, 5, 6, or 12 do not exist in the first 3D data or the second 3D data. Blades in the reference data assigned these IDs are displayed.

[0341] The display control unit 56 may display each blade on the display unit 71 so that the user can distinguish between the blades included in the first 3D data or the second 3D data and the blades included in the reference data. For example, the color of the blades included in the first 3D data or the second 3D data may be different from the color of the blades included in the reference data. The user can check the blades included in the 3D data and the blades not included in the 3D data.

[0342] The parameter generating unit 53 may execute the process shown in Fig. 19 in addition to the process shown in Fig. 34. This allows the parameter generating unit 53 to connect the first 3D data and the second 3D data to each other more accurately.

[0343] Each aspect of the present invention may include the following modifications: The subject includes two or more objects, and the structural information indicates positions at which the two or more objects are arranged.

[0344] In the first modification of the fourth embodiment, the parameter generating unit 53 generates the transformation parameters by using the ID as the structural information, so that the image display device 50 can accurately connect the first 3D data and the second 3D data.

[0345] (Second modified example of the fourth embodiment) A second modified example of the fourth embodiment of the present invention will now be described. In the second modified example of the fourth embodiment, an image display device 50 shown in FIG.

[0346] The object in the first modification of the fourth embodiment described above is a gas turbine blade. On the other hand, the object in the second modification of the fourth embodiment is a combustion chamber of the gas turbine. The combustion chamber includes a fuel injection nozzle and a plate as typical inspection parts. The plate is arranged around the fuel injection nozzle.

[0347] Fuel injection nozzles, like blades, are often arranged in a ring shape. An ID is often assigned to each fuel injection nozzle. Each ID has a value corresponding to the position where the fuel injection nozzle is arranged. Each fuel injection nozzle is arranged on an arbitrary circumference, and the center position of that circle is defined.

[0348] FIG. 36 shows the structure of a combustion chamber CC80. The combustion chamber CC80 has 12 fuel injection nozzles NZ80. The actual number of fuel injection nozzles varies depending on the engine model. The 12 fuel injection nozzles NZ80 are arranged in the front stage of the combustion chamber CC80. A center position CP80 indicates the center of a circle in which the 12 fuel injection nozzles NZ80 are arranged. In a second modification of the fourth embodiment, the first 3D data and the second 3D data are connected to each other by using IDs assigned to the fuel injection nozzles.

[0349] Information INF10 in the dialog box DL10 shown in FIG. 4 includes information about the gas turbine to be inspected and information about the combustion chamber in the gas turbine. For example, the user selects "gas turbine" and then selects "combustion chamber." When the user inputs information indicating that the selection has been confirmed into the image display device 50, the information accepting unit 57 accepts the information selected by the user. At this time, the information accepting unit 57 accepts the character string "combustion chamber."

[0350] For example, the structural information of the combustion chamber indicates that the fuel injection nozzles in each of the first 3D data and the second 3D data are arranged at equal intervals on a circumference having an arbitrary radius of curvature. The structural information indicates that the fuel injection nozzles are the same size. In step S106, the parameter generation unit 53 generates conversion parameters based on the structural information.

[0351] If the engine model is identified, the total number of fuel injection nozzles, the size of each fuel injection nozzle, and the ID of each fuel injection nozzle are known. The structural information may include the total number of fuel injection nozzles, the size of each fuel injection nozzle, and the ID of each fuel injection nozzle. In this case, the parameter generation unit 53 can generate position and orientation transformation parameters for accurately connecting the first 3D data and the second 3D data to each other.

[0352] In step S106, the parameter generating unit 53 executes the process shown in Fig. 34. The process shown in Fig. 34 is also applied when the subject is a combustion chamber.

[0353] FIG. 37 shows an example of an image displayed on the display unit 71. The display control unit 56 displays an image IMG81 on the display unit 71. The image IMG81 includes an area R81. Images of a first 3D shape 3D81, a second 3D shape 3D82, and a third 3D shape 3D83 are displayed in the area R81. The first 3D shape 3D81 is represented by the first 3D data. The second 3D shape 3D82 is represented by the second 3D data. The third 3D shape 3D83 is represented by the reference data.

[0354] The first 3D shape 3D81 includes four fuel injection nozzles. The IDs of the four fuel injection nozzles are 2 to 5. The position of the first 3D data is set to a position that overlaps with the four fuel injection nozzles of the reference data that are assigned IDs 2 to 5.

[0355] The second 3D shape 3D82 includes four fuel injection nozzles. The IDs of the four fuel injection nozzles are 9 to 12. The position of the second 3D data is set to a position that overlaps with the four fuel injection nozzles of the reference data that are assigned IDs 9 to 12.

[0356] The fuel injection nozzles assigned IDs 1, 6, 7, and 8 do not exist in the first 3D data and the second 3D data. The fuel injection nozzles in the reference data assigned these IDs are displayed.

[0357] The display control unit 56 may display each blade on the display unit 71 so that the user can distinguish between the fuel injector nozzles included in the first 3D data or the second 3D data and the fuel injector nozzles included in the reference data. For example, the color of the fuel injector nozzles included in the first 3D data or the second 3D data may be different from the color of the fuel injector nozzles included in the reference data. The user can check the fuel injector nozzles included in the 3D data and the fuel injector nozzles not included in the 3D data.

[0358] In the second modification of the fourth embodiment, the parameter generating unit 53 generates the transformation parameters by using the ID as the structural information, so that the image display device 50 can accurately connect the first 3D data and the second 3D data.

[0359] The first and second modifications of the fourth embodiment are applied to the inspection of a gas turbine of an aircraft engine. These modifications may also be applied to the inspection of tubular structures. For example, these modifications may be applied to the inspection of heat exchanger tubes. The heat exchanger tubes have dozens of thin tubes. An ID may be assigned to each tube. The parameter generator 53 can connect the first 3D data and the second 3D data to each other by using the ID.

[0360] (Fifth embodiment) A fifth embodiment of the present invention will be described. In the fifth embodiment, the structural information is camera trajectory data. The camera trajectory data indicates two or more positions where a camera that generates two or more images used to generate the 3D data is located. The positions correspond to the camera coordinates described above. In other words, the camera trajectory data indicates the trajectory of the camera. The camera trajectory data is associated with each of the first 3D data and the second 3D data. When the endoscope device 1 shown in FIG. 22 is used, the camera trajectory data indicates two or more positions where the tip 20 of the insertion section 2 is located. By using the camera trajectory data, the first 3D data and the second 3D data are connected to each other.

[0361] The image display device 50 shown in Fig. 1 is changed to an image display device 50c shown in Fig. 38. Fig. 38 shows the configuration of the image display device 50c. Description of the same configuration as that shown in Fig. 1 will be omitted.

[0362] 38 includes a control unit 51, a data acquisition unit 52, a parameter generation unit 53, a conversion unit 54, a data generation unit 55, a display control unit 56, and a missing segment calculation unit 60. The image display device 50c does not include the information reception unit 57 and the structure estimation unit 58 shown in FIG.

[0363] Each unit of the image display device 50c may be configured with at least one of a processor and a logic circuit. Each unit of the image display device 50c may include one or more processors. Each unit of the image display device 50c may include one or more logic circuits.

[0364] The storage unit 73 stores first camera trajectory data and second camera trajectory data. The first camera trajectory data is associated with the first 3D data, and the second camera trajectory data is associated with the second 3D data. The data acquisition unit 52 is connected to the storage unit 73 and acquires the first camera trajectory data and the second camera trajectory data from the storage unit 73.

[0365] The missing section calculation unit 60 calculates the position of the missing section by using the first camera trajectory data and the second camera trajectory data. The missing section includes 3D data corresponding to a missing region of the subject. The missing region is not included in the first region or the second region of the subject. The first region corresponds to 3D coordinates included in the first 3D data. The second region corresponds to 3D coordinates included in the second 3D data. The missing section calculation unit 60 can calculate the length of the missing section based on the position of the missing section.

[0366] The processing executed by the image display device 50c will be described using Fig. 39. Fig. 39 shows the procedure of the processing executed by the image display device 50c. Description of the same processing as that shown in Fig. 2 will be omitted.

[0367] The data acquisition unit 52 connects to the storage unit 73 and acquires the first 3D data and the first camera trajectory data from the storage unit 73 (step S140). After step S140, the display control unit 56 displays an image of the first 3D data and the first camera trajectory data on the display unit 71 (step S141).

[0368] After step S141, the data acquisition unit 52 connects to the storage unit 73 and acquires the second 3D data and the second camera trajectory data from the storage unit 73 (step S142). After step S142, the display control unit 56 displays images of the second 3D data and the second camera trajectory data on the display unit 71 (step S143).

[0369] The order of steps S140 to S143 is not limited to the order shown in Fig. 39. For example, steps S142 and S143 may be performed first, followed by steps S140 and S141. Alternatively, steps S140 and S142 may be performed first, followed by steps S141 and S143. Steps S141 and S143 may be omitted.

[0370] After step S143, the parameter generation unit 53 generates transformation parameters based on the relationship between the first camera trajectory data and the second camera trajectory data (step S144). The parameter generation unit 53 generates position and attitude transformation parameters for connecting the first 3D data and the second 3D data to each other by using the following method. The following method can be applied to pipe inspection and blade (moving blade) inspection.

[0371] When the subject is a pipe, the parameter generation unit 53 executes the following process. The parameter generation unit 53 calculates a first line that approximates the camera trajectory indicated by the first camera trajectory data. The parameter generation unit 53 also calculates a second line that approximates the camera trajectory indicated by the second camera trajectory data. The parameter generation unit 53 generates position and orientation transformation parameters for correcting the positions and orientations of the first 3D data and the second 3D data so that the first line and the second line coincide with each other.

[0372] The parameter generation unit 53 may calculate the first straight line by using first camera trajectory data associated with the entire first 3D data.The parameter generation unit 53 may calculate the first straight line by using first camera trajectory data associated with a connection region identified based on a timestamp of the first 3D data.

[0373] The parameter generation unit 53 may calculate the second straight line by using second camera trajectory data associated with the entire second 3D data.The parameter generation unit 53 may calculate the second straight line by using second camera trajectory data associated with a connection region identified based on a timestamp of the second 3D data.

[0374] FIG. 40 shows an example of an image displayed on the display unit 71 before the first 3D data and the second 3D data are connected to each other. The display control unit 56 displays an image IMG91 on the display unit 71. The image IMG91 includes an area R91 and an area R92. An image of the first 3D shape 3D91 is displayed in the area R91. The first 3D shape 3D91 is represented by the first 3D data. An image of the second 3D shape 3D92 is displayed in the area R92. The second 3D shape 3D92 is represented by the second 3D data.

[0375] A first camera trajectory TR91 and a second camera trajectory TR92 are shown in FIG. 40. The first camera trajectory TR91 indicates two or more positions included in the first camera trajectory data. The second camera trajectory TR92 indicates two or more positions included in the second camera trajectory data. It is not necessary for the display control unit 56 to display the first camera trajectory TR91 and the second camera trajectory TR92 on the display unit 71.

[0376] FIG. 41 shows an example of an image displayed on the display unit 71 after the first 3D data and the second 3D data have been connected to each other. In FIG. 41, step S145, which will be described later, is not taken into consideration. Descriptions of parts that are the same as those shown in FIG. 40 will be omitted. The display control unit 56 displays an image IMG92 on the display unit 71. The image IMG92 includes an area R93. Images of the first 3D shape 3D91 and the second 3D shape 3D92 are displayed in the area R93.

[0377] The first 3D shape 3D91 and the second 3D shape 3D92 are arranged so that a first straight line approximating the first camera trajectory TR91 and a second straight line approximating the second camera trajectory TR92 coincide with each other. A straight line L91 is formed by the first straight line and the second straight line.

[0378] When the subject is a blade (moving surface), the parameter generation unit 53 executes the following process. The parameter generation unit 53 calculates a first curve that approximates the camera trajectory indicated by the first camera trajectory data. The parameter generation unit 53 also calculates a second curve that approximates the camera trajectory indicated by the second camera trajectory data. Each of the first curve and the second curve is an arc. The parameter generation unit 53 generates position and orientation transformation parameters for correcting the position and orientation of each of the first 3D data and the second 3D data so that the first curve and the second curve are connected to each other on the circumference of a circle with an arbitrary diameter.

[0379] FIG. 42 shows an example of an image displayed on the display unit 71 before the first 3D data and the second 3D data are connected to each other. The display control unit 56 displays an image IMG101 on the display unit 71. The image IMG101 includes an area R101 and an area R102. An image of the first 3D shape 3D101 is displayed in the area R101. The first 3D shape 3D101 is represented by the first 3D data. An image of the second 3D shape 3D102 is displayed in the area R102. The second 3D shape 3D102 is represented by the second 3D data.

[0380] A first camera trajectory TR101 and a second camera trajectory TR102 are shown in FIG. 42. The first camera trajectory TR101 indicates two or more positions included in the first camera trajectory data. The second camera trajectory TR102 indicates two or more positions included in the second camera trajectory data. It is not necessary for the display control unit 56 to display the first camera trajectory TR101 and the second camera trajectory TR102 on the display unit 71.

[0381] FIG. 43 shows an example of an image displayed on the display unit 71 after the first 3D data and the second 3D data have been connected to each other. In FIG. 43, step S145, which will be described later, is not taken into consideration. Descriptions of parts that are the same as those shown in FIG. 42 will be omitted. The display control unit 56 displays an image IMG102 on the display unit 71. The image IMG102 includes an area R103. Images of the first 3D shape 3D101 and the second 3D shape 3D102 are displayed in the area R103.

[0382] The first 3D shape 3D101 and the second 3D shape 3D102 are arranged so that a first curve approximating the first camera trajectory TR101 and a second curve approximating the second camera trajectory TR102 are arranged on the circumference of a circle. The circle CR101 is composed of the first curve and the second curve.

[0383] The parameter generating unit 53 can generate position and posture transformation parameters according to the object by using the above method. If the object is a pipe, the scale of each of the first 3D data and the second 3D data is not corrected. If the object is a blade (moving surface), the scale of each of the first 3D data and the second 3D data may be corrected.

[0384] After step S144, the missing section calculation unit 60 calculates the missing section in the connection region by using the first camera trajectory data and the second camera trajectory data (step S145). The missing section calculation unit 60 calculates the position of the missing section by using the following method. The following method can be applied to pipe inspection and blade (moving blade) inspection.

[0385] When the subject is a pipe, the missing section calculation section 60 executes the following process: Figure 44 shows a method for calculating missing sections.

[0386] Each of the two or more points P111, two or more points P112, and two or more points P113 indicates the position (3D coordinates) of the camera when the camera generated the image. The two or more points P111 constitute first camera trajectory data and are associated with the first 3D data 3D111. The two or more points P112 constitute second camera trajectory data and are associated with the second 3D data 3D112. The two or more points P113 indicate the position of the camera in an area where no 3D data exists.

[0387] Since there is no 3D data corresponding to two or more points P113, there is no camera trajectory data corresponding to two or more points P113. Two or more points P113 constitute a missing section.

[0388] The first 3D data 3D111 is generated by using two or more images FR111. The second 3D data 3D112 is generated by using two or more images FR112. The two or more images FR113 are not used to generate the first 3D data 3D111 and the second 3D data 3D112. The two or more images FR111, the two or more images FR112, and the two or more images FR113 are included in one video file. Each of the two or more images FR111, the two or more images FR112, and the two or more images FR113 has a frame number.

[0389] Each of the two or more points P111 is associated with the image FR111. Each of the two or more points P112 is associated with the image FR112. Each of the two or more points P113 is not associated with the image FR113.

[0390] Not all of the two or more images FR111 are necessarily used to generate the first 3D data 3D111. Two or more key frames included in the two or more images FR111 are used to generate the first 3D data 3D111. The number of key frames is equal to or less than the total number of the two or more images FR111. Similarly, two or more key frames included in the two or more images FR112 are used to generate the second 3D data 3D112.

[0391] There is a correlation between the number of points P111 and the number of images FR111 (key frames). There is a correlation between the number of points P112 and the number of images FR112 (key frames). The missing section calculation unit 60 calculates the number of points P111 and points P112 per image. The missing section calculation unit 60 calculates the number of points P113 by using the number of points P111 and points P112 per image and the number of images FR113.

[0392] The missing section calculation unit 60 calculates the interval (three-dimensional distance) between points P111 or P112. The missing section calculation unit 60 calculates the length LT111 of the missing section by using the interval and the number of points P113. If the video data used to generate the first 3D data is different from the video data used to generate the second 3D data, the above method is not used.

[0393] When the subject is a blade (moving surface), the missing section calculation unit 60 executes the following process: Figures 45 and 46 show a method for calculating a missing section.

[0394] FIG. 45 shows the relationship between camera trajectory data and 3D data. Two or more points P121, two or more points P122, two or more points P123, and two or more points P124 indicate the position (3D coordinates) of the camera when the camera generated an image. The two or more points P121 constitute first camera trajectory data and are associated with the first 3D data 3D121. The two or more points P122 constitute second camera trajectory data and are associated with the second 3D data 3D122. The two or more points P123 and the two or more points P124 indicate the position of the camera in an area where no 3D data exists.

[0395] Since there is no 3D data corresponding to two or more points P123 and no 3D data corresponding to two or more points P124, there is no camera trajectory data corresponding to two or more points P123 and no camera trajectory data corresponding to two or more points P124. Two or more points P123 constitute a first missing section. Two or more points P124 constitute a second missing section.

[0396] FIG. 46 shows a video file. First 3D data 3D121 is generated by using two or more images FR121. Second 3D data 3D122 is generated by using two or more images FR122. Two or more images FR123 and two or more images FR124 were not used to generate the first 3D data 3D121 and the second 3D data 3D122. Two or more images FR121, two or more images FR122, two or more images FR123, and two or more images FR124 are included in one video file. Each of the two or more images FR121, two or more images FR122, two or more images FR123, and two or more images FR124 has a frame number.

[0397] Each of the two or more points P121 is associated with the image FR121. Each of the two or more points P122 is associated with the image FR122. Each of the two or more points P123 is not associated with the image FR123. Each of the two or more points P124 is not associated with the image FR124.

[0398] Not all of the two or more images FR121 are necessarily used to generate the first 3D data 3D121. Two or more key frames included in the two or more images FR121 are used to generate the first 3D data 3D121. The number of key frames is equal to or less than the total number of the two or more images FR121. Similarly, two or more key frames included in the two or more images FR122 are used to generate the second 3D data 3D122.

[0399] There is a correlation between the number of points P121 and the number of images FR121 (key frames). There is a correlation between the number of points P122 and the number of images FR122 (key frames). The missing section calculation unit 60 calculates the number of points P121 and points P122 per image. The missing section calculation unit 60 calculates the number of points P123 by using the number of points P121 and points P122 per image and the number of images FR123.

[0400] The missing section calculation unit 60 calculates the interval (angle) between point P121 or point P122. The missing section calculation unit 60 calculates the angle AG121 of the first missing section by using that interval and the number of points P113. The missing section calculation unit 60 may calculate the interval (angle) of the blades by using the interval between points P121 and the number of blades in the 3D data 3D121. The missing section calculation unit 60 may calculate the number of blades in the first missing section by using that interval and the angle AG121 of the first missing section.

[0401] The missing section calculation unit 60 calculates the angle AG122 of the second missing section by subtracting the first angle, the second angle, and the angle AG121 from the angle of one revolution of the blade (360 degrees). The first angle corresponds to the range in which the blades are arranged in the 3D data 3D121. The second angle corresponds to the range in which the blades are arranged in the 3D data 3D122. The missing section calculation unit 60 may calculate the number of blades in the second missing section by subtracting the number of blades in the 3D data 3D121, the number of blades in the 3D data 3D122, and the number of blades in the first missing section from the total number of blades. If the video data used to generate the first 3D data is different from the video data used to generate the second 3D data, the above method is not used.

[0402] After step S145, in step S106, the parameter generating unit 53 generates transformation parameters based on the missing section. At this time, the parameter generating unit 53 generates transformation parameters for separating the first 3D data and the second 3D data by the length or angle of the missing section.

[0403] Each aspect of the present invention may include the following modifications: The structural information indicates two or more positions at which the tip 20 of the insertion portion 2 to be inserted into the object having the subject is positioned.

[0404] Each aspect of the present invention may include the following modifications. The structural information includes first position information (first camera trajectory data) and second position information (second camera trajectory data). The first position information indicates two or more positions where the tip 20 is placed to acquire two or more first images. The second position information indicates two or more positions where the tip 20 is placed to acquire two or more second images. In a generation step (step S144), the parameter generation unit 53 generates position transformation parameters and orientation transformation parameters for transforming the first 3D coordinate system and the second 3D coordinate system into a common coordinate system based on the first position information and the second position information. In a transformation step (step S107), the transformation unit 54 transforms the first 3D coordinate system and the second 3D coordinate system into the common coordinate system by using the position transformation parameters and orientation transformation parameters.

[0405] Each aspect of the present invention may include the following variations: the first 3D data is generated by using two or more first images; the second 3D data is generated by using two or more second images; and the two or more first images and the two or more second images form one video file.

[0406] Each aspect of the present invention may include the following modifications. In a calculation step (step S145), the missing section calculation unit 60 calculates the position of the missing area based on the number of two or more first images, the number of two or more second images, and the number of third images. The third image is arranged temporally between two or more first images and two or more second images in the video file. The missing area is a region of the subject that is different from both the first region of the subject and the second region of the subject. The first region corresponds to 3D coordinates included in the first 3D data. The second region corresponds to 3D coordinates included in the second 3D data. In a conversion step (step S107), the conversion unit 54 converts the first 3D coordinate system and the second 3D coordinate system into a common coordinate system based on the position of the missing area.

[0407] In the fifth embodiment, the image display device 50c can connect the first 3D data and the second 3D data to each other and can display a wide range of 3D shape images. Furthermore, the image display device 50c can display a 3D shape image close to the original shape of the subject by taking missing sections into consideration.

[0408] (Modification of the fifth embodiment) A modified example of the fifth embodiment of the present invention will be described. In the above-described fifth embodiment, camera trajectory data generated through image processing is used. On the other hand, in the modified example of the fifth embodiment, the camera position is calculated using sensor data, and data indicating the position is used instead of the camera trajectory data. In the modified example of the fifth embodiment, an image display device 50c shown in FIG. 38 is used.

[0409] For example, an IMU or an insertion length sensor is used. The IMU is disposed at the tip 20 of the insertion portion 2. The insertion length sensor is disposed at the base end of the insertion portion 2 shown in FIG. 22 or on a drum that houses the insertion portion 2. The insertion length sensor detects the length (insertion length) of the portion of the insertion portion 2 that is inserted into the test subject.

[0410] The processing executed by the image display device 50c will be described using Figure 47. Figure 47 shows the procedure of the processing executed by the image display device 50c. Description of the same processing as that shown in Figure 2 will be omitted.

[0411] After step S103, the data acquisition unit 52 connects to the storage unit 73 and acquires sensor data in the connection region from the storage unit 73 (step S120). For example, the data acquisition unit 52 acquires sensor data in the section from the time of the end of the first 3D data to the time of the start of the second 3D data from the storage unit 73. Step S120 is the same as step S120 shown in FIG.

[0412] After step S120, the parameter generating unit 53 generates conversion parameters based on the sensor data acquired in step S120 (step S150).

[0413] For example, the parameter generation unit 53 calculates the trajectory of the tip 20 of the insertion unit 2 by using sensor data from the IMU. The trajectory indicates two or more positions where the tip 20 is located. The parameter generation unit 53 determines the movement of the tip 20 based on the trajectory. Specifically, the parameter generation unit 53 determines whether the tip 20 is moving straight or whether the tip 20 will turn 90 degrees. Furthermore, the missing section calculation unit 60 calculates the length of the missing section by using the method in the fifth embodiment.

[0414] When sensor data from the insertion length sensor is used, the parameter generation unit 53 cannot determine whether the tip 20 is moving straight or whether the tip 20 will bend 90 degrees. The missing section calculation unit 60 can calculate the length of the missing section by using the sensor data from the insertion length sensor. When history information on operations related to bending of the insertion unit 2 in the second modified example of the third embodiment is used, the parameter generation unit 53 can determine whether the tip 20 is moving straight or whether the tip 20 will bend 90 degrees based on the history information. The missing section calculation unit 60 can calculate the shape of the missing section based on the history information.

[0415] When inspecting a gas turbine stator blade, the tip of the scope moves around the circumference of the stator blade disk. Even in this case, the parameter generator 53 can calculate the length of the missing section by using the sensor data from the insertion length sensor.

[0416] Fig. 48 shows an example of an image displayed on the display unit 71. The display control unit 56 displays an image IMG131 on the display unit 71. The image IMG131 includes an area R131. Images of a first 3D shape 3D131 and a second 3D shape 3D132 are displayed in the area R131. The first 3D shape 3D131 is represented by first 3D data. The second 3D shape 3D132 is represented by second 3D data.

[0417] 48, the parameter generation unit 53 determines that the tip 20 will bend 90 degrees. Therefore, the parameter generation unit 53 generates position and orientation transformation parameters for making the central axis of the second 3D shape 3D132 perpendicular to the central axis of the first 3D shape 3D131. The first 3D shape 3D131 and the second 3D shape 3D132 are positioned so that the angle AG131 between them is 90 degrees.

[0418] In the second modification of the third embodiment described above, the parameter generating unit 53 generates the transformation parameters by using structural information. On the other hand, in the modification of the fifth embodiment, the parameter generating unit 53 generates the transformation parameters by using sensor data without using structural information.

[0419] Each aspect of the present invention may include the following modifications: In a calculation step (step S150), the missing section calculation unit 60 calculates the shape of the missing area. In a conversion step (step S107), the conversion unit 54 converts the first 3D coordinate system and the second 3D coordinate system into a common coordinate system based on the shape of the missing area.

[0420] In a modification of the fifth embodiment, the image display device 50c can connect the first 3D data and the second 3D data to each other and display a wide range of 3D shaped images. The image display device 50c can also display 3D shaped images by taking missing sections into consideration.

[0421] (Sixth embodiment) A sixth embodiment of the present invention will now be described. Industrial endoscope devices are used to inspect the interiors of boilers, gas turbines, automobile engines, pipes, etc. for abnormalities (such as scratches and corrosion). In order to enable verification of the content of the inspection performed by the inspector, video is generally recorded during the inspection.

[0422] There is a known technique for reconstructing the 3D shape of an object by using a moving image. The moving image is generated based on an optical image acquired through a monocular optical system. Using this technique, an endoscope system can acquire 3D data of a wide range of objects.

[0423] The 3D data of the inspection target may be used to allow the user to understand which areas have been inspected and which areas have not been inspected. 3D data is generated for the inspected areas, and 3D data is not generated for the uninspected areas. If the inspection target has a unique shape, the user can understand which areas of the 3D data correspond to which areas of the actual inspection target. The user can understand which areas of the inspection target have been inspected by checking the 3D shape shown by the 3D data.

[0424] However, there are few unique structures among typical inspection targets in endoscopic inspections, such as heat exchanger tubes or gas turbines. This makes it difficult for users to understand which regions of the 3D data correspond to which regions of the inspection target. Even when users visually compare the 3D data with reference data of the inspection target, it is difficult for users to understand which regions of the 3D data correspond to which regions of the reference data.

[0425] The sixth embodiment provides a method for allowing a user to identify inspected and uninspected areas even when the object does not have a unique shape. For example, the object may have a structure with a continuous identical shape (e.g., a pipe), or may have multiple structures with the same shape (e.g., blades).

[0426] The sixth embodiment is almost the same as the fourth embodiment. However, in the sixth embodiment, the 3D data is not divided into two. In the following, an example in which the subject is a blade will be described. The 3D shape shown by the 3D data is close to a ring.

[0427] In the sixth embodiment, the image display device 50 shown in Fig. 1 is used. The processing executed by the image display device 50 will be described with reference to Fig. 49. Fig. 49 shows the procedure of the processing executed by the image display device 50.

[0428] The data acquisition unit 52 connects to the storage unit 73 and acquires 3D data from the storage unit 73 (step S200). After step S200, the display control unit 56 displays an image of the 3D data on the display unit 71 (step S201). In the first to fifth embodiments described above, two types of 3D data are acquired from the storage unit 73, and an image of each of the 3D data is displayed on the display unit 71. On the other hand, in the sixth embodiment, one type of 3D data is acquired from the storage unit 73, and an image of the 3D data is displayed on the display unit 71.

[0429] After step S201, the information receiving unit 57 receives reference data (step S202). For example, the communication unit 72 receives the reference data from an external device. The information receiving unit 57 receives the reference data received by the communication unit 72. The reference data includes structural information. An ID is assigned to each of two or more blades having the same shape. The reference data includes the ID of each blade.

[0430] The reference data may be data generated using 3D-CAD, or data representing a two-dimensional shape on any plane, etc. The format of the reference data is not limited to the above examples, as long as the reference data includes structural information and an ID is associated with the reference data.

[0431] An ID does not need to be assigned to a blade before the information accepting unit 57 accepts the reference data. An ID may be assigned to a blade by a user after the information accepting unit 57 accepts the reference data. Alternatively, an ID may be assigned to a blade automatically.

[0432] After step S202, the display control unit 56 displays the image of the reference data received in step S201 on the display unit 71 (step S203).

[0433] The order of steps S200 to S203 is not limited to the order shown in Fig. 49. For example, steps S202 and S203 may be performed first, followed by steps S200 and S201. Alternatively, steps S200 and S202 may be performed first, followed by steps S201 and S203. Steps S201 and S203 may be omitted.

[0434] After step S203, the parameter generating unit 53 generates conversion parameters for converting the 3D coordinate system of the 3D data and the 3D coordinate system of the reference data into a common coordinate system (step S204).

[0435] An example will be described below in which the 3D coordinate system of the reference data is used as the reference coordinate system and as the common coordinate system. In the following example, the parameter generation unit 53 generates position and orientation transformation parameters for matching the position and orientation of the 3D coordinate system of the 3D data with the position and orientation of the 3D coordinate system of the reference data. The parameter generation unit 53 also generates scale transformation parameters for matching the scale of the 3D coordinate system of the 3D data with the scale of the 3D coordinate system of the reference data. However, the method of generating transformation parameters is not limited to the following example. The 3D coordinate system of the 3D data may be used as the common coordinate system. A 3D coordinate system different from both the 3D coordinate system of the 3D data and the 3D coordinate system of the reference data may be used as the common coordinate system.

[0436] In step S204, the parameter generating unit 53 executes the process shown in Fig. 50. Fig. 50 shows the procedure of the process executed by the parameter generating unit 53.

[0437] The parameter generating unit 53 identifies blade regions in the 3D data (step S204a). Each blade region includes one blade.

[0438] After step S204a, the parameter generating unit 53 assigns an ID to the blade region identified in step S204a (step S204b).

[0439] After step S204b, the parameter generating unit 53 calculates transformation parameters for correcting the position, orientation, and scale of the 3D data so that the ID of the blade in the reference data and the ID of the blade region in the 3D data match each other (step S204c). When step S204c is executed, the process shown in Fig. 50 ends.

[0440] FIG. 51 shows an example of an image displayed on the display unit 71. The display control unit 56 displays an image IMG141 on the display unit 71. The image IMG141 includes an area R141 and an area R142. An image of a first 3D shape 3D141 is displayed in the area R141. The first 3D shape 3D141 is represented by 3D data. An image of a second 3D shape 3D142 is displayed in the area R142. The second 3D shape 3D142 is represented by reference data.

[0441] The first 3D shape 3D141 includes 11 blades, with IDs 1 to 4 and 6 to 12. The blade with ID 5 is missing and is not included in the 3D data.

[0442] The second 3D shape 3D142 includes 12 blades, with IDs 1 through 12.

[0443] After step S204, the conversion unit 54 converts the 3D coordinate system of the 3D data and the 3D coordinate system of the reference data into a common coordinate system by using the conversion parameters generated in step S204. That is, the conversion unit 54 converts each of the 3D data and the reference data into 3D data in the common coordinate system (step S205). When the 3D coordinate system of the reference data is used as the common coordinate system, the position, orientation, and scale of the 3D coordinate system of the reference data are not changed, but the position, orientation, and scale of the 3D coordinate system of the 3D data are changed.

[0444] After step S205, the display control unit 56 displays an image of the 3D shape of the 3D data and the reference data on the display unit 71. At this time, the display control unit 56 controls the state of the image so that the user can distinguish between an area that corresponds to the 3D data and an area that does not correspond to the 3D data (step S206). When step S206 is executed, the processing shown in FIG. 49 ends.

[0445] Fig. 52 shows an example of an image displayed on the display unit 71. Description of parts that are the same as those shown in Fig. 51 will be omitted. The display control unit 56 displays an image IMG142 on the display unit 71. The image IMG142 includes an area R143. Images of the 3D shape 3D141 and the 3D shape 3D142 are displayed in the area R143.

[0446] For example, the display control unit 56 displays blades with IDs of 1 to 4 and 6 to 12 in a first color, and displays a blade with ID of 5 in a second color different from the first color. The blade with ID of 5 is not included in the 3D data. The user can confirm the blades included in the reference data and the blades not included in the reference data. In other words, the user can confirm the blades in the reference data that correspond to the 3D data and the blades in the reference data that do not correspond to the 3D data.

[0447] The display control unit 56 may execute processing to highlight areas that correspond to the 3D data or areas that do not correspond to the 3D data. The user may be notified of the respective positions of the areas that correspond to the 3D data and the areas that do not correspond to the 3D data by voice. As long as the user can distinguish between areas that correspond to the 3D data and areas that do not correspond to the 3D data, the method of notifying the user of the areas of the 3D data is not limited to the above method.

[0448] The method of the sixth embodiment may be applied to the inspection of a combustion chamber of a gas turbine or the inspection of a heat exchanger of a gas turbine. Fig. 53 shows the structure of a heat exchanger HE150. The heat exchanger HE150 has two or more heat exchange tubes HT150. Generally, an ID is assigned to each of the two or more heat exchange tubes in the reference data.

[0449] An ID may be assigned to the 3D data of each heat exchange tube. The image display device 50 may execute processing similar to the processing shown in Figures 49 and 50. In this way, the image display device 50 may match the position, orientation, and scale of the 3D data of each heat exchange tube with the position, orientation, and scale of the heat exchange tube in the reference data.

[0450] In the sixth embodiment, the image display device 50 uses the 3D data and the reference data to allow the user to understand the inspected area and the uninspected area. Compared to when only the 3D data is used, the user can easily distinguish the inspected area from the uninspected area. This improves the quality of the inspection report created by the inspector.

[0451] The examination is approved by the manager of the examination department or a member of the requesting organization. Generally, the person approving the examination is not very knowledgeable about endoscopic examinations. By reviewing the data showing the areas that were examined and the areas that were not, the approver can objectively determine whether the examination was performed correctly. This ensures the reliability of the examination.

[0452] Although the preferred embodiments of the present invention have been described above, the present invention is not limited to these embodiments and their modifications. Addition, omission, substitution, and other modifications of the configuration are possible without departing from the spirit of the present invention. Furthermore, the present invention is not limited by the above description, but is limited only by the scope of the appended claims. [Explanation of symbols]

[0453] 1 Endoscopic device 2 Insertion section 3 Main body 4,70 Control unit 5,71 Display section 8 Endoscope Unit 9 CCU 10 Control device 12 Video signal processing circuit 13 ROM 14 RAM 15 Card Interface 16 External device interface 17 Control Interface 18 CPU 20 Tip 28 Image sensor 50, 50a, 50b, 50c Image display device 51,180 Control unit 52,181 Data Acquisition Section 53,182 Parameter Generation Unit 54,183 conversion unit 55,184,188 Data Generation Unit 56,185 Display control unit 57,186 Information Reception Department 58,187 Structure estimation part 59 Adjustment section 60 Missing section calculation unit 72 Communications Department 73 Memory section 74 sensors 1880 Image Acquisition Unit 1881 Conditions Reception Department 1882 Data Calculation Department

Claims

1. a first acquisition step in which a processor acquires three-dimensional data of a subject including one or more structures, the three-dimensional data including information indicating three-dimensional coordinates of at least a portion of the one or more structures; a second acquiring step in which the processor acquires reference data including information indicative of the one or more structures; an associating step in which the processor associates the information included in the three-dimensional data with the information included in the reference data based on information of a common structure, the common structure indicating a structure of the subject that is common to at least a portion of the information included in the three-dimensional data and at least a portion of the information included in the reference data; a display step in which the processor displays at least a portion of the three-dimensional data and at least a portion of the reference data on a display; A three-dimensional image display method comprising:

2. The information included in the reference data indicates the area of ​​the one or more structures. The three-dimensional image display method according to claim 1 .

3. The reference data is a known three-dimensional model.

3. The three-dimensional image display method according to claim 2.

4. The reference data is data that indicates a two-dimensional shape on a plane.

3. The three-dimensional image display method according to claim 2.

5. The reference data is received from an external device. The three-dimensional image display method according to claim 1 .

6. The subject has two or more structures including the one or more structures. The three-dimensional image display method according to claim 1 .

7. The reference data includes identification information corresponding to each of the two or more structures.

7. The three-dimensional image display method according to claim 6.

8. The information of the common structure indicates the identification information of the structure that is common to at least a portion of the information included in the three-dimensional data and at least a portion of the information included in the reference data. The three-dimensional image display method according to claim 7.

9. The object is a turbine, and the two or more structures are blades.

9. The three-dimensional image display method according to claim 8.

10. The object has one structure, and the structure has a continuous structure with the same shape. The three-dimensional image display method according to claim 1 .

11. the three-dimensional coordinates are defined in a first three-dimensional coordinate system; The reference data includes two-dimensional coordinates or three-dimensional coordinates defined in a second coordinate system different from the first three-dimensional coordinate system. The three-dimensional image display method according to claim 1 .

12. The processor converts the first three-dimensional coordinate system and the second coordinate system into a common coordinate system in the associating step. The three-dimensional image display method according to claim 11.

13. In the display step, the processor displays the three-dimensional data and the reference data in the common coordinate system on the display. The three-dimensional image display method according to claim 12.

14. The method further includes an input step in which, after the second acquisition step, the processor receives input of the identification information corresponding to each of the two or more structures. The three-dimensional image display method according to claim 7.

15. the reference data includes identification information corresponding to each of the one or more structures; The associating step includes: a step of identifying by the processor an area of ​​the one or more structures indicated by the information included in the three-dimensional data; an assigning step in which the processor assigns identification information to the area identified in the identifying step; Including, In the associating step, the processor identifies the common structure corresponding to the identification information of the region and the identification information included in the reference data; The identification information of the region and the identification information included in the reference data match each other. The three-dimensional image display method according to claim 1 .

16. In the associating step, the processor corrects at least one of the position, the orientation, and the scale of the three-dimensional data based on the information about the common structure. The three-dimensional image display method according to claim 15.

17. In the display step, the processor displays the information included in the three-dimensional data on the display, and in the display step, displays information on a structure that corresponds to the information included in the reference data but does not correspond to the information included in the three-dimensional data on the display. The three-dimensional image display method according to claim 1 .

18. In the display step, the processor displays an image of the reference data on the display. The three-dimensional image display method according to claim 1 .

19. In the display step, the processor displays the identification information corresponding to each of the two or more structures on the display. The three-dimensional image display method according to claim 7.

20. a processor, the processor comprising: acquiring three-dimensional data of a subject including one or more structures, the three-dimensional data including information indicating three-dimensional coordinates of at least a portion of the one or more structures; acquiring reference data including information indicative of the one or more structures; associate the information included in the three-dimensional data with the information included in the reference data based on information of a common structure, the common structure indicating a structure of the subject that is common to at least a portion of the information included in the three-dimensional data and at least a portion of the information included in the reference data; At least a part of the three-dimensional data and at least a part of the reference data are displayed on a display. Three-dimensional image display device.

21. a first acquisition step of acquiring three-dimensional data of a subject including one or more structures, the three-dimensional data including information indicating three-dimensional coordinates of at least a portion of the one or more structures; a second acquisition step of acquiring reference data including information indicative of the one or more structures; an associating step of associating the information included in the three-dimensional data with the information included in the reference data based on information of a common structure, the common structure indicating a structure of the subject that is common to at least a part of the information included in the three-dimensional data and at least a part of the information included in the reference data; a display step of displaying at least a portion of the three-dimensional data and at least a portion of the reference data on a display; A program that causes a computer to execute the following.

Images