3D data generation method, 3D data generation system, and program

Abstract

To improve reliability of processing of generating three-dimensional data indicating a three-dimensional shape of an interior of a turbine.SOLUTION: A three-dimensional data generation method includes the following steps. A processor acquires two or more images of a component of an interior of a turbine in an image acquisition step. The processor detects two or more corresponding areas being the same area in at least two images in an area detection step. The processor determines whether at least a partial area in each image is a change area or a non-change area in an area determination step. The processor generates three-dimensional data by using the corresponding area determined to be the change area without using the corresponding area determined to be the non-change area in a data generation step.SELECTED DRAWING: Figure 7

Description

[Technical Field]

[0001] The present invention relates to a three-dimensional data generation method, a three-dimensional data generation system, and a program for generating three-dimensional data that indicates the three-dimensional shape of the interior of a turbine. [Background technology]

[0002] Industrial endoscopy systems are used to inspect the interior of industrial equipment, such as boilers, turbines, engines, and pipes, for defects (such as scratches and corrosion). A variety of subjects are targets for inspection using industrial endoscopy systems. Turbines used in aircraft and power generation facilities are particularly important targets for inspection using industrial endoscopy systems.

[0003] Generally, industrial endoscopy systems use monocular optical adapters and measurement-only optical adapters. Monocular optical adapters are used for normal observation of the object. Measurement-only optical adapters are used to reconstruct three-dimensional (3D) information about the object. For example, a measurement-only optical adapter is a stereo optical adapter with two fields of view. Industrial endoscopy systems can measure the size of any abnormalities detected by using the 3D information. Users can also confirm the reconstructed shape of the object (such as unevenness). In this way, 3D information contributes to higher quality and more efficient inspections.

[0004] In recent years, a technology has been developed that uses a monocular optical adapter to acquire images of a subject and then uses those images to reconstruct 3D information about the subject. Such a technology performs 3D reconstruction processing based on changes in the relative movement of the tip of the endoscope with respect to the subject, and reconstructs 3D information.

[0005] Turbines are used in aircraft engines or generators. Turbine blades are a primary subject of inspection using industrial endoscopy equipment. In each of the compression and turbine sections, multiple stages are arranged along the rotational axis within the turbine, each consisting of rotatable rotor blades and fixed stator vanes. Shrouds are located outside the rotor blades.

[0006] Typically, rotor blade inspections involve searching for abnormalities on the rotor blade while it is rotating. In such inspections, the stator blades or shrouds are often in the endoscope's field of view in addition to the rotor blades. Unlike the rotor blades, the stator blades and shrouds do not move. If the stator blades or shrouds are captured together with the rotor blades in an image acquired using a monocular optical adapter, industrial endoscope devices may fail to reconstruct 3D information of the subject.

[0007] Patent Document 1 discloses a technology for restoring 3D information of a subject by using images of moving and stationary objects. The technology uses images acquired by a camera mounted on the vehicle to restore 3D information of the subject around the vehicle while it is moving. [Prior art documents] [Patent documents]

[0008] [Patent Document 1] Japanese Patent Publication No. 2020-126432 Summary of the Invention [Problem to be solved by the invention]

[0009] A monocular camera is used in the technology disclosed in Patent Document 1. The technology assumes that the vehicle on which the camera is mounted is moving and that the subject being subjected to the 3D reconstruction process is stationary. The technology avoids failure in the reconstruction of 3D information by performing the 3D reconstruction process without using the area of ​​a moving object captured in the image. The moving object is, for example, a tree branch or leaf.

[0010] On the other hand, when inspecting rotor blades using an industrial endoscope, the tip of the endoscope is essentially stationary, while the subject of the 3D reconstruction process moves. Therefore, the subject of the 3D reconstruction process using images of the rotor blade differs from the subject of the 3D reconstruction process using images acquired by a camera mounted on a vehicle in terms of subject movement. If the technology disclosed in Patent Document 1 were applied to rotor blade inspection, the area containing the rotor blade would not be used in the 3D reconstruction process. Therefore, the technology cannot be applied to rotor blade inspection.

[0011] The present invention aims to provide a three-dimensional data generation method, a three-dimensional data generation system, and a program that can improve the reliability of the process of generating three-dimensional data that shows the three-dimensional shape of the interior of a turbine. [Means for solving the problem]

[0012] The present invention provides a three-dimensional data generation method for generating three-dimensional data showing a three-dimensional shape inside a turbine, the method comprising: an image acquisition step in which a processor acquires two or more images of components inside the turbine, the components including a first object that is movable inside the turbine and a second object that is stationary inside the turbine; the two or more images are generated by an imaging device having a tubular insertion part that captures light inside the turbine; the insertion part is inserted into the turbine through a hole formed in the turbine; a moving direction of the insertion part when inserted into the turbine is different from a moving direction of the first object; and while the first object is moving, a position of the first object relative to the insertion part is different at each timing when the imaging device generates an image; and a region detection step of detecting two or more corresponding regions that are identical regions of the constituent element in at least two images included in two or more images; a region determination step in which the processor determines whether at least a portion of a region in each of the two or more images is a changed region or a non-changed region, wherein the changed region is a region of the constituent element whose coordinates change in the image generated by the imaging device, and the non-changed region is a region of the constituent element whose coordinates do not change in the image generated by the imaging device; and a data generation step in which the processor generates the three-dimensional data by using the corresponding region of the two or more corresponding regions that is determined to be the changed region, without using the corresponding region of the two or more corresponding regions that is determined to be the non-changed region. The first object rotates inside the turbine by a driving force generated by a driving device, and the three-dimensional data generation method further includes a rotation determination step in which the processor determines whether the first object has rotated once inside the turbine, and a notification step in which the processor executes a process to notify a user of information indicating that the first object has rotated once inside the turbine when the processor determines that the first object has rotated once inside the turbine. A method for generating 3D data. The present invention provides a three-dimensional data generation method for generating three-dimensional data showing a three-dimensional shape inside a turbine, the method including: an image acquisition step in which a processor acquires two or more images of components inside the turbine, the components including a first object movable inside the turbine and a second object stationary inside the turbine; the two or more images are generated by an imaging device having a tubular insertion part that captures light inside the turbine; the insertion part is inserted into the turbine through a hole formed in the turbine; a moving direction of the insertion part when inserted into the turbine differs from a moving direction of the first object; and while the first object is moving, a position of the first object relative to the insertion part varies each time the imaging device generates an image; an area detection step in which the processor detects two or more corresponding areas that are identical areas of the components in at least two images included in the two or more images; and an area detection step in which the processor detects at least a part of the area in each of the two or more images. a region determination step of determining whether a corresponding region of the two or more corresponding regions is a changing region or a non-changing region, where the changing region is a region of the component whose coordinates change in an image generated by the imaging device, and the non-changing region is a region of the component whose coordinates do not change in the image generated by the imaging device; and a data generation step of the processor generating the three-dimensional data by using the corresponding region of the two or more corresponding regions that is determined to be the changing region, without using the corresponding region that is determined to be the non-changing region, wherein the first object rotates inside the turbine by a driving force generated by a driving device, and while the first object is rotating, the processor repeatedly executes the region detection step, the region determination step, and the data generation step, and when the rotation of the first object stops, the processor stops the data generation step and continues the region detection step and the region determination step.

[0013] In the three-dimensional data generation method of the present invention, after the processor detects the two or more corresponding regions in the region detection step, the processor determines whether at least some of the regions are the changed region or the non-changed region in the region determination step.

[0014] In the three-dimensional data generation method of the present invention, after the processor determines in the area determination step whether at least a portion of the area is the changed area or the non-changed area, the processor detects the two or more corresponding areas by using the changed area without using the non-changed area in the area detection step.

[0015] In the three-dimensional data generation method of the present invention, the first object includes a rotor blade, and the second object includes a stator blade or a shroud.

[0016] In the three-dimensional data generating method of the present invention, a part of the first object is hidden by the second object in the two or more images.

[0017] In the three-dimensional data generation method of the present invention, the second object includes an object that hides part of the first object in the two or more images, and an object whose part is hidden by the first object in the two or more images.

[0018] In the three-dimensional data generation method of the present invention, the imaging device generates the two or more images at two or more different times, and in the area determination step, the processor determines whether at least a portion of the area is a changed area or a non-changed area based on the amount of movement of the corresponding area between at least two images included in the two or more images.

[0019] In the three-dimensional data generation method of the present invention, in the area determination step, the processor determines whether at least a portion of the area is a changed area or a non-changed area based on the difference in pixel values ​​between at least two images contained in the two or more images.

[0020] In the three-dimensional data generation method of the present invention, in the area determination step, the processor determines whether at least a portion of the area is a changed area or a non-changed area by determining a subject appearing in one of the two or more images.

[0021] The three-dimensional data generation method of the present invention further includes a position determination step in which the processor determines the position of the insertion part inside the turbine, and a notification step in which, when the position of the insertion part changes, the processor executes a process to notify a user of information indicating the change in position.

[0022] The three-dimensional data generation method of the present invention further includes a calculation step of calculating the area of ​​the non-changing region in an image included in the two or more images before the processor executes the data generation step for the first time, and a notification step of executing a process by the processor to notify a user of a warning when the area is larger than a predetermined value.

[0025] In the three-dimensional data generating method of the present invention, when the first object starts to rotate again, the processor restarts the data generating step.

[0026] In the three-dimensional data generation method of the present invention, the processor determines the rotation state of the first object by using images included in the two or more images.

[0027] In the three-dimensional data generating method of the present invention, the processor determines the rotational state of the first object by monitoring the state of the driving device.

[0028] In the three-dimensional data generation method of the present invention, the insertion part is fixed inside the turbine.

[0029] In the three-dimensional data generating method of the present invention, the imaging device is a borescope.

[0030] The present invention provides a three-dimensional data generation system for generating three-dimensional data showing the three-dimensional shape inside a turbine, the three-dimensional data generation system comprising: an imaging device having a tubular insertion part that captures light inside the turbine, and generating two or more images of components inside the turbine, the components including a first object that is movable inside the turbine and a second object that is stationary inside the turbine, the two or more images being generated by the imaging device, the insertion part being inserted into the turbine through a hole formed in the turbine, the direction of movement of the insertion part when inserted into the turbine being different from the direction of movement of the first object, and the position of the first object relative to the insertion part while the first object is moving being different at each timing when the imaging device generates an image; and a processor. and a three-dimensional data generating device including: the processor acquires the two or more images, detects two or more corresponding regions that are identical regions of the component in at least two images included in the two or more images, and determines whether at least a portion of a region in each of the two or more images is a changed region or a non-changed region, the changed region being a region of the component whose coordinates change in the image generated by the imaging device, and the non-changed region being a region of the component whose coordinates do not change in the image generated by the imaging device, and generates the three-dimensional data by using the corresponding region determined to be the changed region out of the two or more corresponding regions, without using the corresponding region determined to be the non-changed region out of the two or more corresponding regions. The first object rotates inside the turbine by a driving force generated by a driving device, and the processor determines whether the first object has rotated once inside the turbine, and when it determines that the first object has rotated once inside the turbine, executes a process of notifying a user of information indicating that the first object has rotated once inside the turbine. It is a 3D data generation system. The present invention provides a 3D data generation system for generating 3D data showing the 3D shape of the interior of a turbine, the 3D data generation system comprising: an imaging device having a tubular insertion part that captures light from inside the turbine, and generating two or more images of components inside the turbine, the components including a first object that is movable inside the turbine and a second object that is stationary inside the turbine, the two or more images being generated by the imaging device, the insertion part being inserted into the turbine through a hole formed in the turbine, the direction of movement of the insertion part when inserted into the turbine being different from the direction of movement of the first object, and while the first object is moving, the position of the first object relative to the insertion part being different each time the imaging device generates an image; and a 3D data generation device including a processor, the processor performing a region detection step of acquiring the two or more images and detecting two or more corresponding regions that are identical regions of the components in at least two of the images included in the two or more images, a region determination step of determining whether at least a portion of a region in each of the above images is a change region or a non-change region, the change region being a region of the component whose coordinates change in the image generated by the imaging device, and the non-change region being a region of the component whose coordinates do not change in the image generated by the imaging device; a data generation step of generating the three-dimensional data by using the corresponding region among the two or more corresponding regions that is determined to be the change region, without using the corresponding region among the two or more corresponding regions that is determined to be the non-change region; the first object rotates inside the turbine by a driving force generated by a driving device; while the first object is rotating, the processor repeatedly executes the region detection step, the region determination step, and the data generation step; and when the rotation of the first object stops, the processor stops the data generation step and continues the region detection step and the region determination step.

[0031] In the three-dimensional data generation system of the present invention, the imaging device and the three-dimensional data generation device are included in an endoscope device.

[0032] In the three-dimensional data generation system of the present invention, the imaging device is included in an endoscope device, and the three-dimensional data generation device is included in an external device separate from the endoscope device.

[0033] The present invention provides a program for causing a computer to execute a process for generating three-dimensional data showing a three-dimensional shape inside a turbine, the program including: an image acquisition step of acquiring two or more images of components inside the turbine, the components including a first object that is movable inside the turbine and a second object that is stationary inside the turbine; the two or more images are generated by an imaging device having a tubular insertion part that captures light inside the turbine; the insertion part is inserted into the turbine through a hole formed in the turbine; a moving direction of the insertion part when inserted into the turbine is different from a moving direction of the first object; and a position of the first object relative to the insertion part while the first object is moving, the image acquisition step being different at each timing when the imaging device generates an image. a region detection step of detecting two or more corresponding regions that are identical regions of the constituent element in at least two images included in the two or more images; a region determination step of determining whether at least a portion of a region in each of the two or more images is a changed region or a non-changed region, wherein the changed region is a region of the constituent element whose coordinates change in the image generated by the imaging device, and the non-changed region is a region of the constituent element whose coordinates do not change in the image generated by the imaging device; and a data generation step of generating the three-dimensional data by using the corresponding region of the two or more corresponding regions that is determined to be the changed region, without using the corresponding region of the two or more corresponding regions that is determined to be the non-changed region. the first object is rotated inside the turbine by a driving force generated by a driving device, and a rotation determination step of determining whether the first object has rotated once inside the turbine; and a notification step of executing a process of notifying a user of information indicating that the first object has rotated once inside the turbine when it is determined that the first object has rotated once inside the turbine. This is a program for The present invention provides a program for causing a computer to execute a process for generating three-dimensional data showing a three-dimensional shape inside a turbine, the program comprising: an image acquisition step of acquiring two or more images of components inside the turbine, the components including a first object that is movable inside the turbine and a second object that is stationary inside the turbine, the two or more images being generated by an imaging device having a tubular insertion part that captures light inside the turbine, the insertion part being inserted into the turbine through a hole formed in the turbine, the direction of movement of the insertion part when inserted into the turbine being different from the direction of movement of the first object, and while the first object is moving, the position of the first object relative to the insertion part being different at each timing when the imaging device generates an image; an area detection step of detecting two or more corresponding areas that are the same area of ​​the components in at least two images included in the two or more images; and an area detection step of detecting two or more corresponding areas that are the same area of ​​the components in at least two images included in the two or more images, the area detection step being performed when at least a part of the area in each of the two or more images is changed. a region determination step of determining whether a corresponding region is a changed region or a non-changed region, where the changed region is a region of the component whose coordinates change in an image generated by the imaging device, and the non-changed region is a region of the component whose coordinates do not change in an image generated by the imaging device; and a data generation step of generating the three-dimensional data by using the corresponding region among the two or more corresponding regions that is determined to be the changed region, without using the corresponding region among the two or more corresponding regions that is determined to be the non-changed region; the first object rotates inside the turbine by a driving force generated by a driving device; and the program causes the computer to repeatedly execute the region detection step, the region determination step, and the data generation step while the first object is rotating, and stops the data generation step when the rotation of the first object stops, and continues the region detection step and the region determination step. [Effects of the Invention]

[0034] According to the present invention, the three-dimensional data generation method, three-dimensional data generation system, and program can improve the reliability of the process of generating three-dimensional data that shows the three-dimensional shape of the inside of a turbine. [Brief explanation of the drawings]

[0035] [Figure 1] 1 is a perspective view showing the overall configuration of an endoscope apparatus according to a first embodiment of the present invention. [Figure 2] 1 is a block diagram showing the internal configuration of an endoscope apparatus according to a first embodiment of the present invention. [Figure 3] FIG. 1 is a diagram schematically showing an arrangement of rotor blades and stator blades in a turbine according to a first embodiment of the present invention. [Figure 4] FIG. 2 is a diagram schematically showing the arrangement of rotor blades in a turbine according to the first embodiment of the present invention. [Figure 5] 2A to 2C are diagrams showing examples of images acquired by the endoscope device according to the first embodiment of the present invention. [Figure 6] 2 is a block diagram showing the functional configuration of a CPU included in the endoscope device according to the first embodiment of the present invention. FIG. [Figure 7] 5 is a flowchart showing the procedure of a three-dimensional (3D) data generation process according to the first embodiment of the present invention. [Figure 8] FIG. 2 is a diagram showing an example of a feature region in the first embodiment of the present invention. [Figure 9] FIG. 3 is a diagram showing an example of identical feature regions between two images according to the first embodiment of the present invention. [Figure 10] FIG. 3 is a diagram illustrating a method for calculating a motion amount in the first embodiment of the present invention. [Figure 11] FIG. 2 is a diagram illustrating a method for determining whether an image region is a moving region or a still region in the first embodiment of the present invention. [Figure 12] FIG. 2 is a schematic diagram showing a situation in which an image is acquired in the first embodiment of the present invention. [Figure 13] 5 is a flowchart showing the procedure of 3D reconstruction processing in the first embodiment of the present invention. [Figure 14] 3A to 3C are diagrams showing examples of information displayed on a display unit included in the endoscope apparatus according to the first embodiment of the present invention. [Figure 15] 3A to 3C are diagrams showing examples of information displayed on a display unit included in the endoscope apparatus according to the first embodiment of the present invention. [Figure 16] 10 is a flowchart showing the procedure of a 3D data generation process in a first modified example of the first embodiment of the present invention. [Figure 17] FIG. 10 is a diagram showing a method for determining whether an area of ​​an image is a moving area or a still area in the first modified example of the first embodiment of the present invention. [Figure 18] 10 is a flowchart showing the procedure of a 3D data generation process in a first modified example of the first embodiment of the present invention. [Figure 19] 10 is a flowchart showing the procedure of a 3D data generation process in a second modified example of the first embodiment of the present invention. [Figure 20] FIG. 10 is a diagram showing an example of an image and examples of a moving region and a still region in a second modified example of the first embodiment of the present invention. [Figure 21] FIG. 10 is a diagram showing an example of object information in a second modified example of the first embodiment of the present invention. [Figure 22] 10 is a flowchart showing the procedure of processing executed by the endoscope apparatus according to the second modified example of the first embodiment of the present invention. [Figure 23] FIG. 10 is a block diagram showing the functional configuration of a CPU included in an endoscope apparatus according to a second embodiment of the present invention. [Figure 24] 10 is a flowchart showing the procedure of a 3D data generation process according to a second embodiment of the present invention. [Figure 25] 10 is a flowchart showing the procedure of a 3D data generation process according to a second embodiment of the present invention. [Figure 26] 10A and 10B are diagrams illustrating an example of an image and an example of a process for determining the size of a still area in the second embodiment of the present invention. [Figure 27] FIG. 6 is a diagram showing an observation position of a rotor blade in a second embodiment of the present invention. [Figure 28] FIG. 6 is a diagram showing an observation position of a rotor blade in a second embodiment of the present invention. [Figure 29] 10 is a flowchart showing the procedure of a 3D data generation process according to a second embodiment of the present invention. [Figure 30]10 is a flowchart showing the procedure of a 3D data generation process according to a second embodiment of the present invention. [Figure 31] 10 is a flowchart showing the procedure of a 3D data generation process according to a second embodiment of the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0036] Hereinafter, an embodiment of the present invention will be described with reference to the drawings.

[0037] (First embodiment) A first embodiment of the present invention will be described below, in which an example in which a three-dimensional (3D) data generating device is included in an endoscope device will be described.

[0038] The configuration of an endoscope device 1 in a first embodiment will be described with reference to Figures 1 and 2. Figure 1 shows the external appearance of the endoscope device 1. Figure 2 shows the internal configuration of the endoscope device 1.

[0039] The endoscope device 1 shown in FIG. 1 has an insertion section 2, a main body section 3, an operation section 4, and a display section 5. The endoscope device 1 captures an image of a subject and generates an image. The subject is an industrial product. To observe various subjects, the user can replace the optical adapter attached to the tip 20 of the insertion section 2, select a built-in image processing program, and add an image processing program.

[0040] The insertion section 2 is inserted into the interior of the subject. The insertion section 2 is a long, thin tube that is bendable from the tip 20 to the base end. The insertion section 2 captures an image of the subject and outputs an image signal to the main body 3. An optical adapter is attached to the tip 20 of the insertion section 2. For example, a monocular optical adapter is attached to the tip 20. The main body 3 is a control device that includes a storage section for storing the insertion section 2. The operation section 4 accepts user operations on the endoscope device 1. The display section 5 has a display screen, and displays images of the subject acquired by the insertion section 2, operation menus, etc. on the display screen.

[0041] The operation unit 4 is a user interface. The display unit 5 is a monitor (display) such as an LCD (Liquid Crystal Display). The display unit 5 may be a touch panel. In that case, the operation unit 4 and the display unit 5 are integrated.

[0042] The main body 3 shown in FIG. 2 includes an endoscope unit 8, a CCU (Camera Control Unit) 9, and a control device 10.

[0043] The endoscope unit 8 has a light source device and a bending device, not shown. The light source device supplies illumination light necessary for observation to the tip 20. The bending device bends the bending mechanism built into the insertion portion 2.

[0044] The lens 21 and the image sensor 28 are built into the tip 20 of the insertion section 2. The lens 21 is an observation optical system. The lens 21 captures an optical image of the subject formed by the optical adapter. The image sensor 28 is an image sensor. The image sensor 28 photoelectrically converts the optical image of the subject and generates an image signal. The lens 21 and the image sensor 28 form a monocular camera with one viewpoint.

[0045] The CCU 9 drives the imaging element 28. An imaging signal output from the imaging element 28 is input to the CCU 9. The CCU 9 performs preprocessing, including amplification and noise removal, on the imaging signal acquired by the imaging element 28. The CCU 9 converts the preprocessed imaging signal into a video signal such as an NTSC signal.

[0046] The control device 10 has a video signal processing circuit 12, a ROM (Read Only Memory) 13, a RAM (Random Access Memory) 14, a card interface 15, an external device interface 16, a control interface 17, and a CPU (Central Processing Unit) 18.

[0047] The video signal processing circuit 12 performs predetermined video processing on the video signal output from the CCU 9. For example, the video signal processing circuit 12 performs video processing related to improving visibility. For example, the video processing includes color reproduction, gradation correction, noise suppression, and edge enhancement. For example, the video signal processing circuit 12 combines the video signal output from the CCU 9 with a graphic image signal generated by the CPU 18. The graphic image signal includes an image of an operation screen, etc. The video signal processing circuit 12 outputs the combined video signal to the display unit 5.

[0048] The ROM 13 is a non-volatile recording medium that stores a program for the CPU 18 to control the operation of the endoscope device 1. The RAM 14 is a volatile recording medium that temporarily stores information used by the CPU 18 to control the endoscope device 1. The CPU 18 controls the operation of the endoscope device 1 based on the program stored in the ROM 13.

[0049] A memory card 42 is connected to the card interface 15. The memory card 42 is a recording medium that is detachable from the endoscope device 1. The card interface 15 imports the control processing information, image information, etc. stored in the memory card 42 into the control device 10. The card interface 15 also records the control processing information, image information, etc. generated by the endoscope device 1 onto the memory card 42.

[0050] An external device such as a USB device is connected to the external device interface 16. For example, a personal computer (PC) 41 is connected to the external device interface 16. The external device interface 16 transmits information to the PC 41 and receives information from the PC 41. This allows the PC 41 to display information. Furthermore, a user can perform operations related to the control of the endoscope device 1 by inputting instructions to the PC 41.

[0051] A turning tool 43 may be used to rotate rotor blades inside the turbine. The turning tool 43 is a driving device that generates a driving force for rotating the rotor blades. The rotor blades are rotated by the driving force generated by the turning tool 43. The turning tool 43 is connected to the external device interface 16. The external device interface 16 outputs control information for controlling the turning tool 43 to the turning tool 43. The external device interface 16 also inputs status information indicating the status of the turning tool 43 into the control device 10.

[0052] The control interface 17 communicates with the operation unit 4, the endoscope unit 8, and the CCU 9 for operational control. The control interface 17 notifies the CPU 18 of information input by the user to the operation unit 4. The control interface 17 outputs control signals to the endoscope unit 8 for controlling the light source device and the bending device. The control interface 17 outputs control signals to the CCU 9 for controlling the image sensor 28.

[0053] The program executed by the CPU 18 may be recorded on a computer-readable recording medium. The program recorded on this recording medium may be read and executed by a computer other than the endoscope device 1. For example, the program may be read and executed by the PC 41. The PC 41 may control the endoscope device 1 by transmitting control information for controlling the endoscope device 1 to the endoscope device 1 in accordance with the program. Alternatively, the PC 41 may acquire a video signal from the endoscope device 1 and process the acquired video signal.

[0054] As described above, the endoscope device 1 has the imaging element 28 and the CPU 18. The imaging element 28 captures an image of a subject and generates an imaging signal. The imaging signal includes an image of the subject. Therefore, the imaging element 28 captures an image of the subject and generates the image. The image is a two-dimensional image (2D image). The image captured by the imaging element 28 is input to the CPU 18 via the video signal processing circuit 12.

[0055] The insertion portion 2 constitutes an imaging device (camera). The imaging element 28 may be disposed in the main body portion 3, and an optical fiber may be disposed in the insertion portion 2. Light incident on the lens 21 may reach the imaging element 28 through the optical fiber. A borescope may be used as a camera.

[0056] Turbines are used in aircraft engines or generators. There are gas turbines and steam turbines. The structure of a gas turbine is explained below. In the following, a gas turbine will be referred to as a turbine.

[0057] A turbine has a compression section, a combustion chamber, and a turbine section. Air is compressed in the compression section. The compressed air is sent to the combustion chamber. Fuel is continuously burned in the combustion chamber, generating high-temperature, high-pressure gases. The gases expand in the turbine section, generating energy. This energy is used to rotate the compressor, and the remaining energy is extracted. In the compression and turbine sections, rotating blades fixed to the engine's rotating shaft and stationary vanes fixed to the casing are arranged alternately.

[0058] A turbine has components located within its interior space. The components can be moving objects that can move within the turbine or stationary objects that are stationary within the turbine. The moving objects are rotor blades. The stationary objects are stator vanes or shrouds.

[0059] Figure 3 shows a schematic diagram of the arrangement of rotor blades and stator vanes in the compression section of turbine TB10. Figure 3 shows a portion of a cross section of turbine TB10 passing through the rotation axis RA10 of the engine. Turbine TB10 has rotor blade RT10, stator vane ST10, rotor blade RT11, stator vane ST11, rotor blade RT12, stator vane ST12, rotor blade RT13, and stator vane ST13 in the compression section. These rotor blades and stator vanes rotate in a direction DR12 around the rotation axis RA10.

[0060] Air taken into the turbine TB10 flows in a direction DR11. The rotor blade RT10 is located in the low pressure section where the air is taken in. The rotor blade RT13 is located in the high pressure section where the air is discharged.

[0061] Access port AP10 is formed to enable inspection of the interior of turbine TB10 without disassembling turbine TB10. Turbine TB10 has two or more access ports, one of which is shown in Figure 3 as access port AP10. Access port AP10 is a hole formed in turbine TB10.

[0062] The insertion portion 2 constitutes an endoscope. The insertion portion 2 is inserted into the turbine TB10 through an access port AP10. When the insertion portion 2 is inserted into the turbine TB10, the insertion portion 2 moves in a direction DR10. When the insertion portion 2 is withdrawn from the turbine TB10, the insertion portion 2 moves in a direction opposite to the direction DR10. The direction DR10 is different from the direction DR12. Illumination light LT10 is emitted from the tip 20 of the insertion portion 2.

[0063] Figure 4 shows a schematic diagram of an arrangement of two or more rotor blades RT10 viewed from a direction parallel to the rotation axis RA10. In Figure 4, eight rotor blades RT10 are arranged on a circular disk DS10. The rotation axis RA10 passes through the center of the disk DS10. The disk DS10 rotates in a direction DR12. Therefore, the eight rotor blades RT10 rotate in the direction DR12.

[0064] In practice, there may be several tens or even hundreds of blades on a single disk. The number of blades on a disk depends on the engine model and the location of the disk in the low-pressure to high-pressure region.

[0065] The user rotates the disk manually, or the turning tool 43 rotates the disk. The insert 2 is inserted into the turbine TB 10 through the access port AP 10, and the tip 20 is fixed. While the disk is rotating, the user inspects two or more rotor blades to determine whether there are any abnormalities in each rotor blade. This inspection is one of the main inspection items in turbine inspection.

[0066] Fig. 5 shows an example of an image acquired by the endoscope device 1. Image IMG10 shown in Fig. 5 is an image of the high-pressure side compression section. Rotor blade RT14, stator blade ST14, and shroud SH10 are shown in image IMG10.

[0067] When the subject is viewed from the tip 20 of the insertion part 2, the stator vane ST14 is disposed in front of the rotor blade RT14, and the shroud SH10 is disposed on the far side of the rotor blade RT14. The distance between the tip 20 and the stator vane ST14 is smaller than the distance between the tip 20 and the rotor blade RT14. In the region where the shroud SH10 is hidden by the rotor blade RT14, the distance between the tip 20 and the shroud SH10 is larger than the distance between the tip 20 and the rotor blade RT14. A portion of the rotor blade RT14 is hidden by the stator vane ST14.

[0068] The image sensor 28 generates two or more images, each of which is temporally associated with the other images included in the two or more images. For example, each of the two or more images is a still image. A moving image may be used instead of a still image. Each of the two or more frames included in the moving image is associated with one another by a timestamp (time code).

[0069] The RAM 14 stores two or more images generated by the imaging element 28. When the disk is rotating, the relative position of the moving blades with respect to the tip 20 (viewpoint) of the insertion section 2 differs between the two or more images. Alternatively, the relative position and attitude of the moving blades with respect to the tip 20 differs between the two or more images. In other words, the position of the moving blades differs each time the imaging element 28 generates an image. Therefore, the position (two-dimensional coordinates) of the moving blades in the image generated by the imaging element 28 changes. When the disk is stationary, the position (two-dimensional coordinates) of the moving blades in the image generated by the imaging element 28 does not change.

[0070] The RAM 14 also stores parameters required for 3D reconstruction processing. These parameters include internal camera parameters, camera distortion correction parameters, settings, and scale information. The settings are used for various processes to generate three-dimensional data (3D data) that indicates the three-dimensional shape (3D shape) of the subject. The scale information is used to convert the scale of the 3D data to the actual scale of the subject.

[0071] The memory card 42 may store two or more images and the above parameters. The endoscope device 1 may read the two or more images and parameters from the memory card 42 and store the two or more images and parameters in the RAM 14.

[0072] The endoscopic device 1 may perform wireless or wired communication with an external device via the external device interface 16. The external device may be a PC 41, a cloud server, or the like. The endoscopic device 1 may transmit two or more images generated by the image sensor 28 to the external device. The external device may store the two or more images and the above-mentioned parameters. The endoscopic device 1 may receive the two or more images and parameters from the external device and store the two or more images and parameters in the RAM 14.

[0073] 6 shows the functional configuration of CPU 18. CPU 18 functions as control unit 180, image acquisition unit 181, region detection unit 182, region determination unit 183, 3D reconstruction unit 184, and display control unit 185. At least one of the blocks shown in FIG. 6 may be configured by a circuit different from CPU 18.

[0074] The control unit 180 controls the processes executed by the units shown in FIG.

[0075] The image acquisition unit 181 acquires two or more images and the above parameters from the RAM 14. The image acquisition unit 181 may acquire two or more images and the above parameters from the memory card 42 or an external device via the external device interface 16.

[0076] The region detection unit 182 detects two or more characteristic regions in each of the two or more images. The region detection unit 182 also detects the same characteristic region (corresponding region) in the two or more images. The corresponding region is the region of the component element of the turbine.

[0077] For example, if a first feature region in a first image and a second feature region in a second image are identical, the region detection unit 182 associates the first feature region and the second feature region as corresponding regions. If the second feature region is identical to a third feature region in a third image, the region detection unit 182 associates the second feature region and the third feature region as corresponding regions. In this case, the region detection unit 182 detects identical corresponding regions across the three images.

[0078] The region determination unit 183 determines whether a feature region in each of two or more images is a moving region or a still region. As a result, the region determination unit 183 classifies the feature region as a moving region or a still region. The region determination unit 183 may determine whether a feature region included in only a part of each image is a moving region or a still region. The region determination unit 183 may determine whether a feature region included in the entirety of each image is a moving region or a still region. Each image may include both moving regions and still regions. Alternatively, each image may include only moving regions or only still regions.

[0079] The 3D restoration unit 184 performs 3D restoration processing by using corresponding regions determined to be moving regions among the feature regions of two or more images, and generates 3D data. At this time, the 3D restoration unit 184 does not use corresponding regions determined to be still regions among the feature regions of two or more images.

[0080] The 3D data includes three-dimensional coordinates (3D coordinates) of two or more regions of the subject, camera coordinates, and posture information. The camera coordinates indicate the 3D coordinates of the camera that captured each of the two or more images, and are associated with each of the two or more images. The camera coordinates indicate the 3D coordinates of the viewpoint when each image was captured. For example, the camera coordinates indicate the 3D coordinates of the observation optical system of the camera. The posture information indicates the posture of the camera that captured each of the two or more images, and is associated with each of the two or more images. For example, the posture information indicates the posture of the observation optical system of the camera.

[0081] The display control unit 185 controls the processing executed by the video signal processing circuit 12. The CCU 9 outputs a video signal. The video signal includes color data for each pixel of the image generated by the imaging element 28. The display control unit 185 causes the video signal processing circuit 12 to output the video signal output from the CCU 9 to the display unit 5. The video signal processing circuit 12 outputs the video signal to the display unit 5. The display unit 5 displays an image based on the video signal output from the video signal processing circuit 12. As a result, the display control unit 185 displays the image generated by the imaging element 28 on the display unit 5.

[0082] The display control unit 185 displays various types of information on the display unit 5. That is, the display control unit 185 displays various types of information on an image.

[0083] For example, the display control unit 185 generates a graphic image signal of various information. The display control unit 185 outputs the generated graphic image signal to the video signal processing circuit 12. The video signal processing circuit 12 combines the video signal output from the CCU 9 and the graphic image signal output from the CPU 18. As a result, the various information is superimposed on the image. The video signal processing circuit 12 outputs the combined video signal to the display unit 5. The display unit 5 displays the image on which the various information is superimposed.

[0084] Furthermore, the display control unit 185 generates a graphic image signal of 3D data. The display control unit 185 outputs the graphic image signal to the video signal processing circuit 12. Processing similar to the processing described above is executed, and the display unit 5 displays an image of the 3D data. As a result, the display control unit 185 displays the image of the 3D data on the display unit 5.

[0085] Each unit shown in FIG. 6 may be configured with at least one of a processor and a logic circuit. For example, the processor is at least one of a CPU (Central Processing Unit), a DSP (Digital Signal Processor), and a GPU (Graphics Processing Unit). For example, the logic circuit is at least one of an ASIC (Application Specific Integrated Circuit) and an FPGA (Field-Programmable Gate Array). Each unit shown in FIG. 6 may include one or more processors. Each unit shown in FIG. 6 may include one or more logic circuits.

[0086] The computer of the endoscope device 1 may load a program and execute the loaded program. The program includes instructions that define the operation of each unit shown in Fig. 6. In other words, the functions of each unit shown in Fig. 6 may be realized by software.

[0087] The above program may be provided by a "computer-readable recording medium" such as a flash memory. The program may be transmitted from a computer storing the program to the endoscope device 1 via a transmission medium or by transmission waves in the transmission medium. A "transmission medium" that transmits the program is a medium that has the function of transmitting information. Media that have the function of transmitting information include networks (communication networks) such as the Internet and communication lines (communication lines) such as telephone lines. The above program may realize some of the functions described above. Furthermore, the above program may be a difference file (difference program). The functions described above may be realized by combining a program already recorded on a computer with a difference program.

[0088] The following describes characteristic processing of the first embodiment. In the following description, it is assumed that the 3D data is generated by using two or more images acquired by an endoscopic device. The inspection device that acquires two or more images is not limited to an endoscopic device. Any device may be used as long as it has a camera and can acquire images of the internal components of the turbine.

[0089] The control device 10 functions as a 3D data generating device. The 3D data generating device of each aspect of the present invention may be a computer system such as a PC separate from the endoscopic device. The 3D data generating device may be any of a desktop PC, a laptop PC, and a tablet terminal. The 3D data generating device may also be a computer system operating on the cloud.

[0090] The processing executed by the endoscope device 1 will be described with reference to Fig. 7. Fig. 7 shows the procedure of the 3D data generation processing executed by the endoscope device 1.

[0091] When the process shown in FIG. 7 starts, the control unit 180 sets the number n for managing images acquired from the RAM 14 to 0 (step S100).

[0092] After step S100, the control unit 180 increments the number n by 1 to acquire an image (step S101). When step S101 is executed for the first time, the number n is set to 1.

[0093] After step S101, the image acquisition unit 181 acquires the image IMGn indicated by the number n from the RAM 14 (step S102). When step S102 is executed for the first time, the image acquisition unit 181 acquires the image IMG1.

[0094] After step S102, the region detection unit 182 analyzes the image IMGn and detects two or more characteristic regions of the subject appearing in the image IMGn (step S103).

[0095] A characteristic region indicates a corner or edge of an image that has a large image brightness gradient. A characteristic region may be composed of one pixel, which is the smallest unit of an image. Alternatively, a characteristic region may be composed of two or more pixels. The region detection unit 182 detects the characteristic region by using SIFT (Scale-Invariant Feature Transform) or FAST (Features from Accelerated Segment Test), etc.

[0096] An example of characteristic regions is shown in Fig. 8. An image IMGn is shown in Fig. 8. The region detection unit 182 detects characteristic regions P1n, P2n, P3n, P4n, P5n, and P6n in the image IMGn.

[0097] After step S103, the control unit 180 determines whether the number n is 1 (step S104).

[0098] In the first embodiment, the area determination unit 183 calculates the amount of movement of the area of ​​the subject between two images in step S106, which will be described later. Therefore, at least two images captured at different times need to be acquired from the RAM 14. When the control unit 180 determines in step S104 that the number n is 1, step S101 is executed. When the control unit 180 determines in step S104 that the number n is not 1, step S105, which will be described later, is executed.

[0099] When the number n is 2 or greater, two or more characteristic regions have already been detected in each of the images IMG(n-1) and IMGn. The timing at which the image sensor 28 generates the image IMGn is different from the timing at which the image sensor 28 generates the image IMG(n-1). The region detection unit 182 detects the same characteristic region (corresponding region) in the images IMG(n-1) and IMGn (step S105).

[0100] The region detection unit 182 executes the following process in step S105. The region detection unit 182 calculates the degree of correlation of the characteristic regions between the image IMG(n-1) and the image IMGn. If the region detection unit 182 finds a characteristic region with a high degree of correlation between the two images, the region detection unit 182 stores information (corresponding information) of the characteristic region (corresponding region) in the RAM 14. In this way, the region detection unit 182 associates the characteristic regions of the two images with each other. On the other hand, if the region detection unit 182 does not find a characteristic region with a high degree of correlation between the two images, the region detection unit 182 discards the information of the corresponding region between the image IMG(n-1) and the image IMGn.

[0101] FIG. 9 shows an example of identical feature regions between two images. Two or more feature regions in image IMG(n-1) and two or more feature regions in image IMGn are shown in FIG. 9. After image IMGn shown in FIG. 8 is acquired in step S102, number n is incremented by 1 in step S101. Then, image IMGn shown in FIG. 9 is acquired in step S102. Image IMG(n-1) shown in FIG. 9 is the same as image IMGn shown in FIG. 8.

[0102] Correspondence information M1, M2, M3, M4, M5, and M6 indicate identical feature regions between image IMG(n-1) and image IMGn. Correspondence information M1 indicates that feature region P1(n-1) in image IMG(n-1) is identical to feature region P1n in image IMGn. Correspondence information M2 indicates that feature region P2(n-1) in image IMG(n-1) is identical to feature region P2n in image IMGn. Correspondence information M3 indicates that feature region P3(n-1) in image IMG(n-1) is identical to feature region P3n in image IMGn. Correspondence information M4 indicates that feature region P4(n-1) in image IMG(n-1) is identical to feature region P4n in image IMGn. Correspondence information M5 indicates that feature region P5(n-1) in image IMG(n-1) is identical to feature region P5n in image IMGn. Correspondence information M6 indicates that characteristic region P6(n-1) in image IMG(n-1) is the same as characteristic region P6n in image IMGn.

[0103] A feature region in one image is not necessarily identical to a feature region in another image. For example, a feature region detected at the edge of image IMG(n-1) may be out of the camera's field of view when image IMGn is captured. Also, due to blurring and other factors, it may be difficult to associate feature regions between two images. Therefore, the number of feature regions in each image is equal to or greater than the number of feature regions identical to feature regions in other images.

[0104] After step S105, the region determination unit 183 calculates the amount of movement of the region of the subject between the image IMG(n-1) and the image IMGn (step S106).

[0105] The region determination unit 183 executes the following process in step S106. As shown in FIG. 10, the region determination unit 183 divides the region of image IMGn into two or more grid-shaped small regions. In the example shown in FIG. 10, the region of image IMGn is divided into seven regions horizontally and six regions vertically. The region of image IMGn is divided into 42 small regions. The shape of the small regions is not limited to a grid. The shape of the small regions may be a circle, for example. The region determination unit 183 divides the region of image IMG(n-1) into two or more grid-shaped small regions by using a method similar to that described above.

[0106] The region determination unit 183 refers to correspondence information of a feature region corresponding to a specific small region. The correspondence information indicates the same feature region between the two images. The region determination unit 183 calculates a representative amount of movement between the small region of the image IMG(n-1) and the small region of the image IMGn. The small region of the image IMG(n-1) and the small region of the image IMGn are associated with each other in the correspondence information.

[0107] The area determination unit 183 can use a statistic as a representative amount of movement. The statistic may be an average value, a median value, or the like.

[0108] The region determination unit 183 may determine the reliability of the amount of motion based on the deviation of the amount of motion between two or more feature regions included in a small region or the deviation of the direction of motion between two or more feature regions. For example, if there is no consistency in the direction of motion, it is possible that the same feature regions between two images are not correctly associated with each other. In this case, the region determination unit 183 may determine that the reliability of the amount of motion is low. When the amount of motion or direction of motion for a certain small region differs from the amount of motion or direction of motion for all other small regions, the region determination unit 183 may determine that the amount of motion is abnormal and exclude that amount of motion.

[0109] The region determination unit 183 executes the above process by using all small regions of the image IMGn, and calculates a representative amount of motion for each small region.

[0110] After step S106, the region determination unit 183 determines whether the small region of the image IMGn is a moving region or a still region based on the amount of motion (step S107).

[0111] The region determination unit 183 performs the following process in step S107. The region determination unit 183 compares a representative amount of movement of each small region with a threshold value. The threshold value is set in advance. Alternatively, the threshold value is calculated depending on the inspection conditions. When the amount of movement is greater than the threshold value, the region determination unit 183 determines that the small region is a moving region. When the amount of movement is equal to or less than the threshold value, the region determination unit 183 determines that the small region is a still region.

[0112] After step S107, the 3D restoration unit 184 extracts, from the image IMGn, feature regions included in the small regions determined to be movement regions in step S107. As a result, the 3D restoration unit 184 extracts feature regions corresponding to the movement regions (step S108).

[0113] The 3D restoration unit 184 does not extract, from the image IMGn, feature regions included in the small regions determined to be still regions in step S107. In other words, the 3D restoration unit 184 does not extract feature regions corresponding to still regions. Therefore, feature regions corresponding to still regions are not used in the 3D restoration process.

[0114] Steps S107 and S108 will be described in detail with reference to Fig. 11. Two or more feature regions in image IMG(n-1) and two or more feature regions in image IMGn are shown in Fig. 11. Each feature region is the same as the feature region shown in Fig. 9.

[0115] The region determination unit 183 calculates the amount of motion of each small region of the image IMG(n-1) by using the images IMG(n-2) and IMG(n-1). The region determination unit 183 classifies two or more characteristic regions of the image IMG(n-1) into a moving region and a still region MS(n-1) based on the amount of motion. The moving region includes the characteristic regions P2(n-1), P3(n-1), P4(n-1), and P5(n-1).

[0116] The region determination unit 183 calculates the amount of motion of each small region of the image IMGn by using the image IMG(n-1) and the image IMGn. The region determination unit 183 classifies two or more feature regions of the image IMGn into a moving region and a still region MSn based on the amount of motion. The moving region includes the feature regions P2n, P3n, P4n, and P5n.

[0117] The 3D reconstruction unit 184 extracts the feature region corresponding to the moving region in step S108. The 3D reconstruction unit 184 does not extract the feature region corresponding to the still region MS(n-1) or the still region MSn in step S108.

[0118] The same process as above is executed each time a new image is acquired in step S102.

[0119] After step S108, the 3D reconstruction unit 184 executes 3D reconstruction processing by using the movement region and the corresponding feature region (step S109). The 3D reconstruction unit 184 reads parameters necessary for the 3D reconstruction processing from the RAM 14 and uses the parameters for the 3D reconstruction processing.

[0120] The 3D reconstruction unit 184 executes the following process in step S109: Figure 12 shows a schematic diagram of an image acquisition situation when two images are acquired.

[0121] As shown in Fig. 12, first, an image I1 is captured in an imaging state c1 of the camera. Next, an image I2 is captured in an imaging state c2 of the camera. At least one of the imaging position and the imaging attitude is different between the imaging states c1 and c2. In Fig. 12, both the imaging position and the imaging attitude are different between the imaging states c1 and c2.

[0122] In each embodiment of the present invention, it is assumed that images I1 and I2 are acquired by the same endoscope. Furthermore, in each embodiment of the present invention, it is assumed that the parameters of the objective optical system of the endoscope do not change. The parameters of the objective optical system include focal length, distortion, and pixel size of the image sensor. Hereinafter, for convenience, the parameters of the objective optical system are abbreviated as intrinsic parameters. Assuming such conditions, the intrinsic parameters that describe the characteristics of the optical system of the endoscope can be commonly used regardless of the position and orientation of the camera (observation optical system). In each embodiment of the present invention, it is assumed that the intrinsic parameters are acquired at the time of shipment from the factory. Furthermore, in each embodiment of the present invention, it is assumed that the intrinsic parameters are known when the images are acquired.

[0123] For example, images I1 and I2 are still images. Images I1 and I2 may also be specific frames extracted from a video. In each embodiment of the present invention, it is assumed that images I1 and I2 are acquired by a single endoscope. However, the present invention is not limited to this. For example, the present invention can also be applied to a case where 3D data is generated using multiple videos acquired by multiple endoscopes. In this case, images I1 and I2 are acquired using different endoscopic devices, and different internal parameters are required for each endoscope. Even if the internal parameters are unknown, calculations can be performed using the internal parameters as variables. Therefore, the subsequent procedures do not change significantly depending on whether the internal parameters are known or not.

[0124] Figure 12 shows the coordinate P 11 A feature region at coordinates P 12 An example of a feature region detected from image I2 is shown below. 11 The feature region in the coordinate P 12 The characteristic region in corresponds to the region at coordinate P1 on the subject. These characteristic regions are detected in step S103 and associated with each other in step S105.

[0125] Although only one feature region is shown for each image in Figure 12, in reality, two or more feature regions are detected in each image. The number of feature regions detected may vary between images. Each feature region detected from each image is converted into data called a feature quantity. A feature quantity is data that represents the characteristics of a feature region.

[0126] The 3D reconstruction process in step S109 will be described in detail with reference to Fig. 13. Fig. 13 shows the procedure of the 3D reconstruction process.

[0127] The 3D restoration unit 184 reads out the coordinates of feature regions associated with each other between the two images from the RAM 14. The coordinates are a pair of coordinates of feature regions in each image. The 3D restoration unit 184 executes a position and orientation calculation process based on the read coordinates (step S109a). In the position and orientation calculation process, the 3D restoration unit 184 calculates the relative position and orientation between the imaging state c1 of the camera that captured image I1 and the imaging state c2 of the camera that captured image I2. More specifically, the 3D restoration unit 184 calculates the matrix E by solving the following equation (1) that utilizes the epipolar constraint:

[0128]

number

[0129] The matrix E is called the fundamental matrix. The fundamental matrix E is a matrix that holds the relative position and orientation between the imaging state c1 of the camera that captured image I1 and the imaging state c2 of the camera that captured image I2. In equation (1), the matrix p1 is a matrix that contains the coordinates of the feature regions detected from image I1. The matrix p2 is a matrix that contains the coordinates of the feature regions detected from image I2. The fundamental matrix E contains information about the relative position and orientation of the cameras, and therefore corresponds to the extrinsic parameters of the cameras. The 3D reconstruction unit 184 can solve the fundamental matrix E by using a known algorithm.

[0130] As shown in FIG. 12, when the amount of change in camera position (relative position) is t and the amount of change in camera attitude (relative attitude) is R, equations (2) and (3) hold true.

[0131]

number

[0132] In equation (2), the amount of movement in the x-axis direction is t x The amount of movement in the y-axis direction is expressed as t y and the amount of movement in the z-axis direction is t zIn equation (3), the rotation amount α around the x-axis is R x (α), and the rotation amount around the y-axis is R y (β), and the rotation amount around the z-axis is R z After the fundamental matrix E is calculated, an optimization process called bundle adjustment may be performed to improve the accuracy of the reconstruction of the 3D coordinates.

[0133] The 3D restoration unit 184 calculates 3D coordinates (camera coordinates) in the coordinate system of the 3D data by using the calculated amount of change in camera position. For example, the 3D restoration unit 184 defines the 3D coordinates of the camera that acquired image I1. The 3D restoration unit 184 calculates the 3D coordinates of the camera that acquired image I2 based on the 3D coordinates of the camera that acquired image I1 and the amount of change in position of the camera that acquired image I2.

[0134] The 3D restoration unit 184 calculates posture information in the coordinate system of the 3D model by using the calculated posture change amount of the camera. For example, the 3D restoration unit 184 defines posture information of the camera that acquired image I1. The 3D restoration unit 184 generates posture information of the camera that acquired image I2 based on the posture information of the camera that acquired image I1 and the posture change amount of the camera that acquired image I2.

[0135] The 3D restoration unit 184 generates three-dimensional shape (3D shape) data (3D shape data) by executing a position and orientation calculation process (step S109a). The 3D shape data includes 3D coordinates (camera coordinates) at the camera position and orientation information indicating the camera orientation. Furthermore, if a method such as Structure from Motion or visual-SLAM is applied to the position and orientation calculation process (step S109a), the 3D restoration unit 184 further calculates the 3D coordinates of each feature region in step S109a. The 3D shape data generated in step S109a does not include 3D coordinates of regions on the subject other than the feature regions. Therefore, the 3D shape data indicates a sparse 3D shape of the subject.

[0136] The 3D shape data includes the 3D coordinates of each feature region, the camera coordinates described above, and the posture information described above. The 3D coordinates of each feature region are defined in the coordinate system of the 3D data. The 3D coordinates of each feature region are associated with the two-dimensional coordinates (2D coordinates) of each feature region. The 2D coordinates of each feature region are defined in the coordinate system of the image in which each feature region is included. The 2D and 3D coordinates of each feature region are associated with the image in which each feature region is included.

[0137] After step S109a, the 3D restoration unit 184 executes a three-dimensional shape restoration process based on the relative position and orientation (position change amount t and orientation change amount R) of the camera calculated in step S109a (step S109b). In the three-dimensional shape restoration process, the 3D restoration unit 184 generates 3D data of the subject. Methods for restoring the three-dimensional shape of the subject include patch-based multi-view stereo (PMVS) and matching processing using rectified stereo. However, the method is not particularly limited.

[0138] In step S109b, the 3D restoration unit 184 calculates the 3D coordinates of regions on the subject other than the feature regions. The 3D coordinates of each region other than the feature regions are defined in the coordinate system of the 3D data. The 3D coordinates of each region are associated with the 2D coordinates of each region. The 2D coordinates of each region are defined in the coordinate system of the 2D image in which each region is included. The 2D and 3D coordinates of each region are associated with the 2D image in which each region is included. The 3D restoration unit 184 updates the 3D shape data. The updated 3D shape data includes the 3D coordinates of each feature region, the 3D coordinates of each region other than the feature regions, camera coordinates, and posture information. The 3D shape data updated in step S109b includes the 3D coordinates of regions on the subject other than the feature regions in addition to the 3D coordinates of the feature regions. Therefore, the 3D shape data indicates a dense 3D shape of the subject.

[0139] After step S109b, the 3D restoration unit 184 executes a scale conversion process (step S109c) based on the 3D shape data processed in the 3D shape restoration process (step S109b) and the scale information read from the RAM 14. In the scale conversion process, the 3D restoration unit 184 converts the 3D shape data of the subject into 3D coordinate data (3D data) having a length dimension. When step S109c is executed, the 3D restoration process ends.

[0140] To shorten the processing time, step S109b may be omitted, in which case, after step S109a is executed, step S109c is executed without executing step S109b.

[0141] Step S109c may be omitted. In this case, after step S109b is executed, step S109c is not executed and the 3D reconstruction process ends. In this case, the 3D data indicates the relative shape of the object without the dimension of length.

[0142] In order for 3D data to be generated according to the principle shown in Fig. 12, at least a portion of the area of ​​each image must be common to at least a portion of each area of ​​at least one other image. That is, an area of ​​a first image and an area of ​​a second image different from the first image include a common area. The area other than the common area in the first image and the area other than the common area in the second image are different from each other.

[0143] For example, in step S109, the 3D reconstruction unit 184 generates 3D data by using the feature regions of image IMG(n-1) and image IMGn. The feature regions of each image correspond to the movement regions extracted in step S108. When step S109 is executed two or more times, the 3D reconstruction unit 184 combines the 3D data generated in the previous execution of step S109 with the 3D data generated in the current execution of step S109.

[0144] The images I1 and I2 do not have to be two temporally consecutive frames in the video, and there may be one or more frames between the images I1 and I2 in the video.

[0145] The process executed by the endoscope device 1 will be described by using FIG. 7 again.

[0146] After step S109, the control unit 180 determines whether the number n has reached a predetermined number, and then determines whether all images have been acquired (step S110).

[0147] For example, the predetermined number may be the number of the last frame of the video or a number preset in the software. The predetermined number may also be the number of the frame being processed when the user performs the termination operation. In this case, the predetermined number changes depending on when the user performs the termination operation.

[0148] When the control unit 180 determines in step S110 that the number n has not reached the predetermined number, step S101 is executed. In this case, a new image is acquired in step S102, and the above-described processing is repeated. When the control unit 180 determines in step S110 that the number n has reached the predetermined number, the display control unit 185 displays the image of the 3D data generated in step S109 on the display unit 5 (step S111). When step S111 is executed, the 3D data generation processing ends.

[0149] Instead of performing step S111, the endoscope device 1 may store the generated 3D data in the RAM 14 or the memory card 42. Alternatively, the endoscope device 1 may transmit the generated 3D data to an external device, such as the PC 41 or a cloud server.

[0150] After step S111 is executed, the user may specify a measurement position in the 3D data displayed on the display unit 5. The endoscope device 1 may perform measurement by using the 3D data.

[0151] In the above example, after two or more feature regions are detected in step S103, a determination regarding moving regions and still regions is made in step S107. After two or more tentative feature regions are detected in step S103 and a determination regarding moving regions and still regions is made in step S107, a formal feature region may be calculated based on information about the moving regions.

[0152] An example of a user interface related to the 3D data generation process will be described using Figures 14 and 15. Figures 14 and 15 show an example of information displayed on the display unit 5.

[0153] The display control unit 185 displays a dialog box DB10 shown in Fig. 14 on the screen of the display unit 5. The dialog box DB10 includes an area RG10, an area RG11, buttons BT10 and BT11, and a seek bar SB10.

[0154] An image of the subject is displayed in area RG10. In the examples shown in Figures 14 and 15, a moving image of the subject is displayed in area RG10. An image of 3D data is displayed in area RG11.

[0155] The user operates buttons BT10 and BT11 by operating operation unit 4. If display unit 5 is configured as a touch panel, the user operates buttons BT10 and BT11 by touching the screen of display unit 5.

[0156] The user presses button BT10 to load the video from RAM 14. After button BT10 is pressed, frames of the video are displayed in area RG10. Figure 14 shows the dialog box DB10 after button BT10 is pressed.

[0157] The user may perform a predetermined operation on the region RG10 by operating the operation unit 4 or touching the screen of the display unit 5. When the predetermined operation is performed, an instruction to play or pause the video may be input to the endoscope device 1. The dialog box DB10 may include buttons for inputting an instruction to play or pause the video.

[0158] The seek bar SB10 indicates the position of the frame displayed in the region RG10. The user can change the position of the frame in the seek bar SB10 by operating the operation unit 4 or touching the screen of the display unit 5. The user can also specify the frame FR10 at which the 3D reconstruction process starts and the frame FR11 at which the 3D reconstruction process ends by operating the operation unit 4 or touching the screen of the display unit 5.

[0159] In the above example, the user specifies a start frame at which the 3D reconstruction process begins and an end frame at which the 3D reconstruction process ends. The control unit 180 may automatically specify the start and end frames. For example, the control unit 180 may detect a section of a video in which a subject is moving. Alternatively, the control unit 180 may detect a section of a video in which an abnormality such as a scratch is captured. The section includes two or more frames of the video. The control unit 180 may specify the first frame of the section as the start frame and the last frame of the section as the end frame.

[0160] Either the start frame or the end frame may be specified by the user, or either the start frame or the end frame may be specified automatically. The method for setting the section including the frames used in the 3D reconstruction process is not limited to the above example.

[0161] The user presses button BT11 to start the 3D reconstruction process. After button BT11 is pressed, the 3D data generation process shown in Fig. 7 begins. Fig. 15 shows the dialog box DB10 after button BT11 is pressed.

[0162] After button BT11 is pressed, button BT11 changes to button BT12 for interrupting the 3D reconstruction process. While the 3D data generation process is being executed, the user can input an instruction to interrupt the 3D data generation process to the endoscope device 1 by pressing button BT12 at any timing. When the user presses button BT12, the control unit 180 sets the current number n as a predetermined number to be used in step S110. After the process for the current number n is completed, step S111 is executed.

[0163] The seek bar SB10 displays the current number n and the corresponding frame FR12, allowing the user to check the progress of the 3D reconstruction process from frame FR10, where the 3D reconstruction process started, to frame FR12, where the 3D reconstruction process was interrupted.

[0164] The display control unit 185 displays an image 3D10 of the 3D data in the region RG11. The image 3D10 shows 3D data generated by using frames FR10 to FR12. In the example shown in FIG. 15, the 3D data shows the reconstructed 3D shapes of seven rotor blades. In the example shown in FIG. 15, a number is assigned to each rotor blade, and the number is displayed. In the example shown in FIG. 15, eight rotor blades are arranged, and 3D data for the remaining rotor blade has not yet been generated.

[0165] Any method may be used to assign numbers to the blades, and an example of a method for assigning numbers to the blades will be described below.

[0166] For example, the 3D reconstruction unit 184 may use image recognition technology such as machine learning to recognize blades in the image displayed in region RG10, and may assign numbers to each blade according to the number of recognized blades. When the turning tool 43 rotates the blades, the 3D reconstruction unit 184 may acquire information indicating the rotation angle of the blades from the turning tool 43. The 3D reconstruction unit 184 may identify the blades in the image displayed in region RG10 based on the rotation angle of the blades and the number of blades fixed to the outer periphery of the disk. The 3D reconstruction unit 184 may assign numbers to the identified blades. The number of blades fixed to the outer periphery of the disk is known.

[0167] The 3D reconstruction unit 184 may apply shape recognition technology to the generated 3D data to recognize an object having a shape similar to the rotor blade, and may assign a number to the object.

[0168] If the 3D data generation device is a device other than the endoscope device 1, the control unit 180 may transmit two or more images generated by the image sensor 28 to the 3D data generation device via the external device interface 16. The 3D data generation device may receive the two or more images and perform processing similar to the processing shown in FIG.

[0169] The 3D data generation method according to each aspect of the present invention generates 3D data showing the 3D shape of the interior of a turbine. The 3D data generation method includes an image acquisition step, a region detection step, a region determination step, and a data generation step.

[0170] In an image acquisition step (step S102), the CPU 18 acquires two or more images of components inside the turbine. The components include a first object that is movable inside the turbine and a second object that is stationary inside the turbine. The two or more images are generated by an imaging device having a tubular insertion portion 2 that captures light inside the turbine. The insertion portion 2 is inserted into the turbine through a hole (access port AP10) formed in the turbine. The direction of movement of the insertion portion 2 when inserted into the turbine (direction DR10) is different from the direction of movement of the first object (direction DR12). While the first object is moving, the position of the first object relative to the insertion portion 2 changes each time the imaging device generates an image.

[0171] In a region detection step (step S105), CPU 18 detects two or more corresponding regions that are identical regions of the constituent elements in at least two images included in the two or more images. In a region determination step (step S107), CPU 18 determines whether at least a portion of a region in each of the two or more images is a changing region (moving region) or a non-changing region (still region). A changing region is a region of the constituent element whose coordinates change in the image generated by the imaging device. A non-changing region is a region of the constituent element whose coordinates do not change in the image generated by the imaging device. In a data generation step (step S109), CPU 18 generates 3D data by using the corresponding regions of the two or more corresponding regions that are determined to be changing regions, without using the corresponding regions of the two or more corresponding regions that are determined to be non-changing regions.

[0172] The 3D data generation system according to each aspect of the present invention includes an imaging device and a 3D data generation device (controller 10). The imaging device has a tubular insertion section 2 that captures light from inside the turbine and generates two or more images of the components inside the turbine. The 3D data generation device includes a CPU 18. The CPU 18 executes the image acquisition step, region detection step, region determination step, and data generation step described above.

[0173] Each aspect of the present invention may include the following modifications: After CPU 18 detects the two or more corresponding regions in the region detection step (step S105), CPU 18 determines whether at least a portion of the region in each of the two or more images is a changed region or a non-changed region in the region determination step (step S107).

[0174] Each aspect of the present invention may include the following variations: The first object includes a rotor blade The second object includes a stator vane or a shroud.

[0175] Each aspect of the present invention may include the following variations: In the two or more images, a first object is partially obscured by a second object. In the example shown in Figure 5, a rotor blade RT14 is partially obscured by a stator blade ST14.

[0176] Each aspect of the present invention may include the following variations. The second object includes an object that hides a portion of the first object in the two or more images, and an object that is partially hidden by the first object in the two or more images. In the example shown in Figure 5, the stator vane ST14 hides a portion of the rotor blade RT14, and a portion of the shroud SH10 is hidden by the rotor blade RT14.

[0177] Each aspect of the present invention may include the following modifications: The imaging device generates two or more images at two or more different times. In a region determination step (step S107), CPU 18 determines whether at least a portion of a region in each of the two or more images is a changed region or a non-changed region based on the amount of movement of corresponding regions between at least two of the two or more images.

[0178] Each aspect of the invention may include the following variations: The insert 2 is fixed inside the turbine.

[0179] Each aspect of the present invention may include the following variations: The imaging device is a borescope.

[0180] In the first embodiment, the endoscope device 1 generates 3D data showing the 3D shape of the interior of the turbine by using feature regions corresponding to moving regions instead of feature regions corresponding to stationary regions, thereby improving the reliability of the process of generating 3D data.

[0181] The user can check the condition of the rotor blade by checking the image of the 3D data displayed on the display unit 5 or by performing measurements using the 3D data, thereby improving the quality and efficiency of the inspection.

[0182] (First Modification of the First Embodiment) A first modification of the first embodiment of the present invention will be described. In the first embodiment described above, the endoscope device 1 classifies a subject area into a moving area or a stationary area by using the amount of movement of the subject area between two or more images. In the first modification of the first embodiment, the endoscope device 1 classifies a subject area into a moving area or a stationary area by using the brightness difference between the images.

[0183] The processing executed by the endoscope device 1 will be described using Fig. 16. Fig. 16 shows the procedure of the 3D data generation processing executed by the endoscope device 1. The description of the same processing as that shown in Fig. 7 will be omitted.

[0184] After step S105, the region determination unit 183 calculates the brightness difference between the image IMG(n-1) and the image IMGn (step S120).

[0185] In step S120, the region determination unit 183 performs the following process. The region determination unit 183 calculates the difference between the luminance of each pixel in the image IMG(n-1) and the luminance of each pixel in the image IMGn. The region determination unit 183 uses the luminance of the same pixel in the image IMG(n-1) and the image IMGn.

[0186] If the image sensor 28 generates a color image having three channels, the region determination unit 183 may convert the images IMG(n-1) and IMGn into grayscale images. Alternatively, the region determination unit 183 may calculate three difference values ​​by using the values ​​of the three channels, and calculate the statistical value of the three difference values ​​as the brightness difference.

[0187] After step S120, the region determination unit 183 determines whether the feature region of the image IMGn is a moving region or a still region based on the brightness difference (step S107a). After step S107a, step S108 is executed.

[0188] The region determination unit 183 performs the following process in step S107a. The region determination unit 183 compares the brightness difference of each pixel with a threshold. If the brightness difference is greater than the threshold, the region determination unit 183 determines that the pixel is included in a moving region. If the brightness difference is equal to or less than the threshold, the region determination unit 183 determines that the pixel is included in a still region. The region determination unit 183 performs the above process for all pixels of the image IMGn.

[0189] When the tip 20 of the insertion section 2 is moving, the entire area shown in the image moves in the same direction. Therefore, the area determination unit 183 may match the position of the image IMG(n-1) with the position of the image IMGn before calculating the brightness difference.

[0190] Steps S107a and S108 will be described in detail with reference to Fig. 17. Two or more feature regions in image IMG(n-1) and two or more feature regions in image IMGn are shown in Fig. 17. Each feature region is the same as the feature region shown in Fig. 9.

[0191] The region determination unit 183 calculates the brightness difference of each pixel of the image IMG(n-1) by using the images IMG(n-2) and IMG(n-1). The region determination unit 183 classifies two or more characteristic regions of the image IMG(n-1) into a moving region and a still region MS(n-1) based on the brightness difference. The moving region includes the characteristic regions P2(n-1), P3(n-1), P4(n-1), and P5(n-1).

[0192] The region determination unit 183 calculates the brightness difference of each pixel of the image IMGn by using the image IMG(n-1) and the image IMGn. The region determination unit 183 classifies two or more characteristic regions of the image IMGn into a moving region and a still region MSn based on the brightness difference. The moving region includes the characteristic regions P2n, P3n, P4n, and P5n.

[0193] The 3D reconstruction unit 184 does not extract a feature region corresponding to the still region MS(n-1) in step S108. Furthermore, the 3D reconstruction unit 184 does not extract a feature region corresponding to the still region MSn in step S108.

[0194] The same process as above is executed each time a new image is acquired in step S102.

[0195] The order of step S105, step S120, and step S107a is not limited to the order shown in Fig. 16. A modified example of the processing executed by the endoscope device 1 will be described using Fig. 18. Fig. 18 shows the procedure of the 3D data generation processing executed by the endoscope device 1. Description of the same processing as the processing shown in Fig. 16 will be omitted.

[0196] When the control unit 180 determines in step S104 that the number n is not 1, the region determination unit 183 calculates the brightness difference between the image IMG(n-1) and the image IMGn in step S120. After step S120, the region determination unit 183 determines in step S107a whether the feature region of the image IMGn is a moving region or a still region based on the brightness difference. At this time, the feature region of the image IMGn and the feature region of the image IMG(n-1) are not associated with each other.

[0197] After step S107a, the region detection unit 182 detects the same feature region (corresponding region) in the image IMG(n-1) and the image IMGn. At this time, the region detection unit 182 uses information on the feature region determined to be a moving region in step S107a, but does not use information on the feature region determined to be a still region in step S107a (step S105a). After step S105a, step S108 is executed.

[0198] Each aspect of the present invention may include the following modifications: In the region determination step (step S107a), CPU 18 determines whether at least a portion of a region in each of two or more images of the internal components of the turbine is a changing region (moving region) or a non-changing region (still region) based on the difference in pixel values ​​between at least two images included in the two or more images.

[0199] Each aspect of the present invention may include the following modifications: After CPU 18 determines whether at least some areas in the two or more images are changed areas or non-changed areas in the area determination step (step S107a), CPU 18 detects two or more corresponding areas by using changed areas but not non-changed areas in the area detection step (step S105a).

[0200] In a first modification of the first embodiment, the endoscope device 1 can determine moving and stationary regions by using the brightness difference between images.

[0201] (Second Modification of the First Embodiment) A second modification of the first embodiment of the present invention will be described. In the first embodiment and the first modification of the first embodiment described above, the endoscope device 1 classifies the area of ​​the subject into a moving area or a stationary area by using two or more images. In the second modification of the first embodiment, the endoscope device 1 classifies the area of ​​the subject into a moving area or a stationary area by using one image.

[0202] The processing executed by the endoscope device 1 will be described using Fig. 19. Fig. 19 shows the procedure of the 3D data generation processing executed by the endoscope device 1. The description of the processing that is the same as the processing shown in Fig. 7 will be omitted.

[0203] After step S105, the area determination unit 183 applies an image recognition technique to the image IMGn to determine the area of ​​the subject in the image IMGn (step S121).

[0204] The region determination unit 183 performs the following process in step S121. The region determination unit 183 processes the image IMGn and detects rotor blades, stator blades, shrouds, or the like that appear in the image IMGn. The region determination unit 183 may use machine learning, which is known as an image recognition technology. The region determination unit 183 may also use a method that does not use machine learning. Any method may be used to determine a region in an image.

[0205] An example in which the region determination unit 183 uses machine learning will be described. A user assigns the name of an object to a specific region in an image acquired in a previous inspection. The image is a still image or a video frame. Before the process shown in FIG. 19 is performed, the region determination unit 183 learns the name assigned by the user as correct answer data. The correct answer data is also called training data. The region determination unit 183 uses the correct answer data to analyze the features of the image for learning and generate a trained model. When an inspection of a turbine rotor blade is performed, the rotor blade rotates and the stator blade or shroud is fixed. The region determination unit 183 analyzes the features related to the movement of each object.

[0206] An external device may execute the above process and generate a trained model. The external device may be the PC 41, a cloud server, or the like. The endoscope device 1 may acquire the trained model from the external device.

[0207] After the trained model is generated, in step S121, the area determination unit 183 uses the trained model to determine the object appearing in the evaluation image.

[0208] An example will be described in which the region determination unit 183 does not use machine learning. The region determination unit 183 calculates a uniquely designed image feature. The region determination unit 183 performs clustering using the image feature and determines objects appearing in the image. The region determination unit 183 can apply technology such as a support vector machine to the clustering.

[0209] After step S121, the region determination unit 183 refers to information about the region detected in step S121. The region determination unit 183 determines the characteristic region of the rotor blade in the image IMGn as a moving region, and determines the characteristic region of the stator blade or shroud in the image IMGn as a stationary region (step S107b). After step S107b, step S108 is executed.

[0210] Steps S121 and S107b will be described in detail using Figures 20(a) and 20(b). Figure 20(a) shows an image IMGn generated by the image sensor 28. Figure 20(b) shows moving and still regions in the image IMGn.

[0211] In step S121, the area determination unit 183 analyzes the image IMGn and detects the objects OBJ1, OBJ2, and OBJ3. A number according to the characteristics of each object is assigned to each object. For example, the number 1 is assigned to the object OBJ1, the number 2 is assigned to the object OBJ2, and the number 3 is assigned to the object OBJ3. When two or more objects are of the same or similar type, similar numbers may be assigned to the two or more objects. In the example shown in FIG. 21, similar numbers 2-1, 2-2, etc. are assigned to the object OBJ2. The area determination unit 183 references the object information and acquires the name of each object.

[0212] The object information includes an object number and an object name. Figure 21 shows an example of object information. In the object information, the number and the name are associated with each other. The name indicates the name of the object that has the feature corresponding to the number.

[0213] In the object information, the stationary blade is associated with the number 1 of the object OBJ1. Therefore, the area determination unit 183 determines that the name of the object OBJ1 is a stationary blade. In the object information, the rotor blade is associated with the number 2 of the object OBJ2. Therefore, the area determination unit 183 determines that the name of the object OBJ2 is a rotor blade. In the object information, the shroud is associated with the number 3 of the object OBJ3. Therefore, the area determination unit 183 determines that the name of the object OBJ3 is a shroud.

[0214] In step S107b, the region determination unit 183 determines that the characteristic region of the object OBJ2 is a moving region. Also, in step S107b, the region determination unit 183 determines that the characteristic regions of the objects OBJ1 and OBJ3 are moving regions.

[0215] Step S121 may be executed at any timing between the timing when step S102 is executed and the timing when step S107b is executed.

[0216] The order of step S105, step S121, and step S107b is not limited to the order shown in Fig. 19. A modified example of the processing executed by the endoscope device 1 will be described using Fig. 22. Fig. 22 shows the procedure of the 3D data generation processing executed by the endoscope device 1. Description of the same processing as the processing shown in Fig. 19 will be omitted.

[0217] When the control unit 180 determines in step S104 that the number n is not 1, the region determination unit 183 determines the region of the subject in the image IMGn in step S121. After step S121, the region determination unit 183 determines in step S107b the characteristic region of the rotor blade in the image IMGn as a moving region. Also, in step S107b, the region determination unit 183 determines the characteristic region of the stator blade or shroud in the image IMGn as a stationary region. At this time, the characteristic region of the image IMGn and the characteristic region of the image IMG(n-1) are not associated with each other.

[0218] After step S107b, the region detection unit 182 detects the same feature region (corresponding region) in the image IMG(n-1) and the image IMGn. At this time, the region detection unit 182 uses information on the feature region determined to be a moving region in step S107b, but does not use information on the feature region determined to be a still region in step S107b (step S105b). After step S105b, step S108 is executed.

[0219] Each aspect of the present invention may include the following modifications: In the region determination step (step S107b), CPU 18 determines whether at least a portion of a region in each of the two or more images is a changing region (moving region) or a non-changing region (still region) by determining a subject appearing in one of the two or more images.

[0220] Each aspect of the present invention may include the following modifications: After CPU 18 determines whether at least some areas in two or more images are changed areas or non-changed areas in the area determination step (step S107b), CPU 18 detects two or more corresponding areas by using changed areas but not non-changed areas in the area detection step (step S105b).

[0221] In the second modification of the first embodiment, the endoscope device 1 can determine the type of subject in the image to determine whether the area is moving or stationary.

[0222] (Second embodiment) A second embodiment of the present invention will be described. When an examination is being performed, an endoscope device 1 generates images and 3D data of the subject. That is, the endoscope device 1 generates images and 3D data in parallel. To avoid failures in the 3D reconstruction process and promote efficient examination, the endoscope device 1 provides various support functions.

[0223] The CPU 18 shown in Fig. 6 is changed to a CPU 18a shown in Fig. 23. Fig. 23 shows the configuration of the CPU 18a. A description of the same configuration as that shown in Fig. 6 will be omitted.

[0224] CPU 18a functions as control unit 180, image acquisition unit 181, region detection unit 182, region determination unit 183, 3D restoration unit 184, display control unit 185, state determination unit 186, and notification unit 187. At least one of the blocks shown in Fig. 23 may be configured by a circuit different from CPU 18a.

[0225] Each part of the CPU 18a may be composed of at least one of a processor and a logic circuit. Each part of the CPU 18a may include one or more processors. Each part of the CPU 18a may include one or more logic circuits.

[0226] The status determination unit 186 determines the status of the examination. For example, the status of the examination is the image quality of the image generated by the image sensor 28, the size of the still area captured in the image, the movement of the camera, or the progress of observation of the subject. The status determination unit 186 generates examination status information regarding the status of the examination.

[0227] The notification unit 187 executes notification processing to notify the user of the examination status information. For example, the notification unit 187 generates a graphic image signal corresponding to the examination status information. The notification unit 187 outputs the graphic image signal to the video signal processing circuit 12. Processing similar to the processing described above is executed, and the display unit 5 displays a message including the examination status information. As a result, the notification unit 187 displays the examination status information on the display unit 5.

[0228] The notification unit 187 may output sound data to a speaker and cause the speaker to generate a sound corresponding to the inspection status information. The notification unit 187 may output a control signal indicating a vibration pattern to a vibration generator and cause the vibration generator to generate vibrations having a pattern corresponding to the inspection status information. The notification unit 187 may output a control signal indicating a light emission pattern to a light source and cause the light source to emit light having a pattern corresponding to the inspection status information.

[0229] In the following, an example will be described in which a message including test status information is displayed on the display unit 5. In this case, the display control unit 185 may function as the notification unit 187.

[0230] 24 and 25, a first example of processing executed by the endoscope device 1 will be described. Figures 24 and 25 show the procedure of 3D data generation processing executed by the endoscope device 1. Description of processing that is the same as the processing shown in Figure 7 will be omitted.

[0231] For example, the user operates the operation unit 4 to input an instruction to start the 3D data generation process to the endoscope device 1. If the display unit 5 is configured as a touch panel, the user touches the screen of the display unit 5 to input the instruction to the endoscope device 1. The control unit 180 accepts the instruction and starts the 3D data generation process.

[0232] The control unit 180 may start the 3D data generation process when it is determined that a subject captured in an image generated by the image sensor 28 has been stationary for a predetermined period of time. Alternatively, the control unit 180 may start the 3D data generation process when the composition of the image captured to acquire the image matches a predetermined composition. For example, the control unit 180 may check whether the current composition matches the previous composition by using an inspection image captured in a previous inspection. The user may manually adjust the composition of the image to match the predetermined composition or check the composition of the image. When the 3D data generation process starts, the control unit 180 may send control information to the turning tool 43 to cause the turning tool 43 to start rotating.

[0233] After step S102, the state determination unit 186 determines the image quality of the image IMGn (step S130).

[0234] For example, if halation occurs in image IMGn or if image IMGn is dark, 3D restoration processing is difficult. 3D restoration processing is also difficult if the subject in image IMGn has little pattern. 3D restoration processing is also difficult when the blade rotates quickly due to motion blur. When these factors occur, the status determination unit 186 determines that the image quality of image IMGn is low. When these factors do not occur, the status determination unit 186 determines that the image quality of image IMGn is high.

[0235] When the status determination unit 186 determines in step S130 that the image quality of the image IMGn is high, step S103 is executed. When the status determination unit 186 determines in step S130 that the image quality of the image IMGn is low, the status determination unit 186 generates examination status information indicating that the image quality is low or that 3D reconstruction processing is difficult. The notification unit 187 executes notification processing and displays the examination status information on the display unit 5 (step S133). The examination status information functions as a warning. The examination status information may include information prompting the user to change the shooting composition, image setting conditions, etc.

[0236] When step S133 is executed, the control unit 180 may transmit control information to the turning tool 43 to stop the turning tool 43. The turning tool 43 may stop the rotation of the rotor blades based on the control information. When step S133 is executed, the 3D data generation process ends.

[0237] After step S106, the control unit 180 adjusts the rotation speed of the rotor blades based on the amount of movement calculated in step S106 (step S131). After step S131, step S107 is executed.

[0238] The control unit 180 executes the following process in step S131. The control unit 180 calculates the difference between the amount of movement calculated in step S106 and a target amount of movement. The target amount of movement is set in advance. If the difference is greater than a predetermined value, the control unit 180 transmits control information to the turning tool 43 to slow down the rotation speed of the rotor blades. The turning tool 43 slows down the rotation speed of the rotor blades based on the control information.

[0239] When the tip 20 of the insertion portion 2 is close to the subject, the subject moves quickly in the image generated by the image sensor 28. Therefore, the endoscope device 1 needs to reduce the rotation speed of the moving blades. Because the rotation speed of the moving blades is controlled based on the calculated amount of movement, the endoscope device 1 can reduce the influence of the observation composition on the examination.

[0240] After step S106, step S107 may be executed without executing step S131.

[0241] After step S107, the state determination unit 186 determines whether the area of ​​the still region in the image IMGn is large (step S132).

[0242] The state determination unit 186 executes the following process in step S132. For example, the state determination unit 186 calculates the number of pixels in the still region in the image IMGn. The state determination unit 186 compares the calculated number of pixels with a predetermined value. When the number of pixels is greater than the predetermined value, the state determination unit 186 determines that the area of ​​the still region is large. When the number of pixels is equal to or less than the predetermined value, the state determination unit 186 determines that the area of ​​the still region is small. The predetermined value is set in advance. The predetermined value may be changeable.

[0243] An example of step S132 will be described using Figures 26(a) and 26(b). Figure 26(a) shows an image IMGn generated by the image sensor 28. Figure 26(b) shows moving and still regions in the image IMGn.

[0244] Image IMGn includes still regions RG20 and RG21, and a moving region RG22. The proportion of moving region RG22 to the entire image IMGn is small. In this case, the endoscope device 1 is more likely to fail in the 3D reconstruction process.

[0245] When the state determination unit 186 determines in step S132 that the area of ​​the still region is small, step S108 is executed. When the state determination unit 186 determines in step S132 that the area of ​​the still region is large, the state determination unit 186 generates examination status information indicating that 3D reconstruction processing is difficult in step S133. The notification unit 187 executes notification processing in step S133 and displays the examination status information on the display unit 5. The examination status information functions as a warning. The examination status information may include information prompting the user to change the composition of the image.

[0246] When the number n is 2, step S132 is executed for the first time. At this time, step S109 has not yet been executed. When the state determination unit 186 determines in step S132 that the area of ​​the stationary region is large, step S133 is executed as described above, and the 3D data generation process ends. At this time, the control unit 180 may cause the turning tool 43 to stop the rotation of the rotor blade.

[0247] On the other hand, after step S132 has been executed one or more times, step S132 may be executed again, and the state determination unit 186 may determine that the area of ​​the stationary region is large. This means that the stationary region increases during the 3D data generation process. The stationary region increases when the user stops the rotation of the moving blades to change the position of the tip 20 of the insertion unit 2. The condition for stopping the rotation of the moving blades is not limited to the issuance of an instruction by the user. For example, the endoscope device 1 may automatically stop the rotation of the moving blades. The condition for stopping the rotation of the moving blades is not limited to the above example.

[0248] Consider a situation where the stationary area is increased. An inspection of a large blade may be performed. Often, the large blade is located in the low-pressure compressor or turbine section. Only a portion of the blade is in the camera's field of view. In this inspection, the tip 20 is repositioned and two or more images of the blade are taken.

[0249] The inspection of the large rotor blades will be explained in detail using Figures 27(a), 27(b), 28(a), and 28(b), each showing the observation positions of eight rotor blades RT20.

[0250] First, the tip 20 is fixed at the position shown in Fig. 27(a). At this time, illumination light LT20 is irradiated onto the upper part of the rotor blade RT20 as shown in Fig. 27(b). In this state, the rotor blade rotates once in direction DR20. At this time, the rotor blade rotates 360° or more. While the rotor blade is rotating, steps S101 to S110 are repeatedly executed.

[0251] After the rotor blade rotates once, the rotation of the rotor blade is stopped. The position of the tip 20 is changed and the tip 20 is fixed at the position shown in Fig. 28(a). At this time, the illumination light LT20 is irradiated onto the lower part of the rotor blade RT20 as shown in Fig. 28(b). In this state, the rotor blade rotates once in the direction DR20, and steps S101 to S110 are repeatedly executed.

[0252] When the rotation of the rotor blades is stopped to change the position of the tip 20, the entire subject captured in the image generated by the image sensor 28 stops. Therefore, in step S132, the state determination unit 186 determines that the area of ​​the stationary region is large. In this case, the endoscope device 1 may continue the 3D data generation process. That is, step S110 may be executed after step S133 is executed. In this case, the endoscope device 1 stops steps S108 and S109 and continues steps S101 to S107.

[0253] 27(a) to the position shown in FIG. 28(a), the entire subject shown in the image generated by the image sensor 28 moves. Therefore, in step S132, the state determination unit 186 determines that the area of ​​the stationary region is small. In this case, the endoscope device 1 resumes steps S108 and S109, and continues steps S101 to S110.

[0254] 28(a) and is fixed. At this time, the entire subject captured in the image generated by the image sensor 28 stops. Therefore, the state determination unit 186 determines in step S132 that the area of ​​the still region is large. In this case, the endoscope device 1 may stop steps S108 and S109 and continue with steps S101 to S107.

[0255] After that, the rotor blades start rotating again. Therefore, the state determination unit 186 determines in step S132 that the area of ​​the stationary region is small. In this case, the endoscope device 1 restarts steps S108 and S109, and continues steps S101 to S110.

[0256] When inspecting large rotor blades, steps S101 to S107 are repeatedly executed even when rotor blade rotation is stopped. This allows the endoscope device 1 to accumulate two or more images generated by the image sensor 28 and information on the same feature area in the two or more images. When rotor blade rotation is resumed and the 3D reconstruction process in step S109 is executed again, the 3D reconstruction unit 184 can reconstruct the 3D shape of the subject by using the accumulated images and information. Therefore, the endoscope device 1 can generate a single piece of 3D data that indicates the entire 3D shape of two or more rotor blades.

[0257] When the state determination unit 186 determines in step S132 that the number n is 2 or greater and the area of ​​the stationary region is large, the state determination unit 186 may generate inspection state information indicating that the rotation of the rotor blade has stopped in step S133.

[0258] The state determination unit 186 may receive state information indicating the drive state of the turning tool 43 from the turning tool 43. The state determination unit 186 may monitor the state of the rotor blade based on the state information. The state information indicates whether the turning tool 43 is rotating the rotor blade. In step S133, the state determination unit 186 may detect that the rotation of the rotor blade has stopped based on the state information, and generate inspection state information indicating that the rotation of the rotor blade has stopped.

[0259] When the inspection of the entire rotor blade is completed and the state determination unit 186 determines in step S132 that the area of ​​the stationary region is large, the endoscope device 1 may execute step S133 and terminate the 3D data generation process. The endoscope device 1 may use a method of detecting that a subject observed during the ongoing inspection has been observed again, as a method of detecting that the inspection of the entire rotor blade is completed. This method will be described later in a third example.

[0260] The user may input information indicating that the inspection of the entire rotor blade has been completed to the endoscope device 1 by operating the operation unit 4. If the display unit 5 is configured as a touch panel, the user may input the information to the endoscope device 1 by touching the screen of the display unit 5. The state determination unit 186 may detect that the inspection has been completed based on the information.

[0261] 29 and 30, a second example of processing executed by the endoscope device 1 will be described. Figures 29 and 30 show the procedure of 3D data generation processing executed by the endoscope device 1. Description of processing that is the same as the processing shown in Figure 7 will be omitted.

[0262] After step S106, step S131 is executed. Step S131 shown in Fig. 30 is the same as step S131 shown in Fig. 25. After step S131, step S107 is executed. After step S106, step S107 may be executed without executing step S131.

[0263] After step S107, the state determination unit 186 determines the movement of the camera (step S134).

[0264] The state determination unit 186 executes the following process in step S134. For example, the state determination unit 186 determines the movement of the camera by analyzing the image IMGn. When an IMU (Inertial Measurement Unit) that detects the acceleration and angular velocity of the tip 20 is disposed at the tip 20 of the insertion unit 2, the state determination unit 186 may determine the movement of the camera based on values ​​measured by the IMU. When the movement of the camera is slight, the state determination unit 186 may determine that the camera is not moving.

[0265] When the status determination unit 186 determines in step S134 that the camera is not moving, step S108 is executed. When the status determination unit 186 determines in step S134 that the camera is moving, the status determination unit 186 generates examination status information indicating that the camera is moving. The notification unit 187 executes notification processing and displays the examination status information on the display unit 5 (step S135). The examination status information functions as a warning.

[0266] When step S135 is executed, the control unit 180 may transmit control information to the turning tool 43 to stop the turning tool 43. The turning tool 43 may stop the rotation of the rotor blades based on the control information. When step S135 is executed, the 3D data generation process ends.

[0267] As described above, in order to inspect a large rotor blade, the position of the tip 20 may be changed and the rotor blade may be photographed two or more times. When the tip 20 is moving, the state determination unit 186 determines in step S134 that the camera is moving. In this case, the endoscope device 1 may continue the 3D data generation process. That is, step S110 may be executed after step S135 is executed. In this case, the endoscope device 1 stops steps S108 and S109 and continues steps S101 to S107.

[0268] When the state determination unit 186 determines in step S134 that the inspection of the entire rotor blade has been completed and the camera is moving, the endoscope device 1 may execute step S135 and terminate the 3D data generation process. The endoscope device 1 may use a method of detecting that the subject has been observed again during the ongoing inspection as a method of detecting that the inspection of the entire rotor blade has been completed. This method will be described later in a third example.

[0269] As described above, the user may input information indicating that the inspection of the entire rotor blade has been completed to the endoscope device 1 by operating the operation unit 4 or the touch panel. The state determination unit 186 may detect that the inspection has been completed based on that information.

[0270] 29 and 31, a third example of processing executed by the endoscope device 1 will be described. Figures 29 and 31 show the procedure of 3D data generation processing executed by the endoscope device 1. Description of processing that is the same as the processing shown in Figure 7 will be omitted.

[0271] After step S106, step S131 is executed. Step S131 shown in Fig. 31 is the same as step S131 shown in Fig. 25. After step S131, step S107 is executed. After step S106, step S107 may be executed without executing step S131.

[0272] After step S109, the state determination unit 186 determines whether or not the object has been observed again in the ongoing inspection (step S136). When the rotor blade makes one rotation, the object that has already been observed is observed again.

[0273] The state determination unit 186 executes the following process in step S136. For example, the state determination unit 186 receives state information indicating the drive state of the turning tool 43 from the turning tool 43. The state determination unit 186 monitors the state of the rotor blades based on the state information. The state information indicates the rotation angle of the rotor blades. When the rotation angle of the rotor blades is less than 360 degrees, the state determination unit 186 determines that the subject has not been observed again. When the rotation angle of the rotor blades is 360 degrees or more, the state determination unit 186 determines that the subject has been observed again.

[0274] Alternatively, the state determination unit 186 analyzes an image generated by the image sensor 28 and determines whether or not an object having characteristics similar to those of an object already observed is captured in the image. If an object having those characteristics is not captured in the image, the state determination unit 186 determines that the object has not been observed again. If an object having those characteristics is captured in the image, the state determination unit 186 determines that the object has been observed again.

[0275] Alternatively, the state determination unit 186 analyzes the image generated by the imaging element 28 and recognizes the rotor blades that appear in the image. The state determination unit 186 counts the number of recognized rotor blades. The number of rotor blades fixed to the outer periphery of the disk (the design number) is known. The state determination unit 186 determines whether the number of recognized rotor blades has reached the design number. When the number of recognized rotor blades has not reached the design number, the state determination unit 186 determines that the subject has not been observed again. When the number of recognized rotor blades has reached the design number, the state determination unit 186 determines that the subject has been observed again.

[0276] When the status determination unit 186 determines in step S136 that the subject has not been observed again, step S110 is performed. When the status determination unit 186 determines in step S136 that the subject has been observed again, the status determination unit 186 generates inspection status information indicating that the subject has been observed again. The notification unit 187 executes notification processing and displays the inspection status information on the display unit 5 (step S137). The inspection status information functions as a warning. The inspection status information may include information urging the user to end the inspection. As described above, when the position of the tip 20 is changed to inspect a large rotor blade and the rotor blade is photographed more than once, the inspection status information may include information urging the user to change the position of the tip 20.

[0277] When step S137 is executed, the control unit 180 may transmit control information to the turning tool 43 to stop the turning tool 43. The turning tool 43 may stop the rotation of the rotor blades based on the control information. When step S137 is executed, the 3D data generation process ends.

[0278] In the first to third examples, step S106 and step S107 may be changed to step S120 and step S107a, respectively, shown in Fig. 16. In the first to third examples, step S106 and step S107 may be changed to step S121 and step S107b, respectively, shown in Fig. 19.

[0279] Each aspect of the present invention may include the following modifications: In a position determination step (step S134), the CPU 18 determines the position of the insertion portion 2 inside the turbine. When the position of the insertion portion 2 changes, the CPU 18 executes a process of notifying the user of information indicating the change in the position of the insertion portion 2 in a notification step (step S135).

[0280] Each aspect of the present invention may include the following modifications. Before CPU 18 executes data generation step (step S109) for the first time, CPU 18 calculates the area of ​​a non-changing region (static region) in two or more images of the internal components of the turbine in calculation step (step S132). If the calculated area is larger than a predetermined value, CPU 18 executes processing to notify a user of a warning in notification step (step S133).

[0281] Each aspect of the present invention may include the following modifications. A first object (rotor blade) rotates inside the turbine by a driving force generated by a turning tool 43 (drive device). In a rotation determination step (step S136), the CPU 18 determines whether the first object has rotated once inside the turbine. When the CPU 18 determines that the first object has rotated once inside the turbine, the CPU 18 executes a process of notifying a user of information indicating that the first object has rotated once inside the turbine in a notification step (step S137).

[0282] Each aspect of the present invention may include the following modifications. A first object (rotor blade) rotates inside a turbine by a driving force generated by a turning tool 43 (drive device). While the first object is rotating, the CPU 18 repeatedly executes a region detection step (step S105), a region determination step (step S107), and a data generation step (step S109). When the rotation of the first object stops, the CPU 18 stops the data generation step and continues the region detection step and the region determination step.

[0283] Each aspect of the present invention may include the following modifications: When the first object (rotor blade) starts to rotate again, the CPU 18 restarts the data generating step (step S109).

[0284] Aspects of the present invention may include the following variations: The CPU 18 determines the rotational state of the first object (the rotor blade) by using images from two or more images of the internal components of the turbine.

[0285] Each aspect of the present invention may include the following modifications: The CPU 18 determines the rotation state of the first object (rotor blade) by monitoring the state of the turning tool 43 (drive device).

[0286] In the second embodiment, the endoscope device 1 notifies the user of examination status information regarding the status of the examination, thereby enabling the endoscope device 1 to avoid failures in the 3D reconstruction process and promote efficient examination.

[0287] Although the preferred embodiments of the present invention have been described above, the present invention is not limited to these embodiments and their modifications. Addition, omission, substitution, and other modifications of the configuration are possible within the scope of the spirit of the present invention. Furthermore, the present invention is not limited by the above description, but is limited only by the scope of the appended claims. [Explanation of symbols]

[0288] 1 Endoscopic device 2 Insertion section 3 Main body 4 Control section 5 Display section 8 Endoscope Unit 9 CCU 10 Control device 12 Video signal processing circuit 13 ROM 14 RAM 15 Card Interface 16 External device interface 17 Control Interface 18,18a CPU 20 Tip 21 Lens 28 Image sensor 180 Control Unit 181 Image acquisition unit 182 Area detection unit 183 Area Judgment Department 184 3D Reconstruction Department 185 Display control unit 186 Status determination unit 187 Notification Department

Claims

1. A three-dimensional data generation method for generating three-dimensional data representing a three-dimensional shape inside a turbine, comprising: an image acquisition step in which a processor acquires two or more images of components inside the turbine, the components including a first object movable inside the turbine and a second object stationary inside the turbine, the two or more images being generated by an imaging device having a tubular insertion part that captures light inside the turbine, the insertion part being inserted into the turbine through a hole formed in the turbine, the direction of movement of the insertion part when inserted into the turbine being different from the direction of movement of the first object, and while the first object is moving, the position of the first object relative to the insertion part being different each time the imaging device generates an image; a region detection step in which the processor detects two or more corresponding regions that are identical regions of the component element in at least two images included in the two or more images; a region determination step in which the processor determines whether at least a portion of a region in each of the two or more images is a changed region or a non-changed region, the changed region being a region of the component whose coordinates change in the image generated by the imaging device, and the non-changed region being a region of the component whose coordinates do not change in the image generated by the imaging device; a data generation step in which the processor generates the three-dimensional data by using the corresponding regions determined to be the changed regions among the two or more corresponding regions, without using the corresponding regions determined to be the non-changed regions among the two or more corresponding regions; and the first object is rotated inside the turbine by a driving force generated by a driving device; The three-dimensional data generation method includes: a rotation determining step in which the processor determines whether the first object has rotated one revolution inside the turbine; a notification step in which, when the processor determines that the first object has rotated once inside the turbine, the processor executes a process of notifying a user of information indicating that the first object has rotated once inside the turbine; Further having A method for generating three-dimensional data.

2. A three-dimensional data generation method for generating three-dimensional data representing a three-dimensional shape inside a turbine, comprising: an image acquisition step in which a processor acquires two or more images of components inside the turbine, the components including a first object movable inside the turbine and a second object stationary inside the turbine, the two or more images being generated by an imaging device having a tubular insertion part that captures light inside the turbine, the insertion part being inserted into the turbine through a hole formed in the turbine, the direction of movement of the insertion part when inserted into the turbine being different from the direction of movement of the first object, and while the first object is moving, the position of the first object relative to the insertion part being different each time the imaging device generates an image; a region detection step in which the processor detects two or more corresponding regions that are identical regions of the component element in at least two images included in the two or more images; a region determination step in which the processor determines whether at least a portion of a region in each of the two or more images is a changed region or a non-changed region, the changed region being a region of the component whose coordinates change in the image generated by the imaging device, and the non-changed region being a region of the component whose coordinates do not change in the image generated by the imaging device; a data generation step in which the processor generates the three-dimensional data by using the corresponding regions determined to be the changed regions among the two or more corresponding regions, without using the corresponding regions determined to be the non-changed regions among the two or more corresponding regions; and the first object is rotated inside the turbine by a driving force generated by a driving device; While the first object is rotating, the processor repeatedly executes the region detection step, the region determination step, and the data generation step; When the rotation of the first object stops, the processor stops the data generating step and continues the region detecting step and the region determining step. A method for generating three-dimensional data.

3. After the processor detects the two or more corresponding regions in the region detection step, the processor determines whether the at least some of the regions are the changed region or the non-changed region in the region determination step.

3. The three-dimensional data generating method according to claim 1.

4. After the processor determines whether the at least some of the regions are the changed regions or the non-changed regions in the region determination step, the processor detects the two or more corresponding regions by using the changed regions without using the non-changed regions in the region detection step.

3. The three-dimensional data generating method according to claim 1.

5. the first object includes a blade; The second object includes a stator blade or a shroud.

3. The three-dimensional data generating method according to claim 1.

6. In the two or more images, a portion of the first object is hidden by the second object.

3. The three-dimensional data generating method according to claim 1.

7. The second object includes an object that hides a part of the first object in the two or more images, and an object whose part is hidden by the first object in the two or more images. The three-dimensional data generating method according to claim 6.

8. the imaging device generates the two or more images at two or more different times, In the region determination step, the processor determines whether the at least some region is the changed region or the non-changed region based on an amount of movement of the corresponding region between at least two images included in the two or more images.

3. The three-dimensional data generating method according to claim 1.

9. In the region determination step, the processor determines whether the at least part of the region is the changed region or the non-changed region based on a difference in pixel values ​​between at least two images included in the two or more images.

3. The three-dimensional data generating method according to claim 1.

10. In the region determination step, the processor determines whether the at least part of the region is the changed region or the non-changed region by determining a subject appearing in one of the two or more images.

3. The three-dimensional data generating method according to claim 1.

11. a position determining step in which the processor determines a position of the insert within the turbine; a notification step in which, when the position of the insertion section has changed, the processor executes a process of notifying a user of information indicating the change in the position; 3. The three-dimensional data generating method according to claim 1, further comprising:

12. a calculation step of calculating an area of ​​the non-changed region in an image included in the two or more images before the processor executes the data generation step for the first time; a notification step in which the processor executes a process of notifying a user of a warning when the area is greater than a predetermined value; 3. The three-dimensional data generating method according to claim 1, further comprising:

13. When the first object begins to rotate again, the processor resumes the data generating step. The three-dimensional data generating method according to claim 2 .

14. The processor determines a rotational state of the first object by using images included in the two or more images. The three-dimensional data generating method according to claim 2 .

15. The processor determines the rotational state of the first object by monitoring the state of the drive unit. The three-dimensional data generating method according to claim 2 .

16. The insert is secured within the turbine.

3. The three-dimensional data generating method according to claim 1.

17. The imaging device is a borescope 3. The three-dimensional data generating method according to claim 1.

18. A three-dimensional data generation system that generates three-dimensional data representing a three-dimensional shape inside a turbine, an imaging device having a tubular insertion part that captures light inside the turbine and that generates two or more images of components inside the turbine, the components including a first object that is movable inside the turbine and a second object that is stationary inside the turbine, the two or more images being generated by the imaging device, the insertion part being inserted into the turbine through a hole formed in the turbine, the direction of movement of the insertion part when inserted into the turbine being different from the direction of movement of the first object, and the position of the first object relative to the insertion part while the first object is moving being different each time the imaging device generates an image; a three-dimensional data generating device including a processor; and The processor: acquiring the two or more images; detecting two or more corresponding regions that are identical regions of the component element in at least two images included in the two or more images; determining whether at least a portion of an area in each of the two or more images is a changed area or a non-changed area, the changed area being an area of ​​the component whose coordinates change in the image generated by the imaging device, and the non-changed area being an area of ​​the component whose coordinates do not change in the image generated by the imaging device; generating the three-dimensional data by using the corresponding areas determined to be the changed areas among the two or more corresponding areas, without using the corresponding areas determined to be the non-changed areas among the two or more corresponding areas; the first object is rotated inside the turbine by a driving force generated by a driving device; The processor: determining whether the first object has completed one revolution within the turbine; When it is determined that the first object has rotated once inside the turbine, a process is executed to notify a user of information indicating that the first object has rotated once inside the turbine. 3D data generation system.

19. A three-dimensional data generation system that generates three-dimensional data representing a three-dimensional shape inside a turbine, an imaging device having a tubular insertion part that captures light inside the turbine and that generates two or more images of components inside the turbine, the components including a first object that is movable inside the turbine and a second object that is stationary inside the turbine, the two or more images being generated by the imaging device, the insertion part being inserted into the turbine through a hole formed in the turbine, the direction of movement of the insertion part when inserted into the turbine being different from the direction of movement of the first object, and the position of the first object relative to the insertion part while the first object is moving being different each time the imaging device generates an image; a three-dimensional data generating device including a processor; and The processor: acquiring the two or more images; performing a region detection step of detecting two or more corresponding regions that are identical regions of the component element in at least two images included in the two or more images; a region determination step of determining whether at least a portion of a region in each of the two or more images is a changed region or a non-changed region, the changed region being a region of the component whose coordinates change in the image generated by the imaging device, and the non-changed region being a region of the component whose coordinates do not change in the image generated by the imaging device; a data generating step of generating the three-dimensional data by using the corresponding regions determined to be the changed regions among the two or more corresponding regions without using the corresponding regions determined to be the non-changed regions among the two or more corresponding regions; the first object is rotated inside the turbine by a driving force generated by a driving device; While the first object is rotating, the processor repeatedly executes the region detection step, the region determination step, and the data generation step; When the rotation of the first object stops, the processor stops the data generating step and continues the region detecting step and the region determining step. 3D data generation system.

20. The imaging device and the three-dimensional data generating device are included in an endoscope device.

20. The three-dimensional data generation system according to claim 18 or 19.

21. The imaging device is included in an endoscope device, and the three-dimensional data generating device is included in an external device separate from the endoscope device.

20. The three-dimensional data generation system according to claim 18 or 19.

22. A program for causing a computer to execute a process for generating three-dimensional data representing a three-dimensional shape of the interior of a turbine, an image acquisition step of acquiring two or more images of components inside the turbine, the components including a first object movable inside the turbine and a second object stationary inside the turbine, the two or more images being generated by an imaging device having a tubular insertion part that captures light inside the turbine, the insertion part being inserted into the turbine through a hole formed in the turbine, the direction of movement of the insertion part when inserted into the turbine being different from the direction of movement of the first object, and the position of the first object relative to the insertion part being different at each timing when the imaging device generates an image; a region detection step of detecting two or more corresponding regions that are identical regions of the component element in at least two images included in the two or more images; a region determination step of determining whether at least a portion of a region in each of the two or more images is a changed region or a non-changed region, wherein the changed region is a region of the component whose coordinates change in the image generated by the imaging device, and the non-changed region is a region of the component whose coordinates do not change in the image generated by the imaging device; a data generating step of generating the three-dimensional data by using the corresponding regions determined to be the changed regions among the two or more corresponding regions, without using the corresponding regions determined to be the non-changed regions among the two or more corresponding regions; causing the computer to execute the first object is rotated inside the turbine by a driving force generated by a driving device; a rotation determination step of determining whether the first object has rotated once inside the turbine; a notification step of executing a process of notifying a user of information indicating that the first object has rotated once inside the turbine when it is determined that the first object has rotated once inside the turbine; and a program for causing the computer to execute the above.

23. A program for causing a computer to execute a process for generating three-dimensional data representing a three-dimensional shape of the interior of a turbine, an image acquisition step of acquiring two or more images of components inside the turbine, the components including a first object movable inside the turbine and a second object stationary inside the turbine, the two or more images being generated by an imaging device having a tubular insertion part that captures light inside the turbine, the insertion part being inserted into the turbine through a hole formed in the turbine, the direction of movement of the insertion part when inserted into the turbine being different from the direction of movement of the first object, and the position of the first object relative to the insertion part being different at each timing when the imaging device generates an image; a region detection step of detecting two or more corresponding regions that are identical regions of the component element in at least two images included in the two or more images; a region determination step of determining whether at least a portion of a region in each of the two or more images is a changed region or a non-changed region, wherein the changed region is a region of the component whose coordinates change in the image generated by the imaging device, and the non-changed region is a region of the component whose coordinates do not change in the image generated by the imaging device; a data generating step of generating the three-dimensional data by using the corresponding regions determined to be the changed regions among the two or more corresponding regions, without using the corresponding regions determined to be the non-changed regions among the two or more corresponding regions; causing the computer to execute the first object is rotated inside the turbine by a driving force generated by a driving device; The computer, repeatedly executing the area detection step, the area determination step, and the data generation step while the first object is rotating; a program for stopping the data generating step and continuing the area detecting step and the area determining step when the rotation of the first object stops;

Images