Three-dimensional data generation system, three-dimensional data generation method, and program

Abstract

To provide a three-dimensional data generation system, a three-dimensional data generation method, and a program capable of avoiding a failure of processing for generating three-dimensional data showing a three-dimensional shape inside an object or a deterioration in the accuracy of the three-dimensional data.SOLUTION: A processor of a three-dimensional data generation system determines whether two or more sheets of images include a graphics area having graphics information superimposed thereon. When the two or more sheets of images do not include the graphics area, the processor executes first processing. The first processing includes generation processing for generating three-dimensional data by using the two or more sheets of images. When at least one sheet of an image among the two or more sheets of images includes the graphics area, the processor executes a second processing. The second processing includes processing for preventing the graphics area in the at least one sheet of the image from contributing to the generation of the three-dimensional data.SELECTED DRAWING: Figure 6

Description

[Technical field]

[0001] The present invention relates to a three-dimensional data generating system, a three-dimensional data generating method, and a program. [Background technology]

[0002] Industrial endoscopy devices are used to inspect for abnormalities (such as scratches and corrosion) occurring inside industrial equipment such as boilers, turbines, engines, and pipes. A variety of subjects are subject to inspection using industrial endoscopy devices.

[0003] In general, industrial endoscope devices use a monocular optical adapter and a measurement-only optical adapter. The monocular optical adapter is used for normal observation of the subject. The measurement-only optical adapter is used to restore three-dimensional (3D) information of the subject. For example, a measurement-only optical adapter is a stereo optical adapter with two fields of view. By using the 3D information, industrial endoscope devices can measure the size of any abnormalities found. By using the 3D information, users can check the restored shape of the subject (such as unevenness). In this way, 3D information contributes to higher quality and more efficient inspections.

[0004] In recent years, a technology has been developed that uses a monocular optical adapter to obtain two or more images of a subject and restores 3D information of the subject by using the images. Such a technology estimates the change in the relative movement of the tip of the endoscope with respect to the subject, and performs 3D reconstruction processing based on the estimation result to restore the 3D information. With such a technology, it is possible to obtain 3D information without replacing the monocular optical adapter with a stereo optical adapter. This improves the efficiency of the examination.

[0005] Industrial endoscope devices can superimpose characters or icons, called On-Screen Display (OSD) information, on still images and videos. Hereinafter, OSD information will be referred to as graphics information. Graphics information includes the type of optical adapter, image brightness, date, logo, zoom status, etc.

[0006] The position of the image area on which the graphics information is superimposed varies depending on the model of the endoscope device. The type of graphics information also varies depending on the type of endoscope device. For example, in an endoscope device with a long insertion section of 30 m, information on the insertion length indicating the observation position is important, so the insertion length is superimposed and displayed on the image. In an endoscope device with a short insertion section, the insertion length is not displayed. The user can set whether or not to superimpose each item of graphics information on the image by performing operations on the setting screen displayed on the monitor.

[0007] By looking at an image with the graphic information superimposed on it, a user can easily confirm the conditions under which the image was acquired, and therefore the graphic information is useful for reporting the results of an examination or for managing data.

[0008] Patent Document 1 discloses an apparatus that performs 3D reconstruction processing in the following manner. The apparatus uses two or more images acquired with a monocular optical adapter in use. The apparatus also detects characteristic micro-areas (feature areas) in the two or more images and analyzes the movement of the micro-areas. The micro-areas correspond to the same positions of the subject captured in each image. The apparatus performs 3D reconstruction processing according to the movement of the micro-areas. [Prior art documents] [Patent documents]

[0009] [Patent Document 1] JP 2020-134242 A Summary of the Invention [Problem to be solved by the invention]

[0010] When an image on which graphic information is superimposed is used in the above 3D reconstruction process, the following problem occurs. In areas where graphic information is not superimposed, the micro-area moves with the movement of the subject or the tip of the endoscope. However, in areas where graphic information is superimposed (graphics area), the graphic information does not change, so when a micro-area is detected in the graphics area, the micro-area does not move regardless of the movement of the subject or the tip of the endoscope. As a result, the 3D reconstruction process fails in the graphics area, or the 3D information is not accurately restored.

[0011] To solve the above problem, it is assumed that only the area other than the area at the same position as the graphics area is used for 3D reconstruction processing, regardless of whether graphics information is superimposed on the image. However, with this method, the area used for 3D reconstruction processing is narrow even in images without graphics information superimposed, so the area where 3D information is restored is narrow.

[0012] The present invention aims to provide a three-dimensional data generation system, a three-dimensional data generation method, and a program that can avoid failure in the process of generating three-dimensional data indicating the internal three-dimensional shape of an object or a decrease in the accuracy of the three-dimensional data. [Means for solving the problem]

[0013] The present invention is a three-dimensional data generation system that generates three-dimensional data indicating a three-dimensional shape inside an object, the three-dimensional data generation system comprising: an imaging device having a tubular insertion portion that acquires an optical image of the inside of the object and generates two or more images of the object with the insertion portion inserted inside the object; an image processing device that superimposes graphics information on at least one of the two or more images; and a processor, the processor acquires the two or more images, determines whether the two or more images include a graphics area on which the graphics information is superimposed, and executes a first process when it is determined that the two or more images do not include the graphics area, the first process including a generation process that generates the three-dimensional data using the two or more images, and when it is determined that at least one of the two or more images includes the graphics area, the processor executes a second process that includes a process that does not allow the graphics area in the at least one image to contribute to the generation of the three-dimensional data.

[0014] In the three-dimensional data generating system of the present invention, the second process includes a process of setting the graphics area in the at least one image as an invalid area.

[0015] In the three-dimensional data generating system of the present invention, the second process includes the generating process using an area other than the invalid area in the at least one image.

[0016] In the three-dimensional data generation system of the present invention, the second process includes a process of setting the graphics area as the invalid area based on setting information and position information, the setting information indicating an item corresponding to the graphics information superimposed on the at least one image, and the position information indicating the position of the graphics area on which the graphics information corresponding to the item is superimposed.

[0017] In the three-dimensional data generation system of the present invention, the second process includes a process of setting the graphics area as the invalid area based on model information, setting information, and position information, wherein the model information indicates the model of the imaging device, the setting information is prepared for each model of the imaging device and indicates an item corresponding to the graphics information superimposed on the at least one image, and the position information indicates the position of the graphics area on which the graphics information corresponding to the item is superimposed.

[0018] In the three-dimensional data generation system of the present invention, the processor accepts the model information and the setting information inputted through an input device.

[0019] In the three-dimensional data generation system of the present invention, the generation process includes a detection process for detecting feature points from the two or more images, and the second process includes a process for modifying the image in the invalid area so that feature points are not detected from the invalid area, and the generation process using the two or more images.

[0020] In the three-dimensional data generation system of the present invention, the generation process includes a detection process for detecting feature points from the two or more images, and the second process includes a process for excluding the invalid area from the area in which the detection process is performed.

[0021] In the three-dimensional data generation system of the present invention, the second process includes a process of stopping the execution of the generation process.

[0022] In the three-dimensional data generation system of the present invention, the generation process includes a detection process for detecting feature points from the two or more images, and the second process includes a process for deleting the feature points detected from the invalid area.

[0023] In the three-dimensional data generation system of the present invention, two or more images generated by the imaging device and on which the graphics information is not superimposed by the image processing device are stored in a storage medium, and the second processing includes the generation processing using the two or more images stored in the storage medium.

[0024] In the three-dimensional data generation system of the present invention, the processor determines whether or not the two or more images include the graphics area based on setting information indicating whether or not the graphics information is superimposed on each of the two or more images.

[0025] In the three-dimensional data generation system of the present invention, the processor performs image processing on the two or more images to detect the graphics area, and determines whether the two or more images contain the graphics area based on the results of the image processing.

[0026] In the three-dimensional data generation system of the present invention, the graphics information includes at least one of the type of optical adapter attached to the tip of the insertion portion, the brightness of the image generated by the imaging device, a date, a mark, and the magnification of the image generated by the imaging device.

[0027] In the three-dimensional data generation system of the present invention, after the two or more images are stored in a storage medium, the processor acquires the two or more images from the storage medium.

[0028] In the three-dimensional data generation system of the present invention, the insertion part has a lens and an image sensor.

[0029] In the three-dimensional data generation system of the present invention, the lens and the image sensor are built into the tip of the insertion part, the position of the tip is different from one another when the two or more images are generated, and the posture of the tip is different from one another when the two or more images are generated.

[0030] In the three-dimensional data generating system of the present invention, the imaging device, the image processing device, and the processor are included in an endoscope device.

[0031] In the three-dimensional data generation system of the present invention, the imaging device and the image processing device are included in an endoscope device, and the processor is included in an external device separate from the endoscope device.

[0032] In the three-dimensional data generation system of the present invention, the imaging device generates the two or more images based on an optical image formed through a monocular optical adapter attached to the tip of the insertion portion.

[0033] In the three-dimensional data generation system of the present invention, the same region of a component of the object is captured in at least two of the two or more images.

[0034] The present invention is a three-dimensional data generation method for generating three-dimensional data indicating a three-dimensional shape inside an object, the three-dimensional data generation method comprising the steps of: a processor acquiring two or more images of the object, the two or more images being generated by an imaging device, the imaging device having a tubular insertion portion for acquiring an optical image of the inside of the object, and generating the two or more images with the insertion portion inserted inside the object; a processor determining whether the two or more images include a graphics area on which graphics information is superimposed by an image processing device; a processor executing a first process when it is determined that the two or more images do not include the graphics area, the first process including a generation process for generating the three-dimensional data using the two or more images; and a processor executing a second process when it is determined that at least one image of the two or more images includes the graphics area, the second process including a process for not contributing the graphics area in the at least one image to the generation of the three-dimensional data.

[0035] The present invention is a program for causing a computer to execute a process of generating three-dimensional data indicating a three-dimensional shape inside an object, the program causing the computer to execute the following steps: acquiring two or more images of the object, the two or more images being generated by an imaging device, the imaging device having a tubular insertion portion for acquiring an optical image of the inside of the object, and generating the two or more images with the insertion portion inserted inside the object; determining whether the two or more images include a graphics area on which graphics information is superimposed by an image processing device; executing a first process when it is determined that the two or more images do not include the graphics area, the first process including a generation process for generating the three-dimensional data using the two or more images; and executing a second process when it is determined that at least one image of the two or more images includes the graphics area, the second process including a process for not contributing the graphics area in the at least one image to the generation of the three-dimensional data. Effect of the Invention

[0036] According to the present invention, the three-dimensional data generation system, three-dimensional data generation method, and program can avoid failure in the process of generating three-dimensional data indicating the internal three-dimensional shape of an object or a decrease in the accuracy of the three-dimensional data. [Brief description of the drawings]

[0037] [Figure 1] 1 is a perspective view showing an overall configuration of an endoscope apparatus according to a first embodiment of the present invention. [Diagram 2] 1 is a block diagram showing an internal configuration of an endoscope apparatus according to a first embodiment of the present invention. [Diagram 3] FIG. 1 is a diagram showing a schematic arrangement of rotor blades and stator blades in a turbine according to a first embodiment of the present invention. [Figure 4] FIG. 2 is a diagram showing a schematic arrangement of rotor blades in a turbine according to the first embodiment of the present invention. [Diagram 5]2 is a block diagram showing the functional configuration of a CPU included in the endoscope apparatus according to the first embodiment of the present invention. FIG. [Figure 6] 5 is a flowchart showing a procedure of a three-dimensional (3D) data generation process in the first embodiment of the present invention. [Figure 7] FIG. 4 is a diagram showing an example of a superposition state table in the first embodiment of the present invention. [Figure 8] FIG. 4 is a diagram showing an example of a coordinate table in the first embodiment of the present invention. [Figure 9] 5A to 5C are diagrams illustrating examples of positions of graphics areas in the first embodiment of the present invention. [Figure 10] 3 is a diagram showing an example of a screen of a display unit included in the endoscope apparatus according to the first embodiment of the present invention. FIG. [Figure 11] 3A to 3C are diagrams showing examples of graphics areas in an image generated by an imaging element included in the endoscope apparatus according to the first embodiment of the present invention. [Figure 12] FIG. 2 is a schematic diagram showing a situation in which an image is acquired in the first embodiment of the present invention. [Figure 13] 5 is a flowchart showing a procedure of 3D reconstruction processing in the first embodiment of the present invention. [Figure 14] 13 is a diagram showing an example of a screen of a display unit included in an endoscope apparatus according to the second embodiment of the present invention. FIG. [Figure 15] 13 is a diagram showing an example of a screen of a display unit included in an endoscope apparatus according to the second embodiment of the present invention. FIG. [Figure 16] FIG. 11 is a perspective view showing the overall configuration of an endoscope system according to a third embodiment of the present invention. [Figure 17] FIG. 11 is a block diagram showing a functional configuration of an external device according to a third embodiment of the present invention. [Figure 18] 10 is a flowchart showing a procedure of processing executed by an endoscope apparatus according to a third embodiment of the present invention. [Figure 19] 13 is a flowchart showing a procedure of a process executed by an external device according to a third embodiment of the present invention. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0038] Hereinafter, an embodiment of the present invention will be described with reference to the drawings.

[0039] (First embodiment) A first embodiment of the present invention will be described. In the first embodiment, an example in which a three-dimensional (3D) data generating device is included in an endoscope device will be described. In the following, an example in which the inspection object is a turbine will be described. The following example is also applicable to cases in which the inspection object is a boiler, a turbine, an engine, a pipe, or the like.

[0040] The configuration of an endoscope device 1 in a first embodiment will be described with reference to Figures 1 and 2. Figure 1 shows the external appearance of the endoscope device 1. Figure 2 shows the internal configuration of the endoscope device 1.

[0041] The endoscope device 1 shown in Fig. 1 has an insertion section 2, a main body section 3, an operation section 4, and a display section 5. The endoscope device 1 captures an image of a subject within an inspection object and generates an image. The subject is an industrial product. In order to inspect various inspection objects, a user can replace the optical adapter attached to the tip 20 of the insertion section 2, select a built-in image processing program, and add an image processing program.

[0042] The insertion section 2 is inserted into the inside of the object to be examined. The insertion section 2 is a long, thin tube that is bendable from the tip 20 to the base end. The insertion section 2 captures an image of the object and outputs an image signal to the main body section 3. An optical adapter is attached to the tip 20 of the insertion section 2. For example, a monocular optical adapter is attached to the tip 20. The main body section 3 is a control device that includes a storage section that stores the insertion section 2. The operation section 4 accepts user operations on the endoscope device 1. The display section 5 has a display screen, and displays an image of the object acquired by the insertion section 2, an operation menu, and the like on the display screen.

[0043] The operation unit 4 is a user interface. The display unit 5 is a monitor (display) such as an LCD (Liquid Crystal Display). The display unit 5 may be a touch panel. In that case, the operation unit 4 and the display unit 5 are integrated.

[0044] The main body 3 shown in FIG. 2 includes an endoscope unit 8, a CCU (Camera Control Unit) 9, and a control device 10.

[0045] The endoscope unit 8 has a light source device and a bending device, not shown. The light source device supplies illumination light required for observation to the tip 20. The bending device bends the bending mechanism built into the insertion portion 2.

[0046] The lens 21 and the image sensor 28 are built into the tip 20 of the insertion portion 2. The lens 21 is an observation optical system. The lens 21 captures an optical image of a subject formed by the optical adapter. The image sensor 28 is an image sensor. The image sensor 28 photoelectrically converts the optical image of the subject and generates an image signal. The lens 21 and the image sensor 28 form a monocular camera with one viewpoint.

[0047] The CCU 9 drives the imaging element 28. An imaging signal output from the imaging element 28 is input to the CCU 9. The CCU 9 performs pre-processing, including amplification and noise removal, on the imaging signal acquired by the imaging element 28. The CCU 9 converts the pre-processed imaging signal into a video signal such as an NTSC signal.

[0048] The control device 10 has a video signal processing circuit 12, a ROM (Read Only Memory) 13, a RAM (Random Access Memory) 14, a card interface 15, an external device interface 16, a control interface 17, and a CPU (Central Processing Unit) 18.

[0049] The video signal processing circuit 12 performs a predetermined video processing on the video signal output from the CCU 9. For example, the video signal processing circuit 12 performs video processing related to improving visibility. For example, the video processing includes color reproduction, gradation correction, noise suppression, and edge enhancement. For example, the video signal processing circuit 12 synthesizes the video signal output from the CCU 9 with information generated by the CPU 18. The video signal processing circuit 12 outputs the synthesized video signal to the display unit 5.

[0050] When the graphics information is output from the CPU 18, the video signal processing circuit 12 superimposes the graphics information on the video signal. The video signal processing circuit 12 outputs the video signal on which the graphics information is superimposed to the display unit 5. The video signal processing circuit 12 may be composed of at least one of a processor and a logic circuit, which will be described later.

[0051] The ROM 13 is a non-volatile recording medium in which a program is recorded for the CPU 18 to control the operation of the endoscope apparatus 1. The RAM 14 is a volatile recording medium in which information used by the CPU 18 to control the endoscope apparatus 1 is temporarily stored. The CPU 18 controls the operation of the endoscope apparatus 1 based on the program recorded in the ROM 13.

[0052] A memory card 42 is connected to the card interface 15. The memory card 42 is a recording medium that is detachable from the endoscope device 1. The card interface 15 imports the control processing information, image information, etc. stored in the memory card 42 into the control device 10. In addition, the card interface 15 records the control processing information, image information, etc. generated by the endoscope device 1 in the memory card 42.

[0053] An external device such as a USB device is connected to the external device interface 16. For example, a personal computer (PC) 41 is connected to the external device interface 16. The external device interface 16 transmits information to the PC 41 and receives information from the PC 41. This allows the PC 41 to display information. Also, a user can perform operations related to the control of the endoscope device 1 by inputting instructions to the PC 41.

[0054] The control interface 17 communicates with the operation unit 4, the endoscope unit 8, and the CCU 9 for operational control. The control interface 17 notifies the CPU 18 of information input to the operation unit 4 by a user. The control interface 17 outputs control signals for controlling the light source device and the bending device to the endoscope unit 8. The control interface 17 outputs control signals for controlling the image sensor 28 to the CCU 9.

[0055] The program executed by the CPU 18 may be recorded in a computer-readable recording medium. The program recorded in this recording medium may be read and executed by a computer other than the endoscope device 1. For example, the PC 41 may read and execute the program. The PC 41 may control the endoscope device 1 by transmitting control information for controlling the endoscope device 1 to the endoscope device 1 according to the program. Alternatively, the PC 41 may obtain a video signal from the endoscope device 1 and process the obtained video signal.

[0056] The insertion section 2 constitutes an imaging device (camera). The imaging element 28 may be disposed in the main body section 3, and an optical fiber may be disposed in the insertion section 2. Light incident on the lens 21 may reach the imaging element 28 through the optical fiber. A borescope may be used as the camera.

[0057] Turbines are used in aircraft engines or generators. There are gas turbines and steam turbines. The structure of a gas turbine is explained below. Hereinafter, a gas turbine is called a turbine.

[0058] A turbine has a compression section, a combustion chamber, and a turbine section. In the compression section, air is compressed. The compressed air is sent to the combustion chamber. Fuel is continuously burned in the combustion chamber, generating high-temperature and high-pressure gases. The gases expand in the turbine section, generating energy. The energy is used to rotate the compressor, and the remaining energy is extracted. In the compression and turbine sections, rotating blades fixed to the engine's rotating shaft and stationary vanes fixed to the casing are arranged alternately.

[0059] A turbine has components disposed within the interior space of the turbine. The components can be moving objects that can move within the turbine or stationary objects that are stationary within the turbine. The moving objects are rotor blades. The stationary objects are stator vanes or shrouds.

[0060] Figure 3 shows a schematic arrangement of rotor blades and stator vanes in the compression section of turbine TB10. Figure 3 shows a portion of a cross section of turbine TB10 through the rotation axis RA10 of the engine. Turbine TB10 has rotor blade RT10, stator vane ST10, rotor blade RT11, stator vane ST11, rotor blade RT12, stator vane ST12, rotor blade RT13, and stator vane ST13 in the compression section. These rotor blades and stator vanes rotate in a direction DR12 around the rotation axis RA10.

[0061] Air taken into the turbine TB10 flows in a direction DR11. The rotor blade RT10 is disposed in a low pressure section that takes in air. The rotor blade RT13 is disposed in a high pressure section that discharges air.

[0062] An access port AP10 is formed to enable inspection of the interior of the turbine TB10 without disassembling the turbine TB10. The turbine TB10 has two or more access ports, one of which is shown in Figure 3 as access port AP10. The access port AP10 is a hole formed in the turbine TB10.

[0063] The insertion portion 2 constitutes an endoscope. The insertion portion 2 is inserted into the turbine TB10 through an access port AP10. When the insertion portion 2 is inserted into the turbine TB10, the insertion portion 2 moves in a direction DR10. When the insertion portion 2 is pulled out from the turbine TB10, the insertion portion 2 moves in a direction opposite to the direction DR10. The direction DR10 is different from the direction DR12. Illumination light LT10 is emitted from the tip 20 of the insertion portion 2.

[0064] Figure 4 shows a schematic arrangement of two or more rotor blades RT10 viewed in a direction parallel to a rotation axis RA10. In Figure 4, eight rotor blades RT10 are arranged on a circular disk DS10. The rotation axis RA10 passes through the center of the disk DS10. The disk DS10 rotates in a direction DR12. Therefore, the eight rotor blades RT10 rotate in the direction DR12.

[0065] In practice, several tens or even more than 100 blades are arranged on one disk. The number of blades on one disk depends on the type of engine and on the location of the disk in the low-pressure to high-pressure region.

[0066] When inspecting turbine blades, a user rotates the disk manually or a device called a turning tool rotates the disk. An insert 2 is inserted into the turbine TB 10 through an access port AP 10 and a tip 20 is fixed. As the disk rotates, a user inspects two or more blades and determines whether each blade has an abnormality. This inspection is one of the main inspection items in turbine inspection.

[0067] The image sensor 28 generates two or more images. Each of the two or more images is temporally associated with the other images included in the two or more images. For example, each of the two or more images is a still image. A moving image may be used instead of a still image. The two or more images (frames) included in the moving image are associated with each other by a timestamp (time code).

[0068] The RAM 14 stores two or more images generated by the image sensor 28. The RAM 14 also stores parameters required for 3D reconstruction processing. The parameters include internal parameters of the camera, distortion correction parameters of the camera, setting values, scale information, etc. The setting values ​​are used for various processes to generate 3D data showing the 3D shape of the subject. The scale information is used to convert the scale of the 3D data into the actual scale of the subject.

[0069] The memory card 42 may store two or more images and the above parameters. The endoscope device 1 may read the two or more images and the parameters from the memory card 42, and store the two or more images and the parameters in the RAM 14.

[0070] The endoscope device 1 may perform wireless or wired communication with an external device via the external device interface 16. The external device is a PC 41 or a cloud server, etc. The endoscope device 1 may transmit two or more images generated by the imaging element 28 to the external device. The external device may store the two or more images and the above-mentioned parameters. The endoscope device 1 may receive the two or more images and parameters from the external device and store the two or more images and parameters in the RAM 14.

[0071] As described above, the endoscope device 1 has the imaging element 28 and the CPU 18. The imaging element 28 captures an image of a subject and generates an imaging signal. Thus, the imaging element 28 captures an image of the subject and generates the image. The image is a two-dimensional (2D) image. The image captured by the imaging element 28 is input to the CPU 18 via the video signal processing circuit 12.

[0072] Fig. 5 shows a functional configuration of CPU 18. CPU 18 functions as a control unit 180, an image acquisition unit 181, a graphics processing unit 182, a 3D data generation unit 183, a display control unit 184, and an operation processing unit 185. At least one of the blocks shown in Fig. 5 may be configured by a circuit different from CPU 18.

[0073] The control unit 180 controls the processes executed by each unit shown in FIG.

[0074] The image acquisition unit 181 acquires two or more images and the above parameters from the RAM 14. The image acquisition unit 181 may acquire two or more images and the above parameters from the memory card 42 or an external device via the external device interface 16.

[0075] The graphics processing unit 182 judges whether or not two or more images acquired by the image acquisition unit 181 include a graphics area on which graphics information is superimposed. If it is judged that an image includes a graphics area, the graphics processing unit 182 does not allow the graphics area in that image to contribute to the generation of 3D data.

[0076] The 3D data generating unit 183 executes 3D reconstruction processing by using two or more images acquired by the image acquiring unit 181, and generates 3D data.

[0077] The 3D data includes 3D coordinates of two or more points (3D point cloud) of the subject, camera coordinates, and posture information. The 3D data may include a mesh, which is a surface having the 3D point cloud as vertices, and mesh polygon data, which is a collection of texture information associated with the mesh. The camera coordinates indicate the 3D coordinates of a camera that has acquired each of the two or more images, and are associated with each of the two or more images. The camera coordinates are the 3D coordinates of the viewpoint when each image is acquired, and indicate the position of the camera. For example, the camera coordinates indicate the 3D coordinates of the observation optical system of the camera. The posture information indicates the posture of the camera that has acquired each of the two or more images, and is associated with each of the two or more images. For example, the posture information indicates the posture of the observation optical system of the camera.

[0078] The display control unit 184 controls the processing executed by the video signal processing circuit 12. The CCU 9 outputs a video signal. The video signal includes color data of each pixel of an image generated by the imaging element 28. The display control unit 184 causes the video signal processing circuit 12 to output the video signal output from the CCU 9 to the display unit 5. The video signal processing circuit 12 outputs the video signal to the display unit 5. The display unit 5 displays an image based on the video signal output from the video signal processing circuit 12. As a result, the display control unit 184 displays the image generated by the imaging element 28 on the display unit 5.

[0079] The display control unit 184 displays various information on the display unit 5. That is, the display control unit 184 displays various information on an image.

[0080] For example, the display control unit 184 generates various information. The various information includes images and graphics information of an operation screen. The display control unit 184 outputs the generated information to the video signal processing circuit 12. The video signal processing circuit 12 combines the video signal output from the CCU 9 and the information output from the CPU 18. This causes the various information to be superimposed on an image. The video signal processing circuit 12 outputs the combined video signal to the display unit 5. The display unit 5 displays the image on which the various information is superimposed.

[0081] Furthermore, the display control unit 184 generates image information of the 3D data. The display control unit 184 outputs the image information to the video signal processing circuit 12. The same processing as that described above is executed, and the display unit 5 displays an image of the 3D data. As a result, the display control unit 184 displays the image of the 3D data on the display unit 5.

[0082] A user inputs various information to the endoscope device 1 by operating the operation unit 4. The operation unit 4 outputs the information input by the user. The information is input to a control interface 17, which is an input unit. The information is output from the control interface 17 to the CPU 18. The operation processing unit 185 accepts the information input to the endoscope device 1 via the operation unit 4.

[0083] Each unit shown in FIG. 5 may be composed of at least one of a processor and a logic circuit. For example, the processor is at least one of a CPU (Central Processing Unit), a DSP (Digital Signal Processor), and a GPU (Graphics Processing Unit). For example, the logic circuit is at least one of an ASIC (Application Specific Integrated Circuit) and an FPGA (Field-Programmable Gate Array). Each unit shown in FIG. 5 may include one or more processors. Each unit shown in FIG. 5 may include one or more logic circuits.

[0084] The computer of the endoscope device 1 may load a program and execute the loaded program. The program includes instructions that define the operation of each unit shown in Fig. 5. In other words, the functions of each unit shown in Fig. 5 may be realized by software.

[0085] The above program may be provided by a "computer-readable recording medium" such as a flash memory. The program may be transmitted from a computer that holds the program to the endoscope device 1 via a transmission medium or by a transmission wave in the transmission medium. The "transmission medium" that transmits the program is a medium that has a function of transmitting information. Media that have a function of transmitting information include networks (communication networks) such as the Internet and communication lines (communication lines) such as telephone lines. The above program may realize some of the functions described above. Furthermore, the above program may be a difference file (difference program). The functions described above may be realized by a combination of a program already recorded in the computer and a difference program.

[0086] The graphics information is composed of letters, numbers, symbols, marks, etc. generated by a computer. In the following example, the graphics information includes sub-graphics information for two or more items. The two or more items are optical adapter type, brightness, date, company logo, or zoom state. The optical adapter type indicates the type of optical adapter attached to the tip 20. The brightness indicates the brightness of the image on which the graphics information is superimposed. The company logo is a specified mark of the company that manufactures or sells the endoscope device 1. The date indicates the date on which the image was generated. The zoom state indicates the magnification of the image. The sub-graphics information for each item is superimposed on the graphics area of ​​each item.

[0087] The graphics information is included in the header or footer of a video file as metadata, or in EXIF ​​(Exchangeable image file format) information added to a still image file.

[0088] The position of the image area onto which the graphics information is superimposed varies depending on the model of the inspection device that generated the image. The metadata of a moving image file or the EXIF ​​information of a still image file includes model information that indicates the model of the inspection device.

[0089] The diameter or length of the insertion portion 2 may differ depending on the type of the inspection device. Alternatively, the optical adapter may differ depending on the type of the inspection device. The diameter of the insertion portion 2, the length of the insertion portion 2, or information on the optical adapter may be used as type information.

[0090] The metadata of a moving image file or the EXIF ​​information of a still image file includes setting information indicating a setting of graphics information. The setting information includes information indicating whether or not the graphics information is superimposed on each image. In other words, the setting information includes information indicating whether or not each image includes a graphics area. When the setting information includes information indicating that the graphics information is superimposed on each image, the setting information includes information indicating whether or not the sub-graphics information of each item is superimposed on each image.

[0091] The characteristic processing of the first embodiment will be described below. In the following description, it is assumed that the 3D data is generated by using two or more images acquired by an endoscope device. The inspection device that acquires two or more images is not limited to an endoscope device. Any device may be used as long as an image of the internal components of the turbine is acquired by using a device having a camera.

[0092] The control device 10 functions as a 3D data generating device. The 3D data generating device of each aspect of the present invention may be a computer system such as a PC separate from the endoscope device. The 3D data generating device may be any of a desktop PC, a laptop PC, and a tablet terminal. The 3D data generating device may be a computer system that operates on the cloud.

[0093] The process executed by the endoscope device 1 will be described with reference to Fig. 6. Fig. 6 shows the procedure of the 3D data generation process executed by the endoscope device 1.

[0094] In the following, an example will be described in which a video with metadata stored in the header is used, although two or more still images may be used instead of the video.

[0095] The imaging element 28 continuously generates imaging signals. That is, the imaging element 28 generates imaging signals for each frame corresponding to a moving image. A moving image includes two or more frames. Each frame is composed of an image acquired by the imaging element 28. When the imaging element 28 finishes capturing images, a moving image file including the moving image is recorded in the PC 41 or the memory card 42.

[0096] In the following, an example will be described in which the 3D reconstruction process uses a moving image file recorded in the PC 41 or memory card 42. The CPU 18 may execute the 3D reconstruction process in real time at the same time that the imaging element 28 generates a moving image.

[0097] 6 is started, a moving image file is input from the PC 41 to the RAM 14 via the external device interface 16. Alternatively, a moving image file is input from the memory card 42 to the RAM 14 via the card interface 15. The image acquisition unit 181 acquires the moving image included in the moving image file from the RAM 14 (step S100).

[0098] After step S100, the graphics processing unit 182 acquires model information indicating the model of the inspection device from the header of the video file (step S101).

[0099] After step S101, the graphics processing unit 182 acquires setting information indicating whether or not the graphics information is superimposed on the moving image from the header of the moving image file (step S102). The setting information includes information indicating whether or not the graphics information is superimposed on the moving image.

[0100] After step S102, the graphics processing unit 182 refers to the setting information acquired in step S102 and determines whether or not the graphics information is superimposed on the moving image. As a result, the graphics processing unit 182 determines whether or not the moving image includes a graphics area (step S103). When the graphics processing unit 182 determines in step S103 that the graphics information is not superimposed on the moving image, step S108 described later is executed.

[0101] When the graphics processing unit 182 determines in step S103 that the graphics information is superimposed on the moving image, the graphics processing unit 182 acquires information indicating whether or not the sub-graphics information of each item is superimposed on each frame from the setting information (step S104). As described above, when the setting information includes information indicating that the graphics information is superimposed on the moving image, the setting information includes information indicating whether or not the sub-graphics information of each item is superimposed on each frame.

[0102] After step S104, the graphics processing unit 182 refers to the information acquired in step S104 and determines whether or not the sub-graphics information of each item is superimposed on each frame. The graphics processing unit 182 generates a superimposition state table indicating whether or not the sub-graphics information of each item is superimposed on each frame (step S105).

[0103] Fig. 7 shows an example of the superposition state table. The superposition state table shown in Fig. 7 includes model information IF1 and state information SI1 to SI6.

[0104] Model information IF1 indicates the model of the inspection equipment. Status information SI1 to SI6 constitute setting information. Status information SI1 indicates whether or not graphics information is superimposed on the moving image. Status information SI2 to SI6 indicate whether or not sub-graphics information of each item is superimposed on each frame. If status information SI1 indicates that graphics information is superimposed on the moving image, at least one of status information SI2 to SI6 indicates that sub-graphics information of each item is superimposed on each frame.

[0105] The status information SI2 indicates whether or not sub-graphics information indicating the optical adapter type is superimposed on each frame. The status information SI3 indicates whether or not sub-graphics information indicating the brightness of each frame is superimposed on each frame. The status information SI4 indicates whether or not sub-graphics information indicating the date is superimposed on each frame. The status information SI5 indicates whether or not sub-graphics information indicating a company logo is superimposed on each frame. The status information SI6 indicates whether or not sub-graphics information indicating the zoom state of each frame is superimposed on each frame.

[0106] In the example shown in Fig. 7, status information SI1 indicates that graphics information is superimposed on the video. Status information SI2 and SI3 indicate that sub-graphics information for each item is not superimposed on each frame. Status information SI4 to SI6 indicate that sub-graphics information for each item is superimposed on each frame.

[0107] After step S105, the graphics processing unit 182 identifies one or more graphics areas on which the sub-graphics information is superimposed (step S106).

[0108] The graphics processing unit 182 executes the following process in step S106. The graphics processing unit 182 acquires a coordinate table indicating the image coordinates of the area where the sub-graphics information of each item is superimposed. For example, the coordinate table is stored in the ROM 13.

[0109] FIG. 8 shows an example of a coordinate table. The coordinate table includes model information IF2 and coordinate information C1 to C5 associated with each other. The model information IF2 corresponds to the model information IF1 shown in FIG. 7 and indicates the model of the inspection device. The coordinate information C1 to C5 indicate the image coordinates of the area where the sub-graphics information of each item is superimposed. The coordinate information C1 indicates the image coordinates of the area where the sub-graphics information indicating the optical adapter type is superimposed. The coordinate information C2 indicates the image coordinates of the area where the sub-graphics information indicating the brightness of each frame is superimposed. The coordinate information C3 indicates the image coordinates of the area where the sub-graphics information indicating the date is superimposed. The coordinate information C4 indicates the image coordinates of the area where the sub-graphics information indicating the company logo is superimposed. The coordinate information C5 indicates the image coordinates of the area where the sub-graphics information indicating the zoom state of each frame is superimposed. For example, the coordinate information C1 to C5 include the image coordinates of the upper left point of the graphics area of ​​each item, the horizontal width of the graphics area, and the vertical width of the graphics area.

[0110] 9 shows an example of the positions of the graphics areas in image IMG1 generated by the imaging element 28. Image IMG1 includes graphics areas G1 to G5. Sub-graphics information indicating the optical adapter type is superimposed in graphics area G1. Sub-graphics information indicating the brightness of image IMG1 is superimposed in graphics area G2. Sub-graphics information indicating the date is superimposed in graphics area G3. Sub-graphics information indicating the company logo is superimposed in graphics area G4. Sub-graphics information indicating the zoom state of image IMG1 is superimposed in graphics area G5.

[0111] The graphics processing unit 182 acquires coordinate information corresponding to the type information in the superimposition state table and the state information in the superimposition state table from the coordinate table. For example, the graphics processing unit 182 searches the coordinate table for type information IF2 that is the same as type information IF1 shown in Fig. 7. The graphics processing unit 182 acquires at least one of coordinate information C1 to C5 associated with the found type information IF2. Specifically, the graphics processing unit 182 acquires coordinate information corresponding to state information indicating that sub-graphics information is superimposed on each frame.

[0112] In the superimposition state table shown in Fig. 7, state information SI4 to SI6 indicate that the sub-graphics information of each item is superimposed on each frame. The graphics processing unit 182 acquires coordinate information C3 of the date indicated by the state information SI4, acquires coordinate information C4 of the company logo indicated by the state information SI5, and acquires coordinate information C5 of the zoom state indicated by the state information SI6. The graphics processing unit 182 identifies the graphics area indicated by the coordinate information C3 to C5. For example, the graphics processing unit 182 identifies graphics areas G3 to G5 shown in Fig. 9.

[0113] After step S106, the graphics processing unit 182 sets one or more graphics regions identified in step S106 as invalid regions (step S107). Regions other than the invalid regions in the image generated by the imaging element 28 are valid regions used in 3D reconstruction processing.

[0114] As described later, the 3D reconstruction process includes a process of detecting feature points from at least two images. The graphics processing unit 182 changes the image in the invalid area so that feature points are not detected from the invalid area. For example, the graphics processing unit 182 changes the image in the invalid area to a black image. At this time, the graphics processing unit 182 changes the pixel values ​​of the image in the invalid area to values ​​corresponding to a black level. The graphics processing unit 182 may exclude the invalid area from the area where the process of detecting feature points is executed. The graphics processing unit 182 may delete feature points present in the invalid area after detecting feature points regardless of whether the area is a valid area or an invalid area.

[0115] After step S107, the 3D data generator 183 executes 3D reconstruction processing by using two or more frames included in the moving image, and generates 3D data (step S108). The 3D data generator 183 reads parameters required for the 3D reconstruction processing from the RAM 14, and uses the parameters for the 3D reconstruction processing.

[0116] If the graphics processing unit 182 determines in step S103 that the graphics information is not superimposed on the moving image, the entire area of ​​each of the two or more frames is set as the valid area. If the graphics processing unit 182 sets the graphics area as the invalid area in step S107, the area other than the invalid area in each of the two or more frames is set as the valid area. The 3D data generation unit 183 uses the valid area for 3D reconstruction processing.

[0117] After step S108, the display control unit 184 displays an image of the 3D data on the display unit 5 (step S109). When step S109 is executed, the 3D data generation process shown in FIG. 6 ends.

[0118] The graphics processing unit 182 may execute the following process instead of executing steps S101 and S102. The graphics processing unit 182 acquires n frames of a moving image. For example, n is 10. The graphics processing unit 182 divides the area of ​​each frame into small areas in a lattice shape at intervals of p pixels. For example, p is 8.

[0119] The graphics processing unit 182 compares pixel values ​​of pixels having the same coordinates in n frames. For example, the graphics processing unit 182 compares the RGB values ​​of a pixel having specific coordinates in a first frame with the RGB values ​​of a pixel having the specific coordinates in a second frame following the first frame. The graphics processing unit 182 compares the RGB values ​​of a pixel having specific coordinates in the second frame with the RGB values ​​of a pixel having the specific coordinates in a third frame following the second frame. The graphics processing unit 182 repeats this process to detect a small area including a pixel whose RGB values ​​do not change over the n frames. The graphics processing unit 182 determines that the detected small area is a graphics area.

[0120] Fig. 10 shows an example of a screen of the display unit 5. The display unit 5 displays a screen SC1 shown in Fig. 10. The screen SC1 includes a frame FR1, a button BT1, a button BT2, and a seek bar SB1.

[0121] The user operates buttons BT1 and BT2 by operating operation unit 4. When display unit 5 is configured as a touch panel, the user operates buttons BT1 and BT2 by touching the screen of display unit 5.

[0122] The user presses the button BT1 to play the video. After the button BT1 is pressed, a frame FR1 of the video is displayed. Figure 10 shows a screen SC1 that is displayed on the display unit 5 after the button BT1 is pressed.

[0123] The user may perform a predetermined operation on screen SC1 by operating operation unit 4 or touching the screen of display unit 5. When the predetermined operation is performed, an instruction to play or pause the video may be input to endoscope device 1. Screen SC1 may include a button for inputting an instruction to play or pause the video.

[0124] The seek bar SB1 indicates the position of the frame FR1. The user can change the position of the frame in the seek bar SB1 by operating the operation unit 4 or by touching the screen of the display unit 5. In addition, the user can specify a frame at which the 3D reconstruction process starts and a frame at which the 3D reconstruction process ends by operating the operation unit 4 or by touching the screen of the display unit 5.

[0125] In the above example, the user specifies a start frame at which the 3D reconstruction process starts and an end frame at which the 3D reconstruction process ends. The control unit 180 may automatically specify the start frame and the end frame. For example, the control unit 180 may detect a section of a video in which a subject is moving. Alternatively, the control unit 180 may detect a section of a video in which an abnormality such as a scratch is shown. The section includes two or more frames of the video. The control unit 180 may specify the first frame of the section as the start frame, and may specify the last frame of the section as the end frame.

[0126] Only one of the start frame and the end frame may be specified by the user. Alternatively, only one of the start frame and the end frame may be automatically specified. The method of setting the section including the frames used in the 3D reconstruction process is not limited to the above example.

[0127] The user presses the button BT2 to start the 3D reconstruction process. After the button BT2 is pressed, the 3D data generation process shown in FIG. 6 starts.

[0128] A brightness GI1, a date GI2, and a company logo GI3 are superimposed on a frame FR1 as graphics information. The graphics area where these are superimposed is set as an invalid area.

[0129] Fig. 11 shows an example of graphics areas in an image generated by the image sensor 28. Graphics areas R1 to R3 are superimposed on image IMG2. Image IMG2 corresponds to frame FR1 shown in Fig. 10. Graphics area R1 is an area where brightness GI1 shown in Fig. 10 is superimposed. Graphics area R2 is an area where date GI2 shown in Fig. 10 is superimposed. Graphics area R3 is an area where company logo GI3 shown in Fig. 10 is superimposed.

[0130] In all frames included in the video, graphics areas R1 to R3 are set as invalid areas. The position of each graphics area is the same in all frames. In addition, in all frames included in the video, the area excluding graphics areas R1 to R3 from the entire area is set as valid area.

[0131] The 3D data generating unit 183 executes the following process in step S108: Fig. 12 shows a schematic diagram of an image acquisition situation when two images are acquired.

[0132] As shown in Fig. 12, first, an image I1 is acquired in an imaging state c1 of the camera. Next, an image I2 is acquired in an imaging state c2 of the camera. At least one of the imaging position and the imaging attitude is different between the imaging states c1 and c2. In Fig. 12, both the imaging position and the imaging attitude are different between the imaging states c1 and c2.

[0133] In each embodiment of the present invention, it is assumed that the images I1 and I2 are acquired by the same endoscope. In addition, in each embodiment of the present invention, it is assumed that the parameters of the objective optical system of the endoscope do not change. The parameters of the objective optical system are focal length, distortion aberration, pixel size of the image sensor, etc. Hereinafter, the parameters of the objective optical system are abbreviated as internal parameters for convenience. Under such conditions, the internal parameters describing the characteristics of the optical system of the endoscope can be commonly used regardless of the position and posture of the camera (observation optical system). In each embodiment of the present invention, it is assumed that the internal parameters are acquired at the time of factory shipment or product delivery. The internal parameters may be acquired before the start of the inspection at the inspection site. In addition, in each embodiment of the present invention, it is assumed that the internal parameters are known when the image is acquired.

[0134] In each embodiment of the present invention, it is assumed that the image I1 and the image I2 are acquired by one endoscope. However, the present invention is not limited to this. For example, the present invention can be applied to a case where 3D data is generated using multiple videos acquired by multiple endoscopes. In this case, the image I1 and the image I2 are acquired using different endoscope devices, and it is sufficient that different internal parameters are held for each endoscope. Even if the internal parameters are unknown when generating the 3D data, calculations can be performed using the internal parameters as variables to be estimated. Therefore, the subsequent procedures do not change significantly depending on whether the internal parameters are known or not.

[0135] The 3D reconstruction process in step S108 will be described in detail with reference to Fig. 13. Fig. 13 shows the procedure of the 3D reconstruction process.

[0136] First, the 3D data generating unit 183 executes a feature point detection process (step S108a). In the feature point detection process, the 3D data generating unit 183 detects feature points of each of the two images. A feature point indicates a corner, an edge, or the like having a large image brightness gradient among the subject information captured in the image. A feature point may be composed of one pixel, which is the smallest unit of an image. Alternatively, a feature point may be composed of two or more pixels. The 3D data generating unit 183 detects feature points by using a descriptor of image features such as SIFT (Scale-invariant Feature Transform) or FAST (Features from Accelerated Segment Test).

[0137] Figure 12 shows the feature point P 11 is detected, and feature point P 12 The following is an example of the feature point P 11 and feature point P 12 corresponds to the coordinate P1 on the subject. Although only one feature point is shown for each image in Fig. 12, multiple feature points are actually detected in each image. The number of detected feature points may differ between images. Each feature point detected from each image is converted into data called a feature amount using a descriptor. A feature amount is data that represents the characteristics of a feature point.

[0138] After step S108a, the 3D data generation unit 183 executes a feature point association process (step S108b). In the feature point association process, the 3D data generation unit 183 compares the correlation of feature amounts between images for each feature point detected through the feature point detection process (step S108a). When the correlation of feature amounts is compared and feature points with similar feature amounts are found in each image, the 3D data generation unit 183 holds the information in the RAM 14. In this way, the 3D data generation unit 183 associates the feature points of each image with each other. On the other hand, when feature points with similar feature amounts are not found, the 3D data generation unit 183 discards the information of the feature points.

[0139] After step S108b, the 3D data generation unit 183 reads out the coordinates of feature points (feature point pair) of the two images associated with each other from the RAM 14. The coordinates are a pair of coordinates of feature points in each image. The 3D data generation unit 183 executes a position and orientation calculation process based on the read out coordinates (step S108c). In the position and orientation calculation process, the 3D data generation unit 183 calculates the relative position and orientation between the imaging state c1 of the camera that acquired the image I1 and the imaging state c2 of the camera that acquired the image I2. More specifically, the 3D data generation unit 183 calculates the matrix E by solving the following equation (1) that utilizes the epipolar constraint.

[0140]

number

[0141] The matrix E is called a fundamental matrix. The fundamental matrix E is a matrix that holds the relative position and orientation between the imaging state c1 of the camera that acquired the image I1 and the imaging state c2 of the camera that acquired the image I2. In equation (1), the matrix p1 is a matrix that includes the coordinates of the feature points detected from the image I1. The matrix p2 is a matrix that includes the coordinates of the feature points detected from the image I2. The fundamental matrix E includes information about the relative position and orientation of the camera, and therefore corresponds to the external parameters of the camera. The 3D data generating unit 183 can solve the fundamental matrix E by using a known algorithm.

[0142] As shown in FIG. 12, when the amount of change in camera position (relative position) is t and the amount of change in camera attitude (relative attitude) is R, formulas (2) and (3) hold true.

[0143]

number

[0144] In equation (2), the amount of movement in the x-axis direction is t x and the amount of movement in the y-axis direction is t yand the amount of movement in the z-axis direction is t z In equation (3), the amount of rotation around the x-axis, α, is R x (α), and the amount of rotation around the y-axis is R y (β), and the amount of rotation around the z-axis γ is R z After the fundamental matrix E is calculated, an optimization process called bundle adjustment may be performed to improve the accuracy of the reconstruction of the 3D coordinates.

[0145] The 3D data generation unit 183 calculates 3D coordinates (camera coordinates) in the coordinate system of the 3D data by using the calculated amount of change in position of the camera. For example, the 3D data generation unit 183 defines the 3D coordinates of the camera that acquired the image I1. The 3D data generation unit 183 calculates the 3D coordinates of the camera that acquired the image I2 based on the 3D coordinates of the camera that acquired the image I1 and the amount of change in position of the camera that acquired the image I2.

[0146] The 3D data generation unit 183 calculates posture information in the coordinate system of the 3D model by using the calculated posture change amount of the camera. For example, the 3D data generation unit 183 defines posture information of the camera that acquired the image I1. The 3D data generation unit 183 generates posture information of the camera that acquired the image I2 based on the posture information of the camera that acquired the image I1 and the posture change amount of the camera that acquired the image I2.

[0147] The 3D data generation unit 183 generates 3D shape data by executing a position and orientation calculation process (step S108c). The 3D shape data includes 3D coordinates (camera coordinates) at the position of the camera and orientation information indicating the orientation of the camera. When a method such as Structure from Motion or visual-SLAM is applied to the position and orientation calculation process (step S108c), the 3D data generation unit 183 further calculates the 3D coordinates of each feature point in step S108c. The 3D shape data generated in step S108c does not include 3D coordinates of points on the subject other than the feature points. Therefore, the 3D shape data indicates a sparse 3D shape of the subject.

[0148] The 3D shape data includes the 3D coordinates of each feature point, the above camera coordinates, and the above posture information. The 3D coordinates of each feature point are defined in the coordinate system of the 3D data. The 3D coordinates of each feature point are associated with the two-dimensional coordinates (2D coordinates) of each feature point. The 2D coordinates of each feature point are defined in the coordinate system of the image in which each feature point is included. The 2D and 3D coordinates of each feature point are associated with the image in which each feature point is included.

[0149] After step S108c, the 3D data generating unit 183 executes a 3D shape reconstruction process based on the relative position and orientation (position change amount t and orientation change amount R) of the camera calculated in step S108c (step S108d). The 3D data generating unit 183 generates 3D data of the subject in the 3D shape reconstruction process. Methods for restoring the 3D shape of the subject include patch-based multi-view stereo (PMVS) and matching processing using parallelized stereo. However, the means are not particularly limited.

[0150] In step S108d, the 3D data generation unit 183 calculates the 3D coordinates of points on the subject other than the feature points. The 3D coordinates of each point other than the feature points are defined in the coordinate system of the 3D data. The 3D coordinates of each point are associated with the 2D coordinates of each point. The 2D coordinates of each point are defined in the coordinate system of the 2D image in which each point is included. The 2D coordinates and 3D coordinates of each point are associated with the 2D image in which each point is included. The 3D data generation unit 183 updates the 3D shape data. The updated 3D shape data includes the 3D coordinates of each feature point, the 3D coordinates of each point other than the feature points, the camera coordinates, and the posture information. The 3D shape data updated in step S108d includes the 3D coordinates of points on the subject other than the feature points in addition to the 3D coordinates of the feature points. Therefore, the 3D shape data indicates a dense 3D shape of the subject.

[0151] After step S108d, the 3D data generating unit 183 executes a scale conversion process (step S108e) based on the 3D shape data processed in the 3D shape reconstruction process (step S108d) and the scale information read from the RAM 14. In the scale conversion process, the 3D data generating unit 183 converts the 3D shape data of the subject into 3D coordinate data (3D data) having a dimension of length. When step S108e is executed, the 3D reconstruction process shown in FIG. 13 ends.

[0152] To reduce processing time, step S108d may be omitted, in which case, after step S108c is performed, step S108e is performed without performing step S108d.

[0153] Step S108e may be omitted. In this case, after step S108d is executed, step S108e is not executed and the 3D reconstruction process shown in Fig. 13 ends. In this case, the 3D data indicates the relative shape of the object without the dimension of length.

[0154] In order for 3D data to be generated according to the principle shown in FIG. 12, at least a part of the area of ​​each image must be common to at least a part of the area of ​​at least one other image. That is, the area of ​​the first image and the area of ​​the second image different from the first image include a common area. The area other than the common area in the first image and the area other than the common area in the second image are different from each other. The two or more images used to generate the 3D data include at least two images in which the same area of ​​the subject is captured.

[0155] Image I1 and image I2 do not have to be two temporally consecutive frames in a video. There may be one or more frames between image I1 and image I2 in the video.

[0156] As described above, the graphics processing unit 182 sets one or more graphics regions as invalid regions in step S107 shown in Fig. 6. Specifically, the graphics processing unit 182 changes the image in the invalid region so that feature points are not detected from the graphics region. For example, the graphics processing unit 182 changes the image in the invalid region to a black image. Alternatively, the graphics processing unit 182 excludes the invalid region from the region in which the process of detecting feature points is executed. Alternatively, the graphics processing unit 182 deletes feature points that exist in the invalid region after detecting feature points regardless of whether the region is a valid region or an invalid region.

[0157] In the feature point detection process (step S108a) shown in FIG. 13, feature points are not detected in the black image. Alternatively, the feature point detection process (step S108a) is not executed in the invalid area. Alternatively, in the feature point detection process (step S108a), feature points are detected regardless of whether the area is a valid area or an invalid area, and then feature points existing in the invalid area are deleted. That is, feature points are not detected in the invalid area. Therefore, the invalid area is not used in the feature point association process (step S108b), the position and orientation calculation process (step S108c), and the 3D shape reconstruction process (step S108d) shown in FIG. 13. That is, the invalid area does not contribute to the generation of 3D data. The 3D data generation unit 183 generates 3D data of the subject without using the invalid area.

[0158] The 3D data generating unit 183 generates 3D data of the subject by using only the valid area other than the invalid area in the image on which the graphics information is superimposed. Therefore, the endoscope device 1 can generate 3D data with high accuracy.

[0159] 6, the graphics processing unit 182 may stop the execution of the 3D reconstruction process. In this case, the graphics processing unit 182 does not execute the 3D reconstruction process. Therefore, the endoscope device 1 can avoid a failure of the 3D reconstruction process or a decrease in the accuracy of the 3D data.

[0160] The video signal processing circuit 12 may output the first image and the second image to the CPU 18. The graphics information is not superimposed on the first image. The second image is generated by superimposing the graphics information on the first image. The CPU 18 may obtain the first image and the second image from the video signal processing circuit 12. The CPU 18 may record two or more first images and two or more second images in a storage medium. For example, the storage medium is a storage medium or a memory card 42 included in the PC 41.

[0161] The CPU 18 may use two or more second images to execute the 3D data generation process shown in Fig. 6. When the graphics processing unit 182 determines in step S103 that graphics information is superimposed on two or more second images, the graphics processing unit 182 may use two or more first images to execute the 3D data generation process. By using two or more first images in the 3D reconstruction process, the endoscope device 1 can avoid failure of the 3D reconstruction process or deterioration of the accuracy of the 3D data.

[0162] At least one of the two or more images used in the 3D reconstruction process may include graphics information. When the graphics processing unit 182 determines that the graphics area is not superimposed on any of the two or more images, the graphics processing unit 182 may execute the 3D reconstruction process using the two or more images. When the graphics processing unit 182 determines that the graphics area is superimposed on at least one of the two or more images, the graphics processing unit 182 may set the graphics area in the at least one image as an invalid area.

[0163] An endoscope device 1 (3D data generation system) according to each aspect of the present invention has an imaging device, a video signal processing circuit 12 (image processing device), and a CPU 18. The endoscope device 1 generates 3D data showing a 3D shape inside an object. The imaging device has a tubular insertion section 2 that acquires an optical image of the inside of the object. The imaging device generates two or more images of the object with the insertion section 2 inserted inside the object. The video signal processing circuit 12 superimposes graphics information on at least one of the two or more images.

[0164] The CPU 18 acquires two or more images and determines whether the two or more images include a graphics area on which graphics information is superimposed. When it is determined that the two or more images do not include a graphics area, the CPU 18 executes a first process. The first process includes a 3D reconstruction process (generation process) that generates 3D data using the two or more images. When it is determined that at least one image of the two or more images includes a graphics area, the CPU 18 executes a second process. The second process includes a process that does not allow the graphics area in the at least one image to contribute to the generation of 3D data.

[0165] The 3D data generation method according to each aspect of the present invention generates 3D data showing an internal 3D shape of an object. The 3D data generation method includes an image acquisition step, a determination step, a first processing step, and a second processing step.

[0166] In an image acquisition step (step S100), the CPU 18 acquires two or more images of an object. In a determination step (step S103), the CPU 18 determines whether or not the two or more images include a graphics area on which graphics information is superimposed by the video signal processing circuit 12. When it is determined that the two or more images do not include a graphics area, the CPU 18 executes a first process in a first processing step (step S108). The first process includes a 3D reconstruction process that generates 3D data using the two or more images. When it is determined that at least one image of the two or more images includes a graphics area, the CPU 18 executes a second process in a second processing step (step S107). The second process includes a process that does not allow the graphics area in the at least one image to contribute to the generation of 3D data.

[0167] Each aspect of the present invention may include the following modified examples: The second process includes a process of setting a graphics area in at least one image determined to include a graphics area as an invalid area.

[0168] Each aspect of the present invention may include the following variations: The second process includes a 3D reconstruction process that uses non-invalid regions in two or more images.

[0169] Each aspect of the present invention may include the following modified examples. The second process includes a process of setting a graphics area as an invalid area based on setting information and coordinate information (position information). The setting information indicates an item corresponding to graphics information superimposed on at least one image determined to include a graphics area. The coordinate information indicates the position of the graphics area on which the graphics information corresponding to the item is superimposed.

[0170] Each aspect of the present invention may include the following modified examples. The second process includes a process of setting a graphics area as an invalid area based on model information, setting information, and coordinate information (display position information). The model information indicates the model of the imaging device. The setting information is prepared for each model of the imaging device, and indicates an item corresponding to the graphics information superimposed on at least one image determined to include a graphics area. The coordinate information indicates the position of the graphics area on which the graphics information corresponding to the item is superimposed.

[0171] Each aspect of the present invention may include the following variations: The 3D reconstruction process includes a feature point detection process for detecting feature points from two or more images. The second process includes a process for modifying an image in an invalid region so that feature points are not detected from the invalid region, and a 3D reconstruction process using two or more images.

[0172] Each aspect of the present invention may include the following modifications: The 3D reconstruction process includes a feature point detection process for detecting feature points from two or more images. The second process includes a process for excluding invalid areas from an area where the feature point detection process is performed.

[0173] Each aspect of the present invention may include the following modifications: The 3D reconstruction process includes a feature point detection process for detecting feature points from two or more images. The second process includes a process for removing the detected feature points from invalid areas.

[0174] Each aspect of the present invention may include the following modifications: The second process includes a process of stopping the execution of the 3D reconstruction process.

[0175] Each aspect of the present invention may include the following modifications: Two or more images generated by an imaging device and not overlaid with graphics information by the video signal processing circuit 12 are stored in a storage medium. The second process includes a 3D reconstruction process using the two or more images stored in the storage medium.

[0176] Each aspect of the present invention may include the following modifications: CPU 18 determines whether or not two or more images include a graphics area based on setting information indicating whether or not graphics information is superimposed on each of the two or more images.

[0177] Each aspect of the present invention may include the following modifications: CPU 18 performs image processing on two or more images to detect a graphics area. CPU 18 determines whether or not the two or more images include a graphics area based on the results of the image processing.

[0178] Each aspect of the present invention may include the following modifications. The graphics information includes at least one of the type of optical adapter attached to the tip 20 of the insertion section 2, the brightness of the image generated by the imaging device, a date, a mark, and the magnification of the image generated by the imaging device. In addition, the items of the graphics information may differ depending on the model of the imaging device.

[0179] Each aspect of the present invention may include the following variations: After two or more images are stored in the storage medium, the CPU 18 retrieves the two or more images from the storage medium. That is, instead of performing the 3D data generation process at the same time that the imaging device generates the two or more images, the CPU 18 performs the 3D data generation process using the two or more images stored in the storage medium.

[0180] Each aspect of the present invention may include the following modifications: The insertion section 2 has a lens 21 and an imaging element 28 (image sensor).

[0181] Each aspect of the present invention may include the following modifications. The lens 21 and the imaging element 28 are built into the tip 20 of the insertion section 2. The positions of the tip 20 when two or more images are generated are different from each other. The attitudes of the tip 20 when two or more images are generated are different from each other. For example, the two or more images include a first image and a second image. The position of the tip 20 when the second image is generated is different from the position of the tip 20 when the first image is generated. The attitude of the tip 20 when the second image is generated is different from the attitude of the tip 20 when the first image is generated.

[0182] Each aspect of the present invention may include the following modifications: The imaging device, the video signal processing circuit 12, and the CPU 18 are included in the endoscope device 1.

[0183] Each aspect of the present invention may include the following modifications: The imaging device generates two or more images based on an optical image formed through a monocular optical adapter attached to the tip 20 of the insertion portion 2.

[0184] Each aspect of the present invention may include the following variations: The same region of a component of an object is captured in at least two of the two or more images.

[0185] In the first embodiment, when at least one of the two or more images includes a graphics area, the CPU 18 sets the graphics area as an invalid area, so that the endoscope device 1 can avoid a failure in the process of generating 3D data or a decrease in the accuracy of the 3D data.

[0186] Furthermore, the endoscope device 1 can generate 3D data while leaving graphic information useful for reporting examination results or managing data in the image, thereby making it possible for the endoscope device 1 to avoid a decrease in examination efficiency for generating 3D data.

[0187] When at least one of the two or more images includes a graphics area, the CPU 18 generates 3D data by using an area other than the graphics area in each of the two or more images, so that the endoscope device 1 can generate 3D data with high accuracy.

[0188] Second Embodiment A second embodiment of the present invention will be described. In the above-described first embodiment, the graphics processing unit 182 acquires model information indicating the model of the inspection equipment and setting information indicating whether or not the graphics information is superimposed on the video from the header of the video file. On the other hand, in the second embodiment, the user inputs the model information and setting information to the endoscope device 1, and the graphics processing unit 182 accepts the model information and setting information. In the second embodiment, the metadata stored in the header or footer of the video file does not need to include the model information and setting information.

[0189] In the second embodiment, the 3D data generation process shown in Fig. 6 is modified as follows. The user knows in advance the model of the endoscope device 1 as an inspection device. After the video is displayed on the display unit 5, the user inputs model information into the endoscope device 1 by operating the operation unit 4 or the touch panel. In step S101, the graphics processing unit 182 accepts the model information input through the operation unit 4 or the touch panel.

[0190] The user checks the frames of the moving image displayed on the display unit 5 and determines whether or not graphics information is superimposed on the frames. The user operates the operation unit 4 or the touch panel to input setting information to the endoscope device 1. In step S102, the graphics processing unit 182 accepts the setting information input through the operation unit 4 or the touch panel.

[0191] Fig. 14 shows an example of a screen of the display unit 5. The display unit 5 displays a screen SC2 shown in Fig. 14. The screen SC2 includes a frame FR1, a button BT1, a button BT2, a button BT3, and a seek bar SB1. A description of the same parts as those shown in Fig. 10 will be omitted.

[0192] The user presses the button BT3 to input model information and setting information to the endoscope device 1. After the button BT3 is pressed, the display unit 5 displays the screen SC3 shown in FIG. 15. When the image acquisition unit 181 acquires a video from the RAM 14, the display unit 5 may display the screen SC3 without displaying the screen SC2. The screen SC3 includes a pull-down menu PM1 and check boxes CB1 to CB5. The user operates the pull-down menu PM1 and the check boxes CB1 to CB5 by operating the operation unit 4 or touching the screen of the display unit 5.

[0193] The user operates the pull-down menu PM1 to select the model of the inspection equipment that is compatible with the currently used endoscope apparatus 1. The graphics processing unit 182 receives model information indicating the model selected by the user.

[0194] The user operates check boxes CB1 to CB5 to select the sub-graphics information item superimposed on each frame. Check box CB1 indicates the setting of sub-graphics information indicating the brightness of each frame. Check box CB2 indicates the setting of sub-graphics information indicating the company logo. Check box CB3 indicates the setting of sub-graphics information indicating the date. Check box CB4 indicates the setting of sub-graphics information indicating the optical adapter type. Check box CB5 indicates the setting of sub-graphics information indicating the zoom state of each frame.

[0195] Brightness GI1, date GI2, and company logo GI3 are superimposed on a frame FR1 shown in Fig. 14. Therefore, the user operates check boxes CB1 to CB3. The graphics processing unit 182 receives setting information indicating that sub-graphics information of items corresponding to each of the check boxes CB1 to CB3 is superimposed on each frame.

[0196] In step S103, the graphics processing unit 182 refers to the setting information and determines whether or not the graphics information is superimposed on the moving image. When the setting information indicates that sub-graphics information of an item corresponding to at least one of the check boxes CB1 to CB5 is superimposed on each frame, the graphics processing unit 182 determines that the graphics information is superimposed on the moving image. When the setting information indicates that sub-graphics information of an item corresponding to the check boxes CB1 to CB5 is not superimposed on each frame, the graphics processing unit 182 determines that the graphics information is not superimposed on the moving image.

[0197] When the graphics processing unit 182 determines in step S103 that the graphics information is superimposed on the moving image, the graphics processing unit 182 refers to the setting information in step S105 and determines whether or not the sub-graphics information of each item is superimposed on each frame. In step S105, the graphics processing unit 182 generates a superimposition state table indicating whether or not the sub-graphics information of each item is superimposed on each frame.

[0198] Each aspect of the present invention may include the following modifications: The CPU 18 accepts model information and setting information inputted through an input device. For example, the input device is the operation unit 4 or a touch panel.

[0199] In the second embodiment, even if the model information and the setting information are not previously associated with the video, the CPU 18 sets the graphics area as an invalid area based on the model information and the setting information input by the user through the input device, so that the endoscope device 1 can avoid a failure in the process of generating 3D data or a decrease in the accuracy of the 3D data.

[0200] (Third embodiment) A third embodiment of the present invention will be described. In the third embodiment, a device for acquiring an image of a subject and a device for generating 3D data are different. FIG. 16 shows the configuration of an endoscope system 100 (3D data generation system) of the third embodiment. The endoscope system 100 shown in FIG. 16 has an endoscope device 1 and an external device 6. The endoscope device 1 acquires two or more images of a subject and transmits the two or more images to the external device 6. The external device 6 receives the two or more images from the endoscope device 1 and performs 3D reconstruction processing by using the two or more images.

[0201] The configuration of the endoscope device 1 is similar to the configuration shown in Fig. 2. The external device interface 16 communicates with the external device 6. Specifically, the external device interface 16 transmits to the external device 6 two or more images of a subject.

[0202] For example, the external device interface 16 is a wireless module and performs wireless communication with the external device 6. The endoscope device 1 and the external device 6 may be connected by a cable such as a LAN (Local Area Network) cable, and the external device interface 16 may perform communication with the external device 6 via the cable.

[0203] For example, the external device 6 is a mobile terminal such as a tablet terminal. The external device 6 may be a fixed terminal. The form of the external device 6 is not limited to these.

[0204] 17 shows a functional configuration of the external device 6. The external device 6 shown in FIG.

[0205] The data communication unit 60 receives two or more images from the endoscope device 1. For example, the data communication unit 60 is a wireless module and performs wireless communication with the endoscope device 1. The data communication unit 60 may perform communication with the endoscope device 1 via a cable.

[0206] The CPU 61 has a similar configuration to the CPU 18 shown in Fig. 5. The CPU 61 also controls communication that the data communication unit 60 executes with the endoscope device 1. That is, the CPU 61 causes the data communication unit 60 to receive two or more images from the endoscope device 1.

[0207] The CPU 61 may load a program including instructions that define the operation of the CPU 61, and execute the loaded program. In other words, the functions of the CPU 61 may be realized by software. The method of implementing this program is the same as the method of implementing the program that realizes the functions of the endoscope device 1.

[0208] The display unit 62 has a display screen, and displays images, operation menus, etc. on the display screen. The display unit 62 is a monitor (display) such as an LCD.

[0209] The RAM 63 temporarily stores information that the CPU 61 uses to control the external device 6 .

[0210] The process executed by the endoscope device 1 will be described with reference to Fig. 18. Fig. 18 shows the procedure of the process executed by the endoscope device 1.

[0211] The user knows in advance the model of the endoscope apparatus 1 as an inspection device. The user operates the operation unit 4 or the touch panel to input model information to the endoscope apparatus 1. The CPU 18 accepts the model information input through the operation unit 4 or the touch panel (step S110).

[0212] The user operates the operation unit 4 or the touch panel to input information indicating whether or not graphics information is superimposed on the moving image to the endoscope device 1. After step S110, the CPU 18 accepts the information input through the operation unit 4 or the touch panel (step S111).

[0213] The user operates the operation unit 4 or the touch panel to input information indicating whether or not the sub-graphics information of each item is superimposed on each frame to the endoscope device 1. After step S111, the CPU 18 accepts the information input through the operation unit 4 or the touch panel (step S112).

[0214] After step S112, the image sensor 28 starts capturing images and generates a moving image (step S113). When the image sensor 28 finishes capturing images, a moving image file including the moving image is recorded in the PC 41 or the memory card 42.

[0215] After step S113, CPU 18 generates model information including the information received in step S110. CPU 18 also generates setting information including the information received in step S111 and the information received in step S112. CPU 18 records the model information and setting information in the header of the video file (step S114). This associates the model information and setting information with the video.

[0216] After step S114, CPU 18 reads the moving image file from PC 41 via external device interface 16. Alternatively, CPU 18 reads the moving image file from memory card 42 via card interface 15. CPU 18 transmits the moving image file to external device 6 via external device interface 16 (step S115). When step S115 is executed, the process shown in FIG. 18 ends.

[0217] The process executed by the external device 6 will be described with reference to Fig. 19. Fig. 19 shows the procedure of the process executed by the external device 6.

[0218] The CPU 61 causes the data communication unit 60 to receive the moving image file from the endoscope apparatus 1. As a result, the data communication unit 60 receives the moving image file from the endoscope apparatus 1 (step S200).

[0219] After step S200, the CPU 61 stores the video file received from the endoscope device 1 in the RAM 63 (step S201).

[0220] After step S201, step S202 is executed. As shown in Fig. 19, steps S202 to S211 are executed. Steps S202 to S211 are similar to steps S100 to S109 in Fig. 6, respectively. When step S211 is executed, the process shown in Fig. 19 ends.

[0221] The endoscope device 1 may transmit the moving image to the external device 6 at the same time that the imaging element 28 generates the moving image, and the CPU 61 may execute the 3D reconstruction process in real time at the same time that it receives the moving image from the endoscope device.

[0222] Various modifications that can be applied to the endoscope device 1 in the first and second embodiments can be similarly applied to the endoscope system 100 of the third embodiment.

[0223] Each aspect of the present invention may include the following modified examples: The imaging device and the video signal processing circuit 12 (image processing device) are included in the endoscope device 1. The CPU 61 is included in an external device 6 that is separate from the endoscope device 1.

[0224] In the third embodiment, the endoscope system 100 can avoid failure in the process of generating 3D data or deterioration in the accuracy of the 3D data.

[0225] Although the preferred embodiments of the present invention have been described above, the present invention is not limited to these embodiments and their modifications. Addition, omission, substitution, and other modifications of the configuration are possible without departing from the spirit of the present invention. Furthermore, the present invention is not limited by the above description, but is limited only by the scope of the attached claims. [Explanation of symbols]

[0226] 1 Endoscopic device 2 Insertion section 3 Main body 4 Control section 5,62 Display section 6 External device 8 Endoscope Unit 9 CCU 10 Control device 12 Video signal processing circuit 13 ROM 14,63 RAM 15 Card Interface 16 External device interface 17 Control Interface 18,61 CPU 20 Tip 21 Lens 28 Image sensor 60 Data Communications Department 100 Endoscope system 180 Control Unit 181 Image Acquisition Unit 182 Graphics Processing Unit 183 3D Data Generation Department 184 Display control section 185 Operation Processing Unit

Claims

1. A three-dimensional data generation system for generating three-dimensional data representing an internal three-dimensional shape of an object, comprising: an imaging device having a tubular insertion section for acquiring an optical image of the inside of the object, the imaging device generating two or more images of the object in a state in which the insertion section is inserted into the object; an image processing device that superimposes graphics information on at least one of the two or more images; A processor; having The processor, Acquiring the two or more images; determining whether the two or more images include a graphics area on which the graphics information is superimposed; when it is determined that the two or more images do not include the graphics area, a first process is executed, the first process including a generation process of generating the three-dimensional data using the two or more images; When it is determined that at least one image of the two or more images includes the graphics area, a second process is executed, the second process including a process of not allowing the graphics area in the at least one image to contribute to the generation of the three-dimensional data. 3D data generation system.

2. The second process includes a process of setting the graphics area in the at least one image as an invalid area. The three-dimensional data generating system according to claim 1 .

3. The second process includes the generation process using an area other than the invalid area in the two or more images. The three-dimensional data generating system according to claim 2 .

4. the second process includes a process of setting the graphics area as the invalid area based on setting information and position information; the setting information indicates an item corresponding to the graphics information superimposed on the at least one image, The position information indicates a position in the graphics area on which the graphics information corresponding to the item is superimposed. The three-dimensional data generating system according to claim 2 .

5. the second process includes a process of setting the graphics area as the invalid area based on model information, setting information, and position information; the model information indicates a model of the imaging device, the setting information is prepared for each model of the imaging device, and indicates items corresponding to the graphics information superimposed on the at least one image; The position information indicates a position in the graphics area on which the graphics information corresponding to the item is superimposed. The three-dimensional data generating system according to claim 2 .

6. The processor receives the model information and the setting information inputted through an input device. The three-dimensional data generating system according to claim 5 .

7. The generation process includes a detection process for detecting feature points from the two or more images, The second process includes: A process of modifying an image in the invalid area so that feature points are not detected from the invalid area; said generating process using said two or more images; Includes The three-dimensional data generating system according to claim 2 .

8. The generation process includes a detection process for detecting feature points from the two or more images, The second process includes a process of excluding the invalid area from an area where the detection process is performed. The three-dimensional data generating system according to claim 3 .

9. The generation process includes a detection process for detecting feature points from the two or more images, The second process includes a process of deleting the feature points detected from the invalid area. The three-dimensional data generating system according to claim 3 .

10. The second process includes a process of canceling the execution of the generation process. The three-dimensional data generating system according to claim 1 .

11. two or more images generated by the imaging device and on which the graphics information is not superimposed by the image processing device are stored in a storage medium; The second process includes the generation process using the two or more images stored in the storage medium. The three-dimensional data generating system according to claim 1 .

12. The processor determines whether the two or more images include the graphics area based on setting information indicating whether the graphics information is superimposed on each of the two or more images. The three-dimensional data generating system according to claim 1 .

13. The processor, performing image processing for detecting the graphics area on the two or more images; Based on the result of the image processing, it is determined whether the two or more images include the graphics area. The three-dimensional data generating system according to claim 1 .

14. The graphics information includes at least one of the type of the optical adapter attached to the tip of the insertion portion, the brightness of the image generated by the imaging device, a date, a mark, and a magnification of the image generated by the imaging device. The three-dimensional data generating system according to claim 1 .

15. After the two or more images are stored in a storage medium, the processor retrieves the two or more images from the storage medium. The three-dimensional data generating system according to claim 1 .

16. The insert has a lens and an image sensor. The three-dimensional data generating system according to claim 1 .

17. The lens and the image sensor are built into the tip of the insertion portion, the positions of the tip when the two or more images are generated are different from each other; The orientations of the tip when the two or more images are generated are different from each other. The three-dimensional data generating system according to claim 16.

18. The imaging device, the image processing device, and the processor are included in an endoscope device. The three-dimensional data generating system according to claim 1 .

19. The imaging device and the image processing device are included in an endoscope device, and the processor is included in an external device separate from the endoscope device. The three-dimensional data generating system according to claim 1 .

20. The imaging device generates the two or more images based on an optical image formed through a monocular optical adapter attached to the tip of the insertion portion. The three-dimensional data generating system according to claim 1 .

21. The same region of a component of the object is captured in at least two of the two or more images. The three-dimensional data generating system according to claim 1 .

22. A three-dimensional data generation method for generating three-dimensional data representing an internal three-dimensional shape of an object, comprising the steps of: a processor acquiring two or more images of the object, the two or more images being generated by an imaging device having a tubular insertion portion that acquires an optical image of the interior of the object, the imaging device generating the two or more images with the insertion portion inserted into the interior of the object; a step of the processor determining whether or not the two or more images include a graphics area on which graphics information has been superimposed by an image processing device; a step of executing a first process by the processor when it is determined that the two or more images do not include the graphics area, the first process including a generation process of generating the three-dimensional data using the two or more images; executing a second process by the processor when it is determined that at least one image of the two or more images includes the graphics area, the second process including a process of not allowing the graphics area in the at least one image to contribute to generation of the three-dimensional data; A three-dimensional data generating method comprising the steps of:

23. A program for causing a computer to execute a process for generating three-dimensional data representing a three-dimensional shape inside an object, acquiring two or more images of the object, the two or more images being generated by an imaging device having a tubular insertion portion for acquiring an optical image of the inside of the object, the imaging device generating the two or more images in a state in which the insertion portion is inserted into the inside of the object; determining whether the two or more images include a graphics area on which graphics information is superimposed by an image processing device; executing a first process when it is determined that the two or more images do not include the graphics area, the first process including a generation process of generating the three-dimensional data using the two or more images; executing a second process when it is determined that at least one image of the two or more images includes the graphics area, the second process including a process of not allowing the graphics area in the at least one image to contribute to generation of the three-dimensional data; A program for causing the computer to execute the above.

Images