Three-dimensional data generation system, three-dimensional data generation method, and recording medium

Abstract

The invention provides a three-dimensional data generation system, a three-dimensional data generation method, and a recording medium, which can avoid processing failure of generating three-dimensional data representing a three-dimensional shape inside an object or precision reduction of the three-dimensional data. A processor of a three-dimensional data generation system determines whether or not two or more images include a graphic region on which graphic information is superimposed. When the two or more images do not include the graphic region, the processor executes a first process. The first process includes a generation process for generating three-dimensional data using the two or more images. When at least one of the two or more images includes the graphic region, the processor executes a second process. The second process includes a process in which the graphic region in the at least one image does not contribute to the generation of the three-dimensional data.

Description

Technical Field

[0001] The present invention relates to a three-dimensional data generating system, a three-dimensional data generating method and a recording medium. Background Art

[0002] Industrial endoscopes are used to inspect abnormalities (damage, corrosion, etc.) occurring inside industrial equipment such as boilers, turbines, engines, and pipes. Various objects are inspected using industrial endoscopes.

[0003] Generally speaking, industrial endoscope devices use a monocular optical adapter and a measurement-specific optical adapter. The monocular optical adapter is used for normal observation of the subject. The measurement-specific optical adapter is used to restore the three-dimensional (3D) information of the subject. For example, the measurement-specific optical adapter is a stereoscopic optical adapter with two fields of view. By using 3D information, industrial endoscope devices can measure the size of abnormalities that are found. By using 3D information, users can confirm the restored shape (concavities and convexities, etc.) of the subject. In this way, 3D information helps to improve the quality and efficiency of inspections.

[0004] In recent years, a technology has been developed that uses a monocular optical adapter to obtain two or more images of a subject and uses the images to restore 3D information of the subject. In this technology, the change in the relative motion of the distal end of the endoscope with respect to the subject is estimated, and 3D reconstruction processing is performed based on the estimation result to restore 3D information. According to this technology, 3D information can be obtained without switching from a monocular optical adapter to a stereoscopic optical adapter. Therefore, inspection efficiency is improved.

[0005] Industrial endoscopes can superimpose characters or icons on static and dynamic images, which are called On-Screen Display (OSD) information. Hereinafter, OSD information is referred to as graphic information. Graphic information includes the type of optical adapter, image brightness, date, logo, zoom status, etc.

[0006] The position of the image area superimposed with the graphic information varies depending on the model of the endoscope device. In addition, the type of graphic information varies depending on the form of the endoscope device. For example, in an endoscope device having a strip insertion portion with a length of 30m, information indicating the insertion length of the observation position is important, and therefore, the insertion length is superimposed and displayed on the image. In an endoscope device having a short insertion portion, the insertion length is not displayed. The user can set whether to superimpose each item of graphic information on the image by operating the setting screen displayed on the monitor.

[0007] By observing the image with the graphic information superimposed thereon, the user can easily confirm the conditions under which the image was acquired. Therefore, the graphic information is useful in reporting of inspection results or in data management.

[0008] Patent document 1 discloses a device that performs 3D reconstruction processing by the following method. The device uses two or more images obtained in a state where a monocular optical adapter is used. In addition, the device detects a characteristic micro-region (feature region) in the two or more images and analyzes the movement of the micro-region. The micro-region corresponds to the same position of the subject photographed in each image. The device performs 3D reconstruction processing according to the movement of the micro-region.

[0009] Patent Document 1: Japanese Patent Application Publication No. 2020-134242

[0010] When an image superimposed with graphic information is used for the above-mentioned 3D reconstruction process, the following problem arises. In the area where the graphic information is not superimposed, the micro-area moves with the movement of the subject or the front end of the endoscope. However, in the area where the graphic information is superimposed (graphic area), the graphic information does not change. Therefore, when the micro-area is detected in the graphic area, the micro-area does not move regardless of the movement of the subject or the front end of the endoscope. As a result, the 3D reconstruction process fails in the graphic area, or the 3D information is not accurately restored.

[0011] In order to solve the above problem, it is assumed that, regardless of whether or not graphic information is superimposed on the image, only the area other than the area at the same position as the graphic area is used in the 3D reconstruction process. However, in this method, in the image without graphic information superimposed, the area used for the 3D reconstruction process is also small, and therefore, the area where the 3D information is restored is small. Summary of the invention

[0012] An object of the present invention is to provide a three-dimensional data generation system, a three-dimensional data generation method and a recording medium, which can avoid the failure of the process of generating three-dimensional data representing the three-dimensional shape inside an object or the reduction of the accuracy of the three-dimensional data.

[0013] The three-dimensional data generation system of the present invention generates three-dimensional data representing the three-dimensional shape inside an object. The three-dimensional data generation system includes an imaging device, an image processing device, and a processor. The imaging device includes a tubular insertion portion for acquiring an optical image inside the object, and generates two or more images of the object when the insertion portion is inserted into the object. The image processing device superimposes graphic information on at least one of the two or more images. The processor acquires the two or more images and determines whether the two or more images include a graphic area superimposed with the graphic information. When it is determined that the two or more images do not include the graphic area, the processor performs a first process. The first process includes a generation process of generating the three-dimensional data using the two or more images. When it is determined that at least one of the two or more images includes the graphic area, the processor performs a second process. The second process includes a process of making the graphic area in the at least one image not contribute to the generation of the three-dimensional data.

[0014] In the three-dimensional data generating system of the present invention, the second process may include a process of setting the graphic area in the at least one image as an invalid area.

[0015] In the three-dimensional data generating system of the present invention, the second processing may include the generating processing using an area other than the invalid area in the at least one image.

[0016] In the three-dimensional data generation system of the present invention, the second process may also include a process of setting the graphic area as the invalid area based on setting information and position information. The setting information may also indicate an item corresponding to the graphic information superimposed on the at least one image. The position information may also indicate a position of the graphic area superimposed with the graphic information corresponding to the item.

[0017] In the three-dimensional data generation system of the present invention, the second processing may also include a process of setting the graphic area as the invalid area based on model information, setting information and position information. The model information may also indicate the model of the camera. The setting information may also be prepared according to each model of the camera, and may also indicate an item corresponding to the graphic information superimposed on the at least one image. The position information may also indicate the position of the graphic area superimposed with the graphic information corresponding to the item.

[0018] In the three-dimensional data generating system of the present invention, the processor may receive the model information and the setting information inputted through an input device.

[0019] In the three-dimensional data generation system of the present invention, the generation process may also include a detection process of detecting feature points from the two or more images. The second process may also include: a process of changing the image in the invalid area so that feature points are not detected from the invalid area; and the generation process using the two or more images.

[0020] In the three-dimensional data generation system of the present invention, the generation process may include a detection process of detecting feature points from the two or more images. The second process may include a process of excluding the invalid area from the area to be subjected to the detection process.

[0021] In the three-dimensional data generating system of the present invention, the second process may include a process of suspending execution of the generating process.

[0022] In the three-dimensional data generation system of the present invention, the generation process may include a detection process of detecting feature points from the two or more images. The second process may include a process of deleting feature points detected from the invalid area.

[0023] In the three-dimensional data generation system of the present invention, the two or more images generated by the camera device and not superimposed with the graphic information by the image processing device may also be stored in a storage medium. The second processing may also include the generation processing using the two or more images stored in the storage medium.

[0024] In the three-dimensional data generation system of the present invention, the processor may determine whether the two or more images include the graphic area based on setting information indicating whether the graphic information is superimposed on each of the two or more images.

[0025] In the three-dimensional data generation system of the present invention, the processor may also perform image processing to detect the graphic area on the two or more images, and may also determine whether the two or more images contain the graphic area based on the result of the image processing.

[0026] In the three-dimensional data generation system of the present invention, the graphic information may also include at least one of the type of the optical adapter mounted on the front end of the insertion portion, the brightness of the image generated by the camera device, the date, the mark, and the magnification of the image generated by the camera device.

[0027] In the three-dimensional data generating system of the present invention, after the two or more images are stored in a storage medium, the processor may obtain the two or more images from the storage medium.

[0028] In the three-dimensional data generating system of the present invention, the insertion unit may include a lens and an image sensor.

[0029] In the three-dimensional data generation system of the present invention, the lens and the image sensor may be built into the front end of the insertion portion. The position of the front end when the two or more images are generated may also be different from each other. The posture of the front end when the two or more images are generated may also be different from each other.

[0030] In the three-dimensional data generating system of the present invention, the imaging device, the image processing device, and the processor may be included in an endoscope device.

[0031] In the three-dimensional data generating system of the present invention, the imaging device and the image processing device may be included in an endoscope device. The processor may be included in an external device separate from the endoscope device.

[0032] In the three-dimensional data generating system of the present invention, the imaging device may generate the two or more images based on an optical image formed by an optical adapter of a single eye mounted on a distal end of the insertion portion.

[0033] In the three-dimensional data generating system of the present invention, at least two images included in the two or more images may capture a same area of ​​a structural element of the object.

[0034] The present invention is a three-dimensional data generation method for generating three-dimensional data representing the three-dimensional shape inside an object. The three-dimensional data generation method includes the step of a processor acquiring two or more images of the object. The two or more images are generated by an imaging device. The imaging device has a tubular insertion portion for acquiring an optical image inside the object, and generates the two or more images when the insertion portion is inserted into the object. The three-dimensional data generation method includes the following steps: the processor determines whether the two or more images include a graphic area on which graphic information is superimposed by an image processing device. The three-dimensional data generation method includes the following steps: when it is determined that the two or more images do not include the graphic area, the processor performs a first process. The first process includes a generation process for generating the three-dimensional data using the two or more images. The three-dimensional data generation method includes the following steps: when it is determined that at least one of the two or more images includes the graphic area, the processor performs a second process. The second process includes a process for making the graphic area in the at least one image not contribute to the generation of the three-dimensional data.

[0035] The present invention is a computer-readable recording medium, which records a program for causing a computer to execute processing, the program generating three-dimensional data representing the three-dimensional shape inside an object. The processing includes the step of obtaining two or more images of the object. The two or more images are generated by a camera device. The camera device has a tubular insertion portion for obtaining an optical image of the inside of the object, and generates the two or more images when the insertion portion is inserted into the inside of the object. The processing includes the following steps: judging whether the two or more images include a graphic area superimposed with graphic information by an image processing device. The processing includes the following steps: when it is judged that the two or more images do not include the graphic area, executing a first processing. The first processing includes a generation processing of generating the three-dimensional data using the two or more images. The processing includes the following steps: when it is judged that at least one of the two or more images includes the graphic area, executing a second processing. The second processing includes a processing of making the graphic area in the at least one image not contribute to the generation of the three-dimensional data.

[0036] Effects of the Invention

[0037] According to the present invention, a three-dimensional data generation system, a three-dimensional data generation method, and a recording medium can avoid a processing failure of generating three-dimensional data representing a three-dimensional shape inside an object or a reduction in the accuracy of the three-dimensional data. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 It is a perspective view showing the overall structure of the endoscope device according to the first embodiment of the present invention.

[0039] Figure 2 This is a block diagram showing the internal structure of the endoscope device according to the first embodiment of the present invention.

[0040] Figure 3 It is a diagram schematically showing the arrangement of moving blades and stationary blades in a turbine according to the first embodiment of the present invention.

[0041] Figure 4 It is a diagram schematically showing the arrangement of rotor blades in a turbine according to the first embodiment of the present invention.

[0042] Figure 5 This is a block diagram showing the functional configuration of a CPU included in the endoscope apparatus according to the first embodiment of the present invention.

[0043] Figure 6 This is a flowchart showing the procedure of three-dimensional (3D) data generation processing in the first embodiment of the present invention.

[0044] Figure 7 It is a diagram showing an example of a superimposition state table in the first embodiment of the present invention.

[0045] Figure 8 It is a diagram showing an example of a coordinate table in the first embodiment of the present invention.

[0046] Fig. 9 It is a diagram showing an example of the position of the graphic region in the first embodiment of the present invention.

[0047] Fig.10 It is a diagram showing an example of a screen of a display unit included in the endoscope apparatus according to the first embodiment of the present invention.

[0048] Fig.11 It is a diagram showing an example of a graphic region in an image generated by the imaging element included in the endoscope apparatus according to the first embodiment of the present invention.

[0049] Fig.12 It is a schematic diagram showing a state of image acquisition in the first embodiment of the present invention.

[0050] Fig.13 This is a flowchart showing the procedure of 3D reconstruction processing in the first embodiment of the present invention.

[0051] Fig.14 It is a diagram showing an example of a screen of a display unit included in an endoscope apparatus according to a second embodiment of the present invention.

[0052] Fig.15 It is a diagram showing an example of a screen of a display unit included in an endoscope apparatus according to a second embodiment of the present invention.

[0053] Fig.16 It is a perspective view showing the overall structure of an endoscope system according to a third embodiment of the present invention.

[0054] Fig.17 This is a block diagram showing the functional configuration of an external device according to the third embodiment of the present invention.

[0055] Fig.18 This is a flowchart showing the procedure of processing executed by the endoscope device according to the third embodiment of the present invention.

[0056] Fig.19 This is a flowchart showing the procedure of a process executed by the external device according to the third embodiment of the present invention.

[0057] Description of symbols

[0058] 1 Endoscopic device

[0059] 2 Insertion

[0060] 3 Main body

[0061] 4 Operation section

[0062] 5.62 Display unit

[0063] 6 External devices

[0064] 8 Endoscope unit

[0065] 9CCU

[0066] 10 Control Device

[0067] 12 Image signal processing circuit

[0068] 13ROM

[0069] 14. 63RAM

[0070] 15 Card interface

[0071] 16 External device interface

[0072] 17 Control Interface

[0073] 18.61CPU

[0074] 20 Frontend

[0075] 21 Lens

[0076] 28 Camera Components

[0077] 60 Data Communications Department

[0078] 100 Endoscopic System

[0079] 180 Control Department

[0080] 181 Image acquisition unit

[0081] 182 Graphics Processing Department

[0082] 183 3D Data Generation Department

[0083] 184 Display control unit

[0084] 185 Operation Processing Department DETAILED DESCRIPTION

[0085] Hereinafter, embodiments of the present invention will be described with reference to the drawings.

[0086] (First embodiment)

[0087] The first embodiment of the present invention is described. In the first embodiment, an example in which a three-dimensional (3D) data generating device is included in an endoscope device is described. Next, an example in which the inspection object is a turbine is described. The following example can also be applied to a case in which the inspection object is a boiler, a turbine, an engine, or a pipe.

[0088] use Figure 1 and Figure 2 The configuration of the endoscope device 1 according to the first embodiment will be described. Figure 1 The external appearance of the endoscope device 1 is shown. Figure 2 The internal structure of the endoscope device 1 is shown.

[0089] Figure 1 The endoscope device 1 shown has an insertion portion 2, a main body 3, an operation portion 4, and a display portion 5. The endoscope device 1 captures an object in an inspection object and generates an image. The object is an industrial product. The user can replace the optical adapter mounted on the front end 20 of the insertion portion 2, select the built-in image processing program, and add the image processing program to perform inspections of various inspection objects.

[0090] The insertion section 2 is inserted into the inside of the inspection object. The insertion section 2 is an elongated tube that can be bent from the front end 20 to the base end. The insertion section 2 takes an image of the object and outputs an image signal to the main body 3. An optical adapter is mounted on the front end 20 of the insertion section 2. For example, a monocular optical adapter is mounted on the front end 20. The main body 3 is a control device having a storage portion for storing the insertion section 2. The operation section 4 accepts user operations on the endoscope device 1. The display section 5 has a display screen, and displays an image of the object obtained by the insertion section 2 and an operation menu on the display screen.

[0091] The operation unit 4 is a user interface. The display unit 5 is a monitor (display) such as an LCD (Liquid Crystal Display). The display unit 5 may be a touch panel. In this case, the operation unit 4 and the display unit 5 are integrated.

[0092] Figure 2 The main body 3 shown includes an endoscope unit 8 , a CCU (Camera Control Unit) 9 , and a control device 10 .

[0093] The endoscope unit 8 includes a light source device and a bending device (not shown). The light source device supplies illumination light required for observation to the distal end 20. The bending device bends a bending mechanism built into the insertion portion 2.

[0094] The lens 21 and the imaging element 28 are built into the front end 20 of the insertion part 2. The lens 21 is an observation optical system. The lens 21 takes in the optical image of the subject formed by the optical adapter. The imaging element 28 is an image sensor. The imaging element 28 performs photoelectric conversion on the optical image of the subject and generates an imaging signal. The lens 21 and the imaging element 28 constitute a single-eye camera with one viewpoint.

[0095] The CCU 9 drives the imaging element 28. The imaging signal output from the imaging element 28 is input to the CCU 9. The CCU 9 performs preprocessing including amplification and noise removal on the imaging signal acquired by the imaging element 28. The CCU 9 converts the preprocessed imaging signal into a video signal such as an NTSC signal.

[0096] The control device 10 includes a video signal processing circuit 12 , a ROM (Read Only Memory) 13 , a RAM (Random Access Memory) 14 , a card interface 15 , an external device interface 16 , a control interface 17 , and a CPU (Central Processing Unit) 18 .

[0097] The image signal processing circuit 12 performs a predetermined image processing on the image signal output from the CCU 9. For example, the image signal processing circuit 12 performs image processing related to improving visual recognition. For example, the image processing is color reproduction, grayscale correction, noise suppression, and contour emphasis. For example, the image signal processing circuit 12 synthesizes the image signal output from the CCU 9 and the information generated by the CPU 18. The image signal processing circuit 12 outputs the synthesized image signal to the display unit 5.

[0098] When the graphics information is output from the CPU 18, the video signal processing circuit 12 superimposes the graphics information on the video signal. The video signal processing circuit 12 outputs the video signal superimposed with the graphics information to the display unit 5. The video signal processing circuit 12 may be composed of at least one of a processor and a logic circuit described later.

[0099] The ROM 13 is a nonvolatile recording medium recording a program for the CPU 18 to control the operation of the endoscope device 1. The RAM 14 is a volatile recording medium temporarily storing information used by the CPU 18 to control the endoscope device 1. The CPU 18 controls the operation of the endoscope device 1 according to the program recorded in the ROM 13.

[0100] The memory card 42 is connected to the card interface 15. The memory card 42 is a recording medium that can be attached to and detached from the endoscope device 1. The card interface 15 takes the control processing information and image information stored in the memory card 42 into the control device 10. In addition, the card interface 15 records the control processing information and image information generated by the endoscope device 1 in the memory card 42.

[0101] An external device such as a USB device is connected to the external device interface 16. For example, a personal computer (PC) 41 is connected to the external device interface 16. The external device interface 16 sends information to the PC 41 and receives information from the PC 41. Thus, the PC 41 can display information. In addition, the user can perform operations related to the control of the endoscope device 1 by inputting instructions to the PC 41.

[0102] The control interface 17 communicates with the operation unit 4, the endoscope unit 8, and the CCU 9 for operation control. The control interface 17 notifies the CPU 18 of information input by the user to the operation unit 4. The control interface 17 outputs a control signal for controlling the light source device and the bending device to the endoscope unit 8. The control interface 17 outputs a control signal for controlling the imaging element 28 to the CCU 9.

[0103] The program executed by the CPU 18 may also be recorded in a computer-readable recording medium. The program recorded in the recording medium may also be read and executed by a computer other than the endoscope device 1. For example, the program may also be read and executed by the PC 41. The PC 41 may also send control information for controlling the endoscope device 1 to the endoscope device 1 according to the program, thereby controlling the endoscope device 1. Alternatively, the PC 41 may also obtain an image signal from the endoscope device 1 and process the obtained image signal.

[0104] The insertion part 2 constitutes an imaging device (camera). The imaging element 28 may be arranged in the main body 3, and the optical fiber may be arranged in the insertion part 2. The light incident on the lens 21 may reach the imaging element 28 through the optical fiber. A borescope may also be used as a camera.

[0105] Turbines are used in aircraft engines or generators. There are gas turbines, steam turbines, and the like. The structure of a gas turbine will be described below. Hereinafter, a gas turbine will be referred to as a turbine.

[0106] The turbine has a compression section, a combustion chamber, and a turbine section. In the compression section, air is compressed. The compressed air is sent to the combustion chamber. Fuel is continuously burned in the combustion chamber to produce high-temperature and high-pressure gas. The gas expands in the turbine section to generate energy. By using this energy, the compressor rotates and the remaining energy is extracted. In the compression section and the turbine section, the moving blades fixed to the rotating shaft of the engine and the stationary blades fixed to the casing are alternately arranged.

[0107] The turbine has a structural element arranged in the internal space of the turbine. The structural element is a moving object that can move inside the turbine or a stationary object that is stationary inside the turbine. The moving object is a moving blade. The stationary object is a stationary blade or a shroud.

[0108] Figure 3 The arrangement of the moving blades and the stationary blades in the compression section of the turbine TB10 is schematically shown. Figure 3 A portion of a cross section of the turbine TB10 passing through the rotation axis RA10 of the engine is shown. The turbine TB10 has a moving blade RT10, a stationary blade ST10, a moving blade RT11, a stationary blade ST11, a moving blade RT12, a stationary blade ST12, a moving blade RT13, and a stationary blade ST13 in the compression section. These moving blades and stationary blades rotate around the rotation axis RA10 in the direction DR12.

[0109] The air taken into the turbine TB10 flows in the direction DR11. The rotor blade RT10 is arranged at the low-pressure portion of the taken-in air. The rotor blade RT13 is arranged at the high-pressure portion of the delivered air.

[0110] The access port AP10 is formed so that the inside of the turbine TB10 can be inspected without disassembling the turbine TB10. The turbine TB10 has two or more access ports, one of which is located at Figure 3 The access port AP10 is shown in FIG. The access port AP10 is a hole formed in the turbine TB10 .

[0111] The insertion part 2 constitutes an endoscope. The insertion part 2 is inserted into the turbine TB10 through the access port AP10. When the insertion part 2 is inserted into the turbine TB10, the insertion part 2 moves in the direction DR10. When the insertion part 2 is pulled out of the turbine TB10, the insertion part 2 moves in the direction opposite to the direction DR10. The direction DR10 is different from the direction DR12. The illumination light LT10 is emitted from the front end 20 of the insertion part 2.

[0112] Figure 4 The arrangement of two or more rotor blades RT10 as viewed in a direction parallel to the rotation axis RA10 is schematically shown. Figure 4In the embodiment, eight rotor blades RT10 are arranged on a disk-shaped disk DS10. A rotation axis RA10 passes through the center of the disk DS10. The disk DS10 rotates in a direction DR12. Therefore, the eight rotor blades RT10 rotate in a direction DR12.

[0113] In reality, dozens of rotor blades or more than 100 rotor blades are arranged on one disk. The number of rotor blades on one disk depends on the model of the engine and on the position of the disk in the region from the low pressure part to the high pressure part.

[0114] When inspecting the moving blades of the turbine, the user rotates the disk manually, or the disk is rotated by a device called a rotary tool. The insert 2 is inserted into the turbine TB10 through the access port AP10, and the front end 20 is fixed. When the disk rotates, the user inspects more than two moving blades to determine whether each moving blade has an abnormality. This inspection is one of the main inspection items in the inspection of the turbine.

[0115] The imaging element 28 generates two or more images. The two or more images are respectively associated with other images included in the two or more images in time. For example, the two or more images are static images. A dynamic image can also be used instead of a static image. The two or more images (frames) included in the dynamic image are associated with each other through a time stamp (time code).

[0116] RAM 14 stores two or more images generated by the imaging element 28. In addition, RAM 14 stores parameters required for 3D reconstruction processing. The parameters include internal parameters of the camera, distortion correction parameters of the camera, setting values, and scale information. The setting values ​​are used in various processes for generating 3D data representing the 3D shape of the object. The scale information is used to convert the scale of the 3D data into the actual scale of the object.

[0117] The memory card 42 may store two or more images and the above-mentioned parameters. The endoscope device 1 may read the two or more images and parameters from the memory card 42 and store the two or more images and parameters in the RAM 14 .

[0118] The endoscope device 1 may also implement wireless communication or wired communication with an external device via the external device interface 16. The external device is a PC 41 or a cloud server, etc. The endoscope device 1 may also send two or more images generated by the imaging element 28 to the external device. The external device may also store the two or more images and the above-mentioned parameters. The endoscope device 1 may also receive the two or more images and parameters from the external device and store the two or more images and parameters in the RAM 14.

[0119] As described above, the endoscope device 1 includes an imaging element 28 and a CPU 18. The imaging element 28 captures an image of a subject and generates an imaging signal. Therefore, the imaging element 28 acquires an image of the subject generated by capturing the image of the subject. The image is a two-dimensional (2D) image. The image acquired by the imaging element 28 is input to the CPU 18 via the image signal processing circuit 12.

[0120] Figure 5 1 shows the functional configuration of the CPU 18. The CPU 18 functions as a control unit 180, an image acquisition unit 181, a graphic processing unit 182, a 3D data generation unit 183, a display control unit 184, and an operation processing unit 185. Figure 5 At least one of the blocks shown may be constituted by a circuit different from the CPU 18 .

[0121] Control unit 180 Figure 5 The processing performed by each unit shown is controlled.

[0122] The image acquisition unit 181 acquires two or more images and the above-mentioned parameters from the RAM 14. The image acquisition unit 181 may acquire two or more images and the above-mentioned parameters from the memory card 42 or an external device via the external device interface 16.

[0123] The graphics processing unit 182 determines whether the two or more images acquired by the image acquisition unit 181 include a graphics region on which graphics information is superimposed. If it is determined that the image includes a graphics region, the graphics processing unit 182 prevents the graphics region in the image from contributing to the generation of 3D data.

[0124] The 3D data generating unit 183 generates 3D data by executing a 3D reconstruction process using the two or more images acquired by the image acquiring unit 181 .

[0125] 3D data includes 3D coordinates of more than two points (3D point groups) of the subject, camera coordinates, and posture information. 3D data may also include a mesh and mesh polygon data, where the mesh is a surface with 3D point groups at the vertices, and the mesh polygon data is a collection of texture information associated with the mesh. The camera coordinates represent the 3D coordinates of a camera that has acquired more than two images, and are associated with more than two images. The camera coordinates are the 3D coordinates of the viewpoint when each image is acquired, indicating the position of the camera. For example, the camera coordinates represent the 3D coordinates of the observation optical system of the camera. The posture information represents the posture of the camera that has acquired more than two images, and is associated with more than two images. For example, the posture information represents the posture of the observation optical system of the camera.

[0126] The display control unit 184 controls the processing performed by the image signal processing circuit 12. The CCU 9 outputs an image signal. The image signal includes color data of each pixel of the image generated by the imaging element 28. The display control unit 184 causes the image signal processing circuit 12 to output the image signal output from the CCU 9 to the display unit 5. The image signal processing circuit 12 outputs the image signal to the display unit 5. The display unit 5 displays an image based on the image signal output from the image signal processing circuit 12. Thus, the display control unit 184 displays the image generated by the imaging element 28 on the display unit 5.

[0127] The display control unit 184 displays various information on the display unit 5. That is, the display control unit 184 displays various information on the image.

[0128] For example, the display control unit 184 generates various information. The various information is an image and graphic information of the operation screen. The display control unit 184 outputs the generated information to the image signal processing circuit 12. The image signal processing circuit 12 synthesizes the image signal output from the CCU 9 and the information output from the CPU 18. As a result, various information is superimposed on the image. The image signal processing circuit 12 outputs the synthesized image signal to the display unit 5. The display unit 5 displays an image superimposed with various information.

[0129] In addition, the display control unit 184 generates image information of 3D data. The display control unit 184 outputs the image information to the video signal processing circuit 12. The display unit 5 displays the image of 3D data by performing the same processing as described above. Thus, the display control unit 184 displays the image of 3D data on the display unit 5.

[0130] The user inputs various information to the endoscope device 1 by operating the operation unit 4. The operation unit 4 outputs the information input by the user. The information is input to the control interface 17 as an input unit. The information is output from the control interface 17 to the CPU 18. The operation processing unit 185 receives the information input to the endoscope device 1 via the operation unit 4.

[0131] Figure 5Each of the components shown may also be composed of at least one of a processor and a logic circuit. For example, the processor is at least one of a CPU (Central Processing Unit), a DSP (Digital Signal Processor), and a GPU (Graphics Processing Unit). For example, the logic circuit is at least one of an ASIC (Application Specific Integrated Circuit) and an FPGA (Field Programmable Gate Array). Figure 5 Each of the components shown can include one or more processors. Figure 5 Each of the components shown may include one or more logic circuits.

[0132] The computer of the endoscope device 1 can also read the program and execute the read program. The program includes the following provisions: Figure 5 The instructions for the actions of the various parts shown. That is, Figure 5 The functions of the components shown can also be realized by software.

[0133] The above-mentioned program can also be provided by a "computer-readable recording medium" such as a flash memory. The program can also be transmitted to the endoscope device 1 from the computer holding the program via a transmission medium or by a transmission wave in the transmission medium. The "transmission medium" that transmits the program is a medium that has the function of transmitting information. The medium that has the function of transmitting information includes networks (communication networks) such as the Internet and communication lines (communication lines) such as telephone lines. The above-mentioned program can also realize part of the functions described. Furthermore, the above-mentioned program can also be a differential file (differential program). The functions described can also be realized by a combination of a program already recorded in the computer and a differential program.

[0134] The graphic information is composed of characters, numbers, symbols or marks generated by a computer. In the following example, the graphic information includes sub-graphic information of two or more items. The two or more items are optical adapter category, brightness, date, company logo or zoom state. The optical adapter category indicates the category of the optical adapter mounted on the front end 20. Brightness indicates the brightness of the image superimposed with the graphic information. The company logo is a prescribed mark of the company that manufactures or sells the endoscope device 1. The date indicates the day when the image was generated. The zoom state indicates the magnification of the image. The sub-graphic information of each item is superimposed on the graphic area of ​​each item.

[0135] The graphics information is included in the header or footer as metadata of the moving image file, or in EXIF ​​(Exchangeable image file format) information added to the still image file.

[0136] The position of the image region where the graphic information is superimposed differs depending on the model of the inspection device that generated the image. The metadata of the moving image file or the EXIF ​​information of the still image file includes model information indicating the model of the inspection device.

[0137] The diameter or length of the insertion portion 2 may differ depending on the model of the inspection device. Alternatively, the optical adapter may differ depending on the model of the inspection device. The diameter of the insertion portion 2, the length of the insertion portion 2, or the information of the optical adapter may also be used as model information.

[0138] The metadata of a moving image file or the EXIF ​​information of a still image file includes setting information indicating the setting of the graphic information. The setting information includes information indicating whether the graphic information is superimposed on each image. In other words, the setting information includes information indicating whether each image includes a graphic area. When the setting information includes information indicating that the graphic information is superimposed on each image, the setting information includes information indicating whether the sub-graphics information of each item is superimposed on each image.

[0139] Next, the characteristic processing of the first embodiment is described. In the following description, it is assumed that 3D data is generated by using two or more images obtained by an endoscope device. The inspection device that obtains two or more images is not limited to the endoscope device. As long as the image of the structural elements inside the turbine is obtained by using a device with a camera, any device can be used.

[0140] The control device 10 functions as a 3D data generating device. The 3D data generating device of each embodiment of the present invention may also be a computer system such as a PC that is separate from the endoscope device. The 3D data generating device may also be any one of a desktop PC, a notebook PC, and a tablet terminal. The 3D data generating device may also be a computer system that operates on the cloud.

[0141] use Figure 6 The processing executed by the endoscope apparatus 1 will be described. Figure 6 The following shows the procedure of 3D data generation processing executed by the endoscope device 1 .

[0142] Next, an example using a moving image having metadata stored in the header will be described. Two or more still images may be used instead of the moving image.

[0143] The imaging element 28 continuously generates imaging signals. That is, the imaging element 28 generates imaging signals for each frame corresponding to a dynamic image. A dynamic image includes two or more frames. Each frame is composed of an image obtained by the imaging element 28. When the imaging element 28 finishes imaging, a dynamic image file including the dynamic image is recorded in the PC 41 or the memory card 42.

[0144] Next, an example will be described in which a moving image file recorded in the PC 41 or the memory card 42 is used in the 3D reconstruction process. The 3D reconstruction process may be executed in real time by the CPU 18 while the imaging element 28 generates a moving image.

[0145] exist Figure 6 When the process shown starts, a moving image file is input from PC 41 to RAM 14 via external device interface 16. Alternatively, a moving image file is input from memory card 42 to RAM 14 via card interface 15. Image acquisition unit 181 acquires the moving image included in the moving image file from RAM 14 (step S100).

[0146] After step S100 , the graphic processing unit 182 obtains model information indicating the model of the inspection device from the header of the moving image file (step S101 ).

[0147] After step S101, the graphics processing unit 182 obtains setting information indicating whether graphics information is superimposed on the moving image from the header of the moving image file (step S102). The setting information includes information indicating whether graphics information is superimposed on the moving image.

[0148] After step S102, the graphics processing unit 182 refers to the setting information obtained in step S102 to determine whether graphics information is superimposed on the dynamic image. Thus, the graphics processing unit 182 determines whether the dynamic image includes a graphics area (step S103). When the graphics processing unit 182 determines in step S103 that graphics information is not superimposed on the dynamic image, step S108 described later is executed.

[0149] When the graphics processing unit 182 determines in step S103 that the graphics information is superimposed on the dynamic image, the graphics processing unit 182 obtains information indicating whether the sub-graphics information of each item is superimposed on each frame from the setting information (step S104). As described above, when the setting information includes information indicating that the graphics information is superimposed on the dynamic image, the setting information includes information indicating whether the sub-graphics information of each item is superimposed on each frame.

[0150] After step S104, the graphics processing unit 182 refers to the information acquired in step S104 to determine whether the sub-graphics information of each item is superimposed on each frame. The graphics processing unit 182 generates a superimposition state table indicating whether the sub-graphics information of each item is superimposed on each frame (step S105).

[0151] Figure 7 An example of a superposition state table is shown. Figure 7 The superimposed state table shown includes model information IF1 and state information SI1 to SI6.

[0152] Model information IF1 indicates the model of the inspection device. Status information SI1 to SI6 constitute setting information. Status information SI1 indicates whether graphic information is superimposed on the dynamic image. Status information SI2 to SI6 indicate whether sub-graphic information of each item is superimposed on each frame. When status information SI1 indicates that graphic information is superimposed on the dynamic image, at least one of status information SI2 to SI6 indicates that sub-graphic information of each item is superimposed on each frame.

[0153] Status information SI2 indicates whether sub-graphic information indicating the type of optical adapter is superimposed on each frame. Status information SI3 indicates whether sub-graphic information indicating the brightness of each frame is superimposed on each frame. Status information SI4 indicates whether sub-graphic information indicating the date is superimposed on each frame. Status information SI5 indicates whether sub-graphic information indicating the company logo is superimposed on each frame. Status information SI6 indicates whether sub-graphic information indicating the zoom state of each frame is superimposed on each frame.

[0154] exist Figure 7 In the example shown, state information SI1 indicates that graphics information is superimposed on the dynamic image. State information SI2 and SI3 indicate that sub-graphics information of each item is not superimposed on each frame. State information SI4 to SI6 indicate that sub-graphics information of each item is superimposed on each frame.

[0155] After step S105, the graphics processing unit 182 determines one or more graphics areas on which the sub-graphics information is superimposed (step S106).

[0156] The graphic processing unit 182 executes the following processing in step S106 : The graphic processing unit 182 obtains a coordinate table indicating the image coordinates of the region where the sub-graphics information of each item is superimposed. The coordinate table is stored in the ROM 13 , for example.

[0157] Figure 8 An example of a coordinate table is shown. The coordinate table contains model information IF2 and coordinate information C1 to C5 associated with each other. Model information IF2 corresponds to Figure 7The model information IF1 shown indicates the model of the inspection device. The coordinate information C1 to C5 indicates the image coordinates of the area where the sub-graphic information of each item is superimposed. The coordinate information C1 indicates the image coordinates of the area where the sub-graphic information indicating the type of the optical adapter is superimposed. The coordinate information C2 indicates the image coordinates of the area where the sub-graphic information indicating the brightness of each frame is superimposed. The coordinate information C3 indicates the image coordinates of the area where the sub-graphic information indicating the date is superimposed. The coordinate information C4 indicates the image coordinates of the area where the sub-graphic information indicating the company logo is superimposed. The coordinate information C5 indicates the image coordinates of the area where the sub-graphic information indicating the zoom state of each frame is superimposed. For example, the coordinate information C1 to C5 includes the image coordinates of the upper left point of the graphic area of ​​each item, the horizontal width of the graphic area, and the vertical width of the graphic area.

[0158] Fig. 9 An example of the position of the graphic area in the image IMG1 generated by the imaging element 28 is shown. The image IMG1 includes graphic areas G1 to G5. Sub-graphic information indicating the type of the optical adapter is superimposed on the graphic area G1. Sub-graphic information indicating the brightness of the image IMG1 is superimposed on the graphic area G2. Sub-graphic information indicating the date is superimposed on the graphic area G3. Sub-graphic information indicating the company logo is superimposed on the graphic area G4. Sub-graphic information indicating the zoom state of the image IMG1 is superimposed on the graphic area G5.

[0159] The graphics processing unit 182 obtains coordinate information corresponding to the model information of the superimposed state table and the state information of the superimposed state table from the coordinate table. Figure 7 The graphics processing unit 182 obtains at least one of the coordinate information C1 to C5 associated with the found model information IF2. Specifically, the graphics processing unit 182 obtains the coordinate information corresponding to the state information indicating that the sub-graphics information is superimposed on each frame.

[0160] exist Figure 7 In the superimposed state table shown in FIG. 1 , state information SI4 to SI6 indicate that sub-graphics information of each item is superimposed on each frame. The graphics processing unit 182 obtains coordinate information C3 of the date indicated by state information SI4, obtains coordinate information C4 of the company logo indicated by state information SI5, and obtains coordinate information C5 of the zoom state indicated by state information SI6. The graphics processing unit 182 determines the graphic area indicated by the coordinate information C3 to C5. For example, the graphics processing unit 182 determines Fig. 9 The graphic areas G3 to G5 are shown.

[0161] After step S106, the graphic processing unit 182 sets one or more graphic regions determined in step S106 as invalid regions (step S107). In the image generated by the imaging element 28, the region other than the invalid region is a valid region used for 3D reconstruction processing.

[0162] As described later, the 3D reconstruction process includes a process of detecting feature points from at least two images. The graphics processing unit 182 changes the image in the invalid area so that no feature points are detected from the invalid area. For example, the graphics processing unit 182 changes the image in the invalid area to a black image. At this time, the graphics processing unit 182 changes the pixel value of the image in the invalid area to a value corresponding to the black level. The graphics processing unit 182 may also exclude the invalid area from the area where the process of detecting feature points is to be performed. The graphics processing unit 182 may also delete the feature points existing in the invalid area after detecting the feature points regardless of the valid area and the invalid area.

[0163] After step S107, the 3D data generating unit 183 generates 3D data by executing 3D reconstruction processing using two or more frames included in the moving image (step S108). The 3D data generating unit 183 reads parameters required for 3D reconstruction processing from the RAM 14 and uses the parameters for 3D reconstruction processing.

[0164] When the graphics processing unit 182 determines in step S103 that the graphics information is not superimposed on the dynamic image, the entire area of ​​each of the two or more frames is set as the effective area. When the graphics processing unit 182 sets the graphics area as the invalid area in step S107, the area other than the invalid area is set as the effective area in each of the two or more frames. The 3D data generation unit 183 uses the effective area in the 3D reconstruction process.

[0165] After step S108, the display control unit 184 displays the image of the 3D data on the display unit 5 (step S109). When step S109 is executed, Figure 6 The 3D data generation process shown ends.

[0166] The graphics processing unit 182 may also perform the following processing instead of executing step S101 and step S102. The graphics processing unit 182 obtains n frames in the dynamic image. For example, n is 10. The graphics processing unit 182 divides the area of ​​each frame into small areas in a grid shape at intervals of p pixels. For example, p is 8.

[0167] The graphics processing unit 182 compares the pixel values ​​of pixels having the same coordinates in n frames. For example, the graphics processing unit 182 compares the RGB value of a pixel having specific coordinates in the first frame with the RGB value of a pixel having the specific coordinates in the second frame following the first frame. The graphics processing unit 182 compares the RGB value of a pixel having specific coordinates in the second frame with the RGB value of a pixel having the specific coordinates in the third frame following the second frame. The graphics processing unit 182 repeatedly performs this operation to detect a small area including pixels whose RGB values ​​do not change between n frames. The graphics processing unit 182 determines that the detected small area is a graphics area.

[0168] Fig.10 An example of a screen of the display unit 5 is shown. The display unit 5 displays Fig.10 The screen SC1 shown. The screen SC1 includes a frame FR1, a button BT1, a button BT2 and a progress bar SB1.

[0169] The user operates the button BT1 and the button BT2 by operating the operation unit 4. When the display unit 5 is configured as a touch panel, the user operates the button BT1 and the button BT2 by touching the screen of the display unit 5.

[0170] The user presses the button BT1 to reproduce the dynamic image. After the button BT1 is pressed, a frame FR1 of the dynamic image is displayed. Fig.10 A screen SC1 displayed on the display unit 5 after the button BT1 is pressed is shown.

[0171] The user may also perform a predetermined operation on the screen SC1 by operating the operation unit 4 or touching the screen of the display unit 5. When the predetermined operation is performed, an instruction indicating the reproduction or temporary stop of the dynamic image may be input to the endoscope device 1. The screen SC1 may also include a button for inputting an instruction indicating the reproduction or temporary stop of the dynamic image.

[0172] The progress bar SB1 indicates the position of the frame FR1. The user can change the position of the frame in the progress bar SB1 by operating the operation unit 4 or touching the screen of the display unit 5. In addition, the user can specify the frame to start the 3D reconstruction process and the frame to end the 3D reconstruction process by operating the operation unit 4 or touching the screen of the display unit 5.

[0173] In the above example, the user specifies the start frame for starting the 3D reconstruction process and the end frame for ending the 3D reconstruction process. The start frame and the end frame may also be automatically specified by the control unit 180. For example, the control unit 180 may also detect an interval of a dynamic image in which a subject is moving. Alternatively, the control unit 180 may also detect an interval of a dynamic image in which an abnormality such as damage is captured. The interval includes more than two frames of the dynamic image. The control unit 180 may specify the first frame of the interval as the start frame, and may specify the last frame of the interval as the end frame.

[0174] Only one of the start frame and the end frame may be specified by the user. Alternatively, only one of the start frame and the end frame may be automatically specified. The method of setting the interval including the frames used in the 3D reconstruction process is not limited to the above example.

[0175] The user presses button BT2 to start the 3D reconstruction process. After button BT2 is pressed, the Figure 6 The 3D data generation process shown.

[0176] Brightness GI1, date GI2, and company logo GI3 are superimposed on frame FR1 as graphic information. The graphic area superimposed with these is set as an invalid area.

[0177] Fig.11 An example of a graphic region in an image generated by the imaging element 28 is shown. The graphic regions R1 to R3 are superimposed on the image IMG2. The image IMG2 corresponds to Fig.10 The frame FR1 shown. The graphic region R1 is superimposed with Fig.10 The area of ​​brightness GI1 shown in FIG. The graphic area R2 is superimposed with Fig.10 The area of ​​the date GI2 shown. The graphic area R3 is superimposed with Fig.10 The company logo of GI3 is shown in the area.

[0178] In all frames included in the dynamic image, the graphic regions R1 to R3 are set as invalid regions. The positions of the graphic regions are the same in all frames. In addition, in all frames included in the dynamic image, the region other than the graphic regions R1 to R3 from the entire region is set as a valid region.

[0179] The 3D data generating unit 183 executes the following processing in step S108. Fig.12 The following schematically shows the state of image acquisition when acquiring two images.

[0180] like Fig.12As shown, initially, image I1 is acquired in the camera's imaging state c1. Next, image I2 is acquired in the camera's imaging state c2. In the imaging state c1 and the imaging state c2, at least one of the imaging position and the imaging posture is different. Fig.12 In the imaging state c1 and the imaging state c2, both the imaging position and the imaging posture are different.

[0181] In various embodiments of the present invention, it is assumed that image I1 and image I2 are obtained by the same endoscope. In addition, in various embodiments of the present invention, it is assumed that the parameters of the objective optical system of the endoscope do not change. The parameters of the objective optical system are focal length, distortion aberration, and pixel size of the image sensor, etc. Below, for convenience, the parameters of the objective optical system are referred to as internal parameters. When this condition is assumed, the internal parameters that describe the characteristics of the optical system of the endoscope can be used in common regardless of the position and posture of the camera (observation optical system). In various embodiments of the present invention, it is assumed that the internal parameters are obtained when the product leaves the factory or is delivered. The internal parameters can also be obtained before the inspection begins at the inspection site. In addition, in various embodiments of the present invention, it is assumed that the internal parameters are known when the image is obtained.

[0182] In each embodiment of the present invention, it is assumed that image I1 and image I2 are obtained by one endoscope. However, the present invention is not limited to this. For example, the present invention can also be applied when 3D data is generated using multiple dynamic images obtained by multiple endoscopes. In this case, image I1 and image I2 only need to be obtained using different endoscope devices and maintain their own internal parameters for each endoscope. Even if the internal parameters are unknown when generating 3D data, the internal parameters can be used as variables to be estimated to perform calculations. Therefore, the subsequent steps will not change significantly depending on whether the internal parameters are known.

[0183] use Fig.13 The details of the 3D reconstruction process in step S108 will be described. Fig.13 The steps of the 3D reconstruction process are shown.

[0184] First, the 3D data generation unit 183 performs feature point detection processing (step S108a). The 3D data generation unit 183 detects feature points of each of the two images in the feature point detection processing. Feature points represent corners and edges with large image brightness gradients in the subject information captured in the image. Feature points can also be composed of 1 pixel, which is the smallest unit of the image. Alternatively, feature points can also be composed of more than 2 pixels. The 3D data generation unit 183 detects feature points by using image feature descriptors such as SIFT (Scale-invariant Feature Transform) or FAST (Features from Accelerated Segment Test).

[0185] Fig.12 It shows that the feature point P is detected from the image I1 11 , and feature point P is detected from image I2 12 Example. Feature point P 11 and feature point P 12 Corresponds to the coordinate P1 on the subject. Fig.12 In the figure, only one feature point is shown for each image, but in reality, multiple feature points are detected in each image. The number of detected feature points may differ between images. Each feature point detected from each image is converted into data such as a feature quantity using a descriptor. A feature quantity is data that represents the characteristics of a feature point.

[0186] After step S108a, the 3D data generation unit 183 performs feature point association processing (step S108b). In the feature point association processing, the 3D data generation unit 183 compares the correlation of feature quantities between images for each feature point detected by the feature point detection processing (step S108a). When the correlation of feature quantities is compared and feature points with similar feature quantities are found in each image, the 3D data generation unit 183 stores the information in the RAM 14. Thus, the 3D data generation unit 183 associates the feature points of each image with each other. On the other hand, when no feature point with similar feature quantities is found, the 3D data generation unit 183 discards the information of the feature point.

[0187] After step S108b, the 3D data generation unit 183 reads out the coordinates of the feature points (feature point pairs) of the two images associated with each other from RAM14. The coordinates are a pair of coordinates of the feature points in each image. The 3D data generation unit 183 performs position and posture calculation processing based on the read coordinates (step S108c). In the position and posture calculation processing, the 3D data generation unit 183 calculates the relative position and posture between the camera state c1 of the camera that obtained the image I1 and the camera state c2 of the camera that obtained the image I2. More specifically, the 3D data generation unit 183 calculates the matrix E by solving the following equation (1) using the epipolar constraint.

[0188]

[0189] The matrix E is called the basic matrix. The basic matrix E is a matrix that maintains the relative position and posture between the camera state c1 of the camera that obtained the image I1 and the camera state c2 of the camera that obtained the image I2. In equation (1), the matrix p1 is a matrix containing the coordinates of the feature points detected from the image I1. The matrix p2 is a matrix containing the coordinates of the feature points detected from the image I2. The basic matrix E contains information related to the relative position and posture of the camera, and is therefore equivalent to the external parameters of the camera. The 3D data generation unit 183 can solve the basic matrix E by using a well-known algorithm.

[0190] like Fig.12 As shown, when the position change (relative position) of the camera is t and the posture change (relative posture) of the camera is R, equations (2) and (3) hold.

[0191] t=(t x , t y , t z )…(2)

[0192]

[0193] In formula (2), the movement amount in the x-axis direction is expressed as t x , the movement in the y-axis direction is represented by t y , and the movement in the z-axis direction is represented by t z In formula (3), the rotation amount α around the x-axis is expressed as R x (α), the rotation β around the y-axis is expressed as R y (β), and the rotation amount γ around the z-axis is represented by R z (γ) After the basic matrix E is calculated, an optimization process called bundle adjustment may be performed to improve the restoration accuracy of the 3D coordinates.

[0194] The 3D data generation unit 183 calculates the 3D coordinates (camera coordinates) in the coordinate system of the 3D data by using the calculated position change amount of the camera. For example, the 3D data generation unit 183 defines the 3D coordinates of the camera that acquired the image I1. The 3D data generation unit 183 calculates the 3D coordinates of the camera that acquired the image I2 based on the 3D coordinates of the camera that acquired the image I1 and the position change amount of the camera that acquired the image I2.

[0195] The 3D data generation unit 183 calculates the posture information in the coordinate system of the 3D model by using the calculated posture change amount of the camera. For example, the 3D data generation unit 183 defines the posture information of the camera that has acquired the image I1. The 3D data generation unit 183 generates the posture information of the camera that has acquired the image I2 based on the posture information of the camera that has acquired the image I1 and the posture change amount of the camera that has acquired the image I2.

[0196] The 3D data generation unit 183 generates 3D shape data by performing position and posture calculation processing (step S108c). The 3D shape data includes 3D coordinates (camera coordinates) at the position of the camera and posture information representing the posture of the camera. In addition, when methods such as structure from motion or visual SLAM are applied to the position and posture calculation processing (step S108c), the 3D data generation unit 183 also calculates the 3D coordinates of each feature point in step S108c. The 3D shape data generated in step S108c does not include the 3D coordinates of points on the subject other than the feature points. Therefore, the 3D shape data represents the sparse 3D shape of the subject.

[0197] The 3D shape data includes the 3D coordinates of each feature point, the camera coordinates and the posture information. The 3D coordinates of each feature point are defined in the coordinate system of the 3D data. The 3D coordinates of each feature point are associated with the two-dimensional coordinates (2D coordinates) of each feature point. The 2D coordinates of each feature point are defined in the coordinate system of the image containing each feature point. The 2D coordinates and 3D coordinates of each feature point are associated with the image containing each feature point.

[0198] After step S108c, the 3D data generation unit 183 performs 3D shape reconstruction processing (step S108d) based on the relative position and posture (position change t and posture change R) of the camera calculated in step S108c. The 3D data generation unit 183 generates 3D data of the subject in the 3D shape reconstruction processing. As a method for restoring the 3D shape of the subject, PMVS (Patch-based Multi-view Stereo) and matching processing based on parallel stereo are cited. However, there is no particular limitation on the means.

[0199] The 3D data generation unit 183 calculates the 3D coordinates of points on the subject other than the feature points in step S108d. The 3D coordinates of each point other than the feature points are defined in the coordinate system of the 3D data. The 3D coordinates of each point are associated with the 2D coordinates of each point. The 2D coordinates of each point are defined in the coordinate system of the 2D image containing each point. The 2D coordinates and 3D coordinates of each point are associated with the 2D image containing each point. The 3D data generation unit 183 updates the 3D shape data. The updated 3D shape data includes the 3D coordinates of each feature point, the 3D coordinates of each point other than the feature points, the camera coordinates and the posture information. The 3D shape data updated in step S108d also includes the 3D coordinates of points on the subject other than the feature points on the basis of the 3D coordinates of the feature points. Therefore, the 3D shape data represents the dense 3D shape of the subject.

[0200] After step S108d, the 3D data generation unit 183 performs a scale conversion process (step S108e) based on the 3D shape data processed in the 3D shape reconstruction process (step S108d) and the scale information read from the RAM 14. In the scale conversion process, the 3D data generation unit 183 converts the 3D shape data of the object into 3D coordinate data (3D data) having a length dimension. When step S108e is executed, Fig.13 The 3D reconstruction process shown ends.

[0201] In order to shorten the processing time, step S108d may be omitted. In this case, after step S108c is executed, step S108e is executed instead of step S108d.

[0202] Step S108e may also be omitted. In this case, after executing step S108d, Fig.13 The 3D reconstruction process shown ends without executing step S108e. In this case, the 3D data represents the relative shape of the object without a length dimension.

[0203] In order to follow Fig.12To generate 3D data according to the principle shown, at least a portion of the area of ​​each image and at least a portion of the area of ​​each of at least one other image must be the same. That is, the area of ​​the first image and the area of ​​the second image different from the first image include the same area. The area other than the same area in the first image and the area other than the same area in the second image are different from each other. The two or more images used to generate 3D data include at least two images that captured the same area of ​​the subject.

[0204] The image I1 and the image I2 do not need to be two temporally continuous frames in the moving image. In the moving image, one or more frames may exist between the image I1 and the image I2.

[0205] As described above, the graphics processing unit 182 Figure 6 In step S107 shown in FIG. 1 , one or more graphic regions are set as invalid regions. Specifically, the graphic processing unit 182 changes the image in the invalid region so that no feature points are detected from the graphic region. For example, the graphic processing unit 182 changes the image in the invalid region to a black image. Alternatively, the graphic processing unit 182 excludes the invalid region from the region where the feature point detection process is to be performed. Alternatively, after the graphic processing unit 182 detects feature points regardless of the valid region and the invalid region, it deletes the feature points existing in the invalid region.

[0206] exist Fig.13 In the feature point detection process (step S108a) shown in FIG. 1 , feature points in the black image are not detected. Alternatively, feature point detection process (step S108a) is not performed in the invalid area. Alternatively, in the feature point detection process (step S108a), after feature points are detected regardless of the valid area and the invalid area, feature points existing in the invalid area are deleted. That is, feature points in the invalid area are not detected. Therefore, in Fig.13 The invalid area is not used in the feature point association process (step S108b), the position and posture calculation process (step S108c), and the 3D shape reconstruction process (step S108d). That is, the invalid area does not contribute to the generation of 3D data. The 3D data generation unit 183 generates 3D data of the subject without using the invalid area.

[0207] The 3D data generating unit 183 generates 3D data of the subject by using only the valid area other than the invalid area in the image on which the graphic information is superimposed. Therefore, the endoscope apparatus 1 can generate 3D data with high accuracy.

[0208] In the graphics processing unit 182 Figure 6When it is determined in step S103 that graphic information is superimposed on the dynamic image, the graphic processing unit 182 may also stop executing the 3D reconstruction process. In this case, the graphic processing unit 182 does not execute the 3D reconstruction process. Therefore, the endoscope device 1 can avoid the failure of the 3D reconstruction process or the reduction of the accuracy of the 3D data.

[0209] The image signal processing circuit 12 may also output the first image and the second image to the CPU 18. The graphic information is not superimposed on the first image. The second image is generated by superimposing the graphic information on the first image. The CPU 18 may also obtain the first image and the second image from the image signal processing circuit 12. The CPU 18 may also record more than two first images and more than two second images in a storage medium. For example, the storage medium is a storage medium or a memory card 42 possessed by the PC 41.

[0210] The CPU 18 may also use two or more second images to execute Figure 6 When the graphics processing unit 182 determines in step S103 that the graphics information is superimposed on two or more second images, the graphics processing unit 182 may perform the 3D data generation process using two or more first images. The endoscope device 1 can avoid failure of the 3D reconstruction process or reduction in the accuracy of the 3D data by using two or more first images in the 3D reconstruction process.

[0211] At least one of the two or more images used in the 3D reconstruction process may also include graphic information. When the graphic processing unit 182 determines that a graphic area is not superimposed on any one of the two or more images, the graphic processing unit 182 may also perform the 3D reconstruction process using the two or more images. When the graphic processing unit 182 determines that a graphic area is superimposed on at least one of the two or more images, the graphic processing unit 182 may also set the graphic area in the at least one image as an invalid area.

[0212] The endoscope device 1 (3D data generation system) of each embodiment of the present invention includes an imaging device, an image signal processing circuit 12 (image processing device), and a CPU 18. The endoscope device 1 generates 3D data representing the 3D shape of the inside of an object. The imaging device includes a tubular insertion portion 2 for obtaining an optical image of the inside of the object. The imaging device generates two or more images of the object when the insertion portion 2 is inserted into the inside of the object. The image signal processing circuit 12 superimposes graphic information on at least one of the two or more images.

[0213] The CPU 18 obtains two or more images and determines whether the two or more images include a graphic area superimposed with graphic information. When it is determined that the two or more images do not include the graphic area, the CPU 18 performs a first process. The first process includes a 3D reconstruction process (generation process) for generating 3D data using the two or more images. When it is determined that at least one of the two or more images includes a graphic area, the CPU 18 performs a second process. The second process includes a process for making the graphic area in the at least one image not contribute to the generation of 3D data.

[0214] The 3D data generation method according to each aspect of the present invention generates 3D data representing the 3D shape inside an object. The 3D data generation method includes an image acquisition step, a determination step, a first processing step, and a second processing step.

[0215] The CPU 18 obtains two or more images of the object in the image acquisition step (step S100). The CPU 18 determines in the judgment step (step S103) whether the two or more images include a graphic area on which graphic information is superimposed by the image signal processing circuit 12. When it is determined that the two or more images do not include the graphic area, the CPU 18 performs the first processing in the first processing step (step S108). The first processing includes a 3D reconstruction processing for generating 3D data using two or more images. When it is determined that at least one of the two or more images includes a graphic area, the CPU 18 performs the second processing in the second processing step (step S107). The second processing includes a processing for making the graphic area in the at least one image not contribute to the generation of 3D data.

[0216] Each aspect of the present invention may include the following modified examples: The second process includes a process of setting the graphic region in at least one image determined to include the graphic region as an invalid region.

[0217] Each aspect of the present invention may include the following modified examples: The second process includes a 3D reconstruction process using an area other than the invalid area in two or more images.

[0218] Each mode of the present invention may also include the following modified examples. The second process includes a process of setting the graphic area as an invalid area based on the setting information and the coordinate information (position information). The setting information indicates an item corresponding to the graphic information superimposed on at least one image determined to include the graphic area. The coordinate information indicates the position of the graphic area superimposed with the graphic information corresponding to the item.

[0219] Each mode of the present invention may also include the following modified examples. The second processing includes a process of setting the graphic area as an invalid area based on the model information, the setting information and the coordinate information (display position information). The model information indicates the model of the camera device. The setting information is prepared according to each model of the camera device, and indicates an item corresponding to the graphic information superimposed on at least one image judged to include the graphic area. The coordinate information indicates the position of the graphic area superimposed with the graphic information corresponding to the item.

[0220] Each aspect of the present invention may also include the following modified examples. The 3D reconstruction process includes a feature point detection process for detecting feature points from two or more images. The second process includes a process for changing the image in the invalid area so that feature points are not detected from the invalid area, and a 3D reconstruction process using two or more images.

[0221] Each aspect of the present invention may include the following modified examples: The 3D reconstruction process includes a feature point detection process of detecting feature points from two or more images. The second process includes a process of excluding an invalid area from an area to be subjected to the feature point detection process.

[0222] Each aspect of the present invention may include the following modified examples. The 3D reconstruction process includes a feature point detection process of detecting feature points from two or more images. The second process includes a process of deleting feature points detected from an invalid area.

[0223] Each aspect of the present invention may include the following modified examples: The second process includes a process of suspending execution of the 3D reconstruction process.

[0224] Each aspect of the present invention may include the following modified examples. Two or more images generated by the imaging device and not superimposed with graphic information by the video signal processing circuit 12 are stored in a storage medium. The second process includes 3D reconstruction using the two or more images stored in the storage medium.

[0225] Each aspect of the present invention may include the following modified examples: The CPU 18 determines whether two or more images include a graphic area based on setting information indicating whether graphic information is superimposed on each of the two or more images.

[0226] Each aspect of the present invention may include the following modified examples. The CPU 18 performs image processing for detecting a graphic region on two or more images. The CPU 18 determines whether the two or more images include a graphic region based on the result of the image processing.

[0227] Each mode of the present invention may also include the following modified examples. The graphic information includes at least one of the type of the optical adapter mounted on the front end 20 of the insertion portion 2, the brightness of the image generated by the camera, the date, the mark, and the magnification of the image generated by the camera. In addition, the items of the graphic information may also be different depending on the model of the camera.

[0228] Each mode of the present invention may also include the following modified examples. After two or more images are stored in the storage medium, the CPU 18 obtains the two or more images from the storage medium. That is, instead of executing the 3D data generation process while the camera device generates two or more images, the CPU 18 executes the 3D data generation process using the two or more images stored in the storage medium.

[0229] Each aspect of the present invention may include the following modified examples. The insertion portion 2 includes a lens 21 and an imaging element 28 (image sensor).

[0230] Each embodiment of the present invention may also include the following modified examples. The lens 21 and the imaging element 28 are built into the front end 20 of the insertion portion 2. The position of the front end 20 when two or more images are generated is different from each other. The posture of the front end 20 when two or more images are generated is different from each other. For example, the two or more images include a first image and a second image. The position of the front end 20 when the second image is generated is different from the position of the front end 20 when the first image is generated. The posture of the front end 20 when the second image is generated is different from the posture of the front end 20 when the first image is generated.

[0231] Each aspect of the present invention may include the following modified examples. The imaging device, the video signal processing circuit 12 , and the CPU 18 are included in the endoscope device 1 .

[0232] Each aspect of the present invention may include the following modified examples: The imaging device generates two or more images based on an optical image formed by a monocular optical adapter attached to the distal end 20 of the insertion portion 2 .

[0233] Each aspect of the present invention may include the following modified examples: The same region of a structural element of an object is captured in at least two images included in the two or more images.

[0234] In the first embodiment, when at least one of two or more images includes a graphic region, the CPU 18 sets the graphic region as an invalid region. Therefore, the endoscope device 1 can avoid a processing failure in generating 3D data or a reduction in the accuracy of 3D data.

[0235] Furthermore, the endoscope apparatus 1 can generate 3D data while retaining graphic information useful for reporting of inspection results or data management in the image. Therefore, the endoscope apparatus 1 can avoid a decrease in inspection efficiency for generating 3D data.

[0236] When at least one of the two or more images includes a graphic region, the CPU 18 generates 3D data using regions other than the graphic region of each of the two or more images. Therefore, the endoscope apparatus 1 can generate 3D data with high accuracy.

[0237] (Second embodiment)

[0238] A second embodiment of the present invention will be described. In the first embodiment, the graphics processing unit 182 obtains model information indicating the model of the inspection device and setting information indicating whether the graphics information is superimposed on the dynamic image from the header of the dynamic image file. On the other hand, in the second embodiment, the user inputs the model information and setting information to the endoscope device 1, and the graphics processing unit 182 receives the model information and setting information. In the second embodiment, the metadata stored in the header or footer of the dynamic image file does not need to include the model information and setting information.

[0239] In the second embodiment, Figure 6 The 3D data generation process shown is changed as follows. The user knows in advance the model of the endoscope device 1 as the inspection device. After the dynamic image is displayed on the display unit 5, the user inputs the model information to the endoscope device 1 by operating the operation unit 4 or the touch panel. The graphics processing unit 182 receives the model information input through the operation unit 4 or the touch panel in step S101.

[0240] The user checks the frames of the dynamic image displayed on the display unit 5 and determines whether the graphics information is superimposed on the frames. The user operates the operation unit 4 or the touch panel to input setting information to the endoscope device 1. The graphics processing unit 182 receives the setting information input through the operation unit 4 or the touch panel in step S102.

[0241] Fig.14 An example of a screen of the display unit 5 is shown. The display unit 5 displays Fig.14 The screen SC2 shown in FIG. SC2 includes a frame FR1, a button BT1, a button BT2, a button BT3 and a progress bar SB1. Fig.10 The parts shown are the same as the parts described.

[0242] The user presses the button BT3 to input the model information and setting information to the endoscope device 1. After the button BT3 is pressed, the display unit 5 displays Fig.15When the image acquisition unit 181 acquires a moving image from the RAM 14, the display unit 5 may display the screen SC3 instead of the screen SC2. The screen SC3 includes a pull-down menu PM1 and check boxes CB1 to CB5. The user operates the operation unit 4 or touches the screen of the display unit 5 to operate the pull-down menu PM1 and the check boxes CB1 to CB5.

[0243] The user operates the pull-down menu PM1 to select the model of the inspection instrument corresponding to the endoscope apparatus 1 currently in use. The graphic processing unit 182 receives model information indicating the model selected by the user.

[0244] The user operates check boxes CB1 to CB5 to select items of sub-graphic information to be superimposed on each frame. Check box CB1 shows the setting of sub-graphic information indicating the brightness of each frame. Check box CB2 shows the setting of sub-graphic information indicating the company logo. Check box CB3 shows the setting of sub-graphic information indicating the date. Check box CB4 shows the setting of sub-graphic information indicating the type of optical adapter. Check box CB5 shows the setting of sub-graphic information indicating the zoom state of each frame.

[0245] Brightness GI1, date GI2 and company logo GI3 are superimposed on Fig.14 Therefore, the user operates the check boxes CB1 to CB3. The graphics processing unit 182 receives setting information indicating that sub-graphics information of items corresponding to the check boxes CB1 to CB3 are superimposed on each frame.

[0246] In step S103, the graphics processing unit 182 refers to the setting information and determines whether the graphics information is superimposed on the dynamic image. When the setting information indicates that the sub-graphics information of the item corresponding to at least one of the check boxes CB1 to CB5 is superimposed on each frame, the graphics processing unit 182 determines that the graphics information is superimposed on the dynamic image. When the setting information indicates that the sub-graphics information of the item corresponding to the check boxes CB1 to CB5 is not superimposed on each frame, the graphics processing unit 182 determines that the graphics information is not superimposed on the dynamic image.

[0247] When the graphics processing unit 182 determines in step S103 that the graphics information is superimposed on the dynamic image, the graphics processing unit 182 refers to the setting information in step S105 to determine whether the sub-graphics information of each item is superimposed on each frame. In step S105, the graphics processing unit 182 generates a superimposition state table indicating whether the sub-graphics information of each item is superimposed on each frame.

[0248] Each aspect of the present invention may include the following modified examples. The CPU 18 receives model information and setting information inputted via an input device. For example, the input device is the operation unit 4 or a touch panel.

[0249] In the second embodiment, even when the model information and setting information are not pre-associated with the dynamic image, the CPU 18 sets the graphic area as an invalid area according to the model information and setting information input by the user through the input device. Therefore, the endoscope device 1 can avoid the processing failure of generating 3D data or the reduction of the accuracy of 3D data.

[0250] (Third embodiment)

[0251] A third embodiment of the present invention will be described. In the third embodiment, the device for acquiring an image of a subject and the device for generating 3D data are different. Fig.16 The configuration of an endoscope system 100 (3D data generating system) according to the third embodiment is shown. Fig.16 The endoscope system 100 shown includes an endoscope device 1 and an external device 6. The endoscope device 1 acquires two or more images of a subject and transmits the two or more images to the external device 6. The external device 6 receives the two or more images from the endoscope device 1 and performs 3D reconstruction processing by using the two or more images.

[0252] The structure of the endoscope device 1 Figure 2 The external device interface 16 communicates with the external device 6. Specifically, the external device interface 16 sends two or more images of the subject to the external device 6.

[0253] For example, the external device interface 16 is a wireless module and performs wireless communication with the external device 6. The endoscope apparatus 1 and the external device 6 may be connected by a cable such as a LAN (Local Area Network) cable, and the external device interface 16 may communicate with the external device 6 via the cable.

[0254] For example, the external device 6 is a mobile terminal such as a tablet terminal. The external device 6 may also be a fixed terminal. The form of the external device 6 is not limited to this.

[0255] Fig.17 The functional structure of the external device 6 is shown. Fig.17 The external device 6 shown includes a data communication section 60 , a CPU 61 , a display section 62 , and a RAM 63 .

[0256] The data communication unit 60 receives two or more images from the endoscope device 1. For example, the data communication unit 60 is a wireless module, and performs wireless communication with the endoscope device 1. The data communication unit 60 may also perform communication with the endoscope device 1 via a cable.

[0257] CPU61 and Figure 5The CPU 18 shown in FIG. 1 is configured similarly. The CPU 61 controls the communication between the data communication unit 60 and the endoscope device 1 . That is, the CPU 61 causes the data communication unit 60 to receive two or more images from the endoscope device 1 .

[0258] The CPU 61 may also read a program and execute the read program, and the program includes commands that define the operation of the CPU 61. That is, the function of the CPU 61 may also be realized by software. The method of installing the program is the same as the method of installing the program that realizes the function of the endoscope device 1.

[0259] The display unit 62 has a display screen, and displays images, operation menus, etc. on the display screen. The display unit 62 is a monitor (display) such as an LCD.

[0260] The RAM 63 temporarily stores information used by the CPU 61 to control the external device 6 .

[0261] use Fig.18 The processing executed by the endoscope apparatus 1 will be described. Fig.18 The following shows the steps of a process performed by the endoscope apparatus 1 .

[0262] The user knows in advance the model of the endoscope device 1 as the inspection instrument. The user operates the operation unit 4 or the touch panel to input the model information to the endoscope device 1. The CPU 18 receives the model information input through the operation unit 4 or the touch panel (step S110).

[0263] The user operates the operation unit 4 or the touch panel to input information indicating whether or not graphic information is superimposed on the moving image to the endoscope apparatus 1. After step S110, the CPU 18 receives information input via the operation unit 4 or the touch panel (step S111).

[0264] The user operates the operation unit 4 or the touch panel to input information indicating whether sub-graphic information of each item is superimposed on each frame to the endoscope apparatus 1. After step S111, the CPU 18 receives information input via the operation unit 4 or the touch panel (step S112).

[0265] After step S112 , the image pickup device 28 starts capturing images and generates a moving image (step S113 ). When the image pickup device 28 finishes capturing images, a moving image file including the moving image is recorded in the PC 41 or the memory card 42 .

[0266] After step S113, CPU 18 generates model information including the information accepted in step S110. In addition, CPU 18 generates setting information including the information accepted in step S111 and the information accepted in step S112. CPU 18 records the model information and setting information in the header of the dynamic image file (step S114). Thus, the model information and setting information are associated with the dynamic image.

[0267] After step S114, CPU 18 reads the dynamic image file from PC 41 via external device interface 16. Alternatively, CPU 18 reads the dynamic image file from memory card 42 via card interface 15. CPU 18 sends the dynamic image file to external device 6 via external device interface 16 (step S115). When step S115 is executed, Fig.18 The shown processing ends.

[0268] use Fig.19 The processing executed by the external device 6 will be described. Fig.19 The steps of the process executed by the external device 6 are shown.

[0269] The CPU 61 causes the data communication unit 60 to receive the moving image file from the endoscope device 1. Thus, the data communication unit 60 receives the moving image file from the endoscope device 1 (step S200).

[0270] After step S200 , the CPU 61 stores the moving image file received from the endoscope apparatus 1 in the RAM 63 (step S201 ).

[0271] After step S201, step S202 is performed. Fig.19 As shown, execute steps S202 to S211. Steps S202 to S211 are respectively Figure 6 The steps S100 to S109 in are the same. When step S211 is executed, Fig.19 The shown processing ends.

[0272] The endoscope device 1 may transmit the moving image to the external device 6 while the imaging element 28 generates the moving image, and the CPU 61 may perform the 3D reconstruction process in real time while receiving the moving image from the endoscope device.

[0273] Various modifications that can be applied to the endoscope device 1 in the first and second embodiments can be similarly applied to the endoscope system 100 of the third embodiment.

[0274] Each aspect of the present invention may include the following modified examples. The imaging device and the video signal processing circuit 12 (image processing device) are included in the endoscope device 1. The CPU 61 is included in the external device 6 which is separate from the endoscope device 1.

[0275] In the third embodiment, the endoscope system 100 can avoid a processing failure in generating 3D data or a decrease in the accuracy of 3D data.

[0276] The preferred embodiments of the present invention are described above, but the present invention is not limited to these embodiments and their variations. Addition, omission, substitution and other changes of the structure can be made within the scope of the present invention. In addition, the present invention is not limited by the above description, but only by the attached claims.

Claims

1. A three-dimensional data generation system, which generates three-dimensional data representing the three-dimensional shape inside an object, wherein: The three-dimensional data generation system comprises: an imaging device having a tubular insertion portion for acquiring an optical image of the interior of the object, and generating two or more images of the object in a state where the insertion portion is inserted into the interior of the object; an image processing device that superimposes graphic information on at least one of the two or more images; as well as processor, The processor obtains the two or more images, The processor determines whether the two or more images include a graphic area superimposed with the graphic information, When it is determined that the two or more images do not include the graphic area, the processor executes a first process, the first process including a generation process of generating the three-dimensional data using the two or more images. When it is determined that at least one of the two or more images includes the graphic region, the processor executes a second process including a process of making the graphic region in the at least one image not contribute to generation of the three-dimensional data.

2. The three-dimensional data generation system according to claim 1, wherein: The second process includes a process of setting the graphic area in the at least one image as an invalid area.

3. The three-dimensional data generation system according to claim 2, wherein: The second process includes the generation process using an area other than the invalid area in the two or more images.

4. The three-dimensional data generation system according to claim 2, wherein: The second process includes a process of setting the graphic area as the invalid area based on the setting information and the position information. The setting information indicates an item corresponding to the graphic information superimposed on the at least one image, The position information indicates the position of the graphic area on which the graphic information corresponding to the item is superimposed.

5. The three-dimensional data generation system according to claim 2, wherein: The second process includes setting the graphic area as the invalid area based on the model information, the setting information and the position information. The model information indicates the model of the camera device. The setting information is prepared for each model of the imaging device and indicates items corresponding to the graphic information superimposed on the at least one image. The position information indicates the position of the graphic area on which the graphic information corresponding to the item is superimposed.

6. The three-dimensional data generation system according to claim 5, wherein: The processor receives the model information and the setting information inputted through an input device.

7. The three-dimensional data generation system according to claim 2, wherein: The generation process includes a detection process of detecting feature points from the two or more images. The second process comprises: A process of changing the image in the invalid area so that no feature point is detected from the invalid area; and The generation process uses the two or more images.

8. The three-dimensional data generation system according to claim 3, wherein: The generation process includes a detection process of detecting feature points from the two or more images. The second process includes a process of excluding the invalid area from the area where the detection process is to be performed.

9. The three-dimensional data generation system according to claim 3, wherein: The generation process includes a detection process of detecting feature points from the two or more images. The second process includes a process of deleting feature points detected from the invalid area.

10. The three-dimensional data generation system according to claim 1, wherein: The second process includes a process of suspending execution of the generation process.

11. The three-dimensional data generation system according to claim 1, wherein: Two or more images generated by the imaging device and not superimposed with the graphic information by the image processing device are stored in a storage medium, The second process includes the generation process using the two or more images stored in the storage medium.

12. The three-dimensional data generation system according to claim 1, wherein: The processor determines whether the two or more images include the graphic area based on setting information indicating whether the graphic information is superimposed on each of the two or more images.

13. The three-dimensional data generation system according to claim 1, wherein: The processor performs image processing on the two or more images to detect the graphic area. The processor determines whether the two or more images include the graphic area based on the result of the image processing.

14. The three-dimensional data generation system according to claim 1, wherein: The graphic information includes at least one of the type of the optical adapter attached to the distal end of the insertion portion, brightness of the image generated by the imaging device, date, and mark, and magnification of the image generated by the imaging device.

15. The three-dimensional data generation system according to claim 1, wherein: After the two or more images are stored in a storage medium, the processor acquires the two or more images from the storage medium.

16. The three-dimensional data generation system according to claim 1, wherein: The insertion portion has a lens and an image sensor.

17. The three-dimensional data generation system according to claim 16, wherein: The lens and the image sensor are built into the front end of the insertion portion. The positions of the front ends when generating the two or more images are different from each other, The postures of the front end when the two or more images are generated are different from each other.

18. The three-dimensional data generation system according to claim 1, wherein: The imaging device, the image processing device, and the processor are included in an endoscope device.

19. The three-dimensional data generation system according to claim 1, wherein: The imaging device and the image processing device are included in an endoscope device, and the processor is included in an external device separate from the endoscope device.

20. The three-dimensional data generation system according to claim 1, wherein: The imaging device generates the two or more images based on an optical image formed by an optical adapter of a single eye mounted on a distal end of the insertion portion.

21. The three-dimensional data generation system according to claim 1, wherein: In at least two images included in the two or more images, the same region of the structural element of the object is captured.

22. A three-dimensional data generation method, which generates three-dimensional data representing the three-dimensional shape inside an object, wherein: The three-dimensional data generation method comprises the following steps: The processor acquires two or more images of the object, wherein the two or more images are generated by an imaging device having a tubular insertion portion for acquiring an optical image of the interior of the object, and the two or more images are generated in a state where the insertion portion is inserted into the interior of the object; The processor determines whether the two or more images include a graphic area on which graphic information is superimposed by the image processing device; When it is determined that the two or more images do not include the graphic area, the processor executes a first process, the first process including a generation process of generating the three-dimensional data using the two or more images; as well as When it is determined that at least one of the two or more images includes the graphic region, the processor executes a second process including a process of making the graphic region in the at least one image not contribute to generation of the three-dimensional data.

23. A computer-readable recording medium having recorded thereon a program for causing a computer to execute a process for generating three-dimensional data representing a three-dimensional shape inside an object, wherein: The process comprises the following steps: Acquiring two or more images of the object, wherein the two or more images are generated by an imaging device having a tubular insertion portion for acquiring an optical image of the interior of the object, and generating the two or more images in a state where the insertion portion is inserted into the interior of the object; determining whether the two or more images include a graphic area on which graphic information is superimposed by an image processing device; When it is determined that the two or more images do not include the graphic area, executing a first process, the first process including a generation process of generating the three-dimensional data using the two or more images; as well as When it is determined that at least one of the two or more images includes the graphic region, a second process is executed, the second process including a process of making the graphic region in the at least one image not contribute to the generation of the three-dimensional data.

Images