Image processing device, image processing method, and computer-readable recording medium

Abstract

This image processing device 10 comprises: a combined model generation unit 11 that combines a first three-dimensional model indicating an outer surface of a structure having an internal space with a second three-dimensional model indicating an inner surface of the structure, thereby generating a combined model; and a display unit 12 that displays the combined model on a screen.

Description

Image processing device, image processing method, and computer-readable recording medium

[0001] The present disclosure relates to an image processing device and an image processing method for assisting in the detection of specific portions of a structure from an image, and further to a computer-readable recording medium on which a program for realizing these is recorded.

[0002] Traditionally, workers have regularly inspected infrastructure structures such as bridges visually to find defects. However, this type of inspection places a heavy burden on workers and is inefficient, so in recent years, inspection methods using images have been proposed.

[0003] For example, Patent Document 1 discloses a device for detecting defects from an image of an object. The device disclosed in Patent Document 1 detects defects such as cracks from an image of an infrastructure structure and presents the location of the detected defect to a worker.

[0004] Therefore, the device disclosed in Patent Document 1 can reduce the burden on workers and improve the efficiency of inspecting infrastructure structures. Furthermore, the device disclosed in Patent Document 1 is also useful for inspecting structures other than infrastructure structures, such as water reservoirs and spherical tanks for city gas.

[0005] Japanese Patent Application Laid-Open No. 2021-51597

[0006] However, unlike infrastructure structures, structures such as water reservoirs and spherical city gas tanks (gas holders) have large internal spaces. Therefore, if a defect is detected on the exterior, it is necessary to verify the defect on the interior at the location corresponding to the defect on the exterior, the surrounding area, and a location vertically downward from there in order to confirm the effect on the interior.

[0007] However, the device disclosed in Patent Document 1 does not identify the correspondence between the image of the outer surface and the image of the inner surface. Therefore, even if a defect on the inner surface can be detected from the image of the inner surface, the worker must determine where on the outer surface the detected defect on the inner surface corresponds. The worker must identify the position of the image where the defect on the outer surface was detected and the position of the image where the defect on the inner surface was detected based on the diagram of the structure, and determine the correspondence between the defect on the outer surface and the defect on the inner surface. This type of determination places a heavy burden on the worker.

[0008] One example of an objective of the present disclosure is to enable inspection of a specific location in a structure having an internal space from both the external and internal surfaces at the same time.

[0009] In order to achieve the above object, an image processing device according to one aspect of the present disclosure is characterized by comprising: a combined model generation unit that combines a first three-dimensional model representing the outer surface of a structure having an internal space with a second three-dimensional model representing the inner surface of the structure, thereby generating a combined model; and a display unit that displays the combined model on a screen.

[0010] In addition, in order to achieve the above object, an image processing method according to one aspect of the present disclosure is characterized by having a combined model generation step of combining a first three-dimensional model representing the outer surface of a structure having an internal space with a second three-dimensional model representing the inner surface of the structure, thereby generating a combined model; and a display step of displaying the combined model on a screen.

[0011] Furthermore, in order to achieve the above object, a computer-readable recording medium according to one aspect of the present disclosure is characterized in that it records a program including instructions for causing a computer to execute: a combined model generation step of combining a first three-dimensional model representing the outer surface of a structure having an internal space with a second three-dimensional model representing the inner surface of the structure, thereby generating a combined model; and a display step of displaying the combined model on a screen.

[0012] As described above, according to the present disclosure, a specific location in a structure having an internal space can be inspected from both the outer surface and the inner surface at the same time.

[0013] FIG. 1 is a configuration diagram showing a schematic configuration of an example of an image processing device. FIG. 2 is a configuration diagram specifically showing the configuration of an example of an image processing device. FIG. 3 is a diagram showing an example of a generation process of a bonded model in the image processing device. FIG. 4 is a diagram showing an example of a photographing process of image data used for an exterior model and an interior model. FIG. 5 is a diagram showing an example of a bonded model displayed on a screen. FIG. 6 is a diagram showing an example of an image of a specific location and a corresponding location displayed on a screen. FIG. 7 is a flow diagram showing an example of the operation of the image processing device. FIG. 8 is a block diagram showing an example of a computer that realizes the image processing device.

[0014] First Embodiment An image processing device, an image processing method, and a program according to an embodiment will be described below with reference to FIGS.

[0015] [Device Configuration] First, the schematic configuration of an example of an image processing device will be described with reference to Fig. 1. Fig. 1 is a diagram showing the schematic configuration of an example of an image processing device.

[0016] The image processing device 10 shown in Fig. 1 is a device that supports the detection of a specific portion of a structure from an image. As shown in Fig. 1, the image processing device 10 includes a bond model generation unit 11 and a display unit 12.

[0017] The combined model generator 11 combines a first three-dimensional model (hereinafter referred to as the "exterior model") representing the exterior surface of a structure having an internal space with a second three-dimensional model (hereinafter referred to as the "interior model") representing the interior surface of the structure, thereby generating a combined model. The display unit 12 displays the generated combined model on a screen.

[0018] In this way, the image processing device 10 can generate a combined model that shows both the outer and inner surfaces of a structure. Therefore, when a specific location is specified on the outer or inner surface of a structure with an internal space, the combined model makes it easy to confirm the corresponding location on the opposite surface. The image processing device 10 can verify a specific location in a structure with an internal space from both the outer and inner surfaces at the same time.

[0019] Next, the configuration and functions of the image processing device 10 will be specifically described with reference to Figures 2 to 5. Figure 2 is a block diagram specifically showing the configuration of an example of an image processing device. Figure 3 is a diagram showing an example of a process for generating a bond model in the image processing device.

[0020] 2, the image processing device 10 includes a three-dimensional model acquisition unit 13 and a specific location detection unit 14 in addition to the binding model generation unit 11 and display unit 12. The image processing device 10 is also connected to a database 20 and a user terminal device 30 via a network or the like so as to be able to communicate data with each other.

[0021] In the following description, the target structure 40 is assumed to be a closed structure having an internal space. Specifically, the structure 40 shown in Fig. 2 is, for example, a water reservoir that stores purified water before distribution. An opening connecting the external and internal surfaces of the structure 40 is provided on the external surface, and the opening is normally closed by a hatch 42.

[0022] Furthermore, in the example of FIG. 2, the three-dimensional model generating device 50 generates an exterior surface model and an interior surface model of the structure 40 using image data of the structure 40 captured by the imaging device 51 .

[0023] The imaging device 51 may be a digital camera, a time-of-flight (TOF) sensor, a depth camera such as a light detection and ranging (LiDar), or a point cloud scanner.

[0024] When a digital camera is used as the imaging device 51, the 3D model generation device 50 applies, for example, a structure from motion (SfM) technique to image data of the exterior surface of the structure 40 to generate 3D point cloud data that serves as an exterior surface model. Similarly, the 3D model generation device 50 applies the SfM technique to image data of the interior surface of the structure 40 to generate 3D point cloud data that serves as an interior surface model.

[0025] Furthermore, when a depth camera is used as the imaging device 51, each pixel of the image data has depth information. Therefore, the 3D model generation device 50 generates point cloud data from each piece of image data of the exterior surface of the structure 40, and generates an exterior surface model by combining the generated point cloud data. Similarly, the 3D model generation device 50 generates point cloud data from each piece of image data of the interior surface of the structure 40, and generates an interior surface model by combining the generated point cloud data.

[0026] Furthermore, if a point cloud scanner is used as the imaging device 51, each output point has depth information for the scanned position. Therefore, the 3D model generation device 50 registers point cloud data acquired from multiple locations on the exterior surface of the structure 40 based on the depth information for each point and combines the data into a single point cloud. This generates an exterior surface model. Similarly, the 3D model generation device 50 registers point cloud data acquired from multiple locations on the interior surface of the structure 40 and combines the data into a single point cloud data, thereby generating an interior surface model.

[0027] In this way, the exterior surface model and the interior surface model are three-dimensional point cloud data constructed from a plurality of image data obtained by photographing the structure 40. In addition, the image data group, the exterior surface model, and the interior surface model used to generate the three-dimensional point cloud data are stored in the database 20.

[0028] The three-dimensional model acquisition unit 13 acquires the exterior surface model and the interior surface model from the database 20. In the embodiment, the image processing device 10 may also include a three-dimensional model generation device 50 instead of the three-dimensional model acquisition unit 13.

[0029] The combined model generation unit 11 receives an exterior surface model 60 representing the exterior surface of the structure 40 and an interior surface model 61 representing the interior surface of the structure 40 from the three-dimensional model acquisition unit 13. Then, as shown in Fig. 3, the combined model generation unit 11 combines the exterior surface model 60 and the interior surface model 61, thereby generating a combined model 62.

[0030] Furthermore, the combined model generation unit 11 combines the exterior model 60 and the interior model 61 using information indicating the positional relationship between the markers placed on the exterior surface and the interior surface. In Figure 3, 63 indicates a point cloud of the markers represented on the exterior model 60.

[0031] Here, information used to combine the exterior model 60 and the interior model 61 will be described with reference to Fig. 4. Fig. 4 is a diagram showing an example of a photographing process for image data used for the exterior model and the interior model. Note that, although the following describes a case where a digital camera is used as the imaging device 51, the imaging device 51 is not limited to a digital camera.

[0032] 4 , first, an operator 52 photographs the outer and inner surfaces of the structure 40 using an imaging device 51, and generates image data of the outer and inner surfaces. The image data of the outer surface (hereinafter referred to as "outer surface image data") and the image data of the inner surface (hereinafter referred to as "inner surface image data") are sent to a 3D model generation device 50.

[0033] Furthermore, hatches 42 are provided on the outer surface of the structure 40, and markers 43 are placed around the hatches 42. Therefore, part of the outer surface image data includes the markers 43.

[0034] Furthermore, when the hatch 42 is opened, an opening 41 appears, and the outer surface and the inner surface are connected via the opening 41. Therefore, an operator 52 uses the opening 41 to photograph both the marker 43 and a part of the inner surface with an imaging device 51, and generates image data of both images (hereinafter referred to as "outer and inner surface image data"). The outer and inner surface image data is also sent to the 3D model generation device 50.

[0035] Thereafter, the three-dimensional model generating device 50 generates an exterior model using the exterior image data by the above-described method. At this time, the markers 43 appear as a point cloud 63 in the three-dimensional point cloud data that constructs the exterior model (see FIG. 3). The three-dimensional model generating device 50 also generates an interior model using the interior image data and the outer and inner interior image data by the above-described method. As a result, the markers 43 also appear as a point cloud in the three-dimensional point cloud data that constructs the interior model.

[0036] Therefore, the combined model generation unit 11 combines the exterior model and the interior model using the markers 43 (point cloud 63) that appear in the three-dimensional point cloud data that constructs the exterior model and the markers 43 that appear in the three-dimensional point cloud data that constructs the interior model as information that indicates the above-mentioned positional relationship. Specifically, the combined model generation unit 11 combines both models such that the point cloud 63 of the markers 43 that appear in the three-dimensional point cloud data that constructs the exterior model overlaps with the point cloud of the markers 43 that appear in the three-dimensional point cloud data that constructs the interior model.

[0037] The specific location detection unit 14 detects a specific location on the exterior surface model or the interior surface model. Examples of specific locations include locations where defects such as damage, rust, or cracks occur. The specific location detection unit 14 detects the specific location by, for example, mapping the position of a defect detected on the image of the structure 40 onto the exterior surface model or the interior surface model. Examples of mapping techniques include those described in International Publication No. 2020 / 111139.

[0038] Furthermore, when the specific part detection unit 14 detects a specific part in the exterior model, it uses the combined model to identify a part in the interior model that corresponds to the detected specific part. Furthermore, when the specific part detection unit 14 detects a specific part in the interior model, it uses the combined model to identify a part in the exterior model that corresponds to the detected specific part.

[0039] The display unit 12 displays a bonded model 62 on the screen 31, as shown in Fig. 5. Fig. 5 is a diagram showing an example of a bonded model displayed on the screen. In the example of Fig. 5, the screen 31 is the screen of the terminal device 30. The display unit 12 also displays, in the bonded model 62 on the screen 31, a specific location 64 of the detected exterior surface model or interior surface model, and a location 65 of the other model that corresponds to the specific location 64. Specifically, in the example of Fig. 5, the specific location 64 is a crack. In the example of Fig. 5, the corresponding location 65 is indicated by a virtual line.

[0040] The display unit 12 also displays on the screen 31 a virtual line indicating the relationship between the detected specific location 64 and a location 65 corresponding to the specific location 64. Specifically, if a defect occurs on the outer surface, there is a possibility that a defect also occurs on the inner surface in the vertical direction of the defect, and therefore the display unit 12 also displays a virtual line (broken line) indicating the vertical direction of the location 65 corresponding to the specific location 64.

[0041] Furthermore, the display unit 12 can also display one or both of the exterior model 60 and the interior model 61 in a transparent state in the combined model 62 on the screen 31. Here, a transparent state means a state in which the transmittance is greater than 0, and specifically means a transmittance of about 50%. In this way, when one or both of the exterior model 60 and the interior model 61 are made transparent, the relationship between the specific location 64 and the corresponding location 65 is visualized.

[0042] The display unit 12 can also acquire image data of the specific location 64 and the corresponding location 65 from the database 20 using information such as the feature quantities of the specific location 64 and the corresponding location 65 (see, for example, Japanese Patent Application Laid-Open No. 2023-20180). In this case, the display unit 12 can also display images of each acquired image data on the screen 31, as shown in FIG. 6. This aspect is useful for a person in charge of inspecting and managing the structure 40. FIG. 6 is a diagram showing an example of images of the specific location and the corresponding location displayed on the screen.

[0043] The above has been a description of the case where a digital camera is used as the imaging device 51. Here, a description will be given of the case where a point cloud scanner is used as the imaging device 51. An example of a point cloud scanner is one configured by a LiDAR and a digital camera in a fixed relationship. In this case, the point cloud scanner calculates the positional relationship between the 3D point cloud data and the image data each time an image is captured, and generates positional relationship information. The positional relationship information is stored in the database 20 in a state linked to the image data group, the exterior surface model, and the interior surface model. By using the positional relationship information obtained in this manner, the point cloud scanner can achieve the same functionality as a digital camera when used as the imaging device 51.

[0044] Note that methods for calculating the positional relationship between 3D point cloud data and image data include various calibration methods, such as the method disclosed in the following document: (Reference) Joint Camera Intrinsic and LiDAR-Camera Extrinsic Calibration (https: / / arxiv.org / pdf / 2202.13708)

[0045] [Device Operation] Next, an example of the operation of the image processing device 10 will be described with reference to FIG. 7. FIG. 7 is a flow diagram showing an example of the operation of the image processing device. In the following description, reference will be made to FIGS. 1 to 6 as appropriate. In addition, in the embodiment, an image processing method is implemented by operating the image processing device 10. Therefore, the description of the image processing method in the embodiment will be replaced by the following description of the operation of the image processing device 10.

[0046] First, it is assumed that an operator 52 photographs the exterior and interior surfaces of a structure 40 using an imaging device 51 and generates exterior surface image data, interior surface image data, and exterior / interior surface image data. Furthermore, it is assumed that a three-dimensional model generation device 50 generates an exterior surface model and an interior surface model using the image data.

[0047] 7, first, the three-dimensional model acquisition unit 13 acquires from the database 20 an exterior surface model 60 representing the exterior surface of the structure 40 and an interior surface model 61 (see FIG. 3) representing the interior surface of the structure 40 (step A1). Then, the three-dimensional model acquisition unit 13 sends the acquired exterior surface model 60 and interior surface model 61 to the combined model generation unit 11.

[0048] Next, the combined model generating unit 11 combines the exterior surface model 60 and the interior surface model 61 acquired in step A1, thereby generating a combined model 62 (see FIG. 3) (step A2).

[0049] In step A2, the combined model generation unit 11 uses information indicating the positional relationship between the markers 43 arranged on the outer surface and the inner surface to combine the exterior model 60 and the inner surface model 61. Specifically, the combined model generation unit 11 combines the exterior model and the inner surface model using the markers 43 (point cloud 63) that appear in the three-dimensional point cloud data that constructs the exterior model 60 and the markers 43 that appear in the three-dimensional point cloud data that constructs the inner surface model as information indicating the above-mentioned positional relationship.

[0050] Next, the specific part detection unit 14 detects specific parts on the exterior surface model or the interior surface model (step A3). As described above, specific parts include parts where defects such as damage, rust, and cracks have occurred.

[0051] Next, the specific part detection unit 14 uses the combined model 62 to detect a part corresponding to the detected specific part in a model other than the model in which the specific part was detected (step A4).

[0052] Next, the display unit 12 displays the bond model 62 on the screen 31, and further displays the specific location 64 and a location 65 corresponding to the specific location 64 in the bond model 62 (step A5).

[0053] In step A5, the display unit 12 also displays on the screen 31 a virtual line indicating the relationship between the specific location 64 and the corresponding location 65, and a virtual line indicating the vertical direction of the location 65 corresponding to the specific location 64. Furthermore, the display unit 12 can also display on the screen 31 an image of the specific location and an image of the corresponding location 65, as shown in FIG.

[0054] [Effects of the embodiment] As described above, in the embodiment, the outer surface model 60 and the inner surface model 61 of the structure 40 are combined to generate the combined model 62. If a defect occurs on either or both of the outer surface and the inner surface of the structure, the combined model 62 identifies the corresponding location on the opposite surface and presents the defective location and the corresponding location. In this way, according to the embodiment, it becomes possible to verify a specific location in the structure 40 from both the outer surface and the inner surface at the same time.

[0055] [Program] In the embodiments, an example of the program is a program that causes a computer to execute steps A1 to A5 shown in Fig. 7. By installing and executing this program on a computer, the image processing device 10 and the image processing method can be realized. In this case, the processor of the computer functions as the bond model generation unit 11, the display unit 12, the three-dimensional model acquisition unit 13, and the specific location detection unit 14, and performs processing. Examples of the computer include a general-purpose PC, a server computer, a smartphone, and a tablet terminal device.

[0056] In this embodiment, the program may be executed by a computer system constructed by a plurality of computers. In this case, for example, each computer may function as any one of the bond model generation unit 11, the display unit 12, the three-dimensional model acquisition unit 13, and the specific location detection unit 14.

[0057] [Physical Configuration] A computer that implements the image processing device 10 by executing a program according to the embodiment will now be described with reference to Fig. 8. Fig. 8 is a block diagram showing an example of a computer that implements the image processing device.

[0058] 8, the computer 110 includes a CPU (Central Processing Unit) 111, a main memory 112, a storage device 113, an input interface 114, a display controller 115, a data reader / writer 116, and a communication interface 117. These components are connected to each other via a bus 121 so as to be able to communicate data with each other.

[0059] Furthermore, the computer 110 may include a GPU (Graphics Processing Unit) or an FPGA (Field-Programmable Gate Array) in addition to or instead of the CPU 111. In this aspect, the GPU or FPGA can execute the programs in the embodiments.

[0060] The CPU 111 loads a program in the embodiment, which is composed of a group of codes and stored in the storage device 113, into the main memory 112 and executes each code in a predetermined order to perform various calculations. The main memory 112 is typically a volatile storage device such as a DRAM (Dynamic Random Access Memory).

[0061] The program in the embodiment is provided in a state stored in a computer-readable recording medium 120. The program in the embodiment may be distributed over the Internet connected via the communication interface 117.

[0062] Specific examples of the storage device 113 include a hard disk drive and a semiconductor storage device such as a flash memory. The input interface 114 mediates data transmission between the CPU 111 and input devices 118 such as a keyboard and a mouse. The display controller 115 is connected to a display device 119 and controls the display on the display device 119.

[0063] The data reader / writer 116 mediates data transmission between the CPU 111 and the recording medium 120, reads programs from the recording medium 120, and writes processing results from the computer 110 to the recording medium 120. The communication interface 117 mediates data transmission between the CPU 111 and other computers.

[0064] Specific examples of the recording medium 120 include general-purpose semiconductor storage devices such as CF (Compact Flash (registered trademark)) and SD (Secure Digital), magnetic recording media such as flexible disks, or optical recording media such as CD-ROMs (Compact Disk Read Only Memory).

[0065] The image processing device 10 can be realized not by a computer with a program installed, but by hardware corresponding to each part, such as an electronic circuit. Furthermore, the image processing device 10 may be realized in part by a program and in the remaining part by hardware. In the embodiment, the computer is not limited to the computer shown in FIG. 8.

[0066] Some or all of the above-described embodiments can be expressed by (Supplementary Note 1) to (Supplementary Note 15) described below, but are not limited to the following descriptions.

[0067] (Supplementary Note 1) An image processing device comprising: a combined model generation unit that combines a first three-dimensional model showing the outer surface of a structure having an internal space with a second three-dimensional model showing the inner surface of the structure, thereby generating a combined model; and a display unit that displays the combined model on a screen.

[0068] (Supplementary Note 2) The image processing device according to Supplementary Note 1 further comprises a specific location detection unit that detects a specific location on the first three-dimensional model or the second three-dimensional model, and uses the combined model to identify a location corresponding to the detected specific location in a three-dimensional model other than the three-dimensional model in which the specific location was detected, and the display unit displays the detected specific location and a location corresponding to the detected specific location in the combined model.

[0069] (Supplementary Note 3) The image processing device according to Supplementary Note 2, wherein the display unit displays one or both of the first three-dimensional model and the second three-dimensional model in the combined model on the screen in a transparent state.

[0070] (Supplementary Note 4) The image processing device according to Supplementary Note 2, wherein the display unit displays, on the screen, a virtual line indicating a relationship between the detected specific location and a location corresponding to the specific location in the binding model.

[0071] (Supplementary Note 5) The image processing device according to Supplementary Note 1, wherein the combined model generation unit combines the first three-dimensional model and the second three-dimensional model using information indicating a positional relationship between a marker placed on the outer surface and the inner surface.

[0072] (Supplementary Note 6) The image processing device according to Supplementary Note 5, wherein the structure has an opening connecting an outer surface and an inner surface, the first three-dimensional model is three-dimensional point cloud data constructed from a plurality of image data obtained by photographing the outer surface, the second three-dimensional model is three-dimensional point cloud data constructed using a plurality of image data obtained by photographing the inner surface and image data obtained by photographing the marker from the inner surface through the opening, and the combined model generation unit combines the first three-dimensional model and the second three-dimensional model using, as the information, the markers that appear in the three-dimensional point cloud data that constructs the first three-dimensional model and the markers that appear in the three-dimensional point cloud data that constructs the second three-dimensional model.

[0073] (Supplementary Note 7) An image processing method comprising: a combined model generating step of combining a first three-dimensional model showing an outer surface of a structure having an internal space with a second three-dimensional model showing an inner surface of the structure, thereby generating a combined model; and a display step of displaying the combined model on a screen.

[0074] (Supplementary Note 8) The image processing method according to Supplementary Note 7, further comprising a specific location detection step of detecting a specific location on the first three-dimensional model or the second three-dimensional model, and using the combined model, identifying a location corresponding to the detected specific location in a three-dimensional model other than the three-dimensional model in which the specific location was detected, and in the display step, displaying the detected specific location and a location corresponding to the detected specific location in the combined model.

[0075] (Supplementary Note 9) The image processing method according to Supplementary Note 8, wherein in the displaying step, one or both of the first three-dimensional model and the second three-dimensional model are displayed in a transparent state on the screen in the combined model.

[0076] (Supplementary Note 10) The image processing method according to Supplementary Note 8, wherein the display step displays a virtual line on the screen, the virtual line indicating a relationship between the detected specific location and a location corresponding to the specific location in the binding model.

[0077] (Supplementary Note 11) The image processing method according to Supplementary Note 7, wherein the combined model generating step combines the first three-dimensional model and the second three-dimensional model using information indicating a positional relationship between a marker arranged on the outer surface and the inner surface.

[0078] (Supplementary Note 12) The image processing method according to Supplementary Note 11, wherein the structure has an opening connecting an outer surface and an inner surface, the first three-dimensional model is three-dimensional point cloud data constructed from a plurality of image data obtained by photographing the outer surface, and the second three-dimensional model is three-dimensional point cloud data constructed using a plurality of image data obtained by photographing the inner surface and image data obtained by photographing the marker from the inner surface through the opening, and the combined model generation step combines the first three-dimensional model and the second three-dimensional model using, as the information, the markers that appear in the three-dimensional point cloud data that constructs the first three-dimensional model and the markers that appear in the three-dimensional point cloud data that constructs the second three-dimensional model.

[0079] (Supplementary Note 13) A computer-readable recording medium having recorded thereon a program including instructions for causing a computer to execute: a combined model generating step of combining a first three-dimensional model showing the outer surface of a structure having an internal space with a second three-dimensional model showing the inner surface of the structure, thereby generating a combined model; and a display step of displaying the combined model on a screen.

[0080] (Supplementary Note 14) The computer-readable recording medium according to Supplementary Note 13, wherein the program further includes instructions to cause the computer to execute a specific location detection step of detecting a specific location on the first three-dimensional model or the second three-dimensional model, and using the combined model, identifying a location corresponding to the detected specific location in a three-dimensional model other than the three-dimensional model in which the specific location was detected; and wherein the display step displays the detected specific location and a location corresponding to the detected specific location in the combined model.

[0081] (Supplementary Note 15) The computer-readable recording medium according to Supplementary Note 14, wherein in the displaying step, one or both of the first three-dimensional model and the second three-dimensional model are displayed in a transparent state on the screen in the combined model.

[0082] (Supplementary Note 16) The computer-readable recording medium according to Supplementary Note 14, wherein the display step displays a virtual line on the screen, the virtual line indicating a relationship between the detected specific location and a location corresponding to the specific location in the binding model.

[0083] (Supplementary Note 17) The computer-readable recording medium according to Supplementary Note 13, wherein in the combined model generation step, the first three-dimensional model and the second three-dimensional model are combined using information indicating a positional relationship between a marker arranged on the outer surface and the inner surface.

[0084] (Supplementary Note 18) The computer-readable recording medium according to Supplementary Note 17, wherein the structure has an opening connecting an outer surface and an inner surface, the first three-dimensional model is three-dimensional point cloud data constructed from a plurality of image data obtained by photographing the outer surface, and the second three-dimensional model is three-dimensional point cloud data constructed using a plurality of image data obtained by photographing the inner surface and image data obtained by photographing the marker from the inner surface through the opening, and the combined model generation step combines the first three-dimensional model and the second three-dimensional model using, as the information, the markers that appear in the three-dimensional point cloud data that constructs the first three-dimensional model and the markers that appear in the three-dimensional point cloud data that constructs the second three-dimensional model.

[0085] Although the present invention has been described above with reference to the embodiments, the present invention is not limited to the above-described embodiments. Various modifications that can be understood by those skilled in the art can be made to the configuration and details of the present invention within the scope of the present invention.

[0086] This application claims priority based on Japanese Patent Application No. 2024-088503, filed May 31, 2024, the disclosure of which is incorporated herein in its entirety by reference.

[0087] As described above, according to the present disclosure, a specific location in a structure having an internal space can be simultaneously inspected from both the exterior and interior surfaces, which is useful for various systems that detect a specific location of a structure from an image.

[0088] REFERENCE SIGNS LIST 10 Image processing device 11 Bonded model generation unit 12 Display unit 13 Three-dimensional model acquisition unit 14 Specific location detection unit 20 Database 30 Terminal device 31 Screen 40 Structure 41 Opening 42 Hatch 43 Marker 50 Three-dimensional model generation device 51 Imaging device 52 Worker 60 Exterior surface model 61 Interior surface model 62 Bonded model 63 Point cloud indicating marker 64 Specific location 65 Location corresponding to specific location 110 Computer 111 CPU 112 Main memory 113 Storage device 114 Input interface 115 Display controller 116 Data reader / writer 117 Communication interface 118 Input device 119 Display device 120 Recording medium 121 Bus

Claims

1. An image processing device comprising: a combined model generation means for combining a first three-dimensional model showing the outer surface of a structure having an internal space with a second three-dimensional model showing the inner surface of said structure, thereby generating a combined model; and a display means for displaying said combined model on a screen.

2. The image processing device according to claim 1, further comprising a specific location detection means that detects a specific location on the first three-dimensional model or the second three-dimensional model, and uses the combined model to identify a location corresponding to the detected specific location in a three-dimensional model other than the three-dimensional model in which the specific location was detected, and wherein the display means displays the detected specific location and a location corresponding to the detected specific location in the combined model.

3. The image processing device according to claim 2, wherein said display means displays one or both of said first three-dimensional model and said second three-dimensional model in said combined model on said screen in a transparent state.

4. The image processing device according to claim 2, wherein said display means displays on said screen a virtual line indicating the relationship between said detected specific location and a location corresponding to said specific location in said bond model.

5. The image processing device according to claim 1, wherein the combined model generating means combines the first three-dimensional model and the second three-dimensional model using information indicating the positional relationship between the markers placed on the outer surface and the inner surface.

6. The image processing device according to claim 5, wherein the structure has an opening connecting the outer surface and the inner surface, the first three-dimensional model is three-dimensional point cloud data constructed from a plurality of image data obtained by photographing the outer surface, the second three-dimensional model is three-dimensional point cloud data constructed using a plurality of image data obtained by photographing the inner surface and image data obtained by photographing the marker from the inner surface through the opening, and the combined model generation means combines the first three-dimensional model and the second three-dimensional model using, as the information, the marker appearing in the three-dimensional point cloud data constructing the first three-dimensional model and the marker appearing in the three-dimensional point cloud data constructing the second three-dimensional model.

7. An image processing method comprising: combining a first three-dimensional model showing the outer surface of a structure having an internal space with a second three-dimensional model showing the inner surface of said structure, thereby generating a combined model; and displaying said combined model on a screen.

8. The image processing method according to claim 7, further comprising: detecting a specific location on the first three-dimensional model or the second three-dimensional model; using the combined model, identifying a location corresponding to the detected specific location in a three-dimensional model other than the three-dimensional model in which the specific location was detected; and displaying the detected specific location and a location corresponding to the detected specific location in the combined model on a screen.

9. The image processing method according to claim 8, wherein, in displaying the combined model on a screen, one or both of the first three-dimensional model and the second three-dimensional model are displayed in a transparent state on the screen in the combined model.

10. The image processing method according to claim 8, wherein, in displaying the bond model on a screen, a virtual line is displayed on the screen indicating the relationship between the detected specific location in the bond model and a location corresponding to the specific location.

11. The image processing method according to claim 7, wherein in generating the combined model, the first three-dimensional model and the second three-dimensional model are combined using information indicating the positional relationship between markers placed on the outer surface and the inner surface.

12. The image processing method according to claim 11, wherein the structure has an opening connecting the outer surface and the inner surface, the first three-dimensional model is three-dimensional point cloud data constructed from a plurality of image data obtained by photographing the outer surface, and the second three-dimensional model is three-dimensional point cloud data constructed using a plurality of image data obtained by photographing the inner surface and image data obtained by photographing the marker from the inner surface through the opening, and in generating the combined model, the markers appearing in the three-dimensional point cloud data constructing the first three-dimensional model and the markers appearing in the three-dimensional point cloud data constructing the second three-dimensional model are used as the information to combine the first three-dimensional model and the second three-dimensional model.

13. A computer-readable recording medium having a program recorded thereon, the program including instructions for causing a computer to combine a first three-dimensional model showing the exterior surface of a structure having an interior space with a second three-dimensional model showing the interior surface of said structure, thereby generating a combined model, and displaying said combined model on a screen.

14. A computer-readable recording medium as described in claim 13, wherein the program further includes instructions to the computer to: detect a specific location on the first three-dimensional model or the second three-dimensional model; and, using the combined model, identify a location corresponding to the detected specific location in a three-dimensional model other than the three-dimensional model in which the specific location was detected; and, when displaying the combined model on a screen, display the detected specific location and a location corresponding to the detected specific location in the combined model.

15. The computer-readable recording medium according to claim 14, wherein, in displaying the combined model on a screen, one or both of the first three-dimensional model and the second three-dimensional model are displayed in a transparent state on the screen in the combined model.

16. The computer-readable recording medium according to claim 14, wherein, in displaying the binding model on a screen, a virtual line is displayed on the screen indicating the relationship between the detected specific location and a location corresponding to the specific location in the binding model.

17. The computer-readable recording medium according to claim 13, wherein in generating the combined model, the first three-dimensional model and the second three-dimensional model are combined using information indicating the positional relationship between the markers placed on the outer surface and the inner surface.

18. The computer-readable recording medium according to claim 17, wherein the structure has an opening connecting the outer surface and the inner surface, the first three-dimensional model is three-dimensional point cloud data constructed from a plurality of image data obtained by photographing the outer surface, the second three-dimensional model is three-dimensional point cloud data constructed using a plurality of image data obtained by photographing the inner surface and image data obtained by photographing the marker from the inner surface through the opening, and in generating the combined model, the markers appearing in the three-dimensional point cloud data constructing the first three-dimensional model and the markers appearing in the three-dimensional point cloud data constructing the second three-dimensional model are used as the information to combine the first three-dimensional model and the second three-dimensional model.

Images