Device for generating deformation information, method for generating deformation information, and program

Abstract

A degradation information generation device 1X mainly includes a part inference means 15X, a degradation detection means 16X, and a degradation information generation means 17X. The part inference means 15X infers, on the basis of three-dimensional data representing a structure and a first image obtained by partially imaging the structure, parts of the structure included in the first image. The degradation detection means 16X detects the degradation of the structure, on the basis of the first image or a second image obtained by imaging a portion of an imaging range of the first image. The degradation information generation means 17X generates, on the basis of the inference result of the parts and the detection result of the degradation, degradation information indicating at least identification information of a part in which the degradation has occurred.

Description

Deformation information generating device, deformation information generating method, and storage medium

[0001] The present disclosure relates to the technical fields of a deformation information generation device, a deformation information generation method, and a storage medium that generate deformation information of a structure.

[0002] From a safety perspective, it is necessary to periodically inspect structures such as bridges, tunnels, and buildings to determine whether or not there is any damage or deterioration. Furthermore, in order to develop appropriate repair plans, it is necessary to appropriately generate and manage information about the structure's deformation obtained through inspections. Regarding the detection of such deformation, for example, Patent Literature 1 discloses an image processing device that analyzes images of cracked areas and presents a composite image of the cracked areas and the image analysis results.

[0003] Japanese Patent Application Laid-Open No. 2023-78205

[0004] The method of Patent Document 1 makes it possible to obtain composite images and image analysis results of crack locations without requiring expensive equipment and without requiring complicated user operations. On the other hand, Patent Document 1 has a problem in that, when recording deformations, the user manually enters image position information into a field notebook at the inspection site, which results in a large workload for linking deformations to structural parts.

[0005] In view of the above-mentioned problems, one of the objectives of the present disclosure is to provide a deformation information generation device, a deformation information generation method, and a storage medium that generate deformation information that can identify parts of a structure in which a deformation has been detected.

[0006] One aspect of the deformation information generation device is a deformation information generation device having: a part estimation means for estimating parts of the structure contained in a first image based on three-dimensional data representing the structure and the first image that is a partial photograph of the structure; a deformation detection means for detecting deformation of the structure based on the first image or a second image that is a photograph of a portion of the photographing range of the first image; and a deformation information generation means for generating deformation information that indicates at least identification information of the part in which the deformation has occurred based on the part estimation results and the deformation detection results.

[0007] One aspect of the deformation information generating method is a method in which a computer estimates parts of the structure included in a first image based on three-dimensional data representing the structure and a first image in which the structure is partially photographed, detects deformation of the structure based on the first image or a second image in which a part of the photographed range of the first image is photographed, and generates deformation information indicating at least identification information of the deformed parts based on the part estimation results and the deformation detection results. Note that the "computer" includes any electronic device (which may be a processor included in an electronic device) and may be composed of multiple electronic devices.

[0008] One aspect of the storage medium is a storage medium that stores a program that causes a computer to execute the following processes: based on three-dimensional data representing a structure and a first image that partially photographs the structure, estimate parts of the structure contained in the first image; based on the first image or a second image that photographs part of the photographed range of the first image, detect deformation of the structure; and based on the part estimation results and the deformation detection results, generate deformation information that indicates at least identification information of the part in which the deformation has occurred.

[0009] One example of the effect of the present disclosure is that it becomes possible to generate deformation information that can identify parts of a structure in which a deformation has been detected.

[0010] Shows the schematic configuration of the structure inspection system. An example of the hardware configuration of the deformation information generation device is shown. An example of the functional blocks of the processor of the deformation information generation device. (A) Shows an inspection input image taken when the road surface and shoulder are the structures to be inspected. (B) An image obtained by region division based on the part estimation result. (A) A mask image of the deformed area based on the deformation detection result. (B) Shows an image obtained by superimposing the mask image of the deformed area on the image shown in Fig. 4(B) obtained by region division based on the part estimation result. An example of a flowchart regarding the process executed by the deformation information generation device. Shows a functional block diagram of the processor. Shows a functional block diagram of the processor. A specific example of the process of the deformation detection unit is shown. Shows a functional block diagram of the processor. An example of the display of the design drawing of the structure to be inspected including the part where the deformation is detected. Shows a functional block diagram of the processor. (A) An example of a long-distance inspection image having a road surface area including a deformed area corresponding to "crack". (B) An example of a close-up inspection image corresponding to the long-distance inspection image. (C) Shows a long-distance inspection image clearly showing the correspondence determination result. A block diagram of the deformation information generation device. An example of a flowchart executed by the deformation information generation device.

[0011] Hereinafter, embodiments of the deformation information generation device, the deformation information generation method, and the storage medium will be described with reference to the drawings.

[0012] <First Embodiment> (1) System Configuration Fig. 1 shows the schematic configuration of a structure inspection system 100 according to the first embodiment. The structure inspection system 100 is a system that detects deformations occurring in a structure based on an image of the structure to be inspected and manages information regarding the detected deformations. The structure inspection system 100 mainly includes a deformation information generation device 1, an input device 2, an output device 3, a storage device 4, and a camera 5 that photographs the structure to be inspected 6.

[0013] The inspection target structure 6 is any structure that is subject to inspection, and examples of the inspection target structure 6 include bridges, tunnels, buildings, roads, etc. Note that a "structure" refers to any feature or collection of features that has multiple elements (parts). Furthermore, "deformation" includes any type of damage or deterioration that has occurred to the inspection target structure 6, such as cracks, corrosion, dents, peeling, exposed rebar, and water leakage.

[0014] The deformation information generation device 1 references various information stored in the storage device 4 and detects the presence or absence of a deformation of the inspection target structure 6 in the photographed image based on the photographed image of the inspection target structure 6 generated by the camera 5. If a deformation is detected, the deformation information generation device 1 generates deformation information related to the detected deformation. Here, the photographed image of the inspection target structure 6 generated by the camera 5 is an image that the deformation information generation device 1 inputs to the deformation information generation device 1 as an image for inspection of the inspection target structure 6, and is hereinafter also referred to as an "inspection input image Ii." The deformation information generation device 1 may receive the inspection input image Ii from the camera 5, or may acquire the inspection input image Ii from a device (e.g., the storage device 4 or a device capable of data communication with the deformation information generation device 1) or storage medium that stores the inspection input image Ii generated by the camera 5. For example, the deformation information generation device 1 may receive a user input via the input device 2 to select an image to be used as the inspection input image Ii, and acquire the specified image based on the input signal generated by the input device 2 as the inspection input image Ii.

[0015] The deformation information generating device 1 communicates data with the input device 2, output device 3, and storage device 4 via a communication network or by direct wireless or wired communication.

[0016] The input device 2 is an interface that accepts input (user input) from a user who manages the inspection work of the inspection target structure 6. The input device 2 may be, for example, any of various user input interfaces such as a touch panel, a button, a keyboard, a mouse, or a voice input device. The input device 2 supplies an input signal generated based on the user input to the deformation information generation device 1.

[0017] The output device 3 outputs information (which may include deformation information) relating to the inspection of the inspection target structure 6 based on the output signal supplied from the deformation information generation device 1. In this case, the output signal includes at least one of a display signal and an audio signal. The output device 3 displays information based on the display signal supplied from the deformation information generation device 1, and outputs information as audio based on the audio signal supplied from the deformation information generation device 1. Examples of the output device 3 include display devices such as a display or projector, and audio output devices such as speakers.

[0018] The storage device 4 is a memory that stores information used by the deformation information generation device 1. The storage device 4 may be an external storage device such as a hard disk connected to or built into the deformation information generation device 1, or may be a storage medium such as a flash memory. The storage device 4 may also be a server device that performs data communication with the deformation information generation device 1. The storage device 4 may also be composed of multiple devices.

[0019] The storage device 4 functionally includes a structure measurement data storage unit 41 , a deformation detection model information storage unit 42 , and a deformation information storage unit 43 .

[0020] The structure measurement data storage unit 41 stores structure measurement data, which is data obtained by previously measuring the inspection target structure 6 in three dimensions. For example, the structure measurement data is point cloud data obtained by previously measuring the inspection target structure 6 using a distance measurement sensor such as a lidar. Note that this point cloud data may be point cloud data generated from multiple images using SfM (Structure from Motion). Furthermore, in this embodiment, the structure measurement data is assumed to be point cloud data as an example, but is not limited to point cloud data and may be data representing the inspection target structure 6 using any three-dimensional model (wireframe, surface, or solid).

[0021] Furthermore, in the structure measurement data, label information (also called a "part label") that serves as an identifier for the part (element) of the inspection target structure 6 to which the point belongs is linked to each point (i.e., the smallest unit of data representing a position) that represents the inspection target structure 6. For example, a part label that identifies the part to which the point belongs is linked to the data for each point of the structure measurement data.

[0022] The part label may be information identifying the type of part, or may be unique identification information assigned to each part that constitutes the inspection target structure 6, or may indicate both of these. For example, if the inspection target structure 6 is a bridge, the part label may be information identifying the type of bridge part (e.g., main girder, joint, or deck slab), or may indicate a component element number that can identify the bridge location in more detail. Note that the component element number is a number assigned to each component when the bridge is divided into its smallest constituent unit. Note that the part label may include or be linked to information indicating the name of the part that the part label indicates.

[0023] The anomaly detection model information storage unit 42 stores information such as parameters required to construct the anomaly detection model. The anomaly detection model is a machine learning model (engine) that has learned the relationship between an image and the detection results of anomalies contained in the image through machine learning. For example, when an image showing a structure is input, the anomaly detection model is trained to output a detection result of an area (also referred to as a "deformation area") in the image that indicates a deformed portion of the structure. The detection result of the anomaly area output by the anomaly detection model is, for example, information indicating the presence or absence of a deformation for each pixel (or subpixel; the same applies below). Pixels with a deformation may also include information indicating the type of deformation. When the anomaly detection model is configured using a neural network, the anomaly detection model information storage unit 42 stores various parameters (including hyperparameters), such as the layer structure, the neuron structure of each layer, the number and size of filters in each layer, and the weight of each element of each filter. The anomaly detection model may be any machine learning model used, for example, in segmentation such as instance segmentation or anomaly detection.

[0024] Here, we will provide additional information about machine learning when the anomaly detection model is a deep learning model. Machine learning for the anomaly detection model is performed in advance using training data containing multiple records that pair input samples to the anomaly detection model (here, images from the camera 5) with correct answers to be output by the anomaly detection model when the samples are input (e.g., data indicating the presence or absence of an anomaly and the type of an anomaly, pixel by pixel). In this case, the parameters of the anomaly detection model are determined so that the error (loss) between the detection results output by the anomaly detection model when the samples are input to the anomaly detection model and the correct answers is minimized. The algorithm for determining the above parameters to minimize the loss may be any learning algorithm used in machine learning, such as gradient descent or backpropagation. Similarly, for the other machine learning models described below, the learned parameters are obtained by performing learning using training data that records pairs of input samples and correct answers to be output, as described above.

[0025] The deformation information storage unit 43 stores a database that records the deformation information generated by the deformation information generation device 1. The deformation information includes deformation type information indicating the type of deformation and deformation position information indicating the location where the deformation occurred (deformation position), and may further include any information related to the deformation, such as the size and shape of the deformation. The deformation position information includes at least a part label that identifies the part of the inspection target structure 6 where the deformation occurred. The deformation position information may also include information indicating the position within the part indicated by the part label. For example, the deformation position information may include coordinate information of the deformation position expressed in the coordinate system used in the structure measurement data or a coordinate system that can be coordinate converted from that coordinate system (also referred to as a "reference coordinate system").

[0026] The camera 5 is a camera that photographs the inspection target structure 6 and generates an inspection input image Ii. The inspection input image Ii is, for example, an RGB image. The inspection input image Ii generated by the camera 5 may be supplied directly to the deformation information generation device 1, or may be stored in the camera 5 or an external device or storage medium connected to the camera 5 and then supplied to the deformation information generation device 1.

[0027] The configuration of the structure inspection system 100 shown in Figure 1 is one example, and various modifications may be made to this configuration. For example, the deformation information generation device 1 may be configured integrally with at least one of the input device 2, the output device 3, the storage device 4, and the camera 5. In this case, the structure inspection system 100 may be realized by a single device. In another example, the structure inspection system 100 may not include at least one of the input device 2 or the output device 3.

[0028] (2) Hardware Configuration Fig. 2 shows the hardware configuration of the deformation information generation device 1. The hardware of the deformation information generation device 1 includes a processor 11, a memory 12, and an interface 13. The processor 11, the memory 12, and the interface 13 are connected via a data bus 90.

[0029] The processor 11 executes a program stored in the memory 12 to function as a controller (arithmetic unit) that performs overall control of the deformation information generation device 1. The processor 11 is, for example, a processor such as a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), or a TPU (Tensor Processing Unit). The processor 11 may be composed of multiple processors. The processor 11 is an example of a computer.

[0030] The memory 12 is composed of various types of volatile and non-volatile memory, such as RAM (Random Access Memory), ROM (Read Only Memory), and flash memory. The memory 12 also stores programs for executing the processes performed by the deformation information generation device 1. Some of the information stored in the memory 12 may be stored in one or more external storage devices capable of communicating with the deformation information generation device 1, or in a storage medium that is detachable from the deformation information generation device 1. The memory 12 may also function as at least a part of the storage device 4.

[0031] The interface 13 is an interface for electrically connecting the deformation information generation device 1 to other devices. These interfaces may be wireless interfaces such as network adapters for wirelessly transmitting and receiving data to and from other devices, or may be hardware interfaces for connecting to other devices via cables or the like.

[0032] The hardware configuration of the deformation information generation device 1 is not limited to the configuration shown in Fig. 2. For example, the deformation information generation device 1 may include at least one of an input device 2, an output device 3, a storage device 4, and a camera 5.

[0033] (3) Generation of Deformation Information Next, an overview of the deformation information generation process executed by the deformation information generation device 1 will be described. In general, the deformation information generation device 1 generates deformation information indicating at least the part labels where deformation has occurred based on the detection results of the deformation area on the inspection input image Ii using the deformation detection model and the estimation results of the part labels on the inspection input image Ii estimated using structure measurement data. This allows the deformation information generation device 1 to automate the linking of deformations to structure parts without requiring manual part identification work, and efficiently obtain deformation information that allows the identification of deformed parts. This deformation information provides useful support for the user's decision-making regarding repairs, etc., of the inspection target structure 6.

[0034] (3-1) Functional Blocks Figure 3 is an example of functional blocks possessed by the processor 11 of the deformation information generation device 1. Functionally, the processor 11 of the deformation information generation device 1 has a camera position and attitude estimation unit 14, a parts estimation unit 15, a deformation detection unit 16, and a deformation information generation unit 17. Note that in Figure 3, blocks where data is exchanged are connected by solid lines, but the combination of blocks where data is exchanged is not limited to those shown. The same applies to the other functional block diagrams described below.

[0035] The camera position and orientation estimation unit 14 estimates the position and orientation in the reference coordinate system of the camera 5 that acquired the inspection input image Ii, based on the structure measurement data stored in the structure measurement data storage unit 41 and the inspection input image Ii. In this way, the camera position and orientation estimation unit 14 specifies the position and orientation of the camera 5 in the coordinate system used in the structure measurement data. The camera position and orientation estimation unit 14 then supplies the estimated position and orientation of the camera 5 to the part estimation unit 15.

[0036] For example, the camera position and orientation estimation unit 14 identifies correspondences between points in the structure measurement data and pixels in the inspection input image Ii by matching two-dimensional (i.e., pixel-by-pixel) feature amounts extracted from a two-dimensional image of the structure measurement data with two-dimensional feature amounts extracted from the inspection input image Ii, and estimates the position and orientation of the camera 5 based on the identified correspondences. Such camera position and orientation estimation methods are disclosed in, for example, the following document, but are not limited to: "C. Jaramillo, I. Dryanovski, RG Valenti and J. Xiao, "6-DoF pose localization in 3D point-cloud dense maps using a monocular camera," 2013 IEEE International Conference on Robotics and Biomimetics (ROBIO), pp. 1747-1752, 2013."

[0037] The part estimation unit 15 estimates part labels corresponding to the inspection input image Ii (i.e., a two-dimensional image) based on the structure measurement data associated with the part labels and the estimation result of the position and orientation of the camera 5 generated by the camera position and orientation estimation unit 14. In this way, the part estimation unit 15 generates part label estimation results (also referred to as "part estimation results") that indicate the correspondence between pixels in the inspection input image Ii and part labels. The part estimation results may be, for example, data indicating a part label for each pixel in the inspection input image Ii.

[0038] In a first example of a part label estimation method, the part estimation unit 15 associates the inspection input image Ii with the part labels of the structure measurement data by projecting the part labels onto the image plane of the inspection input image Ii based on the estimation result of the position and orientation of the camera 5. In a second example of a part label estimation method, the part estimation unit 15 associates the inspection input image Ii with the part labels of the structure measurement data by rendering the structure measurement data (point cloud data) within the angle of view of the camera 5 as a two-dimensional image based on the estimation result of the position and orientation of the camera 5. The part estimation unit 15 supplies the part label estimation result to the deformation information generation unit 17.

[0039] The deformity detection unit 16 detects deformed areas in the inspection input image Ii based on a machine-learned deformity detection model composed of deformity detection model information stored in the deformity detection model information storage unit 42 and the inspection input image Ii, and supplies the detection results regarding the deformed areas (also referred to as "deformation detection results") to the deformity information generation unit 17. In this case, the deformity detection unit 16 obtains the deformity detection results based on the information output by the deformity detection model when the inspection input image Ii is input to the deformity detection model. For example, the deformity detection results are information indicating the deformity class for each pixel, and correspond to the segmentation results obtained by segmenting the inspection input image Ii based on the deformity class (i.e., a mask image indicating the deformity class for each pixel). Note that the "deformation class" may be a class indicating the presence or absence of a deformity, or may be a class that can identify the type of deformity if a deformity has occurred in addition to the presence or absence of a deformity. The deformity detection results may be the information output by the deformity detection model itself, or may be information that can be derived from the information output by the deformity detection model.

[0040] The deformation information generation unit 17 generates deformation information based on the part estimation results generated by the part estimation unit 15 and the deformation detection results generated by the deformation detection unit 16. For example, if the deformation information generation unit 17 determines that a deformation area has been detected based on the deformation detection results, it identifies a part label corresponding to the deformation area indicated by the deformation detection results based on the part estimation results. Note that if there are multiple part labels corresponding to the deformation area, the deformation information generation unit 17 may, for example, tally the part labels for each pixel in the deformation area, and if there is a part label corresponding to a predetermined percentage or more of the pixels in the deformation area, identify that part label as the part label corresponding to the deformation area. Furthermore, the deformation information generation unit 17 may generate coordinate information for the deformation area in a reference coordinate system, estimate the size of the deformation area, estimate the shape of the deformation area, and so on, based on various image recognition technologies.

[0041] The deformation information generating unit 17 then generates deformation information including deformation type information indicating the type of deformation in the deformation area, deformation position information indicating part labels (and coordinate information), etc., and other information such as the size of the deformation area, and stores the generated deformation information in the deformation information storage unit 43. Note that the deformation information may be stored in the deformation information storage unit 43 in association with any related information such as the inspection input image Ii and photographing date and time information.

[0042] Furthermore, when a deformation detection result indicating that a deformation area does not exist on the inspection input image Ii is obtained, the deformation information generation unit 17 may store deformation information indicating that no deformation has been detected in the deformation information storage unit 43, or may not generate deformation information. In the former example, the deformation information generation unit 17 may generate deformation information indicating that no deformation has occurred in the part label indicated by the part estimation result.

[0043] In a preferred example, the deformation information generation unit 17 may determine the accuracy of the deformation detection result generated by the deformation detection unit 16 based on the part estimation results estimated by the part estimation unit 15. In this case, for example, table information indicating the types of deformation that can occur for each part label is pre-stored in the storage device 4 or the memory 12. The deformation information generation unit 17 then determines whether the type of deformation in the detected deformation area and the part label corresponding to the deformation area are associated in the table information or the like. If the deformation information generation unit 17 determines that the type of deformation in the detected deformation area and the part label corresponding to the deformation area are not associated in the table information or the like, it determines that the deformation detection result for the deformation area was generated due to an erroneous detection and deletes the deformation information for the deformation detection result without storing it in the deformation information storage unit 43. On the other hand, if the deformation information generation unit 17 determines that the type of deformation in the detected deformation area and the part label corresponding to the deformation area are associated in the above-mentioned table information, etc., it determines that the deformation detection result for the deformation area is correct and stores the deformation information related to the deformation detection result in the deformation information storage unit 43.

[0044] For example, consider the case where deformation information is generated that includes deformation type information indicating "rust" and a part label indicating a "concrete member," and where "rust" is not included in the deformation types associated with the "concrete member" in the table information. In this case, the deformation information generation unit 17 determines that the deformation information is due to erroneous detection and deletes the deformation information without storing it in the deformation information storage unit 43. As another example, consider the case where deformation information is generated that includes deformation type information indicating "crack" and a part label indicating a "reinforcing bar member," and where "crack" is included in the deformation types associated with the "reinforcing bar member" in the table information. In this case, the deformation information generation unit 17 determines that the deformation information was generated based on correct detection results and stores the deformation information in the deformation information storage unit 43.

[0045] The components of the camera position and orientation estimation unit 14, the part estimation unit 15, the deformation detection unit 16, and the deformation information generation unit 17 described in FIG. 3 can be realized, for example, by the processor 11 executing a program. Alternatively, the necessary programs may be recorded on any non-volatile storage medium and installed as needed to realize each component. At least some of these components may not necessarily be realized by software programs, but may be realized by any combination of hardware, firmware, and software. At least some of these components may be realized using a user-programmable integrated circuit, such as an FPGA (Field-Programmable Gate Array) or a microcontroller. In this case, a program consisting of the above components may be realized using this integrated circuit. Furthermore, at least a portion of each component may be configured by an ASSP (Application Specific Standard Product), an ASIC (Application Specific Integrated Circuit), or a quantum processor (quantum computer control chip). In this way, each component may be realized by various hardware. The same applies to other embodiments described below. Furthermore, each of these components may be realized by the cooperation of multiple computers, for example, using cloud computing technology.

[0046] (3-2) Specific Examples Here, specific examples of detection of an abnormal region will be described with reference to FIGS. 4(A), 4(B), 5(A), and 5(B).

[0047] Fig. 4(A) shows an inspection input image Ii captured when the road surface and road shoulder (including curbstones) are the inspection target structures 6. The inspection input image Ii shown in Fig. 4(A) includes a road surface area 71 corresponding to the road surface, which is the inspection target structure 6, and a road shoulder area 72 corresponding to the road shoulder, as well as a vegetation area 73 indicating vegetation. Furthermore, the road surface area 71 includes a deformation area 74 where a deformation corresponding to a "crack" has occurred.

[0048] 4B shows an image segmented based on the part estimation results. Here, the part estimation unit 15 generates segments corresponding to part labels indicating the road surface, part labels indicating the road shoulder, and part labels indicating the vegetation, based on the camera position and orientation estimation results generated by the camera position and orientation estimation unit 14 and the structure measurement data.

[0049] FIG. 5(A) is a mask image of a deformed area based on the deformed area detection result generated from the inspection input image Ii shown in FIG. 4(A). Here, the deformed area detection unit 16 acquires the deformed area detection result output by the deformed area detection model when the inspection input image Ii shown in FIG. 4(A) is input to the deformed area detection model. Here, the deformed area detection unit 16 acquires a mask image indicating the deformed area in the inspection input image Ii shown in FIG. 5(A) from the deformed area detection model. In addition, a classification result (class) indicating that the type of deformed area is "crack" is associated with the pixel indicating the deformed area in the above-mentioned mask image.

[0050] FIG. 5(B) shows an image in which the mask image of the deformed area shown in FIG. 5(A) is superimposed on the image shown in FIG. 4(B), which has been segmented based on the part estimation results. Based on the image shown in FIG. 4(B), which has been segmented based on the part estimation results, and the mask image shown in FIG. 5(A), the deformation information generation unit 17 recognizes that the deformed area 74 is included in the segment corresponding to the part label indicating the road surface. Therefore, the deformation information generation unit 17 generates deformation information including at least deformation position information indicating the part label indicating the road surface and deformation type information indicating "cracks." In this way, the deformation information generation unit 17 can preferably generate deformation information that can identify deformed parts.

[0051] (3-3) Processing Flow Figure 6 is an example of a flowchart related to the processing executed by the deformation information generation device 1. For example, when a user input specifying an inspection input image Ii is detected, or when other processing start conditions are satisfied, the deformation information generation device 1 executes the flowchart shown in Figure 8.

[0052] First, the deformation information generation device 1 acquires the inspection input image Ii generated by the camera 5 (step S11). Instead of acquiring the inspection input image Ii directly from the camera 5, the deformation information generation device 1 may acquire the inspection input image Ii stored in the storage device 4 or the like.

[0053] Next, the deformation information generation device 1 estimates the position and orientation of the camera 5 based on the inspection input image Ii and the structure measurement data stored in the structure measurement data storage unit 41 (step S12). The processing of step S12 corresponds to the processing executed by the camera position and orientation estimation unit 14. Then, the deformation information generation device 1 estimates the area of ​​each part of the inspection target structure 6 included in the inspection input image Ii based on the structure measurement data with part labels and the position and orientation estimation result of the camera (step S13). The processing of step S13 corresponds to the processing executed by the part estimation unit 15.

[0054] The deformation information generation device 1 then refers to the deformation detection model information storage unit 42 and detects a deformation area from the inspection input image Ii based on the deformation detection model, which is a machine-learned model (step S14). The processing of step S14 corresponds to the processing executed by the deformation detection unit 16. Note that step S14 may be executed before step S12 and step S13, or may be executed in parallel with them.

[0055] The deformation information generation device 1 then generates and outputs deformation information based on the processing results of steps S13 and S14 (step S15). In this case, the deformation information generation device 1 stores the generated deformation information in the deformation information storage unit 43. The deformation information generation device 1 may also generate a display signal and / or an audio signal based on the generated deformation information and supply the generated signal to the output device 3, thereby displaying and / or outputting information related to the deformation information as audio by the output device 3. The processing of step S15 corresponds to the processing executed by the deformation information generation unit 17.

[0056] (4) Modifications Modifications of the above-described embodiment will now be described. The following modifications may be applied to the above-described embodiment in any combination.

[0057] (Modification 1) The deformation information generation device 1 may perform a process of interpolating missing parts in the part estimation result generated by the part estimation unit 15.

[0058] FIG. 7 shows a functional block diagram of processor 11. Processor 11 in Modification 1 includes an interpolation unit 18. Interpolation unit 18 interpolates the part label of a pixel (target pixel) that is not associated with a part label in the part estimation result generated by part estimation unit 15, based on the part labels of a predetermined number of neighboring pixels that are close to the target pixel. In this case, for example, interpolation unit 18 determines the most common part label among the predetermined number of neighboring pixels as the part label of the target pixel. Note that interpolation unit 18 may also determine the part label of the target pixel based on part labels corresponding to points in the structure measurement data that correspond to the neighboring pixels.

[0059] Generally, when compared on the inspection input image Ii, the points of the structure measurement data are sparser than the pixels of the inspection input image Ii, so the part estimation unit 15 may not be able to associate part labels with some pixels. Even in this case, the interpolation unit 18 can perform the interpolation process described above to interpolate missing parts of the part estimation results so that part labels can be associated with each pixel. The interpolation unit 18 then supplies the interpolated part estimation results to the deformation information generation unit 17. In this case, the deformation information generation unit 17 can generate more accurate deformation information based on the refined association results.

[0060] (Modification 2) The deformation information generation device 1 may select a deformation detection model to be used for detecting a deformation area based on the part estimation result generated by the part estimation unit 15.

[0061] 8 shows a functional block diagram of the processor 11. The processor 11 in Modification 2 includes a deformity detection unit 16A that acquires part estimation results from the part estimation unit 15. The deformity detection model information storage unit 42 stores parameters of multiple deformity detection models that have undergone machine learning to specialize in detecting specific deformities.

[0062] Here, machine learning is performed on each of the deformation detection models so that they can detect different types of deformation, and the parameters of the deformation detection models obtained by machine learning are stored in the deformation detection model information storage unit 42. For example, each deformation detection model is machine-learned using training data including a plurality of records in which an image of a structure in which a target deformation has occurred is used as a sample, and data indicating, for each pixel, an area in which the type of deformation to be detected has occurred is used as the correct answer.

[0063] The deformation detection unit 16A determines the deformation detection model to be used based on the part estimation results acquired from the part estimation unit 15. For example, table information indicating the identification information of the deformation detection model to be applied for each type of part (or the type of expected deformation) is stored in advance in the storage device 4, memory 12, or the like, and the deformation detection unit 16A refers to the table information to determine the deformation detection model to be used from the type of part indicated by the part estimation results.

[0064] Preferably, the anomaly detection unit 16A generates an image (also called a "part extraction image") in which an image area of ​​each part is extracted for each type of part indicated by the part estimation result from the inspection input image Ii. In this case, the anomaly detection unit 16A inputs each part extraction image into an anomaly detection model corresponding to the type of extracted part and acquires information output by the anomaly detection model. This enables the anomaly detection unit 16A to obtain highly accurate anomaly detection results for each type of anomaly.

[0065] FIG. 9 shows a specific example of the processing of the deformity detection unit 16A in Modification 2. First, based on the part estimation results by the part estimation unit 15, the deformity detection unit 16A generates a part extraction image by extracting image areas of parts from the inspection input image Ii for each type of part requiring inspection. In the example of FIG. 9 , the deformity detection unit 16A recognizes the existence of part type A and part type B based on the part estimation results, and generates a part extraction image "Ix" by extracting image areas of parts of part type A from the inspection input image Ii and a part extraction image "Iy" by extracting image areas of parts of part type B from the inspection input image Ii. The deformity detection unit 16A then references table information indicating the identification information of the deformity detection model to be applied for each type of part, and determines that the deformity detection model "Mx" that detects deformity "X" should be used for the part extraction image Ix from which part type A has been extracted, and the deformity detection model "My" that detects deformity "Y" should be used for the part extraction image Iy from which part type B has been extracted. Therefore, the deformity detection unit 16A obtains the deformity detection result for the part extraction image Ix by inputting the part extraction image Ix of part type A into the deformity detection model Mx, and obtains the deformity detection result for the part extraction image Iy by inputting the part extraction image Iy of part type B into the deformity detection model My.

[0066] According to this modified example, the deformation information generating device 1 can obtain highly accurate deformation detection results by using a deformation detection model suitable for each type of part included in the inspection input image Ii.

[0067] Alternatively, instead of the above example, an anomaly detection model to be applied to each type of part may be machine-learned. In this case, the anomaly detection model information storage unit 42 stores parameters of an anomaly detection model that has been machine-learned for each type of part indicated by the part label. Each anomaly detection model is machine-learned using training data that uses images of the target type of part as samples and includes multiple records in which data indicating, for each pixel, the presence or absence of an anomaly in the part and the type of anomaly is used as a correct answer.

[0068] (Modification 3) The structure measurement data stored in the structure measurement data storage unit 41 does not need to include information about part labels.

[0069] FIG. 10 shows a functional block diagram of processor 11 in Modification 3. Processor 11 has a part estimation unit 15A. Part estimation unit 15A generates part estimation results based on unlabeled structure measurement data, the position and orientation estimation result of camera 5, and inspection input image Ii. Part estimation unit 15A generates part estimation results using a part estimation model constructed with reference to part estimation model information stored in part estimation model information storage unit 44 of storage device 4. In this case, part estimation unit 15A identifies partial data (point cloud data) of the structure measurement data that exists within the angle of view of camera 5 based on the position and orientation estimation result of camera 5, and inputs the identified partial data to the part estimation model. Then, part estimation unit 15A acquires the part estimation results output by the part estimation model in this case.

[0070] Here, the part estimation model is a model that has been machine-learned in advance so that, when three-dimensional data (point cloud data) within the camera's field of view corresponding to an image is input, the model outputs a part estimation result indicating the part class for each pixel on the image. The "part class" may be, for example, a label indicating the type of part, a component element number, or any other identifier for the part.

[0071] Furthermore, part estimation unit 15A may generate a part estimation result by further using inspection input image Ii. In this case, the part estimation model is a machine learning model that generates a part estimation result using an image and three-dimensional data corresponding to the image as input. BPNet (Bidirectional Projection Network) is known as an example of the architecture of such a machine learning model. Part estimation unit 15A inputs inspection input image Ii and structure measurement data present within the angle of view of camera 5 to such a part estimation model, and acquires a part estimation result from the part estimation model.

[0072] According to this modification, the parts estimation unit 15A can generate part estimation results in the same way as in the above-described embodiment, even when using structure measurement data to which no labels are assigned.

[0073] (Variation 4) The deformation detection unit 16 of the deformation information generation device 1 may perform deformation detection using the structure measurement data stored in the structure measurement data storage unit 41 in addition to the inspection input image Ii.

[0074] In this case, the deformity detection model is a machine learning model that receives an image and three-dimensional data corresponding to the image as input and generates a deformity detection result that is a result of detecting a deformity in the image. BPNet is a known example of the architecture of such a machine learning model. The deformity detection unit 16 inputs the inspection input image Ii and the structure measurement data corresponding to the inspection input image Ii into the deformity detection model, and obtains the deformity detection result from the deformity detection model.

[0075] In this modification, features of data from different modalities are used in combination, making it possible to detect abnormality areas with higher accuracy.

[0076] (Variant 5) The deformation information generating device 1 may update drawing data showing the design drawings of the inspection target structure 6 based on the deformation information (i.e., the deformation detection results and part estimation results), and store the updated drawing data in the deformation information storage unit 43 in association with the above-mentioned deformation information.

[0077] In this case, the drawing data is, for example, electronic data (images or other display data) of design drawings such as a development view or a bird's-eye view of the inspection target structure 6, and may be stored in the storage device 4 separately from the structure measurement data, or may be generated from the structure measurement data. The drawing data also includes corresponding location information, which indicates the correspondence between the locations of the inspection target structure 6 indicated by the part labels or coordinate positions in the reference coordinate system and the locations on the design drawing. When generating drawing data from structure measurement data, the deformation information generating device 1 may generate corresponding location information based on the structure measurement data. In this case, for example, when converting structure measurement data into a design drawing (e.g., a bird's-eye view), the deformation information generating device 1 generates corresponding location information that associates the coordinate positions in the reference coordinate system or part labels before conversion with the positions on the converted design drawing.

[0078] When the deformation information generating device 1 generates deformation information based on the deformation detection results and part estimation results, it updates the drawing data corresponding to the detected position of the deformation so that the detected position of the deformation and the type of deformation are clearly indicated. For example, the deformation information generating device 1 identifies the detected position of the deformation on the design drawing based on the part label or coordinate information included in the deformation information, and superimposes text information corresponding to the deformation type information included in the deformation information onto the design drawing, correlating it with the identified detected position of the deformation.

[0079] FIG. 11 is a display example of a design drawing of an inspection target structure 6, including a part in which a deformation has been detected. FIG. 11 shows a portion of the inspection target structure 6, which is a bridge, including deck slabs A01-A08, cross beams B01-B10, and main beams B11-B22. Here, A01-A08 and B01-B22 are used as part labels. Based on the generated deformation information, the deformation information generating device 1 recognizes that a deformation of the deformation type "deformation" has been detected in deck slab A03, and superimposes a mark 81 representing the detected deformation on the deck slab A03 where the deformation has occurred, along with text information indicating the type of deformation, "Deformed." This allows the deformation information generating device 1 to appropriately update the drawing data so that the detected deformation is clearly indicated. In addition, the deformation information generating device 1 may determine the size and shape of the mark 81 to be superimposed on the design drawing based on the size information and shape information contained in the deformation information, or may extract a deformation area from the inspection input image Ii and superimpose an image of the extracted deformation area as the mark 81 on the image of the design drawing.

[0080] In this way, according to this modification, the deformation information generating device 1 can add information about the deformation to drawing data representing design drawings such as development drawings, etc. This allows a user viewing the drawing data to easily grasp the location of the deformation on the design drawing.

[0081] Second Embodiment In the second embodiment, the deformation information generation device 1 uses two inspection input images Ii, one of which encompasses the other. This allows the deformation information generation device 1 to more accurately perform deformation detection and part estimation. Hereinafter, an image of the inspection target structure 6 photographed at a close distance (i.e., corresponding to the aforementioned "other photographed range") will be referred to as the "inspection close-up image IiC," and an image of the inspection target structure 6 photographed at a long distance so as to include the photographed range of the inspection close-up image IiC (i.e., corresponding to the aforementioned "one photographed range") will be referred to as the "inspection distant image IiF." The inspection close-up image IiC and the inspection distant image IiF may be generated by the same camera 5 or by two cameras 5. Hereinafter, the deformation information generation device 1 of the second embodiment will be assumed to have the hardware configuration shown in FIG. 2. Furthermore, the same components as those in the first embodiment will be appropriately designated by the same reference numerals, and their description will be omitted.

[0082] 12 shows a functional block configuration of the processor 11 of the deformation information generation device 1. The processor 11 in the second embodiment has a camera position and orientation estimation unit 14B, a part estimation unit 15B, a deformation detection unit 16B, a deformation information generation unit 17B, and a correspondence identification unit 19B.

[0083] The camera position and orientation estimation unit 14B estimates the position and orientation of the camera 5 that captured the inspection distant image IiF based on the structure measurement data stored in the structure measurement data storage unit 41 and the inspection distant image IiF. The part estimation unit 15B estimates part labels corresponding to the inspection distant image IiF based on the structure measurement data associated with the part labels and the estimation result of the position and orientation of the camera 5 generated by the camera position and orientation estimation unit 14B. The processing performed by the camera position and orientation estimation unit 14B and the part estimation unit 15B based on the inspection distant image IiF is the same as the processing performed by the camera position and orientation estimation unit 14 and the part estimation unit 15 based on the inspection input image Ii in the first embodiment.

[0084] The deformation detection unit 16B detects a deformation area in the inspection close-up image IiC based on the machine-learned deformation detection model configured from the deformation detection model information stored in the deformation detection model information storage unit 42 and the inspection close-up image IiC, and supplies the deformation detection result to the deformation information generation unit 17B. The processing performed by the deformation detection unit 16B based on the inspection close-up image IiC is the same as the processing performed by the deformation detection unit 16 based on the inspection input image Ii in the first embodiment.

[0085] The correspondence identification unit 19B identifies the correspondence between the inspection distant image IiF and the inspection close-up image IiC. In this case, the correspondence identification unit 19B identifies an area in the inspection distant image IiF that corresponds to the inspection close-up image IiC based on any image matching technology. The correspondence identification unit 19B then supplies the identification result of the above-mentioned correspondence (also referred to as the "correspondence identification result") to the deformation information generation unit 17B. In this case, for example, the correspondence identification unit 19B supplies information indicating pixels in the inspection distant image IiF that correspond to each pixel of the inspection close-up image IiC to the deformation information generation unit 17B as the correspondence identification result.

[0086] The deformation information generation unit 17B generates deformation information based on the part estimation results generated by the part estimation unit 15B, the deformation detection results generated by the deformation detection unit 16B, and the correspondence identification results generated by the correspondence identification unit 19B. For example, when the deformation information generation unit 17 determines that a deformation area has been detected based on the deformation detection results, it identifies the part label corresponding to the deformation area on the inspection close-up image IiC indicated by the deformation detection results based on the part estimation results indicating the part labels on the inspection close-up image IiF and the correspondence identification result indicating the correspondence between the inspection close-up image IiF and the inspection close-up image IiC. In addition, the deformation information generation unit 17B may generate coordinate information of the deformation area in a reference coordinate system, estimate the size of the deformation area, estimate the shape of the deformation area, and so on, based on various image recognition technologies. The deformation information generation unit 17B then generates deformation information including deformation type information indicating the type of deformation in the deformation area, deformation position information indicating the part labels (and coordinate information), and other information such as the size of the deformation area, and stores this information in the deformation information storage unit 43. Furthermore, if a deformation detection result indicating that the deformation area does not exist on the inspection input image Ii is obtained, the deformation information generation unit 17B may store deformation information indicating that no deformation was detected in the deformation information storage unit 43, or may not generate deformation information. Furthermore, as with the deformation information generation unit 17 of the first embodiment, the deformation information generation unit 17B may correct the deformation detection result by referring to the part labels.

[0087] Fig. 13(A) shows an example of an inspection distant view image IiF having a road surface area 71 including a deformation area 74 corresponding to a "crack," a road shoulder area 72, and a vegetation area 73, and Fig. 13(B) shows an example of an inspection close-up image IiC corresponding to the inspection distant view image IiF shown in Fig. 13(A). Fig. 13(C) shows an inspection distant view image IiF that clearly shows the correspondence relationship identification result. In Fig. 13(C), a frame 85 that is the outer edge of the image area corresponding to the inspection close-up image IiC is clearly shown.

[0088] As shown in Figures 13(A) to 13(C), the inspection close-up image IiC corresponds to a portion of the image area within the inspection distant image IiF, and the correspondence identification unit 19B identifies the image area within the inspection distant image IiF that corresponds to the inspection close-up image IiC. Here, the inspection close-up image IiC is an image of a deformed part (here, the road surface) photographed from a close distance, making it easy for the deformation detection unit 16B to detect deformation. Note that the inspection close-up image IiC is an enlarged image of a limited area of ​​the inspection target structure 6, making it difficult to identify a correspondence with the structure measurement data representing the entire inspection target structure 6. Therefore, the inspection distant image IiF is used for part estimation. The deformation information generation device 1 then identifies part labels corresponding to the deformation area 74 based on the correspondence identification results and part estimation results, and generates deformation information based on the identified part labels, etc.

[0089] Thus, according to the second embodiment, the deformation information generating device 1 can detect the deformation area with high accuracy using the inspection close-up image IiC, while also estimating parts with high accuracy using the inspection distant-view image IiF. Note that the various modifications of the first embodiment can also be applied to the second embodiment in any combination.

[0090] <Third embodiment> Figure 14 is a block diagram of a deformation information generation device 1X. The deformation information generation device 1X mainly has a parts estimation means 15X, a deformation detection means 16X, and a deformation information generation means 17X. Note that the deformation information generation device 1X may be composed of multiple devices.

[0091] The part estimation means 15X estimates parts of the structure included in the first image based on three-dimensional data representing the structure and a first image capturing a portion of the structure. For example, the part estimation means 15X may be the camera position and orientation estimation unit 14 and the part estimation unit 15 (or the camera position and orientation estimation unit 14 and the part estimation unit 15A) in the first embodiment, or the camera position and orientation estimation unit 14B and the part estimation unit 15B in the second embodiment. The first image may be, for example, the inspection input image Ii in the first embodiment or the inspection distant image IiF in the second embodiment.

[0092] The deformation detection means 16X detects deformation of the structure based on the first image or a second image obtained by capturing a portion of the capturing range of the first image. For example, the deformation detection means 16X may be the deformation detection unit 16 or the deformation detection unit 16A in the first embodiment, or the deformation detection unit 16B in the second embodiment. The second image may be, for example, the inspection close-up image IiC in the second embodiment.

[0093] The deformation information generating means 17X generates deformation information indicating at least the identification information of the deformed part based on the part estimation result and the deformation detection result. For example, the deformation information generating means 17X may be the deformation information generating unit 17 in the first embodiment or the deformation information generating unit 17B in the second embodiment.

[0094] 15 is an example of a flowchart executed by the deformation information generation device 1X in the third embodiment. First, the part estimation means 15X estimates the parts of the structure included in the first image based on 3D data representing the structure and a first image capturing a portion of the structure (step S21). Next, the deformation detection means 16X detects deformation of the structure based on the first image or a second image capturing a portion of the first image's capture range (step S22). The deformation information generation means 17X generates deformation information indicating at least the identification information of the deformed parts based on the part estimation results and the deformation detection results (step S23).

[0095] According to the third embodiment, the deformation information generation device 1X can preferably generate deformation information regarding the deformation detection results linked to the identification information of the parts.

[0096] In the above-described embodiments, the program can be stored using various types of non-transitory computer-readable media and supplied to a computer processor or the like. Non-transitory computer-readable media include various types of tangible storage media. Examples of non-transitory computer-readable media include magnetic storage media (e.g., flexible disks, magnetic tapes, hard disk drives), magneto-optical storage media (e.g., magneto-optical disks), CD-ROMs (Read Only Memory), CD-Rs, CD-R / Ws, and semiconductor memories (e.g., mask ROMs, PROMs (Programmable ROMs), EPROMs (Erasable PROMs), flash ROMs, and RAMs (Random Access Memory)). The program may also be supplied to a computer by various types of transitory computer-readable media. Examples of transitory computer-readable media include electrical signals, optical signals, and electromagnetic waves. The transitory computer-readable media can be supplied to a computer via wired communication paths such as electric wires and optical fibers, or via wireless communication paths.

[0097] In addition, some or all of the above embodiments may be described as, but are not limited to, the following supplementary notes.

[0098] [Supplementary Note 1] A deformation information generation device comprising: a parts estimation means for estimating parts of a structure included in a first image based on three-dimensional data representing the structure and a first image that partially captures the structure, a deformation detection means for detecting a deformation of the structure based on the first image or a second image that captures a portion of the capture range of the first image, and a deformation information generation means for generating deformation information indicating at least identification information of the deformed parts based on the part estimation results and the deformation detection results. [Supplementary Note 2] The deformation information generation device according to Supplementary Note 1 further comprises a camera position and orientation estimation means for estimating the position and orientation of a camera that captured the first image based on the three-dimensional data and the first image, wherein the part estimation means estimates the parts included in the first image based on the position and orientation estimation results and identification information of the parts associated with the three-dimensional data. [Supplementary Note 3] The deformation information generation device according to Supplementary Note 1, wherein the deformation detection means detects the deformation based on the first image or the second image and a machine learning model, and the machine learning model is a model that has learned by machine learning the relationship between an image and the detection result of the deformation contained in the image. [Supplementary Note 4] The deformation information generation device according to Supplementary Note 3, wherein the deformation detection means selects the machine learning model based on the estimation result of the part. [Supplementary Note 5] The deformation information generation device according to Supplementary Note 1, wherein the part estimation means generates the estimation result of the part based on the first image, the deformation detection means generates the detection result of the deformation based on the second image, and the deformation information generation means generates the deformation information based on information indicating the correspondence between the first image and the second image, the estimation result of the part, and the detection result of the deformation. [Supplementary Note 6] The deformation information generation device according to Supplementary Note 1, wherein the deformation detection means generates the deformation detection result including information about the type of deformation, and the deformation information generation means generates the deformation information indicating at least identification information of the part where the deformation has occurred and the type of deformation. [Supplementary Note 7] The deformation information generation device according to Supplementary Note 6, wherein the deformation information generation means determines whether the deformation detection result is correct based on the identification information of the part where the deformation has occurred and the type of deformation.[Supplementary Note 8] The deformation information generation device according to Supplementary Note 2, wherein the part estimation means estimates identification information of the part corresponding to each pixel of the first image based on the position and orientation estimation result and identification information of the part associated with the three-dimensional data. [Supplementary Note 9] The deformation information generation device according to Supplementary Note 1, wherein the deformation information generation means adds information indicating the detected position of the deformation in the drawing to drawing data showing a drawing of the structure. [Supplementary Note 10] A deformation information generation method, wherein a computer estimates parts of the structure included in a first image based on three-dimensional data representing the structure and a first image capturing a portion of the structure, detects a deformation of the structure based on the first image or a second image capturing a portion of the capturing range of the first image, and generates deformation information indicating at least identification information of the part in which the deformation has occurred based on the part estimation result and the deformation detection result. [Appendix 11] A storage medium storing a program that causes a computer to execute the following processes: based on three-dimensional data representing a structure and a first image in which the structure is partially photographed, estimate parts of the structure included in the first image; based on the first image or a second image in which part of the photographed range of the first image is photographed; and based on the part estimation results and the deformation detection results, generate deformation information that indicates at least identification information of the parts in which the deformation has occurred.

[0099] Although the present invention has been described above with reference to the embodiments, the present invention is not limited to the above embodiments. Various modifications within the scope of the present invention that would be understood by those skilled in the art can be made to the configuration and details of the present invention. In other words, the present invention naturally includes various modifications and alterations that would be possible for those skilled in the art based on the entire disclosure, including the claims, and the technical ideas. Furthermore, the disclosures of the above-cited patent and non-patent documents are incorporated herein by reference.

[0100] REFERENCE SIGNS LIST 1, 1X Deformation information generating device 2 Input device 3 Output device 4 Storage device 5 Camera 11 Processor 12 Memory 13 Interface 41 Structure measurement data storage unit 42 Deformation detection model information storage unit 43 Deformation information storage unit 90 Data bus 100 Structure inspection system

Claims

1. A parts estimation means for estimating the parts of a structure included in a first image based on three-dimensional data representing the structure and a first image partially taken of the structure, A deformation detection means for detecting deformation of the structure based on the first image or a second image taken of a part of the shooting range of the first image, A deformation information generation means generates deformation information that at least indicates identification information of the part in which the deformation occurred, based on the estimation result of the part and the deformation detection result. A device for generating deformation information.

2. The system further includes a camera position and orientation estimation means that estimates the position and orientation of the camera that captured the first image based on the three-dimensional data and the first image. The deformation information generating apparatus according to claim 1, wherein the part estimation means estimates the part included in the first image based on the position and orientation estimation result and the part identification information associated with the three-dimensional data.

3. The deformation detection means detects the deformation based on the first image or the second image and a machine learning model. The aforementioned machine learning model is a model that has learned the relationship between an image and the detection results of deformations contained in that image. The deformation information generating device according to claim 1.

4. The deformation detection means selects the machine learning model based on the estimation result of the part, as described in claim 3, for the deformation information generation device.

5. The part estimation means generates the part estimation result based on the first image, The deformation detection means generates the deformation detection result based on the second image, The deformation information generating device according to claim 1, wherein the deformation information generating means generates the deformation information based on information indicating the correspondence between the first image and the second image, the estimation result of the part, and the detection result of the deformation.

6. The deformation detection means generates a deformation detection result that includes information about the type of deformation, The deformation information generating device according to claim 1, wherein the deformation information generating means generates deformation information that at least indicates identification information of the part in which the deformation occurred and the type of deformation.

7. The deformation information generating device according to claim 6, wherein the deformation information generating means determines whether the detection result of the deformation is correct or incorrect based on the identification information of the part in which the deformation occurred and the type of deformation.

8. The deformation information generating apparatus according to claim 2, wherein the part estimation means estimates the part identification information corresponding to each pixel of the first image based on the position and orientation estimation result and the part identification information associated with the three-dimensional data.

9. Computers Based on three-dimensional data representing the structure and a first image partially taken of the structure, the parts of the structure included in the first image are estimated. Based on the first image or a second image taken from a portion of the area covered by the first image, deformation of the structure is detected. Based on the estimation results of the part and the detection results of the deformation, deformation information is generated that at least indicates the identification information of the part in which the deformation occurred. Deformation information generation method.

10. Based on three-dimensional data representing the structure and a first image partially taken of the structure, the parts of the structure included in the first image are estimated. Based on the first image or a second image taken from a portion of the area covered by the first image, deformation of the structure is detected. A program that causes a computer to perform a process to generate deformation information that at least indicates the identification information of the part in which the deformation occurred, based on the estimation result of the part and the detection result of the deformation.