Information processing device, information processing method, and information processing program

Abstract

This invention provides an information processing device, an information processing method, and an information processing program that can accurately calculate the difference in damage information. [Solution] The information processing device acquires a first image of a structure and first damage information of the structure extracted from the first image, a second image of the structure taken at a different time than the first image and second damage information of the structure extracted from the second image, extracts multiple feature points from the first image and extracts multiple feature points from the second image, searches for corresponding feature points between the first image and the second image, and based on the information of the corresponding feature points, aligns the first damage information and the second damage information by performing a rough alignment followed by a detailed alignment, and calculates the difference between the first damage information and the second damage information after alignment.

Description

Technical Field

[0001] The present invention relates to an information processing apparatus, an information processing method, and an information processing program, and particularly relates to an information processing apparatus, an information processing method, and an information processing program for processing damage information of a structure extracted from an image.

Background Art

[0002] Social infrastructure structures such as bridges, dams, and tunnels are required to be regularly inspected once every few years. For example, bridges are required to be regularly inspected once every five years. As a technology for supporting this work, a technology for automatically extracting damage (such as cracks and free lime) that appears on the surface of a structure from an image of the structure is known (for example, Patent Documents 1, 2, etc.).

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Patent Document 2

Summary of the Invention

Problems to be Solved by the Invention

[0004] Based on the results of regular inspections, the administrator of the structure determines whether repair is necessary, the timing of repair, etc. At this time, there is a need to determine matters related to repair based not only on the current deterioration state of the structure but also on information on the difference from past inspection results. This is because if there is information on the difference from past inspection results, the future deterioration state can be predicted. However, it is difficult to accurately obtain the difference from past inspection results, so it is not currently being done.

[0005] This invention has been made in view of these circumstances, and aims to provide an information processing device, an information processing method, and an information processing program that can accurately calculate the difference in damage information. [Means for solving the problem]

[0006] (1) An information processing device equipped with a processor, wherein the processor acquires a first image of a structure and first damage information of the structure extracted from the first image, acquires a second image of the structure taken at a different time than the first image and second damage information of the structure extracted from the second image, extracts a plurality of feature points from the first image, extracts a plurality of feature points from the second image, searches for corresponding feature points between the first image and the second image, aligns the first damage information and the second damage information by performing a rough alignment of the first damage information and the second damage information based on the information of the corresponding feature points and then performing a detailed alignment, and calculates the difference between the first damage information and the second damage information after alignment.

[0007] (2) The processor is an information processing device of (1) that performs rigid body alignment as coarse alignment and non-rigid body alignment as detailed alignment.

[0008] (3) The processor is an information processing device of (1) that performs alignment by performing rigid body alignment as coarse alignment and by correcting the difference in lens distortion as detailed alignment.

[0009] (4) Any one of the information processing devices (1) to (3), wherein the first damage information and the second damage information are crack information, and the processor calculates the difference in the width and / or length of the crack.

[0010] (5) The information processing device of (4), which generates corresponding pairs of cracks and calculates the difference in crack widths by calculating the difference in widths between the pairs.

[0011] (6) The information processing device of (4), wherein the processor calculates the difference in crack lengths by generating corresponding pairs of cracks and calculating the length of cracks for which no pairs have been generated.

[0012] (7) The processor is an information processing device of (5) or (6) that generates pairs of cracks from adjacent cracks.

[0013] (8) The information processing device of (5) or (6), wherein the processor generates corresponding pairs of cracks by DP matching.

[0014] (9) An information processing device, one of (1) to (3), wherein the first damage information and the second damage information are information about the damaged area, and the processor calculates the difference between the corresponding damaged areas.

[0015] (10) When a structure is divided and photographed and a plurality of first images and a plurality of first damage information extracted from the plurality of first images are obtained, and a plurality of second images and a plurality of second damage information extracted from the plurality of second images are obtained, the processor extracts a plurality of feature points individually from each first image, extracts a plurality of feature points individually from each second image, searches for corresponding feature points between the corresponding first and second images, aligns the corresponding first and second damage information, and calculates the difference between the corresponding first and second damage information, any one of the information processing devices from (1) to (9).

[0016] (11) The information processing device of (10), wherein the processor acquires synthesis processing information necessary for panoramic synthesis of multiple first images, and synthesizes the difference calculated between the corresponding first and second images based on the synthesis processing information.

[0017] (12) When multiple second images are obtained in which a portion of the imaging area overlaps with the first image, and multiple second damage information is extracted from the multiple second images, the processor individually extracts multiple feature points from each second image, individually searches for corresponding feature points between the first image and each second image, individually aligns the first damage information with each second damage information, individually calculates the difference between the first damage information and each second damage information, and further integrates the differences individually calculated between the first damage information and each second damage information, any one of the information processing devices from (1) to (9).

[0018] (13) A processor is an information processing device which is one of (1) to (9) and extracts a region from which the difference between the first damage information and the second damage information can be calculated between the first image and the second image, and outputs information from the extracted region.

[0019] (14) When the processor outputs information on a region for which a difference can be calculated, it generates an image on the first image showing the region for which a difference can be calculated and outputs it to the display destination, the information processing device of (13).

[0020] (15) When a structure is divided and photographed and multiple first images and multiple first damage information extracted from the multiple first images are obtained, and multiple second images and multiple second damage information extracted from the multiple second images are obtained, the processor panorama stitches together the multiple first images, synthesizes the multiple first damage information based on the synthesis processing information when the multiple first images are panorama stitched together, synthesizes the multiple second images panorama stitches together the multiple second images and synthesizes the multiple second damage information based on the synthesis processing information when the multiple second images are panorama stitched together, extracts multiple feature points from the first image after panorama stitching, extracts multiple feature points from the second image after panorama stitching, searches for corresponding feature points between the first image and the second image after panorama stitching, aligns the synthesized first damage information and second damage information, and calculates the difference between the synthesized first damage information and second damage information, any one of (1) to (9).

[0021] An information processing method for obtaining a first image of a structure and first damage information of the structure extracted from the first image, obtaining a second image of the structure taken at a time different from the first image and second damage information of the structure extracted from the second image, extracting a plurality of feature points from the first image, extracting a plurality of feature points from the second image, searching for corresponding feature points between the first image and the second image, and based on the information of the corresponding feature points, performing rough alignment of the first damage information and the second damage information and then performing detailed alignment to align the first damage information and the second damage information, and calculating the difference between the first damage information and the second damage information after the alignment process.

[0022] An information processing program for causing a computer to perform the following: obtaining a first image of a structure and first damage information of the structure extracted from the first image, obtaining a second image of the structure taken at a time different from the first image and second damage information of the structure extracted from the second image, extracting a plurality of feature points from the first image, extracting a plurality of feature points from the second image, searching for corresponding feature points between the first image and the second image, and based on the information of the corresponding feature points, performing rough alignment of the first damage information and the second damage information and then performing detailed alignment to align the first damage information and the second damage information, and calculating the difference between the first damage information and the second damage information after the alignment process.

Advantages of the Invention

[0023] According to the present invention, the difference in damage information can be accurately calculated.

Brief Description of the Drawings

[0024] [Figure 1] Block diagram showing an example of the hardware configuration of an information processing apparatus [Figure 2] Block diagram of the functions of an information processing apparatus [Figure 3] Diagram showing an example of crack information converted into vector data [Figure 4] Diagram visually representing the search results of corresponding points [Figure 5] Block diagram of the functions of an alignment processing unit [Figure 6] Conceptual diagram for calculating the difference in crack length [Figure 7] Block diagram of the functions of the difference calculation processing unit [Figure 8] Conceptual diagram of crack pair generation [Figure 9] A diagram showing an example of how the results of calculating the difference in crack lengths are displayed. [Figure 10] This figure shows another example of how the calculation results of the difference in crack lengths are displayed. [Figure 11] A diagram showing an example of how the difference in crack width is calculated and displayed. [Figure 12] A flowchart illustrating the procedure for calculating the difference in damage information (crack information). [Figure 13] Flowchart showing the alignment process [Figure 14] A flowchart showing the procedure for calculating the difference. [Figure 15] Conceptual diagram for calculating the difference when damage information is composed of area information. [Figure 16] Conceptual diagram of segmented imaging [Figure 17] Conceptual diagram for correcting lens distortion differences [Figure 18] Block diagram of the functions of an information processing device [Figure 19] A diagram showing the correspondence between current and past images captured in segments. [Figure 20] A flowchart showing the procedure for calculating the difference in damage information. [Figure 21] Conceptual diagram of the shooting range [Figure 22] Block diagram of the functions of an information processing device [Figure 23] Conceptual diagram of the process for calculating the difference in crack lengths. [Figure 24] Conceptual diagram of the process for calculating the difference in crack lengths. [Figure 25] Conceptual diagram of integrated processing [Figure 26] A flowchart showing the procedure for calculating the difference in damage information. [Figure 27]A conceptual diagram illustrating an example where the current scope of damage information extraction differs from the scope of past damage information extraction. [Figure 28] Block diagram of the functions of an information processing device [Figure 29] A diagram showing an example of the output of the region where the difference can be calculated. [Figure 30] Block diagram of the functions of an information processing device [Figure 31] A flowchart showing the procedure for calculating the difference in damage information. [Modes for carrying out the invention]

[0025] Preferred embodiments of the present invention will be described in detail below with reference to the attached drawings.

[0026] [First Embodiment] This section explains the process of extracting cracks from images of a structure and calculating the difference between those extracted and past inspection results. The information about cracks extracted from the images is an example of damage information.

[0027] [Hardware configuration of information processing equipment] Figure 1 is a block diagram showing an example of the hardware configuration of an information processing device.

[0028] The information processing device 10 is composed of a general-purpose computer, such as a personal computer. As shown in Figure 1, the information processing device 10 mainly includes a CPU (Central Processing Unit) 11, ROM (Read Only Memory) 12, RAM (Random Access Memory) 13, auxiliary storage device 14, input device 15, output device 16, input / output interface 17, and communication interface 18.

[0029] CPU 11 is an example of a processor. The computer comprising the information processing device 10 functions as an information processing device when CPU 11 executes a predetermined program (information processing program). The program executed by CPU 11 is stored in ROM 12 or auxiliary storage device 14.

[0030] The auxiliary storage device 14 constitutes the storage unit of the information processing device 10. The auxiliary storage device 14 is composed of, for example, an HDD (Hard Disk Drive), an SSD (Solid State Drive), etc.

[0031] The input device 15 constitutes the operating section of the information processing device 10. The input device 15 is composed of, for example, a keyboard, mouse, touch panel, etc.

[0032] The output device 16 constitutes the display unit of the information processing device 10. The output device 16 is composed of a display such as a liquid crystal display or an organic electro-luminescence display.

[0033] The input / output interface 17 constitutes the connection section of the information processing device 10. The information processing device 10 is connected to external devices via the input / output interface 17.

[0034] The communication interface 18 constitutes the communication unit of the information processing device 10. The information processing device 10 is connected to a network (for example, the Internet) via the communication interface 18.

[0035] [Functions of an information processing device] Figure 2 is a block diagram of the functions of the information processing device.

[0036] Regarding the calculation of the difference in damage information, the information processing device 10 has the functions of an information acquisition unit 21, a feature point extraction processing unit 22, a corresponding point search processing unit 23, a positioning processing unit 24, a difference calculation processing unit 25, a calculation result output processing unit 26, and a calculation result recording processing unit 27. These functions are realized by the CPU 11 executing a predetermined program (information processing program).

[0037] In this embodiment, it is assumed that the crack extraction process has been completed in advance. The extracted crack information (damage information) is associated with the information of the source image and stored in the auxiliary storage device 14. The crack information is recorded, for example, as vector data.

[0038] Figure 3 shows an example of crack information converted into vector data.

[0039] Vectorization involves determining line segments defined at the start and end points of a crack. These line segments become vectors representing the crack (crack vectors). The vectorized crack information includes at least the coordinate information of the start and end points of the crack vector. In this embodiment, it also includes information on the length and width of the crack. The coordinates are set, for example, with one of the feature points extracted from the image as the origin.

[0040] In the table shown in Figure 3, the Group ID (Identification) is information that identifies a group of cracks. A group of cracks is a sequence of crack vectors that can be considered as a single crack. The Vector ID is information that identifies each crack vector.

[0041] In addition to the information provided, various other types of information can be added to the crack information. Furthermore, for branching cracks, the crack information can be recorded in a hierarchical manner (see, for example, Japanese Patent Publication No. 2019-200213).

[0042] The crack extraction process is performed using known methods, such as pre-trained models developed through machine learning.

[0043] Image information includes image data, acquisition date and time information, image width information, image height information, and resolution (pixel resolution) information. The acquisition date and time information is the date and time the image information was acquired. The image width and height information is recorded as the number of pixels in the width and height directions of the image. The unit of resolution is mm / pixel.

[0044] The information acquisition unit 21 acquires information on the cracks to be processed (damage information) and information on the image from which the crack information has been extracted. As described above, in this embodiment, the difference between current and past crack information is calculated. Therefore, the acquired crack information is current and past crack information, and the acquired image information is current and past image information. Current crack information is an example of first damage information. On the other hand, past crack information is an example of second damage information. Also, the image from which current crack information has been extracted (current image) is an example of first image. On the other hand, the image from which past crack information has been extracted (past image) is an example of second image. Current and past images are images of the structure to be inspected taken at different points in time.

[0045] Information about current cracks and information about images from which that current crack information has been extracted (current images) are associated with each other and stored in the auxiliary storage device 14. Similarly, information about past cracks and information about images from which that past crack information has been extracted (past images) are associated with each other and stored in the auxiliary storage device 14. The information acquisition unit 21 reads and acquires information about current cracks and information about images (current images) associated with that current crack information from the auxiliary storage device 14. The information acquisition unit 21 also reads and acquires information about past cracks and information about images (past images) associated with that past crack information from the auxiliary storage device 14.

[0046] Note that the current and past images are photographs taken of approximately the same area on the same surface of the same structure. Therefore, the crack information extracted from each image will be information on cracks in approximately the same area on the same structure.

[0047] The feature point extraction processing unit 22 extracts feature points individually from the current image and past image acquired by the information acquisition unit 21. Feature point extraction is performed using known methods such as AKAZE (Accelerated KAZE) and ORB (Oriented FAST and Rotated BRIEF). Multiple feature points are extracted from each image.

[0048] The corresponding point search processing unit 23 performs a process (so-called matching process) to search for corresponding feature points (corresponding points) between the current image from which feature points have been extracted and past images. This process is performed using known methods such as brute-force or Fast Library for Approximate Nearest Neighbors (FLANN).

[0049] Figure 4 is a visual representation of the results of the correspondence point search. In Figure 4, corresponding feature points between the current image I1 and the past image I2 are connected by straight lines. For example, in Figure 4, feature point CP1 extracted from the current image I1 corresponds to feature point CP2 extracted from the past image I2, and the two are connected by a straight line L.

[0050] The alignment processing unit 24 aligns the current crack information with the past crack information based on the corresponding feature point information between the current image and the past image.

[0051] Figure 5 is a block diagram of the functions of the alignment processing unit.

[0052] As shown in Figure 5, the alignment processing unit 24 has the functions of a coarse alignment processing unit 24A and a detailed alignment processing unit 24B.

[0053] The coarse alignment processing unit 24A performs a rough alignment of the current crack information and past crack information as a preliminary step. That is, it performs a rough alignment. The coarse alignment processing unit 24A performs a rough alignment of the current crack information and past crack information based on the corresponding point information (information on corresponding feature points between the current image and the past image) found by the corresponding point search processing unit 23. In this embodiment, the coarse alignment process is performed by a known rigid body alignment process. For example, processes such as affine transformation and projection transformation can be used as rigid body alignment processes. The coarse alignment processing unit 24A constructs a rigid body deformation model based on the corresponding point information found by the corresponding point search processing unit 23. That is, it constructs a rigid body deformation model that reduces the misalignment of the corresponding points. The coarse alignment processing unit 24A applies the constructed rigid body deformation model to the current crack information or the past crack information to perform a rough alignment (rigid body alignment) of the current crack information and the past crack information. In this embodiment, the model is roughly aligned by applying past crack information. Therefore, when the rigid deformation model is applied to past crack information, a model is constructed that is aligned with the current crack information.

[0054] The detailed alignment processing unit 24B performs detailed alignment of the current crack information after the rough alignment process with the past crack information as a subsequent process. That is, it performs detailed alignment. The detailed alignment processing unit 24B performs detailed alignment of the current crack information after the rough alignment process with the past crack information based on the corresponding point information after the rough alignment process. In this embodiment, the detailed alignment process is performed by performing a known non-rigid alignment process. As a non-rigid alignment process, for example, a process using the TPS (Thin-Plate Spline) method, a process using the CPD (Coherent Point Drift) method, or a process using the FFD (Free-Form Deformation) method can be used. The detailed alignment processing unit 24B constructs a non-rigid deformation model based on the corresponding point information after the rough alignment process. That is, it constructs a non-rigid deformation model that reduces the misalignment of the corresponding points. The detailed alignment processing unit 24B applies the constructed non-rigid deformation model to the current crack information or past crack information after the rough alignment process, and performs detailed alignment (non-rigid alignment) between the current crack information and past crack information after the rough alignment process. In this embodiment, the detailed alignment is performed by applying the model to the past crack information after the rough alignment process. Therefore, when the non-rigid deformation model is applied to the past crack information after the rough alignment process, a model is constructed that is aligned with the current crack information after the rough alignment process.

[0055] As described above, the alignment processing unit 24 performs rough alignment followed by detailed alignment to align the current crack information with the past crack information. This allows for accurate alignment of the current crack information with the past crack information. In other words, while a method that only performs rigid body alignment may result in positional discrepancies, performing non-rigid body alignment afterward corrects any discrepancies that could not be corrected by rigid body alignment alone, thereby achieving high-precision alignment. Furthermore, while performing only non-rigid body alignment may result in excessive deformation, performing rigid body alignment beforehand suppresses excessive deformation, resulting in high-precision alignment overall.

[0056] The difference calculation processing unit 25 calculates the difference between the current crack information after the alignment process and the past crack information.

[0057] Figure 6 is a conceptual diagram of how to calculate the difference in crack length.

[0058] Figure 6(A) shows IC1, which represents current crack information extracted from the current image. Figure 6(B) shows IC2, which represents past crack information extracted from past images. Figure 6(C) shows IC3, which represents the difference calculation result.

[0059] As shown in Figure 6, the difference between current crack information IC1 and past crack information IC2 is calculated as the difference in crack length information IC3. In other words, cracks that are present in one but not in the other are extracted, and the length of the extracted cracks is calculated as the difference. More specifically, the difference in crack length is calculated using the following procedure.

[0060] Figure 7 is a block diagram of the functions of the difference calculation processing unit.

[0061] As shown in Figure 7, the difference calculation processing unit 25 has the functions of a pairing processing unit 25A, a first difference calculation unit 25B, and a second difference calculation unit 25C.

[0062] The pairing processing unit 25A acquires current and past crack information after the alignment process and generates corresponding crack pairs. As described above, the crack information is converted into vector data and recorded as vector information. The pairing processing unit 25A generates pairs (pairs) of adjacent crack vectors.

[0063] Figure 8 is a conceptual diagram of the generation of crack pairs.

[0064] Figure 8(A) provides an overview of some of the current crack information, while Figure 8(B) provides an overview of some of the past crack information. Figure 8(C) is a superimposed view of Figures 8(A) and 8(B).

[0065] While the alignment process aligns corresponding cracks, it is difficult to perfectly match the positions of all cracks. Furthermore, discrepancies may occur in the crack extraction results themselves. Therefore, in this embodiment, pairs of corresponding cracks are generated, and the generated pairs are compared to calculate the difference.

[0066] Pairs are generated by combining adjacent crack vectors when current crack information and past crack information are superimposed. In other words, pairs are generated using the crack vectors that are closest to each other when superimposed.

[0067] For example, in the example shown in Figure 8, if we focus on the crack vector V1-1 in the current crack information, the nearest past crack vector is crack vector V2-1, as shown in Figure 8(C). Therefore, a pair is generated between crack vector V1-1 and crack vector V2-1. In the example shown in Figure 8, a pair is further generated between the current crack vector V1-2 and the past crack vector V2-2. Also, a pair is generated between the current crack vector V1-3 and the past crack vector V2-3. Furthermore, a pair is generated between the current crack vector V1-4 and the past crack vector V2-4. Also, a pair is generated between the current crack vector V1-5 and the past crack vector V2-5. In Figure 8(C), the arrows indicate the correspondence between the current crack vector and the past crack vector.

[0068] On the other hand, in the example shown in Figure 8, there is no past crack vector corresponding to the current crack vector V1-6. ​​In this case, the current crack vector V1-6 is considered an unpaired crack vector. An unpaired crack vector corresponds to a newly generated crack (a grown crack).

[0069] The first difference calculation unit 25B calculates the difference in crack length as the difference in crack information. The first difference calculation unit 25B calculates the difference in crack length based on the current and past crack information for which pairs of crack vectors have been generated. Specifically, it calculates the length of crack vectors for which no pairs have been generated and then calculates the difference in crack length. If the crack information already includes information on the length of the crack vectors, it assumes that the length has already been calculated and retrieves that length information. In the example shown in Figure 8, the crack vector for which no pairs have been generated is the current crack vector V1-6. ​​Therefore, it calculates the length of the current crack vector V1-6 and then calculates the difference in crack length.

[0070] The second difference calculation unit 25C calculates the difference in crack width as the difference in crack information. The second difference calculation unit 25C calculates the difference in crack width based on the current and past crack information from which the pair of crack vectors was generated. Specifically, it calculates the difference in width between pairs of crack vectors. For example, in the example shown in Figure 8, the difference in width between the current crack vector V1-1 and the past crack vector V2-1 pair is calculated as "(width of current crack vector V1-1) - (width of past crack vector V2-1)". Assume that the width of the current crack vector V1-1 is 0.1 mm and the width of the paired past crack vector V2-1 is 0.05 mm. In this case, 0.1 mm - 0.05 mm = 0.05 mm is calculated as the difference in width of this pair.

[0071] The calculation result output processing unit 26 outputs the difference calculation result in a predetermined format to the output device 16. In the information processing device 10 of this embodiment, the output device 16 is composed of a display. Therefore, the calculation result output processing unit 26 displays the difference calculation result on the display, which is the display destination.

[0072] Figure 9 shows an example of how the calculation results for the difference in crack length are displayed.

[0073] As shown in Figure 9, the result of calculating the difference in crack lengths is displayed as an image in which cracks extracted from both the current and past images are represented by lines of different colors or different types. In the example shown in Figure 9, cracks extracted only from the current image are shown as solid red lines, cracks extracted only from the past image are shown as solid green lines, and common cracks extracted from both images are shown as dashed lines (black). The calculation result output processing unit 26 generates an image in which cracks are separated and displayed by lines of different colors or different types, and outputs it to the output device 16.

[0074] Furthermore, "cracks extracted solely from the current image" refer to cracks that were not present in the past. Therefore, these cracks are considered to be either growing (extending) or newly formed.

[0075] On the other hand, "cracks extracted only from past images" are cracks that existed in the past but do not exist now. Unless repairs have been made, it is considered unnatural for such cracks to be output as a result of difference calculation. Therefore, it is preferable to highlight and draw attention to cracks extracted only from past images. For example, they can be displayed with a visually more conspicuous color or line type to draw attention. Alternatively, a warning can be displayed to draw attention.

[0076] The reasons for calculating unnatural differences (differences that contradict the irreversible changes in damage) may include flaws in the photography process or inadequate damage extraction processing.

[0077] The image showing the difference calculation result may be displayed overlaid on the current or past image from which the cracks were extracted, if necessary.

[0078] Figure 10 shows another example of how the calculation results of the difference in crack lengths are displayed. Figure 10 also shows an example of how the display can be switched according to user instructions.

[0079] Figure 10(A) shows an example of displaying image OI1, in which cracks extracted from both current and past images are represented by lines of different colors or different types. This image OI1 is the same as the image shown in Figure 9.

[0080] Figure 10(B) shows an example of displaying image OI2, which represents cracks extracted from the current image as lines. This image OI2 corresponds to the current damage diagram, which traces the current cracks from the current image.

[0081] Figure 10(C) shows an example of displaying image OI3, which represents cracks extracted from past images as lines. This image OI3 corresponds to a past damage diagram, which traces past cracks from past images.

[0082] Figure 10(D) shows an example of displaying image OI4, where only the difference is represented by a line.

[0083] Figure 10(E) shows an example of displaying image OI5, which represents cracks extracted only from the current image from the difference, as lines. In other words, it shows an example of displaying an image that represents only cracks that did not exist in the past, as lines.

[0084] Figure 10(F) shows an example of displaying image OI6, which represents cracks extracted only from past images as lines. In other words, it shows an example of displaying an image that represents only cracks that are not present in the current image as lines.

[0085] The calculation result output processing unit 26 switches these images according to user instructions and outputs them to the output device 16. These images may be displayed overlaid on current or past images from which cracks have been extracted, as needed.

[0086] Figure 11 shows an example of how the calculation results of the difference in crack width are displayed. Figure 11 shows the image OI displayed as the calculation result, and an enlarged image EI of a part of it.

[0087] The example shown in Figure 11 displays the results of calculating the difference in width in addition to the results of calculating the difference in length of the cracks.

[0088] As shown in Figure 11, the calculation of the length difference results in the cracks extracted from both the current and past images being represented by lines of different colors or different types. In addition, the calculation result of the width difference is displayed near the line representing each crack (crack vector). In the example shown in Figure 11, the calculation result of the difference is displayed along with the vector ID of each crack (crack vector).

[0089] The difference calculation results may also be displayed overlaid on the current or past image from which the cracks were extracted.

[0090] Furthermore, regarding differences in width, it is preferable to highlight or otherwise draw the user's attention to areas where the output shows an unnatural difference, i.e., where the current width is smaller than the past width. For example, this could be done by adding a marker or changing the display color to make it stand out more than others.

[0091] The calculation result recording processing unit 27 records the information of the difference calculation result in the auxiliary storage device 14. The calculation result recording processing unit 27 records the calculation result information in the auxiliary storage device 14 in association with the current and past crack information for which the difference was calculated.

[0092] [Procedure for calculating the difference (information processing method)] Figure 12 is a flowchart showing the procedure for calculating the difference in damage information (crack information) by the information processing device of this embodiment.

[0093] First, a process is performed to acquire information about the cracks to be processed, and information about the images from which that crack information has been extracted (step S1). In this embodiment, the difference between the current and past crack information is calculated. Therefore, information about the current and past cracks, as well as information about the current and past images from which that current and past crack information has been extracted, are acquired. In this embodiment, this information is read and acquired from the auxiliary storage device 14.

[0094] Next, a process is performed to extract feature points individually from the acquired current and past images (step S2). Multiple feature points are extracted from each image.

[0095] Next, a process is performed to search for corresponding feature points (corresponding points) between the current and past images (step S3).

[0096] Next, based on the information of the corresponding feature points, a process is performed to align the information of the current crack with the information of the past crack (step S4).

[0097] Figure 13 is a flowchart showing the alignment process.

[0098] First, a rigid body deformation model is constructed based on the information of the corresponding feature points (step S4A).

[0099] Next, the constructed rigid deformation model is used to roughly align the current crack information with the past crack information (rigid alignment) (step S4B). In this embodiment, the constructed rigid deformation model is applied to the past crack information and roughly aligned.

[0100] Next, a non-rigid deformation model is constructed based on the information of the corresponding points after the rough alignment process (step S4C).

[0101] Next, using the constructed non-rigid deformation model, the current crack information after the rough alignment process and the past crack information are aligned in detail (non-rigid alignment) (step S4D). In this embodiment, the constructed non-rigid deformation model is applied to the past crack information after the rough alignment process to perform detailed alignment.

[0102] The alignment process is now complete.

[0103] Next, as shown in Figure 12, a process is performed to calculate the difference between the current crack information after the alignment process and the past crack information (Step S5).

[0104] Figure 14 is a flowchart showing the steps for calculating the difference.

[0105] First, a process is performed to generate corresponding crack pairs between the current crack information after alignment processing and the past crack information (step S5A). In this embodiment, pairs are generated between adjacent crack vectors.

[0106] Next, based on the current and past crack information for which pairs of crack vectors have been generated, the difference in crack lengths is calculated (step S5B). Specifically, the lengths of crack vectors for which no pairs have been generated are calculated, and the difference in crack lengths is then calculated.

[0107] Furthermore, based on the current and past crack information from which the crack vector pairs were generated, the difference in crack width is calculated (step S5C). Specifically, the difference in width is calculated between the pairs of crack vectors.

[0108] The difference calculation process is now complete. Once the difference calculation process is complete, the calculation result is output to the output device 16 (step S6). The calculation result is also recorded in the auxiliary storage device 14 (step S7). The calculation result is recorded in association with the information of the crack from which the difference was calculated.

[0109] As described above, the information processing device of this embodiment allows for the accurate calculation of differences for each crack by performing alignment. In particular, by performing rough alignment followed by detailed alignment as part of the alignment process, the alignment can be performed with high precision. This enables the subsequent difference calculation process to be performed with high precision.

[0110] [Regarding the method for calculating the difference for other types of damage] If the damage information is converted into vector data and recorded as vector information, the difference can be calculated using the information processing device described in the above embodiment. For example, the difference in chalk information can be calculated using the information processing device described in the above embodiment.

[0111] Examples of damage appearing on the surface of structures include, in addition to the cracks mentioned above, spalling, exposed rebar, water leakage (including rust stains), free lime, and corrosion. This information is recorded as area information, and the damage information is recorded.

[0112] If the damage information consists of area information (damaged area information), the difference calculation processing unit 25 calculates the difference in the damage information as follows.

[0113] Figure 15 is a conceptual diagram of the calculation of the difference when the damage information consists of region information.

[0114] Figure 15 shows an example of extracting areas where free lime is present as damaged areas from images taken of a structure.

[0115] Figure 15(A) shows IR1, information on the current free lime generation area (damage area) extracted from the current image. Figure 15(B) shows IR2, information on the past free lime generation area (damage area) extracted from past images. The free lime generation area information extracted from the current image is an example of the first type of damage information. The free lime generation area information extracted from past images is an example of the second type of damage information. Figure 15(C) shows IR3, information on the difference calculation result.

[0116] As shown in Figure 15, when damage information consists of region information, the difference calculation processing unit 25 calculates (extracts) the difference region corresponding to the damaged region. It also calculates the area of ​​that difference region. In the example shown in Figure 15(C), the region calculated as the difference is shown in black.

[0117] When outputting the difference calculation results, it is preferable to display them separately by color and / or line type, similar to the case of cracks. Furthermore, it is preferable to highlight areas where unnatural differences are output that contradict the irreversible changes in damage, for example, areas where the current damage is smaller than the past damage, to draw the user's attention.

[0118] [Variations of the pairing process] As described in the above embodiment, if the damage information is vectorized crack information, the difference can be easily and accurately calculated by generating pairs of corresponding cracks and calculating the difference. In the above embodiment, pairs are generated between adjacent cracks, but the method for generating pairs of corresponding cracks is not limited to this. In addition, for example, a matching method using dynamic programming (DP matching) can be used to generate pairs of corresponding cracks. DP matching is a pattern matching method for one-dimensional data and is a technique used in signal processing matching. This technique is used for vector matching, that is, crack vector matching, to generate pairs of corresponding cracks.

[0119] [Modified version of the information acquisition unit] In the above embodiment, the configuration is such that current and past damage information is acquired from the auxiliary storage device 14. However, the information processing device 10 may be configured to implement a damage extraction function and directly acquire the extraction results. In this case, the extraction process may be performed only for current damage, and past damage information may be read and acquired from the auxiliary storage device 14.

[0120] Furthermore, damage information and image information from which that damage information has been extracted may be acquired from an external device connected via the input / output interface 17. Alternatively, the information may be acquired from an external device connected via a network.

[0121] [Variations of the detailed alignment processing unit] In the above embodiment, a non-rigid alignment process is performed as the detailed alignment process; however, the detailed alignment process is not limited to this. For example, lens distortion can also cause misalignment. Therefore, if the lens distortion differs between the current and past images, detailed alignment is performed by correcting the difference in lens distortion. This eliminates misalignment caused by lens distortion, enabling more accurate alignment. The following describes a method for correcting lens distortion and performing detailed alignment.

[0122] The difference in lens distortion is corrected using the following method: one image is fixed, and the lens distortion variable and the projection transformation matrix variable of the other image are varied to solve an optimization problem that minimizes the shift between corresponding points.

[0123] A camera can be represented by a model that projects three-dimensional coordinates onto an image plane.

[0124]

number

[0125] Here, (X,Y,Z) represents the coordinates of the point in 3D coordinates, and (u,v) represents the coordinates of the point projected onto the image plane. (c x ,c y ) is the principal point (usually the center of the image). x ,f y This is the focal length expressed in pixels.

[0126] Since actual camera lenses have distortion in both the radial and circumferential directions, the above model can be extended as follows:

[0127]

number

[0128] Here, k1, k2, and k3 are the radial distortion coefficients of the lens. p1 and p2 are the circumferential distortion coefficients of the lens.

[0129] Therefore, lens distortion can be expressed by the following equation, where k1, k2, k3, p1, and p2 are variables.

[0130]

number

[0131] The projection transformation matrix is ​​expressed by the following equation:

[0132]

number

[0133] Correcting the difference in lens distortion involves optimizing the lens distortion variable and the projection transformation matrix variable so that the shift in corresponding points is minimized.

[0134] Figure 17 is a conceptual diagram of the correction of lens distortion differences.

[0135] As shown in Figure 17, one image is kept fixed, and the lens distortion variable and projection transformation matrix variable of the other image are changed to solve the optimization problem so that the shift between corresponding points is minimized.

[0136] Figure 17 shows an example of solving an optimization problem by fixing the current image and changing the lens distortion variable and projection transformation matrix variable of past images, so as to minimize the shift between corresponding points.

[0137] Thus, when lens distortion differs between two images, correcting for the difference in lens distortion allows for more accurate alignment. Furthermore, this enables the precise calculation of the difference in damage information.

[0138] [Second Embodiment] When automatically extracting damage from images, high-resolution images are required to accurately detect minute damage. To obtain high-resolution images, the inspection area may be divided into multiple regions and photographed separately. When the inspection area is divided and photographed separately, damage is extracted individually from each image, and the results are combined to obtain the overall extraction result. A panoramic stitching method is used for the integration process.

[0139] This embodiment describes a method for calculating the difference in damage information when inspection points are photographed separately in the present and past.

[0140] [Split shooting] First, let's explain segmented imaging. Figure 16 is a conceptual diagram of segmented imaging. Figure 16 shows an example of inspecting (photographing) the deck of a bridge.

[0141] Generally, inspections of the bridge deck are performed in units of bridge slabs. A bridge slab (GO) is a section of the bridge deck divided by the main girders and transverse girders. If it is not possible to photograph the entire bridge slab (GO) in one shot, or if a high-resolution image cannot be obtained even if it is possible to photograph it in one shot, the shooting area is divided and photographed in multiple shots. In Figure 16, the frame indicated by the symbol AV indicates the range of a single shot. As shown in Figure 16, adjacent areas are photographed so that they partially overlap each other. The reason for photographing them in an overlapping manner is to enable accurate panoramic stitching.

[0142] [Functions of an information processing device] If the inspection area has been photographed in sections both currently and in the past, the information processing device of this embodiment calculates the difference for each of the separately photographed images, and integrates the results of the difference calculations for each image to generate overall difference information.

[0143] Figure 18 is a block diagram of the functions of the information processing device in this embodiment.

[0144] It is assumed that the damage extraction process and the panoramic stitching process have already been completed. As described above, the damage extraction process is performed for each image that was captured in sections. Damage information (information on cracks, free lime, etc.) extracted from each image is recorded in the auxiliary storage device 14 in association with the information of the source image. In addition, the information of each image and the damage information extracted from each image are stored in the auxiliary storage device 14 in association with the information of the panoramic stitched image and the damage information integrated by panoramic stitching. That is, they are stored as a set. The information of the panoramic stitched image includes the information used when stitching each image into a panorama, that is, the information of the stitching parameters (stitching processing information) necessary for processing each image into a panoramic image. The information of the stitching parameters includes information on parameters for image deformation such as affine transformation parameters. Since this type of panoramic stitching process itself is a known technique, a detailed explanation is omitted.

[0145] As shown in Figure 18, the information processing device of this embodiment differs from the information processing device of the first embodiment in that it further has the functions of an integrated processing unit 30.

[0146] The information acquisition unit 21 sequentially acquires information from each of the currently and past images that were captured in segments, as well as information on current and past damage extracted from each of the currently and past images. It also acquires information on the synthesis parameters necessary for processing each image into a panoramic image. This information is read from the auxiliary storage device 14.

[0147] The feature point extraction processing unit 22 extracts feature points individually from each current and past image acquired by the information acquisition unit 21.

[0148] The corresponding point search processing unit 23 searches for corresponding feature points (corresponding points) between the corresponding current and past images.

[0149] Figure 19 shows the correspondence between current and past images captured in segments.

[0150] Figure 19(A) shows the results of current segmented imaging, and Figure 19(B) shows the results of past segmented imaging. Both Figure 19(A) and Figure 19(B) show examples where the subject is divided into eight sections for imaging.

[0151] In this case, the image P1-1, which was taken of the upper left corner region in the present, and the image P2-1, which was taken of the same region in the past, are corresponding images. Similarly, the image P1-2, which was taken of the upper right corner region in the present, and the image P2-2, which was taken of the same region in the past, are corresponding images. In this way, images that were taken of the same region (essentially the same) become corresponding images. The correspondence point search processing unit 23 searches for corresponding points between the corresponding images.

[0152] The alignment processing unit 24 aligns damage information between corresponding images. In this process, the damage information is aligned based on the corresponding point information extracted between the corresponding images. For example, in the example shown in Figure 19, the alignment process is performed between the damage information extracted from the current image P1-1 and the damage information extracted from the past image P2-1 corresponding to the current image P1-1. The alignment process involves a rough alignment process followed by a detailed alignment process.

[0153] The difference calculation processing unit 25 calculates the difference in damage information between corresponding images. For example, in the example shown in Figure 19, the difference is calculated between the damage information extracted from the current image P1-1 and the damage information extracted from the past image P2-1 corresponding to the current image P1-1.

[0154] The integrated processing unit 30 integrates the difference information of damage calculated between corresponding images. The integrated processing unit 30 integrates the difference information using the synthesis parameter information used during panoramic stitching. For example, it integrates the difference information using the synthesis parameter information used when stitching the currently acquired images in sections into a panoramic image. This makes it possible to generate difference information of damage for the entire area to be inspected, corresponding to the panoramic stitched image.

[0155] The calculation result output processing unit 26 outputs the difference calculation result in a predetermined format to the output device 16. In the information processing device 10 of this embodiment, the output device 16 is composed of a display. Therefore, the calculation result output processing unit 26 displays the difference calculation result on the display, which is the display destination.

[0156] The calculation result recording processing unit 27 records the information of the integrated difference calculation result in the auxiliary storage device 14. At this time, the calculation result recording processing unit 27 records the integrated difference calculation result in the auxiliary storage device 14 in association with the integrated current and past damage information.

[0157] [Steps for calculating the difference] Figure 20 is a flowchart showing the procedure for calculating the difference in damage information by the information processing device of this embodiment.

[0158] First, regarding the current and past damage information to be processed, a process is performed to acquire information from each of the segmented images, damage information extracted from each image, and information on the synthesis parameters (Step S11). Next, a process is performed to extract feature points individually from each acquired image (Step S12). Next, a process is performed to search for corresponding feature points (corresponding points) between the corresponding current and past images (Step S13). Next, a process is performed to align the corresponding current and past damage information (Step S14). The alignment process is performed using the information of the corresponding points obtained between the corresponding images. After the rough alignment process, a detailed alignment process is performed. Next, a process is performed to calculate the difference between the corresponding current and past damage information (Step S15). Next, a process is performed to integrate the calculation results of the difference calculated between the corresponding current and past damage information (Step S16). Next, a process is performed to output the integrated difference calculation result (Step S17). Finally, a process is performed to record the integrated difference calculation result (Step S18).

[0159] As described above, the information processing device of this embodiment can accurately calculate the difference when damage information is integrated by panoramic stitching.

[0160] In the above embodiment, it was explained that the panoramic stitching process had already been completed for the currently and past images that were captured in segments. However, the information processing device may be equipped with a panoramic stitching processing function, and the panoramic stitching process may be performed within the device.

[0161] [Third Embodiment] This embodiment describes how to handle cases where the extraction range of damage differs between two pieces of damage information used to calculate the difference. Cases where the extraction range of damage differs between two pieces of damage information used to calculate the difference refer to cases where the shooting range of the images from which each piece of damage information was extracted differs.

[0162] Figure 21 is a conceptual diagram of the shooting range.

[0163] Figure 21 shows an example of inspecting a bridge pier. Figure 21(A) shows the bridge pier as it is now. Figure 21(B) shows the bridge pier as it was when it was inspected in the past.

[0164] In Figure 21, frame F1 shows the range of the image taken of the current bridge pier. Frame F21 shows the range of the first image taken when the bridge pier was inspected in the past, and frame F22 shows the range of the second image.

[0165] As shown in Figure 21, the shooting range of the current image of the bridge pier (the area indicated by frame F1) does not match the shooting range of the first image taken when the bridge pier was inspected in the past (the area indicated by frame F21). Furthermore, the shooting range of the current image of the bridge pier does not match the shooting range of the second image taken when the bridge pier was inspected in the past (the area indicated by frame F22).

[0166] In this way, if the image and shooting range do not match those captured during past inspections, the information processing device calculates the difference in the overlapping area of ​​the images. Furthermore, if there is overlapping area between multiple images, the information processing device integrates the difference information calculated between the multiple images to determine the difference with the past inspection results. For example, in the example shown in Figure 21, the information processing device first calculates the difference in the overlapping area between the damage information extracted from the current image and the damage information extracted from the first past image. Similarly, the information processing device calculates the difference in the overlapping area between the damage information extracted from the current image and the damage information extracted from the second past image. Then, the information processing device integrates each calculation result to generate information on the difference with the past inspection results. In other words, the information processing device mutually complements the information of the missing areas to generate difference information.

[0167] [Functions of an information processing device] Figure 22 is a block diagram of the functions of the information processing device in this embodiment.

[0168] As shown in Figure 22, the information processing device of this embodiment has the functions of an information acquisition unit 21, a feature point extraction processing unit 22, a corresponding point search processing unit 23, a positioning processing unit 24, a difference calculation processing unit 25, an integration processing unit 30, a calculation result output processing unit 26, and a calculation result recording processing unit 27.

[0169] The information acquisition unit 21 acquires current damage information and past damage information for which the difference is calculated, as well as information from the current image from which the current damage information has been extracted and information from the past image from which the past damage information has been extracted. In this embodiment, it is assumed that the damage information extraction process has already been completed. Regarding the past damage information and past image information, information from multiple damage information and past images that have overlapping areas with the current damage information is acquired.

[0170] The feature point extraction processing unit 22 extracts feature points individually from each current and past image acquired by the information acquisition unit 21.

[0171] The correspondence point search processing unit 23 searches for corresponding feature points (corresponding points) individually between the current image and multiple past images.

[0172] The alignment processing unit 24 aligns the current damage information with multiple past damage information individually. In this process, the damage information is aligned based on the corresponding point information extracted between each image. For example, if there are two past damage information items, a first damage information item and a second damage information item, the current damage information is aligned with the first past damage information item, and also with the second past damage information item.

[0173] The difference calculation processing unit 25 calculates the difference in overlapping areas between the current damage information after the alignment process and multiple past damage information.

[0174] Figures 23 and 24 are conceptual diagrams illustrating the process for calculating the difference in crack lengths when the damage information is crack information.

[0175] Figures 23 and 24 show an example where there are two past damage information records, IC21 and IC22, that have overlapping regions with the current damage information record IC11. Figure 23 is a conceptual diagram for calculating the difference between the current damage information record IC11 and the first past damage information record IC21. Figure 24 is a conceptual diagram for calculating the difference between the current damage information record IC11 and the second past damage information record IC22.

[0176] As shown in Figure 23, the current damage information IC11 and the past first damage information IC21 partially overlap. The difference information IC31 is calculated in the overlapping region. In the difference information IC31, the solid lines represent the cracks calculated as differences. In this example, these are cracks that exist only in the present. The dashed lines represent cracks that are common to both the present and the past.

[0177] Similarly, as shown in Figure 24, the current damage information IC11 and the past second damage information IC22 partially overlap. The difference information IC32 is calculated in the overlapping region. In the difference information IC32, the solid lines represent the cracks calculated as differences. In this example, these are cracks that exist only in the present. The dashed lines represent cracks that are common to both the present and the past.

[0178] The integrated processing unit 30 integrates the calculation results of the difference calculated between the current damage information and multiple past damage information.

[0179] Figure 25 is a conceptual diagram of the integrated processing.

[0180] As shown in Figure 25, the first difference information IC31 calculated between the current damage information IC11 and the past first damage information IC21, and the second difference information IC32 calculated between the current damage information IC11 and the past second damage information IC22 are integrated to form the overall difference information IC4.

[0181] In the example shown in Figure 25, the overlapping regions of the first difference information IC31 and the second difference information IC32 are divided into upper and lower sections and cut out. The resulting first difference information IC31A and second difference information IC32A are then combined to generate the overall difference information IC4.

[0182] In addition, other known methods can be used for the integration process. For example, synthesis methods such as alpha blending and Laplacian blending can be employed.

[0183] The calculation result output processing unit 26 outputs the difference calculation result in a predetermined format to the output device 16. In the information processing device 10 of this embodiment, the output device 16 is composed of a display. Therefore, the calculation result output processing unit 26 displays the difference calculation result on the display, which is the display destination.

[0184] The calculation result recording processing unit 27 records the information of the integrated difference calculation result in the auxiliary storage device 14, associating it with current and past damage information.

[0185] [Steps for calculating the difference] Figure 26 is a flowchart showing the procedure for calculating the difference in damage information by the information processing device of this embodiment.

[0186] First, the process of acquiring the current damage information and multiple past damage information to be processed, as well as the image information from which the current and past damage information has been extracted, is performed (Step S21). Next, the process of extracting feature points individually from each acquired image is performed (Step S22). Next, the process of searching for corresponding feature points (corresponding points) individually between the current image and the multiple past images is performed (Step S23). Next, the process of aligning the current damage information with the multiple past damage information is performed (Step S24). The alignment process is performed using the information of corresponding points obtained between each image, and after the rough alignment process, a detailed alignment process is performed. Next, the process of calculating the difference individually between the current damage information and the multiple past damage information is performed (Step S25). Next, the process of integrating the calculation results of the multiple differences calculated between the current damage information and the multiple past damage information is performed (Step S26). Next, the process of outputting the integrated difference calculation result is performed (Step S27). In addition, the process of recording the integrated difference calculation result is performed (Step S28).

[0187] As described above, according to the information processing device of this embodiment, even if the extraction range of past damage information differs from the extraction range of current damage information, it is possible to suppress the occurrence of areas where the difference cannot be calculated between past inspection results and current damage information.

[0188] In the example shown in the above embodiment, the case where past damage information overlaps with each other was explained, but past damage information does not necessarily have to overlap with each other. It is sufficient if it has an overlapping area with the current damage information.

[0189] Furthermore, the technology described in this embodiment can also be applied when calculating differences in each of the images captured in segments. For example, currently, it is possible to calculate the difference between damage information extracted from an image of a single segmented region and damage information extracted from multiple past images.

[0190] [Fourth Embodiment] If the current range for extracting damage information differs from the range for extracting past damage information, the information processing device can only calculate the difference in the overlapping area.

[0191] Figure 27 is a conceptual diagram illustrating an example where the current range of damage information extraction differs from the range of past damage information extraction.

[0192] Figure 27(A) shows the current bridge pier. Figure 27(B) shows the bridge pier as it appeared during a past inspection. In Figure 27, frame F1 indicates the range of the image taken of the current bridge pier. Current damage information is extracted within the range of frame F1. Frame F2 indicates the range of the image taken when the bridge pier was inspected in the past. Past damage information is extracted within the range of frame F2. As shown in Figure 27, the extraction range for current damage information (range of frame F1) and the extraction range for past damage information (range of frame F2) are different. In this case, as shown in Figure 27, the areas indicated by the diagonal lines are the overlapping areas in the current and past damage information. The information processing device can only calculate the difference in these overlapping areas. That is, in the ranges indicated by frames F1 and F2, areas other than those indicated by the diagonal lines are areas from which the difference cannot be calculated. Thus, when the extraction range for current damage information and the extraction range for past damage information are different, it is preferable to clearly indicate to the user the range in the image from which the damage information can be extracted for which the difference can be calculated. This allows for a clear understanding of the areas where differences have been extracted, improving usability.

[0193] [Functions of an information processing device] This section will only explain the function that highlights the areas where differences have been calculated on the image.

[0194] Figure 28 is a block diagram of the functions of the information processing device in this embodiment.

[0195] As shown in Figure 28, the information processing device of this embodiment has, in addition to the functions of the information processing device of the first embodiment, the additional function of the difference calculationable area extraction processing unit 40.

[0196] The difference calculation area extraction processing unit 40 extracts areas (difference calculation areas) where the difference in damage information can be calculated, based on the current and past image information acquired by the information acquisition unit 21. The areas where damage information can be calculated between the current and past images are the overlapping areas. Therefore, the difference calculation area extraction processing unit 40 extracts the areas where the current and past images overlap to extract the difference calculation areas.

[0197] The information on the difference-calculation-possible region extracted by the difference-calculation-possible region extraction processing unit 40 is added to the calculation result output processing unit 26. The calculation result output processing unit 26 generates and outputs an image showing the difference-calculation-possible region information on the current or past image. In this embodiment, the difference-calculation-possible region information is output along with the difference calculation result information.

[0198] Figure 29 shows an example of the output of the region where the difference can be calculated.

[0199] Figure 29 shows an example of displaying the difference calculation results and information on the area where the difference can be calculated overlaid on the current image.

[0200] The calculated difference is the difference in crack length. In the example shown in Figure 29, the calculated difference is displayed in different colors.

[0201] Information on areas where differences can be calculated is shown by enclosing the calculated difference range with a frame W. Furthermore, in this example, areas where differences have not been calculated are shown with diagonal lines. For areas where differences have not been calculated, other methods such as making the image semi-transparent can also be used to distinguish them.

[0202] As described above, according to the information processing device of this embodiment, even if the extraction range of current damage information differs from the extraction range of past damage information, the area from which the difference has been extracted can be clearly identified. This further improves convenience.

[0203] Furthermore, the system can be configured to extract the difference calculation area based on current and past damage information. In this case, the system can be configured to extract the difference calculation area based on current and past damage information after the alignment process. When extracting from images, it is also preferable to perform an alignment process to extract the difference calculation area.

[0204] Furthermore, it is preferable to record the extracted information on the difference calculation area in association with the information on the difference calculation result.

[0205] [Fifth Embodiment] If the inspection area is photographed in separate sections, the information processing device calculates the difference in damage information after the panoramic images are combined.

[0206] [Functions of an information processing device] Figure 30 is a block diagram of the functions of the information processing device.

[0207] The information processing device of this embodiment differs from the information processing device of the first embodiment in that it has the function of a synthesis processing unit 50.

[0208] The information acquisition unit 21 sequentially acquires information from each of the current and past images that were captured in segments, as well as damage information extracted from each of the current and past images.

[0209] The image synthesis processing unit 50 synthesizes the current and past images, which were captured in segments, into a panorama. Since the panorama synthesis process itself is a well-known process, a detailed explanation of it will be omitted here.

[0210] The synthesis processing unit 50 also integrates current and past damage information using the synthesis parameters used when creating the panoramic image. That is, it generates damage information corresponding to the panoramic image.

[0211] The feature point extraction processing unit 22 extracts feature points from the current and past images after panoramic stitching.

[0212] The corresponding point search processing unit 23 searches for corresponding feature points (corresponding points) between the current and past images after panoramic stitching.

[0213] The alignment processing unit 24 aligns damage information between the current and past images after panoramic stitching. In this process, the damage information is aligned based on corresponding point information extracted between the current and past images after panoramic stitching. The alignment process involves a rough alignment process followed by a detailed alignment process.

[0214] The difference calculation processing unit 25 calculates the difference in damage information between the current and past images after panoramic stitching.

[0215] The calculation result output processing unit 26 outputs the difference calculation result in a predetermined format to the output device 16.

[0216] The calculation result recording processing unit 27 records the information of the integrated difference calculation result in the auxiliary storage device 14.

[0217] [Steps for calculating the difference] Figure 31 is a flowchart showing the procedure for calculating the difference in damage information by the information processing device of this embodiment.

[0218] First, regarding the current and past damage information to be processed, a process is performed to acquire information from each of the segmented images and damage information extracted from each image (step S31). Next, a process is performed to combine the acquired current and past images into a panorama (step S32). This generates a single image that captures the entire inspection area. Next, a process is performed to integrate the current and past damage information using the synthesis parameters used during panorama synthesis (step S33). This generates damage information for the entire inspection area corresponding to the panoramic image. Next, a process is performed to extract feature points individually from the current and past images after panoramic synthesis (step S34). Next, a process is performed to search for corresponding feature points (corresponding points) between the current and past images after panoramic synthesis (step S35). Next, a process is performed to align the current and past damage information after integration (step S36). The alignment process is performed using the information of corresponding points found between the current and past images after panoramic synthesis. In addition, a detailed alignment process is performed after the rough alignment process. Next, a process is performed to calculate the difference between the current and past damage information after integration (step S37). Then, a process is performed to output the result of the difference calculation (step S38). In addition, a process is performed to record the result of the difference calculation (step S39).

[0219] As described above, in this embodiment, when the inspection area is photographed in separate sections, the difference in damage information is calculated after panoramic stitching.

[0220] In the above embodiment, the example was given where the inspection area is photographed separately in both the present and the past. However, if only one of them is photographed separately, the synthesis process will only be performed on the inspection results that have been photographed separately. For example, if only past inspection results are photographed separately, the synthesis process will only be performed on the past inspection results.

[0221] Furthermore, although the above embodiment provides the information processing device with a synthesis processing function, the synthesis processing may be performed by another device.

[0222] [Other embodiments] In the above embodiment, the hardware structure that performs various processes of the information processing device is a variety of processors as shown below. These various processors include a CPU, which is a general-purpose processor that executes software (programs) and functions as various processing units; a PLD (Programmable Logic Device), which is a processor whose circuit configuration can be changed after manufacturing, such as an FPGA (Field Programmable Gate Array); and a dedicated electrical circuit, which is a processor with a circuit configuration specifically designed to perform a particular process, such as an ASIC (Application Specific Integrated Circuit).

[0223] A single processing unit may be composed of one of these various processors, or it may be composed of two or more processors of the same or different type (for example, multiple FPGAs, or a combination of a CPU and an FPGA). Alternatively, multiple processing units may be composed of a single processor. Examples of composing multiple processing units with a single processor include, firstly, a configuration where one or more CPUs and software are combined to form a single processor, and this processor functions as multiple processing units, as is typical of computers such as clients and servers. Secondly, a configuration using a processor that realizes the functions of the entire system, including multiple processing units, on a single IC (Integrated Circuit) chip, as is typical of SoCs (System On Chip). Thus, various processing units are configured, in terms of hardware structure, using one or more of the above-mentioned various processors.

[0224] Furthermore, the hardware structure of these various processors is, more specifically, an electrical circuit composed of circuit elements such as semiconductor devices.

[0225] Each of the above-described configurations and functions can be appropriately implemented using any hardware, software, or a combination thereof. For example, the present invention can also be applied to a program that causes a computer to execute the above-described processing steps (processing procedures), a computer-readable recording medium (non-temporary recording medium) that records such a program, or a computer on which such a program can be installed.

[0226] Although examples of the present invention have been described above, it goes without saying that the present invention is not limited to the embodiments described above, and various modifications are possible without departing from the spirit of the invention. [Explanation of symbols]

[0227] 10 Information Processing Devices 11 CPU 12 ROM 13 RAM 14 Auxiliary storage 15 Input device 16 Output device 17 Input / Output Interfaces 18 Communication Interfaces 21 Information Acquisition Department 22 Feature Point Extraction Processing Unit 23 Corresponding Point Search Processing Unit 24 Alignment Processing Unit 24A Coarse alignment processing unit 24B Detailed alignment processing unit 25 Difference Calculation Processing Unit 25A Pairing Processing Unit 25B 1st difference calculation section 25C 2nd difference calculation section 26 Calculation Result Output Processing Unit 27 Calculation Result Recording Processing Unit 30 Integrated Processing Unit 40. Processing unit for extracting areas where differences can be calculated. 50 Synthesis Processing Unit A frame indicating the shooting range for a single shot when shooting AV in split-screen mode. An enlarged image of a portion of the image displayed as the EI calculation result. F1 A frame indicating the shooting range of the image taken of the current bridge piers. F2 A frame indicating the shooting range of images taken during past inspections of the bridge pier. F21 A frame indicating the shooting range of the first image taken during a past inspection of the bridge pier. F22 A frame indicating the shooting range of the second image taken during a past inspection of the bridge pier. GO Square I1 Current Image I2 Past Images IC1 Current crack information IC2 Past crack information IC3 information on the difference in crack length. IC4 overall difference information IC11 Current Damage Information IC21 Past damage information IC22 Past Second Damage Information IC31 First Difference Information IC31A First difference information after extracting the overlapping region IC32 Second Difference Information IC32A Second difference information after extracting the overlapping region IR1 Information on the current damaged area (area where free calcium is generated) IR2 Information on past damaged areas (areas where free calcium was generated) Information on the calculation results of the difference in IR3 damaged areas. L is a straight line connecting corresponding feature points. Image showing the result of calculating the difference in crack length (OI). OI1 Image showing the result of calculating the difference in crack length. Image showing the result of calculating the difference in crack length for OI2. Image showing the result of calculating the difference in crack length for OI3. Image showing the result of calculating the difference in crack length for OI4. Image showing the result of calculating the difference in crack length for OI5. Image showing the result of calculating the difference in crack length (OI6). P1-1 Current image captured in segmented shots P1-2 Current images taken in split shot P2-1 Past images captured in segments P2-2 Past images captured in segments V1-1 Current crack vector V1-1 Current crack vector V1-2 Current crack vector V1-3 Current crack vector V1-4 Current crack vector V1-5 Current crack vector V1-6 Current crack vector V2-1 Past crack vectors V2-2 Past crack vectors V2-3 Past crack vectors V2-4 Past crack vectors V2-5 Past crack vectors W is a frame indicating the range in which the difference was calculated. Procedure for calculating the difference in damage information (crack information) from S1 to S7 S4A~S4D Alignment Processing Procedure Procedure for calculating the difference between S5A and S5C S11-S18 Procedure for calculating the difference in damage information S21-S28 Procedure for calculating the difference in damage information S31-S39 Procedure for calculating the difference in damage information

Claims

1. An information processing device equipped with a processor, The aforementioned processor, A first image of the structure and first damage information of the structure extracted from the first image are obtained. A second image of the structure taken at a different time than the first image and second damage information of the structure extracted from the second image are obtained. Extract multiple feature points from the above image, Multiple feature points are extracted from the two images mentioned above. Search for corresponding feature points between the first image and the second image, Based on the information of the corresponding feature points, the first damage information and the second damage information are roughly aligned, and then detailed alignment is performed to align the first damage information and the second damage information. The difference between the first damage information and the second damage information after alignment is calculated. Information processing device.

2. The aforementioned processor, As a rough alignment, rigid body alignment is performed. As part of the detailed alignment, non-rigid alignment is performed. The information processing apparatus according to claim 1.

3. The aforementioned processor, As a rough alignment, rigid body alignment is performed. As part of the detailed alignment process, the alignment is performed by correcting for differences in lens distortion. The information processing apparatus according to claim 1.

4. The first damage information and the second damage information are crack information, The processor calculates the difference in the width and / or length of the crack. The information processing apparatus according to any one of claims 1 to 3.

5. The processor generates corresponding pairs of cracks and calculates the difference in crack widths by calculating the difference in widths between the pairs. The information processing apparatus according to claim 4.

6. The processor generates corresponding pairs of cracks and calculates the difference in crack lengths by calculating the length of cracks for which no pairs have been generated. The information processing apparatus according to claim 4.

7. The processor generates pairs of cracks from adjacent cracks. The information processing apparatus according to claim 5 or 6.

8. The processor generates corresponding pairs of cracks by DP matching. The information processing apparatus according to claim 5 or 6.

9. The first damage information and the second damage information are information about the damaged area. The processor calculates the difference between the corresponding damaged regions. The information processing apparatus according to any one of claims 1 to 3.

10. When a plurality of first images are obtained by dividing the structure and taking photographs of it, and a plurality of first damage information is extracted from the plurality of first images, and a plurality of second images are obtained by dividing the structure and taking photographs of it, and a plurality of second damage information is extracted from the plurality of second images, The aforementioned processor, Multiple feature points are individually extracted from each previous image, Multiple feature points are individually extracted from each of the two aforementioned images. Search for corresponding feature points between the corresponding first image and the second image, Align the corresponding first damage information and the second damage information, The difference between the corresponding first damage information and the second damage information is calculated. The information processing apparatus according to any one of claims 1 to 9.

11. The aforementioned processor, The necessary synthesis processing information for combining multiple first images into a panorama is obtained. The difference calculated between the corresponding first image and second image is combined based on the synthesis processing information. The information processing apparatus according to claim 10.

12. When multiple second images in which a portion of the imaging area overlaps with the first image, and multiple second damage information extracted from the multiple second images are obtained, The aforementioned processor, Multiple feature points are individually extracted from each of the two aforementioned images. The corresponding feature points are individually searched between the first image and each of the second images. The first damage information and each of the second damage information are individually aligned, The difference is calculated individually between the first damage information and each of the second damage information. Furthermore, The differences calculated individually between the first damage information and each of the second damage information are integrated. The information processing apparatus according to any one of claims 1 to 9.

13. The aforementioned processor, A region is extracted between the first image and the second image in which the difference between the first damage information and the second damage information can be calculated. Output information about the extracted region. The information processing apparatus according to any one of claims 1 to 9.

14. The aforementioned processor, When outputting information on the region where the difference can be calculated, an image is generated showing the region where the difference can be calculated on the first image, and this image is output to the display destination. The information processing apparatus according to claim 13.

15. When a plurality of first images are obtained by dividing the structure and taking photographs of it, and a plurality of first damage information is extracted from the plurality of first images, and a plurality of second images are obtained by dividing the structure and taking photographs of it, and a plurality of second damage information is extracted from the plurality of second images, The aforementioned processor, Multiple images are combined into a panorama, Based on the synthesis processing information obtained when multiple first images are combined into a panorama, multiple first damage information is synthesized. Multiple of the above second images are combined into a panorama, Based on the synthesis processing information obtained when multiple second images are combined into a panoramic image, multiple second damage information is synthesized. Multiple feature points are extracted from the first image after panoramic stitching. Multiple feature points are extracted from the second image after panoramic stitching. The first image and the second image after panoramic stitching are searched for corresponding feature points. The first damage information and the second damage information after synthesis are aligned, The difference between the first damage information and the second damage information after synthesis is calculated. The information processing apparatus according to any one of claims 1 to 9.

16. A first image of the structure and first damage information of the structure extracted from the first image are obtained. A second image of the structure taken at a different time than the first image and second damage information of the structure extracted from the second image are obtained. Extract multiple feature points from the above image, Multiple feature points are extracted from the two images mentioned above. Search for corresponding feature points between the first image and the second image, Based on the information of the corresponding feature points, the first damage information and the second damage information are roughly aligned, and then detailed alignment is performed to align the first damage information and the second damage information. The difference between the first damage information and the second damage information after alignment is calculated. Information processing methods.

17. A first image of the structure and first damage information of the structure extracted from the first image are obtained. A second image of the structure taken at a different time than the first image and second damage information of the structure extracted from the second image are obtained. Extract multiple feature points from the above image, Multiple feature points are extracted from the two images mentioned above. Search for corresponding feature points between the first image and the second image, Based on the information of the corresponding feature points, the first damage information and the second damage information are roughly aligned, and then detailed alignment is performed to align the first damage information and the second damage information. The difference between the first damage information and the second damage information after alignment is calculated. To make a computer do it. Information processing program.

Images