Information processing apparatus, information processing method, and computer-readable non-transitory recording medium

Abstract

The present application provides an information processing device, an information processing method and an information processing program capable of accurately calculating the difference of damage information. The information processing device acquires a first image obtained by photographing a structure and first damage information of the structure extracted from the first image, acquires a second image obtained by photographing the structure at a different time point from the first image and second damage information of the structure extracted from the second image, extracts a plurality of feature points from the first image, extracts a plurality of feature points from the second image, searches for corresponding feature points between the first image and the second image, performs coarse alignment on the first damage information and the second damage information based on the information of the corresponding feature points, and then performs detailed alignment, thereby aligning the first damage information and the second damage information, and calculating the difference of the aligned first damage information and the second damage information.

Description

Technical Field

[0001] This invention relates to information processing apparatus, information processing method, and information processing program, and particularly to an information processing apparatus, information processing method, and information processing program for processing damage information of structures extracted from images. Background Technology

[0002] Social infrastructure structures such as bridges, dams, and tunnels are required to undergo periodic inspections every few years. For example, bridges are required to undergo periodic inspections every 5 years. As a technology to support this operation, there are known techniques for automatically extracting damage (cracks, loose lime, etc.) appearing on the surface of the structure from images obtained by photographing the structure (e.g., Patent Documents 1 and 2).

[0003] Previous technical documents

[0004] Patent documents

[0005] Patent Document 1: Japanese Patent Application Publication No. 2020-52786

[0006] Patent Document 2: Japanese Patent Application Publication No. 2020-96325 Summary of the Invention

[0007] The technical problem to be solved by the invention

[0008] Structural managers determine the need for repairs and the appropriate timing based on the results of periodic inspections. At this stage, it's not only necessary to assess the current state of structural deterioration but also to consider information about the differences between past and current inspection results to determine repair-related matters. This is because the existence of information about these differences allows for the prediction of future deterioration. However, due to the difficulty in accurately calculating these differences, repairs are not currently being carried out.

[0009] The present invention was made in view of the following circumstances, and its object is to provide an information processing apparatus, information processing method and information processing program capable of accurately calculating differential damage information.

[0010] means for solving technical problems

[0011] (1) An information processing apparatus comprising a processor, the processor performing the following processing: acquiring a first image of a structure obtained by photographing it and first damage information of the structure extracted from the first image; acquiring a second image of the structure obtained by photographing it at a different time point than the first image and second damage information of the structure extracted from the second image; extracting multiple feature points from the first image; extracting multiple feature points from the second image; searching for corresponding feature points between the first image and the second image; performing coarse alignment of the first damage information and the second damage information based on the information of the corresponding feature points; performing detailed alignment, thereby aligning the first damage information and the second damage information; and calculating the difference between the aligned first damage information and the second damage information.

[0012] (2) The information processing apparatus according to (1), wherein the processor performs the following processing: as coarse alignment, rigid body alignment is performed, and as detailed alignment, non-rigid body alignment is performed.

[0013] (3) The information processing apparatus according to (1), wherein the processor performs the following processing: as coarse alignment, rigid body alignment is performed, and as detailed alignment, alignment is performed by correcting the difference in lens distortion.

[0014] (4) The information processing apparatus according to any one of (1) to (3), wherein the first damage information and the second damage information are crack information, and the processor calculates the difference between the width and / or length of the crack.

[0015] (5) According to the information processing apparatus of (4), wherein the processor generates pairs of corresponding cracks, calculates the difference in width between the pairs, and thereby calculates the difference in width of the cracks.

[0016] (6) According to the information processing device described in (4), wherein the processor generates corresponding crack pairs, calculates the length of cracks that have not generated pairs, and thereby calculates the difference in crack length.

[0017] (7) The information processing apparatus according to (5) or (6), wherein the processor generates a pair of cracks between adjacent cracks.

[0018] (8) The information processing apparatus according to (5) or (6), wherein the processor generates corresponding crack pairs by DP matching.

[0019] (9) The information processing apparatus according to any one of (1) to (3), wherein the first damage information and the second damage information are information about the damage regions, and the processor calculates the difference between the corresponding damage regions.

[0020] (10) The information processing apparatus according to any one of (1) to (9), wherein, when acquiring a plurality of first images obtained by segmenting and photographing a structure and a plurality of first damage information extracted from the plurality of first images, and acquiring a plurality of second images obtained by segmenting and photographing a structure and a plurality of second damage information extracted from the plurality of second images, the processor performs the following processing: extracting a plurality of feature points from each of the first images, extracting a plurality of feature points from each of the second images, searching for corresponding feature points between the corresponding first images and the second images, aligning the corresponding first damage information and the second damage information, and calculating the difference between the corresponding first damage information and the second damage information.

[0021] (11) The information processing apparatus according to (10), wherein the processor performs the following processing: acquiring synthesis processing information required for panoramic synthesis of multiple first images, and synthesizing the difference calculated between the corresponding first image and the second image based on the synthesis processing information.

[0022] (12) The information processing apparatus according to any one of (1) to (9), wherein, when a plurality of second images in which a portion of the shooting area is repeated with the first image and a plurality of second damage information extracted from the plurality of second images are acquired, the processor performs the following processing: extracting a plurality of feature points from each of the second images, searching for corresponding feature points between the first image and each of the second images, aligning the first damage information with each of the second damage information, calculating the difference between the first damage information and each of the second damage information, and then merging the differences calculated between the first damage information and each of the second damage information.

[0023] (North) An information processing apparatus according to any one of (1) to (9), wherein the processor performs the following processing: extracting a region between a first image and a second image from which the difference between the first damage information and the second damage information can be calculated, and outputting information of the extracted region.

[0024] (14) The information processing apparatus according to (13), wherein the processor performs the following processing: when outputting information about the region where the difference can be calculated, generating an image showing the region where the difference can be calculated on the first image, and outputting it to a display destination.

[0025] (15) The information processing apparatus according to any one of (1) to (9), wherein, when acquiring a plurality of first images obtained by segmenting and photographing a structure and a plurality of first damage information extracted from the plurality of first images, and acquiring a plurality of second images obtained by segmenting and photographing a structure and a plurality of second damage information extracted from the plurality of second images, the processor performs the following processing: performing panoramic synthesis on the plurality of first images, synthesizing the plurality of first damage information based on the synthesis processing information when performing panoramic synthesis on the plurality of first images, performing panoramic synthesis on the plurality of second images, synthesizing the plurality of second damage information based on the synthesis processing information when performing panoramic synthesis on the plurality of second images, extracting a plurality of feature points from the panoramic synthesized first image, extracting a plurality of feature points from the panoramic synthesized second image, searching for corresponding feature points between the panoramic synthesized first image and the second image, aligning the synthesized first damage information and the second damage information, and calculating the difference between the synthesized first damage information and the second damage information.

[0026] (16) An information processing method, wherein: a first image of a structure is obtained by taking a picture of the structure and first damage information of the structure is extracted from the first image; a second image of the structure is obtained by taking a picture of the structure at a different time point than the first image and second damage information of the structure is extracted from the second image; multiple feature points are extracted from the first image and multiple feature points are extracted from the second image; corresponding feature points are searched between the first image and the second image; based on the information of the corresponding feature points, the first damage information and the second damage information are coarsely aligned and then finely aligned, thereby aligning the first damage information and the second damage information; and the difference between the aligned first damage information and the second damage information is calculated.

[0027] (17) An information processing program that enables a computer to: acquire a first image of a structure taken by photographing it and first damage information of the structure extracted from the first image; acquire a second image of the structure taken at a different time point than the first image and second damage information of the structure extracted from the second image; extract multiple feature points from the first image; extract multiple feature points from the second image; search for corresponding feature points between the first image and the second image; perform coarse alignment of the first damage information and the second damage information based on the information of the corresponding feature points, and then perform detailed alignment, thereby aligning the first damage information and the second damage information; and calculate the difference between the aligned first damage information and the second damage information.

[0028] Invention Effects

[0029] According to the present invention, the difference in damage information can be accurately calculated. Attached Figure Description

[0030] Figure 1 This is a block diagram illustrating an example of the hardware structure of an information processing device.

[0031] Figure 2 It is a block diagram of the functions of an information processing device.

[0032] Figure 3 This is a diagram representing an example of crack information in vector data.

[0033] Figure 4 It is a graph that clearly represents the search results for the corresponding points.

[0034] Figure 5 This is a block diagram showing the functions of the position processing unit.

[0035] Figure 6 This is a conceptual diagram for calculating the difference in crack length.

[0036] Figure 7 This is a block diagram showing the functions of the differential calculation and processing unit.

[0037] Figure 8 This is a conceptual diagram of the generation of crack pairs.

[0038] Figure 9 This is a diagram showing an example of the calculation results of the difference in crack length.

[0039] Figure 10 This is another example of a graph showing the calculation results of the difference in crack length.

[0040] Figure 11 This is a diagram showing an example of the calculation results for the difference in crack width.

[0041] Figure 12 This is a flowchart representing the sequence of differential calculations for damage information (crack information).

[0042] Figure 13 This is a flowchart showing the order of bit processing.

[0043] Figure 14 This is a flowchart showing the sequence of difference calculations.

[0044] Figure 15 This is a conceptual diagram of the difference calculation when the damage information is composed of information from the region.

[0045] Figure 16 This is a concept image of segmented shooting.

[0046] Figure 17 This is a conceptual diagram of the correction of lens distortion.

[0047] Figure 18 It is a block diagram of the functions of an information processing device.

[0048] Figure 19 It is a diagram that shows the correspondence between present and past images obtained by segmenting and photographing.

[0049] Figure 20 This is a flowchart representing the sequence of differential calculations for damage information.

[0050] Figure 21 This is a concept diagram of the shooting range.

[0051] Figure 22 It is a block diagram of the functions of an information processing device.

[0052] Figure 23 This is a conceptual diagram of the calculation and processing of the difference in crack length.

[0053] Figure 24 This is a conceptual diagram of the calculation and processing of the difference in crack length.

[0054] Figure 25 This is a conceptual diagram of the merge process.

[0055] Figure 26 This is a flowchart representing the sequence of differential calculations for damage information.

[0056] Figure 27 This is a concept diagram illustrating an example where the current range of damage information extraction differs from the range of past damage information extraction.

[0057] Figure 28 It is a block diagram of the functions of an information processing device.

[0058] Figure 29 This is a diagram representing an example of how the output of a difference region can be calculated.

[0059] Figure 30 It is a block diagram of the functions of an information processing device.

[0060] Figure 31 This is a flowchart representing the sequence of differential calculations for damage information. Detailed Implementation

[0061] The preferred embodiments of the present invention will now be described in detail with reference to the accompanying drawings.

[0062] [First Implementation Method]

[0063] Here, we will take the case of extracting cracks from images obtained from photographs of a structure, and calculating the difference between the extracted cracks and past inspection results as an example. The crack information extracted from the image is an example of damage information.

[0064] [Hardware Structure of Information Processing Device]

[0065] Figure 1 This is a block diagram illustrating an example of the hardware structure of an information processing device.

[0066] Information processing device 10 is, for example, composed of a general-purpose computer such as a personal computer. Figure 1 As shown, the information processing device 10 mainly includes a CPU (Central Processing Unit) 11, a ROM (Read Only Memory) 12, a RAM (Random Access Memory) 13, an auxiliary storage device 14, an input device 15, an output device 16, an input / output interface 17, and a communication interface 18.

[0067] CPU 11 is an example of a processor. The computer constituting the information processing device 10 functions as an information processing device by executing a prescribed program (information processing program) by the CPU 11. The program executed by the CPU 11 is stored in ROM 12 or auxiliary storage device 14.

[0068] The auxiliary storage device 14 constitutes the storage unit of the information processing device 10. The auxiliary storage device 14 is, for example, composed of an HDD (HardDisk Drive) or an SSD (Solid State Drive).

[0069] The input device 15 constitutes the operation unit of the information processing device 10. The input device 15 may be, for example, a keyboard, a mouse, a touch panel, etc.

[0070] The output device 16 constitutes the display unit of the information processing device 10. The output device 16 may be, for example, a liquid crystal display (LCD), an organic EL display (OLED), or the like.

[0071] The input / output interface 17 forms the connection part of the information processing device 10. The information processing device 10 is connected to external devices via the input / output interface 17.

[0072] The communication interface 18 constitutes the communication unit of the information processing device 10. The information processing device 10 is connected to a network (e.g., the Internet) via the communication interface 18.

[0073] [Functions of an information processing device]

[0074] Figure 2 It is a block diagram of the functions of an information processing device.

[0075] Regarding the differential calculation of damage information, the information processing device 10 has the functions of an information acquisition unit 21, a feature point extraction processing unit 22, a corresponding point search processing unit 23, an alignment processing unit 24, a differential calculation processing unit 25, a calculation result output processing unit 26, and a calculation result recording processing unit 27. These functions are implemented by the CPU 11 executing a prescribed program (information processing program).

[0076] Furthermore, in this embodiment, the crack extraction process is assumed to be completed beforehand. The extracted crack information (damage information) is stored in the auxiliary storage device 14 in association with the information of the image from the extraction source. The crack information is recorded, for example, as vector data.

[0077] Figure 3 This is a diagram representing an example of crack information in vector data.

[0078] Vector digitization refers to the process of determining the line segment of a crack, defined by its start and end points. The resulting line segment becomes the vector representing the crack (crack vector). The information about the crack in vector digitization includes at least the coordinates of the crack vector's start and end points. In this embodiment, it also includes information about the crack's length and width. The coordinates are, for example, set as the origin from one of the feature points extracted from the image.

[0079] In addition, Figure 3 In the table shown, the group ID (Identification) is information that identifies the group of cracks. A crack group is a sequence of crack vectors that are considered as a single crack. The vector ID is information that identifies each crack vector.

[0080] In addition to the information on the crack, various other information can be added. Furthermore, for branched cracks, the crack information can be recorded in layers (for example, see Japanese Patent Application Publication No. 2019-200213).

[0081] Crack extraction and processing utilizes well-known methods, such as learned models from machine learning.

[0082] In addition to image data, image information includes the date and time of acquisition, image width, image height, and resolution (pixel resolution). The date and time of acquisition refers to the date and time when the image information was obtained. The image width and height information are recorded as the number of pixels in the width and height directions, respectively. The unit of resolution is mm / pixel.

[0083] The information acquisition unit 21 acquires information about the crack (damage information) that is the object of processing, and information about the image from which the information about the crack is extracted. As described above, in this embodiment, the difference between the current and past crack information is calculated. Therefore, the acquired crack information is information about both the current and past cracks, and the acquired image information is information about both the current and past images. The current crack information is an example of first damage information. On the other hand, the past crack information is an example of second damage information. Furthermore, the image from which the current crack information is extracted (the current image) is an example of a first image. On the other hand, the image from which the past crack information is extracted (the past image) is an example of a second image. The current and past images are images obtained by taking pictures of the structure of the object of inspection at different time points.

[0084] Information about current cracks and images (current images) containing that information are stored together in the auxiliary storage device 14. Similarly, information about past cracks and images (past images) containing that information are stored together in the auxiliary storage device 14. The information acquisition unit 21 reads and acquires information about current cracks and images (current images) associated with that information from the auxiliary storage device 14. The information acquisition unit 21 also reads and acquires information about past cracks and images (past images) associated with that information from the auxiliary storage device 14.

[0085] Furthermore, both the current and past images were taken from the same surface of the same structure within roughly the same area. Therefore, the crack information extracted from each image represents crack information within roughly the same area of ​​the same structure.

[0086] The feature point extraction processing unit 22 extracts feature points from the current image and the past image acquired by the information acquisition unit 21, respectively. Feature point extraction is performed using known methods such as AKAZE (Accelerated KAZE) and ORB (Oriented Fast and Rotated BRIEF). Multiple feature points are extracted from each image.

[0087] The corresponding point search processing unit 23 performs a process (so-called matching process) that searches for corresponding feature points (corresponding points) between the current image and the past image where feature points have been extracted. This process is performed using known methods such as the Brute-Force algorithm or the Fast Library for Approximate Nearest Neighbors (FLANN).

[0088] Figure 4 It is a graph that clearly represents the search results for the corresponding points. Figure 4 Feature points in the current image I1 and the past image I2 are represented by connecting them with straight lines. For example, in Figure 4 In the image, feature point CP1 extracted from the current image I1 corresponds to feature point CP2 extracted from the past image I2, and the two are connected by a straight line L.

[0089] The alignment processing unit 24 aligns the current crack information with the past crack information based on the information of the feature points corresponding between the current image and the past image.

[0090] Figure 5 This is a block diagram showing the functions of the bit processing unit 24.

[0091] like Figure 5 As shown, the alignment processing unit 24 has the functions of a coarse alignment processing unit 24A and a detailed alignment processing unit 24B.

[0092] The coarse alignment processing unit 24A performs coarse alignment between the current crack information and the past crack information as a preliminary processing step. That is, it performs approximate alignment. The coarse alignment processing unit 24A performs coarse alignment between the current crack information and the past crack information based on the information of corresponding points (information of feature points corresponding to the current image and the past image) searched by the corresponding point search processing unit 23. In this embodiment, coarse alignment is performed using known rigid body alignment processing. For example, affine transformation or projection transformation can be used for rigid body alignment. The coarse alignment processing unit 24A constructs a rigid body deformation model based on the information of corresponding points searched by the corresponding point search processing unit 23. That is, it constructs a rigid body deformation model that reduces the offset of the corresponding points. The coarse alignment processing unit 24A applies the constructed rigid body deformation model to the current crack information or the past crack information, performing coarse alignment (rigid body alignment) between the current crack information and the past crack information. In this embodiment, coarse alignment is performed by applying it to the past crack information. Therefore, when applying information about past cracks, the rigid body deformation model is constructed to align with information about current cracks.

[0093] The detailed alignment processing unit 24B, as a subsequent processing step, performs detailed alignment between the current crack information after coarse alignment processing and the past crack information. That is, it performs detailed alignment. Based on the corresponding point information after coarse alignment processing, the detailed alignment processing unit 24B performs detailed alignment between the current crack information after coarse alignment processing and the past crack information. In this embodiment, detailed alignment processing is performed using a known non-rigid body alignment process. Examples of non-rigid body alignment processes include those utilizing the TPS (Thin-Plate Spline) method, the CPD (Coherent Point Drift) method, and the FFD (Free-Form Deformation) method. Based on the corresponding point information after coarse alignment processing, the detailed alignment processing unit 24B constructs a non-rigid body deformation model. That is, it constructs a non-rigid body deformation model that reduces the offset of the corresponding points. The detailed alignment processing unit 24B applies the constructed non-rigid deformation model to the current crack information or past crack information after coarse alignment processing, and performs detailed alignment (non-rigid alignment) between the current crack information after coarse alignment processing and the past crack information. In this embodiment, detailed alignment is performed on the past crack information after coarse alignment processing. Therefore, when the non-rigid deformation model is applied to the past crack information after coarse alignment processing, it constructs a model that aligns with the current crack information after coarse alignment processing.

[0094] As described above, after coarse alignment, the alignment processing unit 24 performs detailed alignment, aligning the current crack information with the past crack information. This allows for accurate alignment of the current crack information with the past crack information. In other words, while rigid body alignment alone may result in positional shifts, subsequent non-rigid body alignment corrects for these misalignments, achieving high-precision alignment. Furthermore, non-rigid body alignment alone may subject to unreasonable deformation, but prior rigid body alignment suppresses this deformation, enabling high-precision alignment as a whole.

[0095] The differential calculation and processing unit 25 calculates the difference between the current crack information after alignment processing and the past crack information.

[0096] Figure 6 This is a conceptual diagram for calculating the difference in crack length.

[0097] Figure 6 (A) represents the information IC1 of the current crack extracted from the current image. Figure 6 (B) represents the information about past cracks extracted from past images, IC2. Figure 6(C) represents the information IC3 of the difference calculation result.

[0098] like Figure 6 As shown, the difference between the current crack information IC1 and the past crack information IC2 is calculated as the crack length difference information IC3. That is, cracks that exist in one crack but not in the other crack are extracted, and the length of the extracted cracks is calculated as the difference. More specifically, the crack length difference is calculated in the following order.

[0099] Figure 7 This is a block diagram of the functions of the differential calculation and processing unit 25.

[0100] like Figure 7 As shown, the difference calculation processing unit 25 has the functions of a pairing processing unit 25A, a first difference calculation unit 25B, and a second difference calculation unit 25C.

[0101] The pairing processing unit 25A acquires information about current and past cracks after alignment processing and generates corresponding crack pairs. As described above, the crack information is digitized into vector data and recorded as vector information. The pairing processing unit 25A generates pairs (pairing) between adjacent crack vectors.

[0102] Figure 8 This is a conceptual diagram of the generation of crack pairs.

[0103] Figure 8 (A) is a top-view diagram showing a portion of the information about the current crack. Figure 8 (B) is a top-down view showing part of the information about past cracks. Additionally, Figure 8 (C) is to Figure 8 (A) and Figure 8 (B) The overlapping diagram.

[0104] While alignment processing can bring the positions of corresponding cracks together, it is difficult to ensure that all cracks are perfectly aligned. Furthermore, the crack extraction results themselves can sometimes be inaccurate. Therefore, in this embodiment, pairs of corresponding cracks are generated, these pairs are compared, and the differences are calculated.

[0105] When the information of the current crack coincides with the information of the past crack, the crack vectors that are close to each other are combined to generate pairs. That is, in the case of coincidence, pairs are generated by the crack vectors located closest to each other.

[0106] For example, in Figure 8 In the example shown, if we focus on the crack vector V1-1 in the current crack information, then as follows Figure 8As shown in (C), the crack vector located at the nearest past position is crack vector V2-1. Therefore, a pair is generated from crack vector V1-1 and crack vector V2-1. Figure 8 In the example shown, a pair is further generated from the current crack vector V1-2 and the past crack vector V2-2. Additionally, a pair is generated from the current crack vector V1-3 and the past crack vector V2-3. Furthermore, a pair is generated from the current crack vector V1-4 and the past crack vector V2-4. Finally, a pair is generated from the current crack vector V1-5 and the past crack vector V2-5. Figure 8 In (C), the arrows indicate the correspondence between the current crack vector and the past crack vector.

[0107] On the other hand, Figure 8 In the example shown, there is no past crack vector corresponding to the current crack vector V1-6. ​​In this case, the current crack vector V1-6 becomes an unpaired crack vector. An unpaired crack vector is equivalent to a newly generated crack (a growing crack).

[0108] The first difference calculation unit 25B calculates the difference in crack length as a difference in crack information. The first difference calculation unit 25B calculates the difference in crack length based on information about current and past cracks that have generated crack vector pairs. Specifically, it calculates the length of crack vectors for which no pairs have been generated, and calculates the difference in crack length. Furthermore, as crack information, if the length of the crack vectors is available, it is assumed that the length has been calculated, and this length information is acquired. Figure 8 In the example shown, the crack vector that was not generated is the current crack vector V1-6. ​​Therefore, the length of the current crack vector V1-6 is calculated, and the difference in crack length is calculated.

[0109] The second difference calculation unit 25C calculates the difference in crack width as a difference in crack information. The second difference calculation unit 25C calculates the difference in crack width based on information about the present and past cracks that have generated crack vector pairs. Specifically, it calculates the difference in width based on the crack vector pairs. For example, in... Figure 8 In the example shown, the difference in width between the current crack vector V1-1 and the paired past crack vector V2-1 is calculated as "(width of current crack vector V1-1) - (width of past crack vector V2-1)". Let's assume the width of the current crack vector V1-1 is 0.1 mm and the width of the paired past crack vector V2-1 is 0.05 mm. In this case, 0.1 mm - 0.05 mm = 0.05 mm is calculated as the difference in width between the pairs.

[0110] The calculation result output processing unit 26 outputs the difference calculation result to the output device 16 in a prescribed form. In the information processing apparatus 10 of this embodiment, the output device 16 is a display. Therefore, the calculation result output processing unit 26 displays the difference calculation result on the display, which is the display destination.

[0111] Figure 9 This is a diagram showing an example of the calculation results of the difference in crack length.

[0112] like Figure 9 As shown, the calculation results of the difference in crack length are displayed, showing images of the crack extracted from both the present and past images, represented by lines of different colors or types. Figure 9 In the example shown, solid red lines represent cracks extracted only from the current image, solid green lines represent cracks extracted only from past images, and dashed (black) lines represent common cracks extracted from both images. Thus, the calculation result output processing unit 26 generates an image displaying cracks differentiated by lines of different colors or types, and outputs it to the output device 16.

[0113] Furthermore, "cracks extracted only from current images" refers to cracks that did not exist in the past. Therefore, this crack is considered to be a growing (extending) or newly formed crack.

[0114] On the other hand, "cracks extracted only from past images" refers to cracks that existed in the past but do not exist now. It is considered unnatural to output such cracks as the result of difference calculations unless repairs are performed. Therefore, it is preferable to set a structure that emphasizes cracks extracted only from past images to draw attention. For example, it could be set to use a visually more striking color or line type to draw attention. Alternatively, it could be set to use a warning display to draw attention.

[0115] Furthermore, the reason for calculating unnatural differences (differences that are opposite to the irreversible changes in damage) is believed to be imperfect imaging and imperfect damage extraction and processing.

[0116] If needed, it can also be configured to overlay the image of the difference calculation result with the current or past image of the extracted crack.

[0117] Figure 10 This is another example of a graph showing the calculation results of the difference in crack length. Figure 10 This is an example of switching the display based on user instructions.

[0118] Figure 10(A) represents an example of displaying image OI1, which uses lines of different colors or different types of lines to represent cracks extracted from both present and past images. This image OI1 and... Figure 9 The images shown are the same.

[0119] Figure 10 Image (B) represents an example of an image OI2 showing the crack extracted from the current image, represented by lines. This image OI2 is equivalent to a current damage map depicting the current crack based on the current image.

[0120] Figure 10 (C) represents an example of an image OI3 showing cracks extracted from past images, represented by lines. This image OI3 is equivalent to a past damage map depicting past cracks based on past images.

[0121] Figure 10 (D) represents an example of the case where the image OI4 is shown using lines to represent only the differences.

[0122] Figure 10 (E) represents an example of an image OI5 where lines represent cracks extracted only from the present image in the difference. That is, it represents an example of an image where lines represent only cracks that did not exist in the past.

[0123] Figure 10 (F) represents an example of an image OI6 where lines represent cracks extracted only from past images in the difference. That is, it represents an example of an image where lines represent only cracks that are not currently present.

[0124] The calculation result output processing unit 26 switches these images according to instructions from the user and outputs them to the output device 16. If necessary, it can also be configured to overlay these images with current or past images of the extracted cracks.

[0125] Figure 11 This is a diagram showing an example of the calculation results representing the difference in crack width. Figure 11 The image OI, representing the result of the calculation, and a portion of it, magnified, are shown in the image EI.

[0126] exist Figure 11 The example shown illustrates not only the calculation results of the difference in crack length but also the calculation results of the difference in crack width.

[0127] like Figure 11As shown, the cracks extracted from both the present and past images are represented by lines of different colors or types, as the result of the length difference calculation. Furthermore, the result of the width difference calculation is displayed near the lines representing each crack (crack vector). Figure 11 In the example shown, the results of the difference calculation are displayed together with the vector IDs of each crack (crack vector).

[0128] Alternatively, it can be configured to overlay the results of the difference calculation with the extracted current or past images of the crack.

[0129] Furthermore, regarding width differences, it is preferable to emphasize unnatural differences, such as areas where the current width is smaller than the past width, to attract user attention. For example, consider adding markers or changing the display color for a more prominent display.

[0130] The calculation result recording and processing unit 27 records the calculation result information of the difference in the auxiliary storage device 14. The calculation result recording and processing unit 27 records the calculation result information in association with the current and past crack information of the calculated difference in the auxiliary storage device 14.

[0131] [The order of difference calculation and processing (information processing method)]

[0132] Figure 12 This is a flowchart illustrating the sequence of differential calculations of damage information (crack information) by the information processing apparatus of this embodiment.

[0133] First, processing is performed to acquire information about the crack being processed and to extract information about the image containing that crack information (step S1). In this embodiment, the difference between the current and past crack information is calculated. Therefore, information about the current and past cracks, as well as information about the current and past images containing that current and past crack information, are acquired. In this embodiment, this information is read from and acquired from the auxiliary storage device 14.

[0134] Next, feature points are extracted from the acquired current and past images (step S2). Multiple feature points are extracted from each image.

[0135] Next, the process of searching for corresponding feature points (corresponding points) between the current and past images is performed (step S3).

[0136] Next, based on the information of the corresponding feature points, the current crack information is aligned with the past crack information (step S4).

[0137] Figure 13 This is a flowchart showing the order of bit processing.

[0138] First, based on the information of the corresponding feature points, a rigid body deformation model is constructed (step S4A).

[0139] Next, using the constructed rigid body deformation model, coarse alignment (rigid body alignment) is performed between the current crack information and the past crack information (step S4B). In this embodiment, the constructed rigid body deformation model is applied to the past crack information for coarse alignment.

[0140] Next, based on the information of the corresponding points after coarse alignment processing, a non-rigid deformation model is constructed (step S4C).

[0141] Next, using the constructed non-rigid deformation model, the information of the current crack after coarse alignment processing is matched in detail with the information of the past crack (non-rigid alignment) (step S4D). In this embodiment, the constructed non-rigid deformation model is applied to the information of the past crack after coarse alignment processing for detailed alignment.

[0142] Based on the above, the alignment process is complete.

[0143] Next, as Figure 12 As shown, the difference between the current crack information and the past crack information after the alignment process is calculated (step S5).

[0144] Figure 14 This is a flowchart showing the sequence of difference calculations.

[0145] First, a process is performed to generate corresponding crack pairs between the current crack information after alignment processing and the past crack information (step S5A). In this embodiment, pairs are generated from close crack vectors.

[0146] Next, based on the information of the present and past cracks that have generated crack vector pairs, the difference in crack length is calculated (step S5B). Specifically, the lengths of crack vectors that have not generated pairs are calculated, and the difference in crack length is calculated.

[0147] Furthermore, based on the information of the present and past cracks that have generated crack vector pairs, the difference in crack width is calculated (step S5C). Specifically, the difference in width is calculated based on the crack vector pairs.

[0148] Based on the above, the differential calculation is completed. When the differential calculation is complete, the calculation result is output to the output device 16 (step S6). In addition, the calculation result is recorded in the auxiliary storage device 14 (step S7). The calculation result is recorded in association with the information of the differentially calculated crack.

[0149] As described above, the information processing apparatus according to this embodiment can accurately calculate the difference for each crack by performing alignment. In particular, as an alignment process, after coarse alignment, detailed alignment is performed, thereby enabling high-precision alignment. As a result, subsequent difference calculation processing can be performed with high precision.

[0150] [Calculation method for differences in other types of damage]

[0151] When damage information is vectorized and recorded as vector information, its difference can be calculated using the information processing apparatus described in the above embodiments. For example, choke information can have its difference calculated using the information processing apparatus described in the above embodiments.

[0152] In addition to cracks, other types of damage that appear on the surface of a structure include: peeling, exposed rebar, water leakage (including rust), loose lime, and corrosion. This information is recorded as area information to document the damage.

[0153] When the damage information is composed of region information (information about the damaged region), the differential calculation processing unit 25 calculates the difference in the damage information as follows.

[0154] Figure 15 This is a conceptual diagram of the difference calculation when the damage information is composed of regional information.

[0155] Figure 15 This example illustrates a situation where a region with free limestone is extracted from an image of a structure as a damaged area.

[0156] Figure 15 (A) shows information IR1 of the current free lime occurrence area (damage area) extracted from the current image. Additionally, Figure 15 (B) shows information IR2 of the past free lime occurrence area (damage area) extracted from a past image. The information of the free lime occurrence area extracted from the current image is an example of first damage information. In addition, the information of the free lime occurrence area extracted from the past image is an example of second damage information. Figure 15 (C) shows the information IR3 of the difference calculation results.

[0157] As shown in Figure 25, when the damage information is composed of regional information, the differential calculation processing unit 25 calculates (extracts) the differential region corresponding to the damage region. Additionally, it calculates the area of ​​this differential region. Figure 15 In the example shown in (C), the region calculated as a difference is indicated by blacking out.

[0158] When outputting the calculation results of the difference, it is preferable to display them by color and / or line type, similar to the case of cracks. Furthermore, it is preferable to emphasize areas where the output shows unnatural differences that contradict irreversible changes in damage, such as areas where the current damage is smaller than in the past, to draw the user's attention.

[0159] [Variations on pairing processing]

[0160] As described in the above embodiments, when the damage information is vector-based crack information, the difference can be calculated simply and accurately by generating pairs of corresponding cracks and calculating the difference. In the above embodiments, a structure for generating pairs of adjacent cracks is assumed, but the method for generating pairs of corresponding cracks is not limited to this. For example, a matching method based on dynamic programming (DP matching) can be used to generate pairs of corresponding cracks. DP matching refers to a graphical matching method for one-dimensional data, a matching technique used in signal processing. This technique is applied to vector matching processing, i.e., crack vector matching processing, to generate pairs of corresponding cracks.

[0161] [A variation of the information acquisition department]

[0162] In the above embodiment, a structure is configured to obtain current and past damage information from the auxiliary storage device 14. However, it is also possible to configure a structure in which a damage extraction function is installed in the information processing device 10 to directly obtain the extraction result. In this case, it is also possible to configure a structure that only performs extraction processing on current damage and reads and obtains past damage information from the auxiliary storage device 14.

[0163] Alternatively, the structure can be configured to acquire damage information and extract an image of the damage information from an external device connected via input / output interface 17. Alternatively, the structure can be configured to acquire the information from an external device connected via a network.

[0164] [Detailed Examples of Modifications to the Alignment Processing Unit]

[0165] In the above embodiment, a structure for non-rigid alignment processing was provided as a detailed alignment process, but the detailed alignment process is not limited to this. For example, lens distortion can also cause positional shifts. Therefore, when there is a difference in lens distortion between the current and past images, detailed alignment is performed by correcting the difference in lens distortion. As a result, positional shifts caused by lens distortion can be eliminated, and higher accuracy alignment can be achieved. The method for detailed alignment by correcting lens distortion will be described below.

[0166] The difference in lens distortion is corrected using the following method: One image is fixed, while the variables of lens distortion and projection transformation matrix of the other image are changed to minimize the offset of corresponding points, thus solving the optimization problem.

[0167] A camera device can be represented by a model that projects three-dimensional coordinates onto an image plane.

[0168] [Mathematical Expression 1]

[0169]

[0170] x'=x / z

[0171] y'=y / z

[0172] u = f x *x'+c x

[0173] v = f y *y'+c y

[0174] Here, (X, Y, Z) represent the coordinates of a point in the three-dimensional coordinate system, and (u, v) represent the coordinates of a point projected onto the image plane. (c x c y ) is the principal point (usually the image center). f x f y Focal length is expressed in pixels.

[0175] Since the lenses of actual camera devices have distortion in both the radial and circumferential directions, the above model is extended as follows.

[0176] [Mathematical Expression 2]

[0177]

[0178] x'=x / z

[0179] y'=y / z

[0180] x″=x'(1+k1r 2 +k2r 4 +k3r 6 )+2p1x'y'+p2(r 2 +2x' 2 )

[0181] y″=y'(1+k1r 2 +k2r 4 +k3r 6 )+p1(r 2 +2y' 2 )+2p2x'y'

[0182] where r 2 =x' 2 +y' 2

[0183] u = f X * x "+c x

[0184] v = f y *y″+c y

[0185] Here, k1, k2, and k3 are the distortion coefficients in the radial direction of the lens. p1 and p2 are the distortion coefficients in the circumferential direction of the lens.

[0186] Therefore, lens distortion is expressed by the following formula, with k1, k2, k3, p1, and p2 as variables.

[0187] [Mathematical Expression 3]

[0188] x″=x'(1+k1r 2 +k2r 4 +k3r 6 )+2p1x'y'+p2(r 2 +2x' 2 )

[0189] y″=y'(1+k1r 2 +k2r 4 +k3r 6 )+p1(r 2 +2y' 2 )+2p2x'y'

[0190] where r 2 =x' 2 +y' 2

[0191] u = f x * x "+c X

[0192] v = f y *y″+c y

[0193] The projection transformation matrix is ​​represented by the following formula.

[0194] [Mathematical Expression 4]

[0195]

[0196] The lens distortion difference is corrected in a way that minimizes the offset of the corresponding point, thereby optimizing the variables of the lens distortion and the projection transformation matrix.

[0197] Figure 17 This is a conceptual diagram of the correction of lens distortion.

[0198] like Figure 17 As shown, an optimization problem is solved by keeping one image unchanged and changing the variables of lens distortion and projection transformation matrix of another image to minimize the offset of the corresponding points.

[0199] Figure 17 This example illustrates an optimization problem where the current image is fixed while the variables of lens distortion and projection transformation matrix of the past image are changed to minimize the offset of the corresponding points.

[0200] In this way, even when there are differences in lens distortion between two images, higher-precision alignment can be achieved by correcting the difference in lens distortion. Furthermore, this allows for the accurate calculation of the difference in damage information.

[0201] [Second Implementation]

[0202] In the automatic damage extraction from images, high-resolution images are required for high-precision detection of even minute damage. To obtain high-resolution images, the inspection site is sometimes segmented into multiple regions for imaging. When the inspection site is segmented for imaging, damage is extracted separately from each image, and the results are merged to obtain the overall extraction result. The merging process uses a panoramic synthesis method.

[0203] In this embodiment, a method for calculating the difference in damage information when the inspection area is segmented for imaging in the present and the past will be explained.

[0204] [Segmented shooting]

[0205] First, let's explain the split-shooting method. Figure 16 This is a concept image of segmented shooting. Figure 16 This refers to an example of inspecting (photographing) the bridge deck.

[0206] Generally, bridge deck inspections are conducted in grid units. A grid (G0) is an area on the bridge deck divided by main beams and crossbeams. In cases where a single shot cannot capture the entirety of a grid G0, or where even if shooting is possible, a high-resolution image cannot be obtained, the shooting area is divided and multiple shots are taken. Figure 16 In the image, the box indicated by the symbol AV represents the area captured in a single shot. For example... Figure 16As shown, the images were taken with adjacent areas partially overlapping each other. This overlapping method was used to achieve high-precision panoramic stitching.

[0207] [Functions of an information processing device]

[0208] In cases where inspection areas are segmented and photographed in the past and present, the information processing apparatus of this embodiment calculates the difference between each image obtained from the segmented photographs, merges the calculation results of the difference between each image, and generates overall difference information.

[0209] Figure 18 This is a block diagram illustrating the functions of the information processing apparatus of this embodiment.

[0210] Furthermore, the damage extraction and panoramic synthesis processes are assumed to be completed. As described above, damage extraction is performed on each image obtained from segmented photography. The damage information (information on cracks, free lime, etc.) extracted from each image is recorded in the auxiliary storage device 14 in association with the information of the source image. In addition, the information of each image, the damage information extracted from each image, the information of the panoramic synthesis image, and the damage information merged through panoramic synthesis are stored in the auxiliary storage device 14 in association. That is, they are stored in groups. The information of the panoramic synthesis image includes information used when performing panoramic synthesis on each image, that is, information on the synthesis parameters required for panoramic synthesis processing on each image (synthesis processing information). The synthesis parameter information includes information on parameters used for image deformation, such as affine transformation parameters. Furthermore, since this panoramic synthesis process itself is a known technique, its detailed description is omitted.

[0211] like Figure 18 As shown, the information processing apparatus of this embodiment differs from the information processing apparatus of the first embodiment in that it also has the function of a merging processing unit 30.

[0212] The information acquisition unit 21 sequentially acquires information from each current and past image obtained through segmentation, as well as current and past damage information extracted from each current and past image. Additionally, it acquires information on the synthesis parameters required for panoramic synthesis processing of each image. This information is read from the auxiliary storage device 14.

[0213] The feature point extraction processing unit 22 extracts feature points from the current and past images acquired by the information acquisition unit 21.

[0214] The corresponding point search processing unit 23 searches for corresponding feature points (corresponding points) between the corresponding current and past images.

[0215] Figure 19It is a diagram that shows the correspondence between present and past images obtained by segmenting and photographing.

[0216] Figure 19 (A) represents the current result of split-shooting. Figure 19 (B) indicates the result of past segmented shooting. Figure 19 (A) and Figure 19 (B) represents an example of a situation where the object is divided into eight parts for shooting.

[0217] In this case, image P1-1, which is now captured in the upper left corner region, and image P2-1, which was captured in the past in the same region, become corresponding images. Similarly, image P1-2, which is now captured in the upper right corner region, and image P2-2, which was captured in the past in the same region, become corresponding images. Thus, images captured in the same region (which are essentially the same) become corresponding images. The corresponding point search processing unit 23 searches for corresponding points between the corresponding images.

[0218] The alignment processing unit 24 aligns the damage information between corresponding images. At this time, the damage information is aligned based on information about corresponding points extracted between the corresponding images. For example, in... Figure 19 In the example shown, alignment processing is performed between the damage information extracted from the current image P1-1 and the damage information extracted from the past image P2-1 corresponding to the current image P1-1. After coarse alignment processing, detailed alignment processing is performed.

[0219] The difference calculation processing unit 25 calculates the difference in damage information between corresponding images. For example, in Figure 19 In the example shown, the difference is calculated between the damage information extracted from the current image P1-1 and the damage information extracted from the past image P2-1 corresponding to the current image P1-1.

[0220] The merging processing unit 30 merges the difference information of damage information calculated between corresponding images. The merging processing unit 30 merges the difference information using information from the synthesis parameters used during panoramic synthesis. For example, it merges the difference information using information from the synthesis parameters used when performing panoramic synthesis on current images obtained from segmented captures. As a result, it is possible to generate difference information of damage information for the entire region being inspected, corresponding to the panoramic synthesized image.

[0221] The calculation result output processing unit 26 outputs the calculation result of the difference to the output device 16 in a prescribed format. In the information processing apparatus 10 of this embodiment, the output device 16 is configured as a display. Therefore, the calculation result output processing unit 26 displays the calculation result of the difference on the display, which is the display destination.

[0222] The calculation result recording and processing unit 27 records the calculation result information of the merged difference in the auxiliary storage device 14. At this time, the calculation result recording and processing unit 27 records the calculation result of the merged difference in the auxiliary storage device 14 in association with the merged current and past damage information.

[0223] [The order of difference calculations]

[0224] Figure 20 This is a flowchart illustrating the sequence of differential calculations of damage information by the information processing apparatus of this embodiment.

[0225] First, for the current and past damage information to be processed, the information of each image obtained by segmentation and capture, the damage information extracted from each image, and the information of the synthesis parameters are processed (step S11). Next, feature points are extracted from each of the acquired images (step S12). Next, corresponding feature points (corresponding points) are searched between the corresponding current and past images (step S13). Next, alignment processing is performed between the corresponding current and past damage information (step S14). The alignment processing is performed using the information of the corresponding points obtained between the corresponding images. In addition, detailed alignment processing is performed after coarse alignment processing. Next, the difference between the corresponding current and past damage information is calculated (step S15). Next, the calculation results of the difference between the corresponding current and past damage information are merged (step S16). Next, the calculation result of the merged difference is output (step S17). In addition, the calculation result of the merged difference is recorded (step S18).

[0226] As described above, the information processing apparatus according to this embodiment can calculate the difference with high accuracy when merging damage information through panoramic synthesis.

[0227] Furthermore, in the above embodiment, the structure is described as having completed the panoramic synthesis process for the current and past images obtained by segmentation and shooting. However, it can also be described as having a structure in which the information processing device has a panoramic synthesis processing function and performs the panoramic synthesis process within the device.

[0228] [Third Implementation Method]

[0229] In this embodiment, the handling of the case where the extraction range of damage differs between the two damage information pieces used for differential calculation is explained. The case where the extraction range of damage differs between the two damage information pieces used for differential calculation refers to the situation where the image range from which each damage information is extracted is different.

[0230] Figure 21 This is a concept diagram of the shooting range.

[0231] Figure 21 Examples illustrating the condition of inspecting bridge piers. Figure 21 (A) represents the current bridge pier. Figure 21 (B) indicates the bridge pier during past inspections.

[0232] exist Figure 21 In the image, box F1 represents the area captured by the current bridge pier. Box F21 represents the area captured by the first image taken during a previous inspection of the bridge pier, and box F22 represents the area captured by the second image.

[0233] like Figure 21 As shown, the image range of the current bridge pier (shown in box F1) is inconsistent with the image range of the first image taken during a previous inspection of the same bridge pier (shown in box F21). Furthermore, the image range of the current bridge pier is also inconsistent with the image range of the second image taken during a previous inspection (shown in box F22).

[0234] Thus, when the images captured during past inspections do not match the captured area, the information processing device performs difference calculations within the overlapping areas of the images. Furthermore, when overlapping areas exist between multiple images, the information processing device merges the difference information calculated across multiple images to determine the difference compared to past inspection results. For example, in... Figure 21 In the example shown, the information processing device first calculates the difference between the overlapping regions of damage information extracted from the current image and damage information extracted from a previous first image. Similarly, the information processing device calculates the difference between the overlapping regions of damage information extracted from the current image and damage information extracted from a previous second image. Then, the information processing device combines the various calculation results to generate information that differs from the previous inspection results. That is, the information processing device complements the information of insufficient regions to generate information that differs.

[0235] [Functions of an information processing device]

[0236] Figure 22 This is a block diagram illustrating the functions of the information processing apparatus of this embodiment.

[0237] like Figure 22 As shown, the information processing device of this embodiment has the functions of an information acquisition unit 21, a feature point extraction processing unit 22, a corresponding point search processing unit 23, an alignment processing unit 24, a difference calculation processing unit 25, a merging processing unit 30, a calculation result output processing unit 26, and a calculation result recording processing unit 27.

[0238] The information acquisition unit 21 acquires current damage information and past damage information for calculating the difference, as well as information on the current image from which the current damage information is extracted and information on the past image from which the past damage information is extracted. Furthermore, in this embodiment, the damage extraction process is considered to be a completed process. For the past damage information and past image information, information on multiple damage information and past images having regions that overlap with the current damage information is acquired.

[0239] The feature point extraction processing unit 22 extracts feature points from the current and past images acquired by the information acquisition unit 21.

[0240] The corresponding point search processing unit 23 searches for corresponding feature points (corresponding points) between the current image and multiple past images.

[0241] The alignment processing unit 24 performs alignment between the current damage information and multiple past damage information. At this time, the damage information is aligned based on information about corresponding points extracted between each image. For example, if there are two types of past damage information, namely first damage information and second damage information, alignment is performed between the current damage information and the past first damage information, and also between the current damage information and the past second damage information.

[0242] The differential calculation processing unit 25 calculates the difference between the current damage information after alignment processing and the overlapping regions between multiple past damage information.

[0243] Figure 23 and Figure 24 This is a conceptual diagram of the calculation and processing of the difference in crack length when the damage information is crack information.

[0244] Figure 23 and Figure 24 This is an example of a situation where there are two past damage information IC21 and IC22 that have overlapping regions relative to the current damage information IC11. Figure 23 This is a conceptual diagram illustrating the calculation of the difference between the current damage information IC11 and the previous first damage information IC21. Figure 24 This is a conceptual diagram illustrating the calculation of the difference between the current damage information IC11 and the past second damage information IC22.

[0245] like Figure 23 As shown, the current damage information IC11 and the previous first damage information IC21 locally overlap. Differential information IC31 is calculated in the overlapping region. Furthermore, in the differential information IC31, the lines represented by solid lines represent cracks calculated as a differential. In this example, these are lines representing only currently existing cracks. Additionally, the lines represented by dashed lines represent cracks common to both the current and past damage.

[0246] Similarly, as Figure 24 As shown, the current damage information IC11 and the past second damage information IC22 locally overlap. Differential information IC32 is calculated in the overlapping region. Furthermore, in the differential information IC32, the lines represented by solid lines represent cracks calculated as a differential. In this example, these are lines representing only currently existing cracks. Additionally, the lines represented by dashed lines represent cracks common to both the current and past damage.

[0247] The merging processing unit 30 merges the calculation results of the differences calculated between the current damage information and multiple past damage information.

[0248] Figure 25 This is a conceptual diagram of the merge process.

[0249] like Figure 25 As shown, the information IC31, which is the first difference calculated between the current damage information IC11 and the past first damage information IC21, and the information IC32, which is the second difference calculated between the current damage information IC11 and the past second damage information IC22, are merged into the overall difference information IC4.

[0250] exist Figure 25 In the example shown, in the first difference information IC31 and the second difference information IC32, the overlapping regions are cut out in two along the top and bottom, and the cut-out first difference information IC31A and second difference information IC32A are synthesized to generate the overall difference information IC4.

[0251] In addition to this, known methods can also be used in the merging process. For example, synthetic methods such as α-mixing and Laplace fusion can be employed.

[0252] The calculation result output processing unit 26 outputs the calculation result of the difference to the output device 16 in a prescribed format. In the information processing apparatus 10 of this embodiment, the output device 16 is configured as a display. Therefore, the calculation result output processing unit 26 displays the calculation result of the difference on the display, which is the display destination.

[0253] The calculation result recording and processing unit 27 records the information of the merged difference calculation result in conjunction with the current and past damage information in the auxiliary storage device 14.

[0254] [The order of difference calculations]

[0255] Figure 26 This is a flowchart illustrating the sequence of differential calculations of damage information by the information processing apparatus of this embodiment.

[0256] First, the process involves acquiring current damage information and multiple past damage information as the processing objects, as well as extracting information from the images containing the current and past damage information (step S21). Next, the process involves extracting feature points from each acquired image (step S22). Next, the process involves searching for corresponding feature points (corresponding points) between the current image and the multiple past images (step S23). Next, the process involves aligning the current damage information with the multiple past damage information (step S24). This alignment process utilizes the information from the corresponding points obtained between the images, performing detailed alignment after coarse alignment. Next, the process involves calculating differences between the current damage information and the multiple past damage information (step S25). Next, the calculation results of the multiple differences calculated between the current damage information and the multiple past damage information are merged (step S26). Next, the merged difference calculation result is output (step S27). Finally, the merged difference calculation result is recorded (step S28).

[0257] As described above, the information processing apparatus according to this embodiment can suppress the occurrence of areas where the difference between past and present damage information cannot be calculated, even when the extraction range of past damage information is different from the extraction range of present damage information.

[0258] Furthermore, in the examples shown in the above embodiments, the case of repeated past damage information was used as an example, but past damage information does not necessarily have to be repeated. It is sufficient that there are overlapping areas between the past and present damage information.

[0259] Furthermore, the techniques described in this embodiment can also be applied to calculating differences among images obtained through segmentation. For example, for damage information extracted from an image obtained by currently capturing a segmented region, differences can be calculated using damage information extracted from multiple past images.

[0260] [Fourth Implementation Method]

[0261] When the current range of damage information extraction differs from the range of past damage information extraction, the information processing device can calculate the difference only in the areas where the two overlap.

[0262] Figure 27 This is a concept diagram illustrating a situation where the current range of damage information extraction differs from the range of damage information extraction in the past.

[0263] Figure 27 (A) represents the current bridge pier. Figure 27 (B) indicates the bridge pier during past inspections. Figure 27 In the diagram, box F1 represents the photographic area of ​​the image taken of the current bridge pier. Current damage information is extracted within the area of ​​box F1. Conversely, box F2 represents the photographic area of ​​the image taken during past inspections of the same bridge pier. Past damage information is extracted within the area of ​​box F2. Figure 27 As shown, the current range for extracting damage information (the range of box F1) differs from the previous range for extracting damage information (the range of box F2). In this case, as... Figure 27 As shown, the area indicated by the diagonal lines in both current and past damage information becomes a repeated area. Furthermore, the information processing device can calculate the difference only in this repeated area. That is, within the ranges shown in boxes F1 and F2, except for the area indicated by the diagonal lines, the areas where the difference cannot be calculated are all regions. Therefore, when the extraction range of current damage information differs from the extraction range of past damage information, it is preferable to clearly indicate to the user the range where the difference can be calculated in the image from which the damage information has been extracted. This allows for a clear understanding of the areas where the difference can be extracted, improving convenience.

[0264] [Functions of an information processing device]

[0265] Here, we will only explain the function of explicitly indicating the regions on the image where the difference has been calculated.

[0266] Figure 28 This is a block diagram illustrating the functions of the information processing apparatus of this embodiment.

[0267] like Figure 28 As shown, the information processing apparatus of this embodiment, in addition to the functions of the information processing apparatus in the first embodiment, also has the function of calculating the differential region extraction processing unit 40.

[0268] The differential region extraction processing unit 40 extracts regions (regions capable of calculating difference information) based on information from current and past images acquired by the information acquisition unit 21. Regions where damage information can be calculated between current and past images are those that overlap. Therefore, the differential region extraction processing unit 40 extracts overlapping regions between current and past images, thus extracting regions capable of calculating difference information.

[0269] The information about the computeable difference region extracted by the computeable difference region extraction processing unit 40 is added to the calculation result output processing unit 26. The calculation result output processing unit 26 generates an image showing the computeable difference region information on the current or past image and outputs it. In this embodiment, the information about the computeable difference region is output along with the information of the difference calculation result.

[0270] Figure 29 This is a diagram representing an example of how the output of a difference region can be calculated.

[0271] Figure 29 This example shows a scenario where the results of the difference calculation and information about the difference region are overlaid on the current image.

[0272] The result of the difference calculation is the result of the difference calculation of the crack length. Figure 29 In the example shown, the calculated differences are displayed using different colors.

[0273] Information about the regions where the difference can be calculated is displayed by enclosing the calculated difference range with a box W. Additionally, in this example, areas where the difference has not been calculated are indicated by diagonal lines. For areas where the difference has not been calculated, other methods such as making the image semi-transparent can also be used to distinguish them.

[0274] As described above, the information processing apparatus according to this embodiment can clearly determine the region from which the difference can be extracted, even when the current range of damage information extraction differs from the range of past damage information extraction. This significantly improves convenience.

[0275] Furthermore, for structures capable of calculating differential regions, extraction can be based on current and past damage information. In this case, the structure can be configured to extract the differential regions based on current and past damage information after alignment processing. Even when extracting from an image, alignment processing is preferred for extracting the differential regions.

[0276] Furthermore, the extracted information that can be used to calculate the difference region is preferably recorded in association with the information of the difference calculation result.

[0277] [Fifth Implementation Method]

[0278] When the inspection area is segmented for imaging, the information processing device calculates the difference in damage information after panoramic synthesis.

[0279] [Functions of an information processing device]

[0280] Figure 30 It is a block diagram of the functions of an information processing device.

[0281] The information processing apparatus of this embodiment differs from the information processing apparatus of the first embodiment described above in that it has the function of a synthesis processing unit 50.

[0282] The information acquisition unit 21 sequentially acquires information from current and past images obtained by segmentation and capture, as well as damage information extracted from current and past images.

[0283] The compositing unit 50 performs panoramic compositing on the current and past images obtained from segmented shooting. Since panoramic compositing is a well-known process, its detailed description is omitted here.

[0284] The compositing processing unit 50 also uses the compositing parameters from the panoramic compositing process to merge current and past damage information. That is, it generates damage information corresponding to the panoramic compositing image.

[0285] The feature point extraction processing unit 22 extracts feature points from the current and past images after panoramic synthesis.

[0286] The corresponding point search processing unit 23 searches for corresponding feature points (corresponding points) between the current and past images after panoramic synthesis.

[0287] The alignment processing unit 24 aligns the damage information between the current and past images after panoramic synthesis. At this time, the damage information is aligned based on the information of corresponding points extracted between the current and past images after panoramic synthesis. In the alignment processing, coarse alignment processing is performed followed by detailed alignment processing.

[0288] The differential calculation processing unit 25 calculates the difference in damage information between the current and past images after panoramic synthesis.

[0289] The calculation result output processing unit 26 outputs the calculation result of the difference to the output device 16 in a specified form.

[0290] The calculation result recording and processing unit 27 records the calculation result information of the merged difference in the auxiliary storage device 14.

[0291] [The order of difference calculations]

[0292] Figure 31This is a flowchart illustrating the sequence of differential calculations of damage information by the information processing apparatus of this embodiment.

[0293] First, for the current and past damage information to be processed, the information of each image obtained by segmentation and the damage information extracted from each image are processed (step S31). Next, the acquired current and past images are subjected to panoramic synthesis (step S32). As a result, an image of the entire inspection area is generated. Next, the current and past damage information is merged using the synthesis parameters used in panoramic synthesis (step S33). As a result, the overall damage information of the inspection area is generated corresponding to the panoramic synthesized image. Next, feature points are extracted from the panoramic synthesized current and past images respectively (step S34). Next, the corresponding feature points (corresponding points) are searched between the panoramic synthesized current and past images (step S35). Next, the alignment is performed between the merged current and past damage information (step S36). The alignment is performed using the information of the corresponding points obtained between the panoramic synthesized current and past images. In addition, detailed alignment is performed after coarse alignment. Next, the process of calculating the difference between the merged present and past damage information is performed (step S37). Next, the calculation result of the difference is output (step S38). Finally, the calculation result of the difference is recorded (step S39).

[0294] As described above, in the information processing apparatus of this embodiment, when the inspection area is segmented and photographed, the difference in damage information is calculated after panoramic synthesis.

[0295] Furthermore, in the above embodiments, the example described is based on the case where the inspection points are segmented and photographed in both the present and past. However, in the case where only one side is segmented and photographed, only the inspection results captured in the segmented photographs are composited. For example, if only the past inspection results are segmented and photographed, only the past inspection results are composited.

[0296] In addition, in the above embodiments, the information processing device is equipped with the function of synthesis processing, but it can also be configured to perform synthesis processing by other devices.

[0297] [Other Implementation Methods]

[0298] In the above embodiments, the hardware structure of the information processing device for performing various processes is as follows: various processors. Among these processors are general-purpose processors that execute software (programs) and function as various processing units, namely CPUs (Central Processing Units); processors such as FPGAs (Field Programmable Gate Arrays) whose circuit structures can be changed after manufacturing, namely PLDs (Programmable Logic Devices); and processors such as ASICs (Application Specific Integrated Circuits) that have circuit structures specifically designed for performing specific processes, namely dedicated circuits.

[0299] A processing unit can be composed of one of these various processors, or it can be composed of two or more processors of the same or different types (e.g., multiple FPGAs, or a combination of CPU and FPGA). Alternatively, a single processor can be used to construct multiple processing units. As examples of using a single processor to construct multiple processing units, firstly, there are configurations such as client or server computers, where a combination of one or more CPUs and software is used to construct a single processor, which functions as multiple processing units. Secondly, there are configurations such as SoCs (System on Chip), where a single IC (Integrated Circuit) chip implements the overall system functionality including multiple processing units. In this way, one or more of the aforementioned processors are used as hardware structures to construct various processing units.

[0300] Furthermore, more specifically, the hardware architecture of these various processors is a circuit composed of circuit elements such as semiconductor components.

[0301] The aforementioned structures and functions can be appropriately implemented by any hardware, software, or a combination of both. For example, this invention can also be applied to a program that causes a computer to perform the above-described processing steps (processing sequence), a computer-readable recording medium (non-temporary recording medium) containing such a program, or a computer capable of installing such a program.

[0302] The examples of the present invention have been described above. However, it should be noted that the present invention is not limited to the above-described embodiments, and various modifications can be made without departing from the spirit of the present invention.

[0303] Symbol Explanation

[0304] 10. Information processing device

[0305] 11 CPU

[0306] 12 ROM

[0307] 13 RAM

[0308] 14. Auxiliary storage device

[0309] 15 Input devices

[0310] 16 Output devices

[0311] 17 Input / Output Interfaces

[0312] 18 Communication Interface

[0313] 21 Information Acquisition Department

[0314] 22 Feature Point Extraction and Processing Unit

[0315] 23 Corresponding Point Search Processing Department

[0316] 24 Alignment Processing Unit

[0317] 24A Coarse Alignment Processing Unit

[0318] 24B Detailed Alignment Processing Unit

[0319] 25. Differential Calculation and Processing Department

[0320] 25A Pairing Processing Unit

[0321] 25B First Difference Calculation Unit

[0322] 25C Second Difference Calculation Unit

[0323] 26. Calculation Result Output Processing Unit

[0324] 27. Calculation Result Recording and Processing Department

[0325] 30. Merging Processing Department

[0326] 40. Capable of calculating differential region extraction processing unit

[0327] 50 Synthesis Processing Department

[0328] AV represents the frame of a single shot in the case of split shooting.

[0329] E1 is an enlarged image that will be displayed as part of the calculation results.

[0330] F1 represents the frame of the image captured by photographing the current bridge pier.

[0331] F2 represents the bounding box of the images captured during past bridge pier inspections.

[0332] F21 represents the frame containing the area captured in the first image taken during a previous inspection of the bridge piers.

[0333] F22 represents the frame of the second image taken during a previous inspection of the bridge piers.

[0334] GO Square

[0335] I1 Now Image

[0336] I2 Past Image

[0337] IC1's current crack information

[0338] IC2 information about past cracks

[0339] Information on the difference in length of IC3 crack

[0340] IC4 overall differential information

[0341] Current damage information for IC11

[0342] IC21's first damage information

[0343] IC22 Past Second Damage Information

[0344] IC31 First Differential Information

[0345] Information from the first difference after IC31A cuts out the overlapping region

[0346] IC32 Second Differential Information

[0347] The second differential information after IC32A cuts out the overlapping region

[0348] Information on the current damaged area (area where free lime occurs) of IR1

[0349] Information on past damage areas (areas where free lime occurred) for IR2

[0350] Information on the calculation results of the difference in the IR3 damage region

[0351] L is the straight line connecting the corresponding feature points.

[0352] The image shows the result of the OI calculation as the difference in crack length.

[0353] The image shows the calculation result of OI1 as the difference in crack length.

[0354] The image shows the result of OI2 as the difference in crack length.

[0355] The image shows the calculation results of OI3 as the difference in crack length.

[0356] The image shows the calculation results of OI4 as the difference in crack length.

[0357] The image shows the calculation results of OI5 as the difference in crack length.

[0358] The image shows the calculation results of OI6 as the difference in crack length.

[0359] P1-1 The current image obtained by segmentation and shooting.

[0360] The current image obtained by splitting the image into P1-2 segments.

[0361] P2-1 Past images obtained through segmentation photography

[0362] P2-2 Past images obtained through segmentation photography

[0363] V1-1 Current Crack Vector

[0364] V1-2 Current Crack Vector

[0365] V1-3 Current Crack Vector

[0366] V1-4 Current Crack Vector

[0367] V1-5 Current Crack Vector

[0368] V1-6 Current Crack Vector

[0369] V2-1 Past Crack Vector

[0370] V2-2 Past Crack Vector

[0371] V2-3 Past Crack Vector

[0372] V2-4 Past Crack Vector

[0373] V2-5 Past Crack Vector

[0374] W represents the box from which the difference is calculated.

[0375] The order of calculation and processing of the differences in damage information (crack information) from S1 to S7

[0376] The order of bit processing for S4A to S4D

[0377] The order of calculation and processing of differences in S5A to S5C

[0378] The order of calculation and processing of the difference in damage information S11 to S18

[0379] The order of calculation and processing of the difference in damage information (S21-S28)

[0380] The order of calculation and processing of the difference in damage information (S31-S39)

Claims

1. An information processing apparatus comprising a processor, wherein, The processor performs the following processing: Acquire a first image of the structure obtained by photographing it, and extract first damage information of the structure from the first image. A second image of the structure, taken at a different time point than the first image, is obtained, along with second damage information of the structure extracted from the second image. Extract multiple feature points from the first image. Extract multiple feature points from the second image. Search for corresponding feature points between the first image and the second image. Based on the information of the corresponding feature points, a first model for coarse alignment is constructed. The first model is applied to perform coarse alignment of corresponding feature points between the first image and the second image, and also to perform coarse alignment of the first damage information and the second damage information. Based on the information of the corresponding feature points after coarse alignment, a second model for detailed alignment is constructed. Using the second model, the first damage information and the second damage information are aligned in detail. Calculate the difference between the first damage information and the second damage information after detailed alignment.

2. The information processing apparatus according to claim 1, wherein, The processor performs the following processing: As a coarse alignment, rigid body alignment is performed. As described in the detailed alignment, non-rigid body alignment is performed.

3. The information processing apparatus according to claim 1, wherein, The processor performs the following processing: As a coarse alignment, rigid body alignment is performed. As a detailed alignment, alignment is performed by correcting the difference in lens distortion.

4. The information processing apparatus according to any one of claims 1 to 3, wherein, The first damage information and the second damage information are information about cracks. The processor calculates the difference between the width and / or length of the crack.

5. The information processing apparatus according to claim 4, wherein, The processor generates corresponding pairs of cracks, calculates the difference in width between the pairs, and thus calculates the difference in crack width.

6. The information processing apparatus according to claim 4, wherein, The processor generates corresponding crack pairs, calculates the length of cracks that do not generate pairs, and thereby calculates the difference in crack length.

7. The information processing apparatus according to claim 5, wherein, The processor generates pairs of cracks between adjacent cracks.

8. The information processing apparatus according to claim 5, wherein, The processor generates corresponding crack pairs through DP matching.

9. The information processing apparatus according to any one of claims 1 to 3, wherein, The first damage information and the second damage information are information about the damaged area. The processor calculates the differences between the corresponding damaged regions.

10. The information processing apparatus according to any one of claims 1 to 3, wherein, In the case of acquiring multiple first images obtained by segmenting and photographing the structure, multiple first damage information extracted from the multiple first images, and acquiring multiple second images obtained by segmenting and photographing the structure, and multiple second damage information extracted from the multiple second images. The processor performs the following processing: Multiple feature points are extracted from each of the first images. Multiple feature points are extracted from each of the second images. Search for corresponding feature points between the first image and the second image. Alignment is performed between the corresponding first damage information and the second damage information. Calculate the difference between the corresponding first damage information and the second damage information.

11. The information processing apparatus according to claim 10, wherein, The processor performs the following processing: Obtain the compositing information required for panoramic synthesis of multiple first images. Based on the synthesis processing information, the difference calculated between the corresponding first image and second image is synthesized.

12. The information processing apparatus according to any one of claims 1 to 3, wherein, In the case of acquiring multiple second images that overlap with the first image in a portion of the captured area, and multiple pieces of second damage information extracted from the multiple second images. The processor performs the following processing: Multiple feature points are extracted from each of the second images. Search for corresponding feature points between the first image and each of the second images. Alignment is performed between the first damage information and each of the second damage information. The difference between the first damage information and each of the second damage information is calculated respectively. and then, The differences calculated between the first damage information and each of the second damage information are combined.

13. The information processing apparatus according to any one of claims 1 to 3, wherein, The processor performs the following processing: Extract the region between the first image and the second image that allows for the calculation of the difference between the first damage information and the second damage information. Output the information of the extracted region.

14. The information processing apparatus according to claim 13, wherein, The processor performs the following processing: If information about the region from which the difference can be calculated is output, an image showing the region from which the difference can be calculated is generated on the first image, and this image is output to a display destination.

15. The information processing apparatus according to any one of claims 1 to 3, wherein, In the case of acquiring multiple first images obtained by segmenting and photographing the structure, multiple first damage information extracted from the multiple first images, and acquiring multiple second images obtained by segmenting and photographing the structure, and multiple second damage information extracted from the multiple second images. The processor performs the following processing: A panoramic composite is created from multiple of the first images. Based on the synthesis processing information from panoramic synthesis of multiple first images, multiple first damage information are synthesized. Panoramic synthesis is performed on multiple second images. Based on the synthesis processing information from panoramic synthesis of multiple second images, multiple pieces of second damage information are synthesized. Multiple feature points are extracted from the first image after panoramic synthesis. Multiple feature points are extracted from the second image after panoramic synthesis. Search for corresponding feature points between the first image and the second image after panoramic synthesis. The synthesized first damage information and the second damage information are aligned. Calculate the difference between the synthesized first damage information and the second damage information.

16. The information processing apparatus according to any one of claims 1 to 3, wherein, The first damage information and the second damage information are obtained before the coarse alignment.

17. An information processing method, wherein, Acquire a first image of the structure obtained by photographing it, and extract first damage information of the structure from the first image. A second image of the structure, taken at a different time point than the first image, is obtained, along with second damage information of the structure extracted from the second image. Extract multiple feature points from the first image. Extract multiple feature points from the second image. Search for corresponding feature points between the first image and the second image. Based on the information of the corresponding feature points, a first model for coarse alignment is constructed. The first model is applied to perform coarse alignment of corresponding feature points between the first image and the second image, and also to perform coarse alignment of the first damage information and the second damage information. Based on the information of the corresponding feature points after coarse alignment, a second model for detailed alignment is constructed. Using the second model, the first damage information and the second damage information are aligned in detail. Calculate the difference between the first damage information and the second damage information after detailed alignment.

18. The information processing method according to claim 17, wherein, The first damage information and the second damage information are obtained before the coarse alignment.

19. A computer-readable, non-transitory recording medium that records a program that enables a computer to perform the following steps: Acquire a first image of the structure obtained by photographing it and extract first damage information of the structure from the first image; Acquire a second image of the structure taken at a different time point than the first image, and extract second damage information of the structure from the second image; Extract multiple feature points from the first image; Extract multiple feature points from the second image; Search for corresponding feature points between the first image and the second image; Based on the information of the corresponding feature points, a first model for coarse alignment is constructed. The first model is applied to perform coarse alignment of the feature points corresponding to the first image and the second image, and to perform coarse alignment of the first damage information and the second damage information. Based on the information of the corresponding feature points after coarse alignment, a second model for detailed alignment is constructed. The second model is applied to perform detailed alignment of the first damage information and the second damage information; as well as Calculate the difference between the first damage information and the second damage information after detailed alignment.

20. An information processing method, wherein, Acquire a first image of the structure obtained by photographing it, and extract first damage information of the structure from the first image. A second image of the structure, taken at a different time point than the first image, is obtained, along with second damage information of the structure extracted from the second image. Information about the corresponding feature points is obtained between multiple feature points extracted from the first image and multiple feature points extracted from the second image. Based on the information of the corresponding feature points, a first model for coarse alignment is constructed. The first model is applied to perform coarse alignment of corresponding feature points between the first image and the second image, and also to perform coarse alignment of the first damage information and the second damage information. Based on the information of the corresponding feature points after coarse alignment, a second model for detailed alignment is constructed. The second model is applied to perform detailed alignment of the first damage information and the second damage information.

Images