Bridge apparent state change intelligent detection method based on images acquired by inspection robot

Abstract

The invention discloses a bridge apparent state change intelligent detection method based on images acquired by an inspection robot, and belongs to the technical field of bridge intelligent detection. According to the method, after a reference image library containing geometric refinement and radiation normalization processing is constructed and a most similar reference frame is retrieved, geometric and illumination dual alignment is carried out, a structural similarity index is adopted to calculate a multi-scale image difference and an adaptive threshold is adopted to determine a candidate change area, and finally transient interference is filtered through multi-frame time sequence consistency verification. And outputting real change information. According to the method, the problems of insufficient illumination robustness, limited geometric alignment precision and difficulty in filtering transient interference in the prior art are solved, sub-pixel-level alignment is realized, the false alarm rate is effectively reduced, the detection accuracy and efficiency are improved, and technical support is provided for bridge intelligent operation and maintenance.

Description

Technical Field

[0001] This invention belongs to the field of bridge intelligent inspection technology, specifically relating to an intelligent detection method for changes in the appearance of bridges based on images acquired by an inspection robot. Background Technology

[0002] Traditional bridge inspections mainly rely on manual visual inspection or bridge inspection vehicles, which suffers from low efficiency, numerous blind spots, and high risks associated with working at heights. In recent years, inspection robots (such as drones and wall-climbing robots) have been gradually applied to the regular inspection of key components of bridges, such as main beams, piers, and cable towers.

[0003] Inspection robots, equipped with high-definition cameras, can periodically collect a large amount of bridge surface image data. However, accurately identifying real changes in the bridge's appearance from these massive amounts of inspection images, while effectively filtering out various environmental interferences, has become a core bottleneck restricting the development of intelligent bridge operation and maintenance.

[0004] Currently, technologies for detecting changes in bridge appearance face numerous significant challenges. In direct pixel difference methods, this approach is extremely sensitive to changes in lighting. When inspection robots collect bridge images at different times and under different weather conditions, even if the structural appearance shows no defects, variations in ambient lighting alone can generate a large number of false difference signals. Regarding feature matching, conventional algorithms exhibit significantly reduced stability on complex bridge surfaces (such as reflective areas on water, textured concrete surfaces, and highly reflective metal components). Furthermore, existing methods lack a unified geometric reference framework for bridges, making precise alignment of images collected at different times difficult due to slight deviations in viewing angle.

[0005] More importantly, bridge inspection scenarios contain numerous transient interference sources, including: vehicles temporarily crossing the bridge, birds flying by, reflections from the water surface beneath the bridge, and changes in light and shadow caused by cloud cover. Although these transient factors appear significantly different in a single frame image, they are not actual structural defects of the bridge. If not effectively filtered, they will seriously affect the accuracy of the detection results. Therefore, there is an urgent need for an intelligent detection method for changes in the apparent condition of bridges based on images acquired by inspection robots. Summary of the Invention

[0006] Based on the above shortcomings, this invention provides an intelligent detection method for changes in the appearance of bridges based on images acquired by an inspection robot. It aims to solve the problems of insufficient robustness to illumination, limited geometric alignment accuracy, and difficulty in filtering transient interference such as vehicles and birds in existing bridge inspection technologies, so as to realize the integrated detection of bridge inspection images by "baseline establishment - precise alignment - difference calculation - intelligent judgment".

[0007] The technical solution adopted in this invention is as follows: An intelligent detection method for changes in the apparent state of bridges based on images acquired by an inspection robot, comprising the following steps:

[0008] Step S1: Construct a benchmark image library of bridge appearance status: Perform geometric refinement and radiometric normalization on the as-built inspection images, establish an effective region mask, and construct a multi-level image pyramid and feature index.

[0009] Step S2: Retrieve the most similar reference frame: Based on the global feature vector, quickly retrieve the reference frame that is most similar to the current inspection image in the vector index library, and obtain its corresponding geometric parameters, effective region mask and lookup table;

[0010] Step S3: Perform double alignment processing: Perform geometric alignment and illumination alignment on the current inspection image in sequence, transform it to the reference coordinate system and complete radiometric normalization;

[0011] Step S4: Calculate multi-scale image differences: Use structural similarity index to calculate the difference map between the aligned image and the reference image, and use sliding window statistics to achieve adaptive threshold determination;

[0012] Step S5: Spatiotemporal consistency verification and output. Perform multi-frame temporal consistency verification on candidate change regions, filter transient interference, and output the coordinates, confidence level, visualization results, and event logs of the real change regions.

[0013] Furthermore, step S1 includes the following sub-steps:

[0014] Step S1.1: Geometric refinement processing. The as-built inspection images are refined by using homography transformation or structure-of-motion reconstruction (SOR) methods to perform planar partitioning, establishing an independent geometric reference frame for each planar region. Simultaneously, the effective region mask for each reference image is recorded. It is used to filter out invalid areas, including: sky, vehicles, pedestrians, and water reflections;

[0015] Step S1.2: Radiation normalization processing, calculate the 1% and 99th percentiles P1 and P99 of the image grayscale histogram, normalize it through the histogram stretching mapping function, and store the mapping relationship as a lookup table LUT;

[0016] The expression for the histogram stretching mapping function is:

[0017]

[0018] In the formula, These are the original image pixel values; and These are the 1% and 99th percentiles of the image's grayscale histogram, respectively. These are the normalized pixel values;

[0019] This mapping relationship is stored as a lookup table (LUT), whose expression is:

[0020]

[0021] In the formula, The function restricts the result to Within the range.

[0022] Step S1.3: Generate a shadow-free reference image A multi-frame median fusion algorithm is used to generate an illumination-invariant reference map from N frames of images of the same scene under different lighting conditions.

[0023] The expression for multi-frame median fusion is as follows:

[0024]

[0025] In the formula, N frames of images of the same scene under different lighting conditions;

[0026] Alternatively, the Mertens exposure fusion algorithm can be used, which achieves weighted fusion. Its expression is:

[0027]

[0028] In the formula, the weights It is determined by the product of three factors: contrast, saturation, and exposure quality.

[0029] Step S1.4: Construct a multi-level image pyramid: Using Gaussian downsampling, an L-level pyramid is generated, the expression of which is:

[0030]

[0031] In the formula, Original image; For the first Image of a pyramid; The kernel weights are 5×5 Gaussian, satisfying the normalization condition. ;

[0032] Step S1.5: Feature Extraction and Index Construction: A deep convolutional neural network is used to extract global feature vectors, and a vector index is built using a vector index library. The expression for extracting feature vectors is:

[0033]

[0034] In the formula, For feature extraction networks, For network parameters, The normalized 128-dimensional feature vector satisfies ;

[0035] The deep convolutional neural network uses a ResNet34 network. The fully connected classification layer of the original network is removed, and the average pooling layer output up to the last convolutional block is retained to obtain a 512-dimensional feature vector. Then, a fully connected layer is used to reduce the dimensionality of the 512-dimensional feature vector to 128-dimensional. This fully connected layer contains 512 input neurons and 128 output neurons. The ReLU activation function is used for non-linear transformation, and the output 128-dimensional feature vector is subjected to L2 normalization.

[0036] Furthermore, in step S2, the reference frame retrieval uses cosine similarity as a metric, and performs an approximate nearest neighbor search in the vector index library, the expression of which is:

[0037]

[0038] In the formula, As a benchmark image library, This is the feature vector of the current inspection image. for The transpose of , The feature vector of the reference image, The most similar reference frame obtained from the retrieval.

[0039] Furthermore, step S3 includes the following sub-steps:

[0040] Step S3.1: Geometric Alignment: First, align the current inspection image... Extract local feature points and compare them with the baseline image. Perform feature matching to obtain an initial set of matching point pairs. ,in, As the reference image coordinates, The coordinates of the image to be aligned. This represents the total number of feature point pairs that were successfully matched between the two images; Represents the transpose of a matrix;

[0041] Then, based on the coplanar constraint, the homography projection transformation equation is established, and its expression is:

[0042]

[0043] In the formula, Scale factor; It is a 3×3 homography matrix;

[0044] Expanding the projection transformation equation into a linear form:

[0045]

[0046] Constructing an overdetermined system of equations using the least squares method: ,

[0047] Wherein, the coefficient matrix Each pair of matching points contributes two rows:

[0048]

[0049] Parameter vector ;

[0050] Next, robust estimation is performed using the RANSAC algorithm. The iterative process is as follows: Four sets of matching points are randomly selected to calculate the initial homography matrix. The interior point set is evaluated using the reprojection error, and its expression is:

[0051]

[0052] In the formula, Using homogeneous coordinates, This is a perspective division operation;

[0053] like It is then determined to be an interior point. The threshold value is the interior point threshold.

[0054] go through After several iterations, the set of largest interior points is selected for final least-squares optimization to obtain the optimal homography matrix. ;

[0055] Then, the current image is projected onto the reference coordinate system through a cascaded transformation, expressed as:

[0056]

[0057] In the formula, These are the geometric parameters corresponding to the reference image; The optimal homography matrix between the current image and the reference image is obtained by using the reference mask. filter Invalid regions in;

[0058] Step S3.2: Illumination Alignment: Using histogram matching, the geometrically aligned image is aligned. According to the lookup table of the reference image Radiation normalization is performed, and the expression is:

[0059] .

[0060] Furthermore, step S4 includes the following sub-steps:

[0061] Step S4.1: Calculate the normalized image using the structural similarity index. Compared with the baseline image The difference diagram; where the expression for the structural similarity index is:

[0062]

[0063] In the formula, and Image blocks and The local mean, and These are the local variances, For covariance, and Here is the stability constant. For pixel dynamic range, , ;

[0064] Local statistics are obtained through a Gaussian weighted window. The calculation is expressed as follows:

[0065]

[0066]

[0067]

[0068] In the formula, Window size; The Gaussian weighting coefficients satisfy... ;

[0069] The expression for the difference plot is defined as follows:

[0070] ;

[0071] Step S4.2: Adaptive threshold determination is achieved through sliding window statistics, marking candidate change regions; wherein, the adaptive threshold determination adopts the sliding window statistical method, dividing the difference map into sizes. For a local window, calculate the local mean and standard deviation within the window using the following expression:

[0072]

[0073]

[0074] The expression for the adaptive threshold is:

[0075]

[0076] In the formula, The threshold coefficient is estimated online based on N days of unchanged historical samples, using a 3... The principle or Bayesian update method is determined;

[0077] Step S4.3: Candidate Change Region Generation: Based on online estimation using N days of historical unchanged samples, a 3D model is employed. Based on the principle or Bayesian update method, if the difference value of M consecutive windows in a certain region exceeds the corresponding threshold, the connected region formed by these consecutive windows is marked as a candidate change region. Through the connected component analysis algorithm, adjacent candidate windows are merged to form a complete region, and tiny regions with an area smaller than the minimum effective area threshold are eliminated. Finally, the bounding box coordinates and pixel range of the candidate change region are output.

[0078] Furthermore, step S5 includes the following sub-steps:

[0079] Step S5.1: Multi-frame temporal consistency verification: Spatiotemporal consistency verification requires that the positions of the same region... Must be in continuous A change is considered real only if all candidate markers are triggered in the newly inspected frame. The expression for temporal consistency determination is:

[0080]

[0081] In the formula, and The first Frame difference map and threshold, ; For indicator functions;

[0082] The expression for determining actual change is:

[0083]

[0084] In the formula, The time-series consistency threshold is set to... or ;

[0085] Step S5.2: Transient interference filtering: If a candidate region is in the... Frame-triggered change detection, but in the first frame... The frame is restored to a state consistent with the reference frame, that is:

[0086]

[0087] The region is then determined to be a transient disturbance and is not included in the actual changes. The final output is as follows:

[0088] 1) Coordinates of the changing region polygon: Extract the contour of the changing region through connected component analysis and output the sequence of polygon vertex coordinates. ,in, The number of vertices;

[0089] 2) Confidence score: Taking into account the intensity of differences, temporal consistency, and spatial continuity, its expression is:

[0090]

[0091] In the formula, This represents the average regional difference. The area of ​​the region of change; The minimum effective area threshold, These are the weighting coefficients;

[0092] 3) Visual overlay: Overlays the changed area onto the original image as a semi-transparent overlay;

[0093] 4) Event Log: Includes timestamp, location number, description of changes, and confidence level assessment.

[0094] This invention also provides an intelligent detection system for bridge appearance changes based on images acquired by an inspection robot. It implements the process described above for an intelligent detection method for bridge appearance changes based on images acquired by an inspection robot, including a baseline establishment module, a baseline matching module, a double alignment module, a difference calculation module, and a spatiotemporal verification output module executed sequentially. These modules work together to achieve intelligent detection of real changes in the bridge's appearance.

[0095] The benchmark establishment module performs geometric refinement, radiometric normalization, shadowless reference image generation, multi-level image pyramid construction, and feature index establishment on the completed inspection images, and outputs a benchmark image library containing geometric parameters, effective region masks, illumination correction lookup tables, and feature vector indexes.

[0096] The benchmark matching module extracts the global feature vector of the current inspection image, performs an approximate nearest neighbor search in the feature index of the benchmark image library based on the cosine similarity metric, and outputs the most similar benchmark frame and its corresponding geometric parameters, effective region mask and illumination correction lookup table.

[0097] The dual alignment module first achieves geometric alignment between the current inspection image and the reference frame through homography transformation, and then performs histogram matching based on the illumination correction lookup table of the reference frame to achieve illumination alignment, and outputs the normalized aligned image.

[0098] The difference calculation module: uses a structural similarity index to calculate the difference map between the normalized image and the reference frame, implements adaptive threshold determination through sliding window statistics, and outputs candidate change regions;

[0099] The spatiotemporal verification output module performs multi-frame temporal consistency verification and transient interference filtering on candidate change regions, and outputs the polygon coordinates, confidence scores, visual overlay plots, and event logs of the real change regions.

[0100] The present invention also provides a computer system, comprising: a processor and a computer-readable storage medium communicatively connected to the processor, wherein the computer-readable storage medium stores computer instructions, and when the computer instructions are executed by the processor, implement the steps of the intelligent detection method for changes in the apparent state of a bridge based on images acquired by an inspection robot as described above.

[0101] The present invention also provides a computer program product, characterized in that, when the computer program / instruction is executed by a processor, it implements the steps of the intelligent detection method for changes in the apparent state of a bridge based on images acquired by an inspection robot, as described above.

[0102] The beneficial effects and advantages of this invention are as follows:

[0103] (1) Eliminating transient interference: This invention requires candidate change regions to be continuously triggered in multiple consecutive frames of images through a spatiotemporal consistency verification mechanism, effectively filtering transient interference such as vehicles, birds, water surface reflections, and changes in light and shadow, and significantly reducing the false alarm rate;

[0104] (2) Strong robustness to illumination: The present invention adopts a method combining offline radiation normalization and online histogram matching to establish a shadowless reference map as an illumination invariant benchmark, effectively overcoming the impact of illumination changes under different time periods and weather conditions;

[0105] (3) Subpixel-level alignment accuracy: This invention uses RANSAC-based homography transformation for geometric refinement, combined with multi-level image pyramids and feature indexes, to achieve subpixel-level spatial alignment accuracy, providing a reliable geometric basis for subsequent difference calculation.

[0106] In summary, this invention solves the problems of poor illumination robustness, low alignment accuracy, and difficulty in removing transient interference in existing technologies, and significantly improves the accuracy and reliability of intelligent bridge inspection in complex environments. Attached Figure Description

[0107] Figure 1 Flowchart of the intelligent detection method for bridge appearance changes based on images acquired by an inspection robot according to the present invention. Detailed Implementation

[0108] The present invention will now be described in further detail with reference to embodiments and accompanying drawings, but the scope of the invention is not limited thereto. Those skilled in the art can make modifications in form and detail, and all such modifications are within the protection scope of the present invention.

[0109] Example 1

[0110] A smart detection method for changes in the apparent state of bridges based on images acquired by an inspection robot, comprising the following steps:

[0111] Step S1: Construct a benchmark image library based on the images acquired by the inspection robot.

[0112] Step S1.1: Perform geometric refinement on the as-built inspection images in the reference image library. Use homography or lightweight motion reconstruction methods (such as COLMAP) for planar partitioning refinement. For each planar region in the scene, establish an independent geometric reference frame. Let the original image be... The geometrically refined image is The transformation relation expression is:

[0113]

[0114] In the formula, This is the homography matrix corresponding to this planar region. Simultaneously, the effective region mask for each reference image is recorded. ,in Indicates an invalid area (sky, vehicles, pedestrians, water reflections, etc.). Indicates the effective monitoring area.

[0115] Step S1.2: Perform radiometric normalization on the reference image. First, calculate the quantile parameters of the image's gray-level histogram, whose expression is:

[0116]

[0117] In the formula, The cumulative distribution function is... This refers to the image resolution. and They were obtained respectively and .

[0118] The expression for the histogram stretching mapping function is:

[0119]

[0120] The expression for discretizing this mapping relationship into a 256-level lookup table (LUT) is as follows:

[0121]

[0122] Step S1.3: Generate a shadow-free reference image. A multi-frame median fusion algorithm is used to generate an illumination-invariant reference image from N frames of images of the same scene acquired under different lighting conditions. The expression is as follows:

[0123]

[0124] Alternatively, the Mertens exposure fusion algorithm can be used, which achieves weighted fusion. Its expression is:

[0125]

[0126] In the formula, the weights It is determined by the product of three factors: contrast, saturation, and exposure quality.

[0127] Step S1.4: Construct a multi-level Gaussian image pyramid. Assume the original image resolution is... ,generate The first pyramid Level image resolution is ,in , .

[0128] The expression for the downsampling process is:

[0129]

[0130] In the formula, the 5×5 Gaussian kernel Defined as:

[0131]

[0132] Step S1.5: Extract global feature vectors and build an index. The expression for extracting 128-dimensional global features using a pre-trained deep convolutional neural network (such as ResNet34) is as follows:

[0133]

[0134] In the formula, This is the output of the intermediate layer of the feature extraction network. These are the parameters for pre-trained network. An IVF-PQ index structure is built using the Faiss library, supporting fast approximate nearest neighbor search for large-scale image databases.

[0135] Step S2: Retrieve the most similar reference frame.

[0136] For the current inspection image Extracting global feature vectors An approximate nearest neighbor search is performed in the Faiss index to retrieve the top-1 most similar baseline frame. The expression is as follows:

[0137]

[0138] In the formula, the similarity measure is cosine similarity, and its expression is:

[0139]

[0140] The search results include a baseline frame. and their corresponding geometric parameters Effective area mask Lighting Correction Lookup Table .

[0141] Step S3: Perform double alignment.

[0142] Step S3.1: Geometric Alignment. Align the current inspection image. Extract local feature points (such as ORB or SIFT) and compare them with a reference image. Perform feature matching.

[0143] Get the initial set of matching point pairs ,in, As the reference image coordinates, The coordinates of the image to be aligned. This represents the total number of feature point pairs that were successfully matched between the two images; This represents the transpose of a matrix.

[0144] Then, based on the coplanar constraint, the homography projection transformation equation is established, and its expression is:

[0145]

[0146] In the formula, Scale factor; It is a 3×3 homography matrix;

[0147] Expanding the projection transformation equation into a linear form:

[0148]

[0149] Constructing an overdetermined system of equations using the least squares method: ,

[0150] Wherein, the coefficient matrix Each pair of matching points contributes two rows:

[0151]

[0152] Parameter vector ;

[0153] Next, robust estimation is performed using the RANSAC algorithm. The iterative process is as follows: Let the... The four sets of matching points randomly selected in the next iteration are: The initial homography matrix is ​​calculated using direct linear transformation (DLT). The expression for calculating the reprojection error for all matching points is:

[0154]

[0155] In the formula, Using homogeneous coordinates, perspective division The condition for determining an interior point is: Count the number of interior points .go through After the second iteration, the largest interior set is selected. ,in .

[0156] The expression for final least squares optimization based on the maximum interior set is:

[0157]

[0158] In the formula, Representative process The homography matrix after the next iteration

[0159] The expression for projecting the current image onto the reference coordinate system through a cascaded transformation is:

[0160]

[0161] Bilinear interpolation is used to process non-integer coordinates, and a mask is applied to filter out invalid regions. The expression is as follows:

[0162]

[0163] Step S3.2: Lighting Alignment. Align the geometrically aligned image. According to the lookup table of the reference image Histogram matching is performed using the following expression:

[0164]

[0165] This operation ensures that the grayscale distribution of the aligned image is consistent with the reference image, eliminating the influence of lighting differences.

[0166] Step S4: Calculate multi-scale image differences.

[0167] Step S4.1: Calculate the Structural Similarity Index (SSIM): using... Gaussian weighted sliding window (e.g., 11×11) is used to calculate local statistics. Let the window center be... The set of pixels within the window is The Gaussian weight kernel is:

[0168]

[0169] After normalization, it satisfies .

[0170] The expression for calculating local statistics is:

[0171]

[0172]

[0173]

[0174]

[0175]

[0176] The structural similarity index, defined by the product of three components—luminance, contrast, and structure—is expressed as follows:

[0177]

[0178] in:

[0179]

[0180]

[0181]

[0182] In the formula, ; ; The expression for the difference plot is defined as follows:

[0183]

[0184] Step S4.2: Adaptive threshold determination. For the difference map... Divide the dimensions as We perform sliding statistics on a local window. Let the coordinates of the top-left corner of the window be... The set of differences within the window is The expressions for calculating the local mean and standard deviation are:

[0185]

[0186]

[0187] The expression for the adaptive threshold is:

[0188]

[0189] Threshold coefficient Based on historically unchanged sample statistics, the method used is 3. In principle .

[0190] The expression for determining window changes is:

[0191]

[0192] In the formula, The threshold is the percentage of pixels exceeding the threshold within the window;

[0193] Step S4.3: Generation of candidate change regions: If continuous Adjacent windows satisfy If the region is identified as a candidate change region, then adjacent candidate windows are merged to form a complete region through a connected component analysis algorithm. Small regions with an area smaller than the minimum effective area threshold are eliminated, and finally the bounding box coordinates and pixel range of the candidate change region are output.

[0194] Step S5: Spatiotemporal consistency verification and output.

[0195] Step S5.1: Multi-frame temporal consistency verification. For locations marked as candidate change regions... In continuous Verification is performed in the new inspection image. Let the first frame... The difference between the frames is shown in the figure. The corresponding threshold is The timing consistency determination expression is:

[0196]

[0197] The expression for determining actual change is:

[0198]

[0199] In the formula, The time-series consistency threshold is typically set to [value missing]. (Requires triggering in all frames) or (allow (Triggered in 2 frames).

[0200] Step S5.2: Transient interference filtering. If a candidate region is in the... Frame-triggered change detection, but in the first frame... The expression for restoring a frame to a state consistent with the reference frame is:

[0201]

[0202] If the area is considered to be subject to transient interference (such as passing vehicles, birds, or changes in light and shadow), it will not be included in the calculation of actual changes.

[0203] Step S5.3: Output Result Generation. For the actual change regions that pass the spatiotemporal consistency check, output the following information:

[0204] (1) Coordinates of the changing region polygon: The contour of the changing region is extracted by connected component analysis, and the vertex coordinate sequence of the polygon is output. ,in, The number of vertices;

[0205] (2) Confidence score: Taking into account the intensity of difference, temporal consistency and spatial continuity, its expression is:

[0206]

[0207] In the formula, This represents the average regional difference. For the area of ​​the changing region, The minimum effective area threshold, These are the weighting coefficients;

[0208] (3) Visual overlay: The changed area is overlaid on the original image as a semi-transparent overlay;

[0209] (4) Event log: includes timestamp, location number, description of changes, and confidence assessment.

[0210] Example 2

[0211] like Figure 1 As shown, an intelligent detection method for changes in the apparent state of bridges based on images acquired by an inspection robot is applied to a bridge structural health monitoring scenario. The steps are as follows:

[0212] Step S1: Construct a benchmark image library based on the images obtained from the inspection robot.

[0213] An inspection robot (model DJI Dock 2) equipped with a 4K high-definition camera was used to periodically collect images of key parts of the bridge, such as the main beam, piers, and supports, to obtain a baseline image library containing 2,000 images with a resolution of 3840×2160 pixels.

[0214] Step S1.1: Perform geometric refinement on the reference image, use COLMAP for sparse reconstruction, and establish independent homography transformation parameters for each planar region. Record the effective region mask. It filters out the sky area (approximately 15% of the pixels), the water surface reflection area (approximately 8% of the pixels), and the temporary obstacle area.

[0215] Step S1.2: Radiometric normalization. Calculate the 1% and 99th percentiles for each baseline map, typical values. , Generate a 256-level lookup table (LUT) for storage.

[0216] Step S1.3: Generate a shadow-free reference map. From 30 frames of images under different lighting conditions at each monitoring point, a light-invariant reference map is generated using median fusion.

[0217] Step S1.4: Construct a 3-level Gaussian image pyramid with a scale factor of 0.5. The original resolution is 3840×2160, level 1 is 1920×1080, and level 2 is 960×540.

[0218] Step S1.5: Extract a 128-dimensional global feature vector using a ResNet34 network. Specifically, remove the fully connected classification layer of the original ResNet34 network and retain the average pooling layer output up to the last convolutional block (conv5_x) to obtain a 512-dimensional feature vector. Then, reduce the dimensionality of the 512-dimensional feature vector to 128-dimensional through a fully connected layer containing 512 input neurons and 128 output neurons. Use the ReLU activation function for non-linear transformation and perform L2 normalization on the output 128-dimensional feature vector. Finally, use the IVF1024-PQ32 index structure from the Faiss library to index the feature vector. The IVF (inverted index) uses 1024 cluster centers, and PQ (product quantization) divides the 128-dimensional vector into 32 subspaces for compression encoding, achieving millisecond-level retrieval response.

[0219] Step S2: Retrieve the most similar reference frame.

[0220] A 128-dimensional feature vector was extracted from the current inspection image, and a Top-1 search was performed in the Faiss index. The average search time was 2.3ms, and the search accuracy was 99.2%. The matching baseline frame and its geometric parameters were then obtained. Mask Lookup table .

[0221] Step S3: Perform double alignment.

[0222] Step S3.1: Geometric Alignment. ORB feature extraction is used, with a maximum of 5000 feature points. BF matching is performed with the baseline frame, typically resulting in approximately 2800 matching point pairs. RANSAC parameter settings: number of iterations. Next, interior point threshold Pixels. The optimal homography matrix is ​​obtained after optimization:

[0223]

[0224] Control point reprojection error after alignment A pixel is equivalent to approximately 0.15 mm of the actual surface area of ​​a structure.

[0225] Step S3.2: Lighting Alignment. Align the geometrically aligned image according to... Perform histogram matching to eliminate lighting differences.

[0226] Step S4: Calculate multi-scale image differences.

[0227] Step S4.1: Use Gaussian weighted window ( Calculate SSIM, stability constant , Calculate the difference map pixel by pixel within the effective area. .

[0228] Step S4.2: Adaptive threshold determination. Sliding window size. Pixels, step size 16 pixels. Threshold coefficient based on statistical analysis of unchanged samples over the past 30 days. Local mean in typical scenarios Standard deviation Adaptive threshold . Threshold for the proportion of pixels exceeding the threshold within the window Number of consecutive windows .

[0229] Step S4.3: Candidate Change Region Generation: For each local window, first calculate the number of pixels within the window that exceed the adaptive threshold T, and calculate the proportion of pixels exceeding the threshold. If this proportion exceeds the threshold γ=0.3, the window is marked as a suspected change window. Further, if a region has M = 3 consecutive adjacent windows that all meet the above conditions, the entire region covered by these consecutive windows is marked as a candidate change region. Through connected component analysis, adjacent candidate windows are merged to form a complete region, and regions with areas smaller than [a certain size] are eliminated. The algorithm calculates the bounding box coordinates and pixel range of a tiny region of pixels, ultimately outputting the candidate region of change.

[0230] Step S5: Spatiotemporal consistency verification and output.

[0231] Step S5.1: Multi-frame timing consistency verification. Set the timing filtering window. Frame, Timing Consistency Threshold (Triggered in at least 2 out of 3 frames).

[0232] Step S5.2: Transient interference filtering. After spatiotemporal verification, the following are filtered out: vehicle obstruction areas (approximately 12 locations per day), bird interference (approximately 3 locations per day), and changes in water surface reflection (approximately 8 locations per day).

[0233] Step S5.3: Output Results. For detected areas of real change (such as new cracks or coating peeling), output polygon coordinates, confidence scores (typical values ​​0.85-0.95), overlay visualizations, and event logs. The system's false alarm rate is reduced from 40% in traditional methods to below 5%, with an effective change detection rate of 96.5%.

[0234] In the specific application scenario of this embodiment, the 4K high-definition camera equipped on the inspection robot has the following technical parameters: a 1-inch CMOS sensor, 20 million effective pixels, a lens focal length of 24-70mm equivalent focal length, an aperture range of f / 2.8-f / 11, a shutter speed of 1 / 8000s-30s, and an ISO sensitivity of 100-12800. The image acquisition interval is set to once a week, with each acquisition covering all 156 monitoring points on the bridge.

[0235] The construction of the benchmark image library took three months, during which 8,000 historical images were collected. After geometric refinement, radiometric normalization, and quality screening, 2,000 high-quality benchmark images were finally added to the library. The metadata accompanying each benchmark image includes: acquisition timestamp (accurate to milliseconds), GPS coordinates (accuracy ±0.5m), camera attitude parameters (pitch angle ±0.1°, yaw angle ±0.2°), ambient light intensity (lux value), and weather condition label.

[0236] In the geometric refinement stage, COLMAP was used for incremental motion reconstruction of the structure, extracting approximately 850,000 sparse point clouds, with the reconstruction error (reprojection RMSE) controlled within 0.8 pixels. Considering the planar characteristics of the bridge structure, the scene was divided into 32 independent planar regions, each with its own independent homography transformation parameters, ensuring a planar fitting error of no more than 5 mm.

[0237] The effective region mask was generated using the DeepLabV3+ semantic segmentation network. The training dataset contained 5000 manually annotated images, and the segmentation categories included: sky, water surface, vehicles, pedestrians, vegetation, buildings, and bridge structures. The segmentation accuracy (mIoU) reached 92.3%, with the recognition accuracy for bridge structures reaching 98.7%.

[0238] In the radiometric normalization stage, the 1% and 99th percentiles of the gray-level histograms were calculated for images under different lighting conditions. Statistical analysis showed that images acquired on sunny mornings... The mean is 8.2. The mean value is 247.5; under cloudy conditions. The mean is 15.6. The mean value is 235.2; under backlight conditions in the evening. The mean is 22.4. The mean is 251.8. The lookup table (LUT) is generated using a piecewise linear interpolation method to ensure the continuity and monotonicity of the mapping.

[0239] The generation of the shadow-free reference image involved selecting 30 representative images from each monitoring point collected at different seasons and time periods, and employing a multi-frame median fusion algorithm. For areas with persistent shadows (such as the projection area of ​​bridge piers), an additional Mertens exposure fusion algorithm was introduced for compensation processing, effectively eliminating approximately 87% of the fixed shadow interference.

[0240] The multi-level image pyramid was constructed using a 3-level Gaussian pyramid structure with a scale factor of 0.5. The original image resolution was 3840×2160 pixels, the first-level pyramid resolution was 1920×1080 pixels, and the second-level pyramid resolution was 960×540 pixels. The Gaussian filter kernel size was 5×5, and the standard deviation was [missing information]. The introduction of the pyramid structure enables feature retrieval to quickly locate features at a coarse scale and accurately match features at a fine scale, improving overall retrieval efficiency by approximately 3.2 times.

[0241] Global feature vector extraction employs a pre-trained ResNet34 network, extracting the output features of the fourth residual block (conv4_x). After global average pooling, a 512-dimensional feature vector is obtained, which is then reduced to 128 dimensions through a fully connected layer. The feature vector is L2 normalized and stored in float16 format to save storage space. The feature index of 2000 benchmark images uses the IVF1024, PQ32 structure from the Faiss library, with an index file size of approximately 12MB, supporting single retrieval completion within 2.3ms.

[0242] During the online detection phase, a 128-dimensional global feature vector is first extracted from each newly acquired image, and an approximate nearest neighbor search is performed in the Faiss index. The similarity threshold for the search results is set to 0.85; matches below this threshold trigger a manual review process. After a six-month trial, a total of 1,872 newly acquired images were processed, achieving a retrieval accuracy rate of 99.2% and a false match rate of only 0.8%, primarily occurring between adjacent monitoring points.

[0243] In the geometric alignment stage, ORB feature descriptors were used, with an upper limit of 5000 feature points extracted and a non-maximum suppression radius of 8 pixels. Feature matching employed brute-force matching (BF Matcher) combined with cross-validation, resulting in approximately 4200 initial matching point pairs. After filtering using Lowe's ratio test (threshold 0.75), approximately 2800 high-confidence matches were retained.

[0244] The RANSAC algorithm parameters are set as follows: maximum number of iterations. Next, interior point threshold The resolution is 99.99% with a minimum inlier ratio of 0.3. In a typical scenario, RANSAC requires approximately 350 iterations to converge, with a final inlier ratio of approximately 78%, meaning approximately 2184 matching points are identified as inliers.

[0245] The homography matrix is ​​solved using the Singular Value Decomposition (SVD) method, with final least-squares optimization based on the maximum interior set. Taking a specific detection as an example, the optimal homography matrix is:

[0246]

[0247] The geometric meaning of this matrix is ​​as follows: the image rotation angle is approximately 0.71°, the horizontal translation is 45.6 pixels, the vertical translation is 32.2 pixels, and the scale change is less than 0.2%. After alignment, the reprojection error is calculated on all inliers, with a mean of 0.35 pixels, a standard deviation of 0.15 pixels, and a maximum value of 1.82 pixels, which meets the sub-pixel alignment accuracy requirements.

[0248] Illumination alignment employs a histogram specification method to adjust the grayscale distribution of the geometrically aligned image to match that of the reference image. Specifically, this involves first calculating the cumulative distribution function of the two images. and Then establish a mapping relationship. To ensure the smoothness of the mapping, cubic spline interpolation is used to smooth the lookup table.

[0249] During the difference map calculation stage, the SSIM parameters are set as follows: Gaussian window size 11×11 pixels, window standard deviation... Pixel, stability constant , To address the texture characteristics of bridge structure surfaces, SSIM calculations were extended to the YUV color space, with evaluations performed on the luminance channel (weight 0.6) and chrominance channel (weight 0.4) to enhance sensitivity to color changes such as rust discoloration and coating aging.

[0250] The sliding window parameters for adaptive threshold determination are set as follows: window size Pixels, sliding step size 16 pixels (50% overlap). Threshold coefficient. The determination was based on statistical analysis of 420 unchanged samples over the past 30 days, using a normal distribution assumption. In principle, Verification showed that this threshold setting kept the false positive rate of a single frame image below 2.1%.

[0251] The statistical parameters for typical monitoring scenarios are as follows: local mean of the difference map. The distribution range is 0.02-0.08, with a mean of approximately 0.05; local standard deviation The distribution range is 0.008-0.035, with a mean of approximately 0.02. Therefore, the adaptive threshold... The typical value is approximately 0.11. The threshold for the percentage of pixels exceeding the threshold within the window is set to... A window is marked only if at least 30% of its pixels exceed a threshold. Number of consecutive windows. That is, three consecutive adjacent windows must meet the conditions to be considered as a candidate change region.

[0252] During the spatiotemporal consistency verification phase, the time filtering window is set to A frame, meaning the same area must consistently show changes in three consecutive inspections. The timing consistency threshold is set to... This means that a change is considered real if it is triggered in at least two out of three frames. This setting allows for a certain tolerance for missed detections while effectively filtering out occasional interference.

[0253] Transient interference filtering effect statistics: During the 6-month operation period, the system detected a total of 12,847 original candidate change areas. After spatiotemporal consistency verification, 11,923 transient interferences were filtered out, and 924 real change areas were finally confirmed. The main sources of transient interference included: vehicle occlusion (46.2%, approximately 12 per day), bird flight (11.7%, approximately 3 per day), changes in water surface reflection (31.4%, approximately 8 per day), changes in cloud shadow (8.3%, approximately 2 per day), and other interference (2.4%).

[0254] The final output of the change detection results includes: polygon coordinates of the changed area, confidence score, and event log. The visualization overlay uses a semi-transparent red overlay to mark the changed area, and includes arrows to indicate the direction of change and numerical values ​​to indicate the degree of change.

[0255] The event logs are stored in JSON format, and each record includes: timestamp (ISO 8601 format), monitoring point number (156 in total), coordinates of the bounding box of the changed area, polygon vertex sequence, confidence score, reference image path, current image path, difference map path, processing time, and other fields. The log files are stored in a rolling order by date, with approximately 2MB of data per day, and support database import and visualization analysis.

[0256] System performance evaluation results: On a test set of 1000 images, the change detection accuracy was 94.8%, the recall was 96.5%, and the F1 score was 95.6%. The false positive rate was reduced from 42.3% in the traditional pixel difference method to 4.7%, a reduction of 88.9%. The total processing time for a single image was approximately 237ms: feature extraction 15ms, index retrieval 2.3ms, geometric alignment 45ms, illumination alignment 8ms, SSIM calculation 120ms, threshold determination 35ms, and output generation 12ms, meeting the requirements for near real-time processing.

[0257] This embodiment fully verifies the effectiveness and practicality of the method of the present invention in actual bridge inspection scenarios, and provides reliable technical support for intelligent bridge operation and maintenance.

Claims

1. A method for intelligent detection of bridge appearance changes based on images acquired by an inspection robot, characterized in that, Includes the following steps: Step S1: Construct a benchmark image library of bridge appearance status: Perform geometric refinement and radiometric normalization on the as-built inspection images, establish an effective region mask, and construct a multi-level image pyramid and feature index. Step S2: Retrieve the most similar reference frame: Based on the global feature vector, quickly retrieve the reference frame that is most similar to the current inspection image in the vector index library, and obtain its corresponding geometric parameters, effective region mask and lookup table; Step S3: Perform double alignment processing: Perform geometric alignment and illumination alignment on the current inspection image in sequence, transform it to the reference coordinate system and complete radiometric normalization; Step S4: Calculate multi-scale image differences: Calculate the difference map between the aligned image and the reference image using structural similarity index, and achieve adaptive threshold determination through sliding window statistics; Step S5: Spatiotemporal consistency verification and output. Perform multi-frame temporal consistency verification on candidate change regions, filter transient interference, and output the coordinates, confidence level, visualization results, and event logs of the real change regions.

2. The intelligent detection method for bridge appearance state changes based on images acquired by an inspection robot according to claim 1, characterized in that, Step S1 includes the following sub-steps: Step S1.1: Geometric refinement processing. The as-built inspection images are refined by using homography transformation or structure-of-motion reconstruction (SOR) methods to perform planar partitioning, establishing an independent geometric reference frame for each planar region. Simultaneously, the effective region mask for each reference image is recorded. It is used to filter out invalid areas, including: sky, vehicles, pedestrians, and water reflections; Step S1.2: Radiation normalization processing, calculate the 1% and 99th percentiles P1 and P99 of the image grayscale histogram, normalize it through the histogram stretching mapping function, and store the mapping relationship as a lookup table LUT; The expression for the histogram stretching mapping function is: In the formula, These are the original image pixel values; and These are the 1% and 99th percentiles of the image's grayscale histogram, respectively. These are the normalized pixel values; This mapping relationship is stored as a lookup table (LUT), whose expression is: In the formula, The function restricts the result to Within the range. Step S1.3: Generate a shadow-free reference image A multi-frame median fusion algorithm is used to generate an illumination-invariant reference map from N frames of images of the same scene under different lighting conditions. The expression for multi-frame median fusion is as follows: In the formula, N frames of images of the same scene under different lighting conditions; Alternatively, the Mertens exposure fusion algorithm can be used, which achieves weighted fusion. Its expression is: In the formula, the weights It is determined by the product of three factors: contrast, saturation, and exposure quality. Step S1.4: Construct a multi-level image pyramid: Using Gaussian downsampling, an L-level pyramid is generated, the expression of which is: In the formula, Original image; For the first Image of a pyramid; The kernel weights are 5×5 Gaussian, satisfying the normalization condition. ; Step S1.5: Feature Extraction and Index Construction: A deep convolutional neural network is used to extract global feature vectors, and a vector index is built using a vector index library. The expression for extracting feature vectors is: In the formula, For feature extraction networks, For network parameters, The normalized 128-dimensional feature vector satisfies ; The deep convolutional neural network uses a ResNet34 network. The fully connected classification layer of the original network is removed, and the average pooling layer output up to the last convolutional block is retained to obtain a 512-dimensional feature vector. Then, a fully connected layer is used to reduce the dimensionality of the 512-dimensional feature vector to 128-dimensional. This fully connected layer contains 512 input neurons and 128 output neurons. The ReLU activation function is used for non-linear transformation, and the output 128-dimensional feature vector is subjected to L2 normalization.

3. The intelligent detection method for changes in the apparent state of a bridge based on images acquired by an inspection robot, as described in claim 2, is characterized in that... In step S2, the reference frame retrieval uses cosine similarity as a metric and performs an approximate nearest neighbor search in the vector index library. The expression for this search is: In the formula, As a benchmark image library, This is the feature vector of the current inspection image. for The transpose of , The feature vector of the reference image, The most similar reference frame obtained from the retrieval.

4. The intelligent detection method for bridge appearance changes based on images acquired by an inspection robot according to claim 3, characterized in that, Step S3 includes the following sub-steps: Step S3.1: Geometric Alignment: First, align the current inspection image... Extract local feature points and compare them with the baseline image. Perform feature matching to obtain an initial set of matching point pairs. ,in, As the reference image coordinates, The coordinates of the image to be aligned. This represents the total number of feature point pairs that were successfully matched between the two images; Represents the transpose of a matrix; Then, based on the coplanar constraint, the homography projection transformation equation is established, and its expression is: In the formula, Scale factor; It is a 3×3 homography matrix; Expanding the projection transformation equation into a linear form: Constructing an overdetermined system of equations using the least squares method: , Wherein, the coefficient matrix Each pair of matching points contributes two rows: parameter vector ; Next, robust estimation is performed using the RANSAC algorithm. The iterative process is as follows: Four sets of matching points are randomly selected to calculate the initial homography matrix. The interior point set is evaluated using the reprojection error, and its expression is: In the formula, Using homogeneous coordinates, This is a perspective division operation; like It is then determined to be an interior point. The threshold value is the interior point threshold. go through After several iterations, the set of largest interior points is selected for final least-squares optimization to obtain the optimal homography matrix. ; The current image is then projected onto the reference coordinate system through a cascaded transformation, the expression of which is: In the formula, These are the geometric parameters corresponding to the reference image; The optimal homography matrix between the current image and the reference image is obtained by using the reference mask. filter Invalid regions in; Step S3.2: Illumination Alignment: Using histogram matching, the geometrically aligned image is aligned. According to the lookup table of the reference image Radiation normalization is performed, and its expression is as follows: 。 5. The intelligent detection method for bridge appearance state changes based on images acquired by an inspection robot according to claim 4, characterized in that, Step S4 includes the following sub-steps: Step S4.1: Calculate the normalized image using the structural similarity index. Compared with the baseline image The difference diagram; where the expression for the structural similarity index is: In the formula, and Image blocks and The local mean, and These are the local variances, For covariance, and It is the stability constant. For pixel dynamic range, , ; Local statistics are obtained through a Gaussian weighted window. The calculation is expressed as follows: In the formula, Window size; The Gaussian weighting coefficients satisfy... ; The expression for the difference plot is defined as follows: ； Step S4.2: Adaptive threshold determination is achieved through sliding window statistics, marking candidate change regions; wherein, the adaptive threshold determination adopts the sliding window statistical method, dividing the difference map into sizes. For a local window, the local mean and standard deviation within the window are calculated using the following expression: The expression for the adaptive threshold is: In the formula, This is the threshold coefficient; Step S4.3: Candidate Change Region Generation: Based on online estimation using N days of historical unchanged samples, a 3D model is employed. Based on the principle or Bayesian update method, if the difference value of M consecutive windows in a certain region exceeds the corresponding threshold, the connected region formed by these consecutive windows is marked as a candidate change region. Through the connected component analysis algorithm, adjacent candidate windows are merged to form a complete region, and tiny regions with an area smaller than the minimum effective area threshold are eliminated. Finally, the bounding box coordinates and pixel range of the candidate change region are output.

6. The intelligent detection method for changes in the apparent state of a bridge based on images acquired by an inspection robot, as described in claim 5, is characterized in that... Step S5 includes the following sub-steps: Step S5.1: Multi-frame temporal consistency verification: Spatiotemporal consistency verification requires that the positions of the same region... Must be in continuous A change is considered real only if all candidate markers are triggered in the newly inspected frame. The expression for temporal consistency determination is: In the formula, and The first Frame difference map and threshold, ; For indicator functions; The expression for determining actual change is: In the formula, This is the time-series consistency threshold. or ; Step S5.2: Transient interference filtering: If a candidate region is in the... Frame-triggered change detection, but in the first frame... The frame is restored to a state consistent with the reference frame, that is: The region is then determined to be a transient disturbance and is not included in the actual changes. The final output is as follows: 1) Coordinates of the changing region polygon: Extract the contour of the changing region through connected component analysis and output the sequence of polygon vertex coordinates. ,in, The number of vertices; 2) Confidence score: Taking into account the intensity of differences, temporal consistency, and spatial continuity, its expression is: In the formula, This represents the average regional difference. The area of ​​the region of change; The minimum effective area threshold, These are the weighting coefficients; 3) Visual overlay: Overlays the changed area onto the original image as a semi-transparent overlay; 4) Event Log: Includes timestamp, location number, description of changes, and confidence level assessment.

7. A bridge appearance state change intelligent detection system based on images acquired by an inspection robot, implementing the process of the bridge appearance state change intelligent detection method based on images acquired by an inspection robot as described in any one of claims 1-6, characterized in that, The system includes a baseline establishment module, a baseline matching module, a double alignment module, a difference calculation module, and a spatiotemporal verification output module, which are executed sequentially. These modules work together to achieve intelligent detection of real changes in the appearance of the bridge. The benchmark establishment module performs geometric refinement, radiometric normalization, shadowless reference image generation, multi-level image pyramid construction, and feature index establishment on the completed inspection images, and outputs a benchmark image library containing geometric parameters, effective region masks, illumination correction lookup tables, and feature vector indexes. The benchmark matching module extracts the global feature vector of the current inspection image, performs an approximate nearest neighbor search in the feature index of the benchmark image library based on the cosine similarity metric, and outputs the most similar benchmark frame and its corresponding geometric parameters, effective region mask and illumination correction lookup table. The dual alignment module first achieves geometric alignment between the current inspection image and the reference frame through homography transformation, and then performs histogram matching based on the illumination correction lookup table of the reference frame to achieve illumination alignment, and outputs the normalized aligned image. The difference calculation module: uses a structural similarity index to calculate the difference map between the normalized image and the reference frame, implements adaptive threshold determination through sliding window statistics, and outputs candidate change regions; The spatiotemporal verification output module performs multi-frame temporal consistency verification and transient interference filtering on candidate change regions, and outputs the polygon coordinates, confidence scores, visual overlay plots, and event logs of the real change regions.

8. A computer system, characterized in that, include: The processor and a computer-readable storage medium communicatively connected to the processor, the computer-readable storage medium storing computer instructions that, when executed by the processor, implement the steps of the intelligent detection method for changes in the apparent state of a bridge based on images acquired by an inspection robot as described in any one of claims 1-6. The steps of the intelligent detection method for changes in the appearance of bridges based on images acquired by an inspection robot.

9. A computer program product, characterized in that, When the computer program / instructions are executed by the processor, they implement the steps of the intelligent detection method for changes in the apparent state of a bridge based on images acquired by an inspection robot, as described in any one of claims 1-6.

Images