Road and bridge expansion joint defect detection system based on image recognition

Abstract

The invention discloses a road and bridge expansion joint defect detection system based on image recognition. Real-time image data of a road and a bridge are acquired based on a cruise image acquisition sensor; the method comprises the following steps: performing coordinate system matching on real-time image data and a standard image, eliminating a visual angle difference by adopting an SIFT (Scale Invariant Feature Transform) feature point extraction algorithm and homography matrix transformation, comparing and aligning the real-time image data and the standard image through a double-encoder-decoder network, and judging whether a defect exists in a suspicious defect area or not by using a MobileNetV3 classification model. If the defects exist, a ResNet-50 model is adopted to classify the types and severity of the defects, and cross validation is carried out on defect detection results and vibration characteristics of the bridge and the road. The method can have high classification precision, provides an accurate decision basis for maintenance and management of roads and bridges, guarantees the detection efficiency, and improves the detection accuracy.

Description

Technical Field

[0001] The present invention relates to the technical field of image recognition, in particular to a road bridge expansion joint defect detection system based on image recognition. Background Art

[0002] Roads and bridges are key components of transportation infrastructure, and expansion joints are crucial structural features within them. They effectively accommodate deformation caused by temperature fluctuations and vehicle loads, preventing damage from stress concentration. Traditional methods for detecting defects in road and bridge expansion joints rely primarily on manual inspections. Inspectors inspect expansion joints using visual inspection and simple measuring tools (such as steel rulers and magnifying glasses). This method is inefficient and difficult to rapidly inspect on a large scale. Summary of the Invention

[0003] The purpose of the present invention is to solve the above problems and to design a road and bridge expansion joint defect detection system based on image recognition.

[0004] To achieve the above-mentioned purpose, the technical solution of the present invention is as follows: further, in the above-mentioned road and bridge expansion joint defect detection system based on image recognition, the road and bridge expansion joint defect detection system comprises: A real-time image acquisition module is used to acquire real-time image data of roads and bridges based on cruise image acquisition sensors; An image difference alignment module is used to match the coordinate system of the real-time image data with the standard image, and use the SIFT feature point extraction algorithm and homography matrix transformation to eliminate the perspective difference to obtain aligned real-time image data; A defect area extraction module is used to compare the aligned real-time image data with the standard image through a dual encoder-decoder network to generate difference patch data, and to remove interference areas of vehicles and shadows in the difference patch data in combination with the HRNet semantic segmentation model to obtain suspected defect areas; A defect result judgment module is used to use the MobileNetV3 classification model to determine whether the suspected defect area has a defect. If a defect exists, the ResNet-50 model is used to classify the defect type and severity to obtain a defect detection result; The defect result verification module is used to cross-verify the defect detection result with the vibration characteristics of the bridge and the road to obtain the target defect detection result.

[0005] Furthermore, in the above-mentioned road and bridge expansion joint defect detection system based on image recognition, the real-time image acquisition module includes the following submodules: The first path planning submodule is used to plan a round-trip bridge image acquisition path from the bridge head to the bridge tail in the bridge area according to the span and structural form of the bridge; The second path planning submodule is used to plan a road collection path near each expansion joint position along the driving direction of the road in the road area; The real-time image acquisition submodule is used to set the acquisition frame rate and parameters, acquire image data based on the round-trip bridge image acquisition path and the road acquisition path, and obtain corresponding positioning information and time information to obtain real-time image data.

[0006] Furthermore, in the above-mentioned road and bridge expansion joint defect detection system based on image recognition, the image difference alignment module includes the following submodules: The extreme point search submodule is used to construct a Gaussian pyramid of the image using the SIFT feature point extraction algorithm, generate blurred images of different scales by convolving Gaussian kernels of different scales with the image, and find extreme points in the blurred images to obtain candidate SIFT feature points; The key point acquisition submodule is used to accurately locate candidate SIFT feature points, determine the position and scale of key points by fitting a three-dimensional quadratic function, delete points with low contrast and points with strong edge response, and obtain key points; The gradient direction calculation submodule is used to take a 16×16 neighborhood window in the scale space where the key point is located with the key point as the center, calculate the gradient magnitude and gradient direction of each pixel in the window, and obtain a direction histogram; The real-time image alignment submodule is used to match the feature point set of the direction histogram with the feature point set of the standard image using a two-way matching method to obtain aligned real-time image data.

[0007] Furthermore, in the above-mentioned road and bridge expansion joint defect detection system based on image recognition, the defect area extraction module includes the following submodules: The model determines the submodule, which is used to use the encoder to extract high-level semantic features and low-level detail features of the image through multi-layer convolution and pooling operations; the decoder restores the feature map extracted by the encoder to the size of the original image and generates a difference map of the same size as the input image; The difference calculation submodule is used to input the aligned real-time image data and the standard image into the dual encoder-decoder network respectively, calculate the difference between the two in the feature space after the encoder extracts the features, and generate difference pattern data through the decoder.

[0008] Furthermore, in the above-mentioned road and bridge expansion joint defect detection system based on image recognition, the defect area extraction module includes the following submodules: The probability calculation submodule is used to input the generated difference patch data into the trained HRNet semantic segmentation model and obtain the probability that each pixel in the model output belongs to a different semantic category; The interference removal submodule is used to delete vehicles and shadows in the difference image data according to a preset category threshold, retain the expansion joint area with defects, and obtain the suspicious defect area.

[0009] Furthermore, in the above-mentioned road and bridge expansion joint defect detection system based on image recognition, the defect result judgment module includes the following submodules: The defect judgment submodule is used to input the suspected defect area into the trained MobileNetV3 model, use the model to output the probability of the existence of a defect in the area, and when the probability is greater than a set threshold, it is judged that the suspected defect area has a defect; The result classification submodule is used to determine if a defect exists and input it into the ResNet-50 model for defect type and severity classification. The specific type and severity of the defect are determined based on the maximum probability value to obtain the defect detection result.

[0010] Furthermore, in the above-mentioned road and bridge expansion joint defect detection system based on image recognition, the defect result verification module includes the following submodules: The frequency acquisition submodule is used to set cross-validation rules between vibration characteristics and image detection results, and collect vibration characteristic frequencies of bridges and roads through vibration sensors; The result verification submodule is used to verify the image detection results and the vibration characteristic frequency to obtain the final target defect detection results.

[0011] Furthermore, in the method for implementing the above-mentioned road and bridge expansion joint defect detection system based on image recognition, the method includes the following steps: Acquire real-time image data of roads and bridges based on cruise image acquisition sensors; Matching the coordinate system of the real-time image data with the standard image, using SIFT feature point extraction algorithm and homography matrix transformation to eliminate the perspective difference, and obtaining aligned real-time image data; The aligned real-time image data is compared with the standard image through a dual encoder-decoder network to generate difference patch data, and the interference areas of vehicles and shadows in the difference patch data are eliminated in combination with the HRNet semantic segmentation model to obtain suspicious defect areas; Use the MobileNetV3 classification model to determine whether the suspected defect area has defects. If defects exist, use the ResNet-50 model to classify the defect type and severity to obtain the defect detection results; The defect detection results are cross-validated with the vibration characteristics of the bridge and the road to obtain target defect detection results.

[0012] Furthermore, in the method for implementing the above-mentioned road and bridge expansion joint defect detection system based on image recognition, the method includes the following steps: The first path planning submodule is used to plan a round-trip bridge image acquisition path from the bridge head to the bridge tail in the bridge area according to the span and structural form of the bridge; The second path planning submodule is used to plan a road collection path near each expansion joint position along the driving direction of the road in the road area; The real-time image acquisition submodule is used to set the acquisition frame rate and parameters, acquire image data based on the round-trip bridge image acquisition path and the road acquisition path, and obtain corresponding positioning information and time information to obtain real-time image data.

[0013] Furthermore, in the method for implementing the above-mentioned road and bridge expansion joint defect detection system based on image recognition, the method includes the following steps: The extreme point search submodule is used to construct a Gaussian pyramid of the image using the SIFT feature point extraction algorithm, generate blurred images of different scales by convolving Gaussian kernels of different scales with the image, and find extreme points in the blurred images to obtain candidate SIFT feature points; The key point acquisition submodule is used to accurately locate candidate SIFT feature points, determine the position and scale of key points by fitting a three-dimensional quadratic function, delete points with low contrast and points with strong edge response, and obtain key points; The gradient direction calculation submodule is used to take a 16×16 neighborhood window in the scale space where the key point is located with the key point as the center, calculate the gradient magnitude and gradient direction of each pixel in the window, and obtain a direction histogram; The real-time image alignment submodule is used to match the feature point set of the direction histogram with the feature point set of the standard image using a two-way matching method to obtain aligned real-time image data.

[0014] Its beneficial effects are as follows: real-time image data of roads and bridges is acquired based on a cruise image acquisition sensor; the real-time image data is coordinate-matched with a standard image, and the SIFT feature point extraction algorithm and homography matrix transformation are used to eliminate perspective differences to obtain aligned real-time image data; the aligned real-time image data and the standard image are compared through a dual encoder-decoder network to generate difference patch data, and the interference areas of vehicles and shadows in the difference patch data are removed by combining the HRNet semantic segmentation model to obtain suspected defect areas; the MobileNetV3 classification model is used to determine whether the suspected defect areas have defects. If defects are present, the ResNet-50 model is used to classify the defect type and severity to obtain defect detection results; the defect detection results are cross-validated with the vibration characteristics of the bridge and road to obtain the target defect detection results. 1. It can accurately extract suspected defect areas from complex image data, effectively eliminate interference factors such as vehicles and shadows, and provide a reliable data foundation for subsequent defect classification. 2. Compared with traditional manual collection methods, it greatly improves the efficiency and comprehensiveness of data collection, reduces labor costs and time costs, and reduces the risk of inspection personnel working in dangerous environments. 3. High classification accuracy can provide accurate decision-making for road and bridge maintenance and management, ensuring both detection efficiency and accuracy. 4. It can effectively avoid misjudgments that may occur with a single detection method, further improving the credibility of the detection results. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Various other advantages and benefits will become apparent to those skilled in the art by reading the following detailed description of the preferred embodiment.The accompanying drawings are only for the purpose of illustrating the preferred embodiment and are not to be considered as limiting the present invention.

[0016] Figure 1 Schematic diagram of a first embodiment of a road and bridge expansion joint defect detection system based on image recognition in an embodiment of the present invention; Figure 2 Schematic diagram of a second embodiment of a road and bridge expansion joint defect detection system based on image recognition in an embodiment of the present invention; Figure 3 This is a schematic diagram of a third embodiment of a road and bridge expansion joint defect detection system based on image recognition in an embodiment of the present invention. DETAILED DESCRIPTION

[0017] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0018] Those skilled in the art will appreciate that, unless otherwise stated, the singular forms "a", "an", "" and "the" used herein may also include plural forms. It should be further understood that the term "comprising" used in the specification of the present invention refers to the presence of features, integers, steps, operations, elements and / or components, but does not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or groups thereof.

[0019] The present invention will be described in detail below with reference to the accompanying drawings. Figure 1 As shown, a road bridge expansion joint defect detection system based on image recognition includes: 101. A real-time image acquisition module, used to acquire real-time image data of roads and bridges based on a cruise image acquisition sensor; Specifically, this embodiment further includes a first path planning submodule for planning a round-trip bridge image acquisition path from the bridge head to the bridge tail in the bridge area according to the span and structural form of the bridge; The second path planning submodule is used to plan a road collection path near each expansion joint position along the driving direction of the road in the road area; The real-time image acquisition submodule is used to set the acquisition frame rate and parameters, collect image data based on the round-trip bridge image acquisition path and the road acquisition path, and obtain the corresponding positioning information and time information to obtain real-time image data.

[0020] 102. Image difference alignment module, used to match the coordinate system of real-time image data with the standard image, using SIFT feature point extraction algorithm and homography matrix transformation to eliminate the perspective difference, and obtain aligned real-time image data; Specifically, this embodiment also includes an extreme point search submodule for constructing a Gaussian pyramid of an image using a SIFT feature point extraction algorithm, generating blurred images of different scales by convolving Gaussian kernels of different scales with the image, and searching for extreme points in the blurred image to obtain candidate SIFT feature points. The key point acquisition submodule is used to accurately locate candidate SIFT feature points, determine the position and scale of key points by fitting a three-dimensional quadratic function, delete points with low contrast and points with strong edge response, and obtain key points; The gradient direction calculation submodule is used to take a 16×16 neighborhood window in the scale space where the key point is located, with the key point as the center, calculate the gradient magnitude and gradient direction of each pixel in the window, and obtain a direction histogram; The real-time image alignment submodule is used to match the feature point set of the direction histogram with the feature point set of the standard image using a two-way matching method to obtain aligned real-time image data.

[0021] 103. Defect area extraction module, which is used to compare and align real-time image data with standard images through a dual encoder-decoder network to generate difference patch data. HRNet semantic segmentation model is used to remove interference areas caused by vehicles and shadows in the difference patch data to obtain suspected defect areas. Specifically, this embodiment also includes a model determination submodule for extracting high-level semantic features and low-level detail features of the image through multi-layer convolution and pooling operations using the encoder; the decoder restores the feature map extracted by the encoder to the size of the original image and generates a difference map of the same size as the input image; The difference calculation submodule is used to input the aligned real-time image data and the standard image into the dual encoder-decoder network respectively, calculate the difference between the two in the feature space after the encoder extracts the features, and generate the difference pattern data through the decoder.

[0022] The probability calculation submodule is used to input the generated difference patch data into the trained HRNet semantic segmentation model and obtain the probability that each pixel in the model output belongs to a different semantic category; The interference removal submodule is used to delete vehicles and shadows in the difference image data according to a preset category threshold, retain the expansion joint area with defects, and obtain the suspicious defect area.

[0023] 104. Defect result judgment module, used to use MobileNetV3 classification model to determine whether there is a defect in the suspected defect area. If there is a defect, ResNet-50 model is used to classify the defect type and severity to obtain the defect detection result; Specifically, this embodiment also includes a defect judgment submodule for inputting a suspected defect area into a trained MobileNetV3 model, using the model to output the probability of a defect in the area, and when the probability is greater than a set threshold, it is determined that a defect exists in the suspected defect area; The result classification submodule is used to determine if a defect exists and input it into the ResNet-50 model for defect type and severity classification. The specific type and severity of the defect are determined based on the maximum probability value to obtain the defect detection result.

[0024] 105. The defect result verification module is used to cross-verify the defect detection results with the vibration characteristics of the bridge and the road to obtain the target defect detection results.

[0025] Specifically, this embodiment also includes a frequency acquisition submodule for setting cross-validation rules between vibration characteristics and image detection results, and collecting vibration characteristic frequencies in bridges and roads through vibration sensors; The result verification submodule is used to verify the image detection results and the vibration characteristic frequency to obtain the final target defect detection results.

[0026] Its beneficial effects include: 1. It can accurately extract suspected defect areas from complex image data, effectively eliminating interfering factors such as vehicles and shadows, and providing a reliable data foundation for subsequent defect classification. 2. Compared with traditional manual collection methods, it greatly improves the efficiency and comprehensiveness of data collection, reduces labor and time costs, and reduces the risk of inspectors working in dangerous environments. 3. Its high classification accuracy provides accurate decision-making for road and bridge maintenance and management, ensuring both detection efficiency and accuracy. 4. It can effectively avoid misjudgments that may occur with a single detection method, further improving the credibility of detection results.

[0027] See also Figure 2 In a DMTO device catalyst balance monitoring system based on rapid calculation, the real-time image acquisition module includes: The first path planning submodule is used to plan a round-trip bridge image acquisition path from the bridge head to the bridge tail in the bridge area according to the span and structural form of the bridge; The second path planning submodule is used to plan a road collection path near each expansion joint position along the driving direction of the road in the road area; The real-time image acquisition submodule is used to set the acquisition frame rate and parameters, collect image data based on the round-trip bridge image acquisition path and the road acquisition path, and obtain the corresponding positioning information and time information to obtain real-time image data.

[0028] See also Figure 3 In a DMTO device catalyst balance monitoring system based on fast accounting, the image difference alignment module includes: The extreme point search submodule is used to construct a Gaussian pyramid of the image using the SIFT feature point extraction algorithm. It generates blurred images of different scales by convolving Gaussian kernels of different scales with the image, and then searches for extreme points in the blurred image to obtain candidate SIFT feature points. The key point acquisition submodule is used to accurately locate candidate SIFT feature points, determine the position and scale of key points by fitting a three-dimensional quadratic function, delete points with low contrast and points with strong edge response, and obtain key points; The gradient direction calculation submodule is used to take a 16×16 neighborhood window in the scale space where the key point is located, with the key point as the center, calculate the gradient magnitude and gradient direction of each pixel in the window, and obtain a direction histogram; The real-time image alignment submodule is used to match the feature point set of the direction histogram with the feature point set of the standard image using a two-way matching method to obtain aligned real-time image data.

[0029] Specifically, the embodiment of the present invention can also be implemented in the following ways: 1. Real-time image data acquisition (1) Installation and debugging of cruise image acquisition sensor Slide rail bracket installation Selection of installation location: Based on the structural characteristics of roads and bridges, for bridges, priority should be given to locations such as the inspection passages on both sides of the bridge beam or the outside of the guardrails, which are stable and do not affect the normal stress and traffic flow of the bridge; for roads, it can be installed at appropriate locations on the shoulders or isolation belts on both sides of the road.

[0030] High-resolution camera installation and parameter setting Parameter Settings: Set the camera's exposure time, gain, frame rate, and other parameters based on the lighting conditions and image acquisition speed requirements of the inspection environment. During bright daylight hours, use shorter exposure times and lower gains to reduce image noise. During low light hours or on cloudy days, increase exposure time and gain appropriately to ensure moderate image brightness. The frame rate setting should match the pan / tilt head rotation speed and the slide rail movement speed to ensure continuous and uniform image acquisition during camera movement. A frame rate of at least 15 frames per second is recommended.

[0031] PTZ installation and control PTZ: Choose a high-precision, low-noise motorized PTZ that supports 360° continuous horizontal rotation and ±90° vertical rotation, with automatic homing and preset settings. The PTZ's load-bearing capacity should be greater than the combined weight of the camera and accessories to ensure it can stably support the camera's rotation.

[0032] Positioning device installation and calibration Positioning device selection: A combined positioning system, including a GPS module and an inertial navigation system (INS), is used to achieve high-precision positioning. The GPS module is used to obtain global position information, while the INS is used to provide continuous position and attitude information in areas where GPS signals are blocked or weak (such as under bridges and in tunnels).

[0033] (2) Image acquisition process Inspection Route Planning: Develop detailed inspection routes based on the layout of roads and bridges to ensure coverage of all expansion joints requiring inspection. For bridges, a round-trip inspection route is planned from the bridgehead to the bridgehead based on the bridge's span and structural form. For roads, appropriate collection paths are planned near each expansion joint, following the direction of travel, to avoid missing critical areas.

[0034] 2. Image Data Coordinate System Matching and Perspective Difference Elimination (1) Standard image acquisition and preprocessing Standard image acquisition: When the expansion joint is in good condition, use the same equipment and parameters as for real-time image acquisition to capture images of the expansion joint from multiple angles (such as the front, 45° angles to the left and right), and positions (different distances, different heights) to establish a standard image library. The number of standard images for each expansion joint should be no less than 10 to cover its condition from different perspectives.

[0035] Preprocessing steps: Preprocess the collected standard images, including image denoising (using median filtering, Gaussian filtering, etc. to remove noise in the image), image enhancement (using histogram equalization, etc. to improve the contrast of the image), image cropping (removing irrelevant background areas in the image and retaining only the area of ​​interest where the expansion joint is located) and grayscale conversion (converting color images into grayscale images to reduce the amount of computation required for subsequent processing).

[0036] (2) SIFT feature point extraction and matching SIFT feature point extraction process Scale-space extrema detection: A Gaussian pyramid is constructed for the image. Blurred images at different scales are generated by convolving the image with Gaussian kernels of different scales. Gaussian blurred images at adjacent scales are subtracted to form a Difference of Gaussian pyramid (DOG pyramid). In the DOG pyramid, each pixel is compared with 26 pixels at adjacent scales (8 neighbors at the current scale, 9 neighbors at the previous scale, and 9 neighbors at the next scale) to identify extreme points in scale space, which serve as candidate SIFT feature points.

[0037] Keypoint Localization: Accurately locate candidate extreme points, fitting a three-dimensional quadratic function to determine their location and scale. Low-contrast points and those with strong edge responses are removed to improve the stability and reliability of feature points. Specifically, the contrast value of a keypoint is calculated. If the contrast value falls below a set threshold (0.04), the keypoint is removed. Furthermore, the determinant and trace of the Hessian matrix are calculated to determine if the keypoint is located on an edge. Keypoints with strong edge responses are removed (setting an edge response threshold, such as 10).

[0038] Direction assignment: Centered on the keypoint, a 16×16 neighborhood window is taken in the scale space where the keypoint resides. The gradient magnitude and gradient direction of each pixel within the window are calculated. The gradient magnitude is weighted using a Gaussian weighting function to enhance the contribution of pixels near the keypoint. The gradient direction is divided into eight direction intervals, and the sum of the gradient magnitudes within each interval is calculated to form an 8-dimensional direction histogram. The direction of the histogram's peak is the primary direction of the keypoint. The direction of the second-largest peak is also recorded as the secondary direction of the keypoint to improve the rotational invariance of the feature point.

[0039] Feature point matching: A two-way matching method is used, that is, matching is performed from the feature point set of the real-time image to the feature point set of the standard image, and then reverse matching is performed from the feature point set of the standard image to the feature point set of the real-time image. Only feature point pairs that are successfully matched in both directions are retained to reduce the number of false matches.

[0040] (3) Homography matrix transformation Homography Matrix Calculation: Based on successfully matched feature point pairs, the least squares method is used to solve the homography matrix. The homography matrix is ​​a 3×3 non-singular matrix that describes the projective transformation relationship between two planes. Since at least four pairs of non-collinear feature points are required to solve the homography matrix, when the number of matched feature point pairs exceeds four, RANSAC (Random Sample Consensus Algorithm) is used for robust estimation to eliminate the influence of mismatched point pairs and improve the calculation accuracy of the homography matrix.

[0041] Image Transformation: Using the calculated homography matrix, the live image is transformed into the same coordinate system as the standard image to eliminate the difference in viewing angle. During the transformation process, the pixel values ​​of the transformed image are interpolated using a bilinear interpolation algorithm to ensure smoothness and continuity.

[0042] 3. Extraction of Suspected Defect Areas (1) Dual Encoder-Decoder Network Construction and Difference Pattern Generation Network Architecture Design: The dual encoder-decoder network consists of two symmetrical encoders and decoders, one for processing live images and the other for processing standard images. The encoder uses a deep convolutional neural network (VGGNet, the first few layers of ResNet) to extract high-level semantic features and low-level detail features of the image through multiple layers of convolution and pooling. The decoder uses deconvolution, upsampling, and skip connections to gradually restore the feature maps extracted by the encoder to the original image size, generating a difference map of the same size as the input image.

[0043] Network training: Labeled images of intact and defective expansion joints are used as training data, with the mean squared error (MSE) used as the loss function. The network parameters are optimized using a backpropagation algorithm. During training, data augmentation techniques (such as random rotation, flipping, scaling, and noise addition) are used to expand the training data to improve the network's generalization capabilities.

[0044] Difference patch generation: The aligned live image and standard image are fed into a dual encoder-decoder network. After feature extraction by the encoder, the difference between the two is calculated in feature space. The decoder then generates difference patch data. In the difference patch data, larger pixel values ​​indicate a greater difference between the live image and the standard image in that area, indicating a possible defect.

[0045] (2) HRNet semantic segmentation model removes interference areas Model Training: The HRNet (High-Resolution Network) semantic segmentation model maintains high-resolution feature maps, enabling accurate segmentation of different semantic regions within an image. The HRNet model was trained using a dataset containing annotations of vehicles, shadows, and expansion joints. The dataset was obtained by manually annotating images collected from actual inspection scenarios. The annotation categories included vehicles, shadows, expansion joint backgrounds, and suspected defect areas.

[0046] Interference Region Elimination: The generated difference image data is fed into the trained HRNet semantic segmentation model, which outputs the probability of each pixel belonging to a different semantic category. Based on a preset category threshold (vehicle and shadow categories are considered interference regions when their probabilities are greater than 0.8), vehicle and shadow interference regions are eliminated from the difference image data, retaining only the expansion joints that may contain defects, resulting in the identification of suspected defect regions.

[0047] 4. Defect Classification and Detection Result Generation (1) Determining the Existence of Defects in the MobileNetV3 Lightweight Classification Model Model Advantages and Structure: MobileNetV3 is a lightweight convolutional neural network that utilizes depthwise separable convolutions and an efficient network structure. It boasts low computational overhead and fast inference speed, making it suitable for rapidly determining the presence of defects in suspected defect areas in embedded devices or real-time detection systems. The model structure comprises an input layer, multiple convolutional blocks (each of which includes depthwise separable convolutions, activation functions, batch normalization, and other operations), a global average pooling layer, and a fully connected layer.

[0048] Training Data and Methods: Images of suspected defect areas are extracted as training data, which are classified into two categories: defective and non-defective. Training data is obtained by cropping and labeling defective and normal areas from historical inspection data. Data augmentation is also performed to increase the data volume.

[0049] Defect Presence Determination: Images of suspected defect areas, processed by the HRNet semantic segmentation model, are fed into the trained MobileNetV3 model. The model outputs the probability of a defect in that area. If the probability is greater than the set threshold (0.5), the suspected defect area is considered defective and proceeds to the next step of defect type and severity classification. Otherwise, the area is considered non-defective.

[0050] (2) Classification of defect types and severity of the ResNet-50 high-precision model Model Features and Structure: ResNet-50 (ResidualNetwork-50) is a 50-layer residual neural network. By introducing residual connections, it solves the vanishing gradient problem in deep neural network training. It can extract deep semantic features of images and achieve high classification accuracy. The model structure includes an input layer, multiple residual blocks (each residual block contains a convolutional layer, a batch normalization layer, and residual connections), a global average pooling layer, and a fully connected layer. The number of output nodes in the fully connected layer is the sum of the number of defect types and severity categories (cracks, rust, and misalignment, each with three severity levels: mild, moderate, and severe, for a total of nine categories).

[0051] Training Strategy: The ResNet-50 model is trained using a high-quality, annotated dataset containing a variety of defect types and severities. This dataset is derived from defect images captured during actual inspections and carefully annotated by professionals to ensure the accuracy of the defect type and severity for each sample.

[0052] Classification process: Images of suspected areas identified as defective by the MobileNetV3 model are fed into the ResNet-50 model for defect type and severity classification. The model outputs a probability distribution for each defect type and severity. Based on the maximum probability, the model determines the defect type (crack, corrosion, misalignment) and severity (mild, moderate, severe), generating the defect detection results.

[0053] 5. Vibration Feature Cross-Validation (1) Vibration feature collection Sensor installation: Install vibration sensors, such as accelerometers or displacement sensors, near the expansion joints of roads and bridges. The sensors should be installed on both sides of the expansion joint, no more than 0.5 meters from the edge of the expansion joint.

[0054] Acquisition parameter settings: Set the sensor's acquisition frequency, acquisition time, and range based on the expansion joint's vibration characteristics and testing requirements. The acquisition frequency should be no less than 100Hz to ensure the capture of high-frequency components of the vibration signal. The acquisition time should be determined based on the vehicle's speed and the length of the expansion joint.

[0055] Signal preprocessing: The collected vibration signals are preprocessed, including DC drift removal (using a high-pass filter to remove the DC component in the signal), denoising (using a band-pass filter to remove external interference noise), and signal normalization (normalizing the signal amplitude to the range of [-1, 1]) to improve signal quality and usability.

[0056] (2) Cross-validation method Verification rule setting: Based on historical data and actual engineering experience, cross-validation rules are set for vibration signatures and image inspection results. For example, if image inspection results indicate a crack defect in an expansion joint with a moderate severity, the corresponding vibration signal displacement root mean square peak value should be greater than the set threshold A, and the spectrum amplitude within a specific frequency range (5-10Hz) should be greater than threshold B. If these conditions are not met, the image inspection results are revised and the presence, type, and severity of the defect are reassessed.

[0057] Result Fusion: The image inspection results and vibration signature verification results are combined to obtain the final target defect detection result. This fusion strategy uses a weighted voting method, assigning different weights based on the reliability of image inspection and vibration signature verification. When the two results are consistent, the final result is directly determined. If the two results are inconsistent, further analysis and testing (such as increasing the number of tests or using other detection methods) are used to determine the final result.

[0058] The above shows and describes the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The above embodiments and descriptions are merely preferred examples of the present invention and are not intended to limit the present invention. Various changes and improvements may be made to the present invention without departing from the spirit and scope of the present invention, and all such changes and improvements fall within the scope of the present invention. The scope of protection claimed in the present invention is defined by the appended claims and their equivalents.

Claims

1. The road and bridge expansion joint defect detection system based on image recognition is characterized by: The road bridge expansion joint defect detection system includes: A real-time image acquisition module is used to acquire real-time image data of roads and bridges based on cruise image acquisition sensors; An image difference alignment module is used to match the coordinate system of the real-time image data with the standard image, and use the SIFT feature point extraction algorithm and homography matrix transformation to eliminate the perspective difference to obtain aligned real-time image data; A defect area extraction module is used to compare the aligned real-time image data with the standard image through a dual encoder-decoder network to generate difference patch data, and to remove interference areas of vehicles and shadows in the difference patch data in combination with the HRNet semantic segmentation model to obtain suspected defect areas; A defect result judgment module is used to use the MobileNetV3 classification model to determine whether the suspected defect area has a defect. If a defect exists, the ResNet-50 model is used to classify the defect type and severity to obtain a defect detection result; The defect result verification module is used to cross-verify the defect detection result with the vibration characteristics of the bridge and the road to obtain the target defect detection result.

2. The road and bridge expansion joint defect detection system based on image recognition according to claim 1 is characterized in that: The real-time image acquisition module includes the following submodules: The first path planning submodule is used to plan a round-trip bridge image acquisition path from the bridge head to the bridge tail in the bridge area according to the span and structural form of the bridge; The second path planning submodule is used to plan a road collection path near each expansion joint position along the driving direction of the road in the road area; The real-time image acquisition submodule is used to set the acquisition frame rate and parameters, acquire image data based on the round-trip bridge image acquisition path and the road acquisition path, and obtain corresponding positioning information and time information to obtain real-time image data.

3. The road and bridge expansion joint defect detection system based on image recognition according to claim 1 is characterized in that: The image difference alignment module includes the following submodules: The extreme point search submodule is used to construct a Gaussian pyramid of the image using the SIFT feature point extraction algorithm, generate blurred images of different scales by convolving Gaussian kernels of different scales with the image, and find extreme points in the blurred images to obtain candidate SIFT feature points; The key point acquisition submodule is used to accurately locate candidate SIFT feature points, determine the position and scale of key points by fitting a three-dimensional quadratic function, delete points with low contrast and points with strong edge response, and obtain key points; The gradient direction calculation submodule is used to take a 16×16 neighborhood window in the scale space where the key point is located with the key point as the center, calculate the gradient magnitude and gradient direction of each pixel in the window, and obtain a direction histogram; The real-time image alignment submodule is used to match the feature point set of the direction histogram with the feature point set of the standard image using a two-way matching method to obtain aligned real-time image data.

4. The road and bridge expansion joint defect detection system based on image recognition according to claim 1 is characterized in that: The defect area extraction module includes the following submodules: The model determines the submodule, which is used to extract high-level semantic features and low-level detail features of the image through multi-layer convolution and pooling operations using the encoder; The decoder restores the feature map extracted by the encoder to the size of the original image and generates a difference map of the same size as the input image; The difference calculation submodule is used to input the aligned real-time image data and the standard image into the dual encoder-decoder network respectively, calculate the difference between the two in the feature space after the encoder extracts the features, and generate difference pattern data through the decoder.

5. The road and bridge expansion joint defect detection system based on image recognition according to claim 1 is characterized in that: The defect area extraction module also includes the following submodules: The probability calculation submodule is used to input the generated difference patch data into the trained HRNet semantic segmentation model and obtain the probability that each pixel in the model output belongs to a different semantic category; The interference removal submodule is used to delete vehicles and shadows in the difference image data according to a preset category threshold, retain the expansion joint area with defects, and obtain the suspicious defect area.

6. The road and bridge expansion joint defect detection system based on image recognition according to claim 1 is characterized in that: The defect result judgment module includes the following submodules: The defect judgment submodule is used to input the suspected defect area into the trained MobileNetV3 model, use the model to output the probability of the existence of a defect in the area, and when the probability is greater than a set threshold, it is judged that the suspected defect area has a defect; The result classification submodule is used to determine if a defect exists and input it into the ResNet-50 model for defect type and severity classification. The specific type and severity of the defect are determined based on the maximum probability value to obtain the defect detection result.

7. The road and bridge expansion joint defect detection system based on image recognition according to claim 1 is characterized in that: The defect result verification module includes the following submodules: The frequency acquisition submodule is used to set cross-validation rules between vibration characteristics and image detection results, and collect vibration characteristic frequencies of bridges and roads through vibration sensors; The result verification submodule is used to verify the image detection results and the vibration characteristic frequency to obtain the final target defect detection results.

8. A method for implementing the road and bridge expansion joint defect detection system based on image recognition as claimed in claim 1, characterized in that: The method comprises the following steps: Acquire real-time image data of roads and bridges based on cruise image acquisition sensors; Matching the coordinate system of the real-time image data with the standard image, using SIFT feature point extraction algorithm and homography matrix transformation to eliminate the perspective difference, and obtaining aligned real-time image data; The aligned real-time image data is compared with the standard image through a dual encoder-decoder network to generate difference patch data, and the interference areas of vehicles and shadows in the difference patch data are eliminated in combination with the HRNet semantic segmentation model to obtain suspicious defect areas; Use the MobileNetV3 classification model to determine whether the suspected defect area has defects. If defects exist, use the ResNet-50 model to classify the defect type and severity to obtain the defect detection results; The defect detection results are cross-validated with the vibration characteristics of the bridge and the road to obtain target defect detection results.

9. A method for implementing the road and bridge expansion joint defect detection system based on image recognition as claimed in claim 1, characterized in that: The method comprises the following steps: The first path planning submodule is used to plan a round-trip bridge image acquisition path from the bridge head to the bridge tail in the bridge area according to the span and structural form of the bridge; The second path planning submodule is used to plan a road collection path near each expansion joint position along the driving direction of the road in the road area; The real-time image acquisition submodule is used to set the acquisition frame rate and parameters, acquire image data based on the round-trip bridge image acquisition path and the road acquisition path, and obtain corresponding positioning information and time information to obtain real-time image data.

10. A method for implementing the road and bridge expansion joint defect detection system based on image recognition as claimed in claim 1, characterized in that: The method comprises the following steps: The extreme point search submodule is used to construct a Gaussian pyramid of the image using the SIFT feature point extraction algorithm, generate blurred images of different scales by convolving Gaussian kernels of different scales with the image, and find extreme points in the blurred images to obtain candidate SIFT feature points; The key point acquisition submodule is used to accurately locate candidate SIFT feature points, determine the position and scale of key points by fitting a three-dimensional quadratic function, delete points with low contrast and points with strong edge response, and obtain key points; The gradient direction calculation submodule is used to take a 16×16 neighborhood window in the scale space where the key point is located with the key point as the center, calculate the gradient magnitude and gradient direction of each pixel in the window, and obtain a direction histogram; The real-time image alignment submodule is used to match the feature point set of the direction histogram with the feature point set of the standard image using a two-way matching method to obtain aligned real-time image data.

Images