Highway pavement damage self-recognition method and system based on deep learning

Abstract

The application discloses a highway pavement damage self-recognition method and system based on deep learning, and relates to the technical field of road damage detection. The method comprises the following steps: continuously collecting images along the highway to construct an original pavement image sequence; performing multi-scale feature analysis on the sequence to draw a feature map; traversing the feature map to detect and generate a candidate damage area, and obtaining an initial recognition result through bidirectional analysis; constructing a pavement three-dimensional space based on three-dimensional mapping of the image sequence, synchronously positioning the result, and determining the damage spatial position; and generating a pavement condition report through position information inversion verification. The technical problem that the existing pavement damage detection is easily affected by light, noise and scale changes in the continuous collection scene, resulting in missed detection and false detection of the damage area, and cannot accurately position, is solved, the technical effects of improving the accuracy of damage recognition through multi-scale feature analysis and bidirectional verification of the candidate area, and realizing accurate positioning and automatic evaluation of the damage through three-dimensional space mapping and inversion verification, are achieved.

Description

Technical Field

[0001] This invention relates to the field of road damage detection technology, specifically to a method and system for self-identification of highway pavement damage based on deep learning. Background Technology

[0002] The health of highway pavements directly affects driving safety, ride comfort, and road lifespan. Traditional pavement damage detection mainly relies on manual visual inspection or semi-automated inspection vehicles. These methods suffer from low efficiency, high subjectivity, poor safety, and difficulty in data quantification and archiving, and can no longer meet the high-frequency, high-precision maintenance and management needs of modern large-scale road networks. With technological advancements, automated visual inspection technology based on vehicle-mounted cameras has gradually become a research hotspot. Early automated methods relied heavily on digital image processing techniques, such as threshold segmentation, edge detection, and texture analysis, to identify damage through predefined rules or features. However, these methods are extremely sensitive to changes in ambient lighting, pavement stains, and shadow interference, exhibiting poor robustness, high false positive and false negative rates, and difficulty in distinguishing fine-grained damage categories.

[0003] In recent years, some deep learning-based methods have been able to learn and identify damage from images end-to-end, significantly outperforming traditional methods. However, most current solutions still have significant limitations: First, they typically only perform identification at the two-dimensional image level, lacking the ability to accurately locate damage in the real-world three-dimensional space, and cannot accurately obtain key geometric information such as the length, width, and depth of the damage, which is the core basis for making scientific maintenance decisions. Second, when faced with targets with varied shapes and large scale differences, such as cracks, single-scale feature extraction networks are prone to missing small-scale damage or inaccurately locating large-scale damage. Furthermore, the identification process is mostly unidirectional forward reasoning, lacking a closed-loop verification mechanism for the reliability and consistency of the identification results. False alarms may occur in complex scenarios, leading to insufficient reliability of the detection results and making it difficult to directly apply them to actual maintenance decisions. Summary of the Invention

[0004] This application provides a method and system for self-identification of road surface damage based on deep learning, which solves the technical problem that existing road surface damage detection is easily affected by lighting, noise and scale changes in continuous acquisition scenarios, resulting in missed detection and false detection of damaged areas, and inability to accurately locate them.

[0005] The first aspect of this application provides a deep learning-based method for self-identification of road surface damage, the method comprising:

[0006] Images are continuously acquired along the direction of highway travel to construct an original road surface image sequence. Multi-scale feature analysis is performed on the original road surface image sequence to generate a multi-scale feature map. Region detection is performed by traversing the multi-scale feature map to generate multiple candidate damage regions for bidirectional analysis, obtaining initial damage identification results. Three-dimensional mapping is performed based on the original road surface image sequence to construct an original road surface three-dimensional space. The initial damage identification results are synchronized to the original road surface three-dimensional space for localization to determine the spatial location information of the damage. Inversion verification is performed according to the damage spatial location information. If the verification passes, a highway road surface condition report is constructed.

[0007] A second aspect of this application provides a deep learning-based self-identification system for highway pavement damage, the system comprising:

[0008] Image acquisition unit: continuously acquires images along the road travel direction to construct an original road surface image sequence; Feature analysis unit: performs multi-scale feature analysis based on the original road surface image sequence to draw a multi-scale feature map; Region detection unit: traverses the multi-scale feature map to perform region detection, generates multiple candidate damage regions for bidirectional analysis, and obtains initial damage identification results; Damage localization unit: performs three-dimensional mapping based on the original road surface image sequence to construct an original road surface three-dimensional space, synchronizes the initial damage identification results to the original road surface three-dimensional space for localization, and determines the spatial location information of the damage; Inversion verification unit: performs inversion verification according to the damage spatial location information, and when the verification is successful, constructs a road surface condition report.

[0009] One or more technical solutions provided in this application have at least the following technical effects or advantages:

[0010] First, road surface images are continuously acquired during highway driving, forming a complete road surface image sequence. Then, multi-scale feature extraction is performed on the image sequence to construct feature maps reflecting damage characteristics at different scales. Next, region detection is performed on the multi-scale feature maps to screen and analyze potential road surface damage areas, obtaining preliminary damage identification results through bidirectional analysis. Then, three-dimensional modeling is performed using the continuous image sequence, mapping the identified damage results onto the three-dimensional space of the road surface to achieve precise spatial localization of the damage. Finally, inversion verification is performed based on the spatial location information of the damage to ensure the accuracy of the detection results. After verification, a detailed road surface condition assessment report is automatically generated, providing a reliable basis for maintenance decisions. Attached Figure Description

[0011] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0012] Figure 1 A schematic diagram of the process for a deep learning-based self-identification method for road surface damage provided in this application embodiment.

[0013] Figure 2 A schematic diagram of the structure of a deep learning-based highway pavement damage self-identification system provided in an embodiment of this application.

[0014] Figure labeling: Image acquisition unit 11, feature analysis unit 12, region detection unit 13, damage localization unit 14, inversion verification unit 15. Detailed Implementation

[0015] To further illustrate the technical means and effects of the present invention in achieving its intended purpose, the following detailed description of the specific implementation methods, structures, features, and effects of the present invention, in conjunction with the accompanying drawings and preferred embodiments, is provided below.

[0016] Example 1, as Figure 1 As shown, this application provides a deep learning-based method for self-identification of highway pavement damage, wherein the method includes:

[0017] Images are continuously acquired along the direction of travel on the highway to construct an original road surface image sequence.

[0018] In this embodiment, forward-looking or downward-looking industrial cameras, high-definition cameras, etc., are fixedly installed on the inspection vehicle and connected to supplementary lighting, speed measuring devices, and positioning devices. Subsequently, as the inspection vehicle travels at a constant speed along the road, the cameras are continuously triggered to acquire road surface image frames according to a preset sampling frequency. The sampling frequency is determined based on the vehicle speed and the desired spatial resolution, ensuring that adjacent frames have a certain overlap in the road surface coverage area. Each image frame simultaneously records a timestamp, vehicle speed, mileage information, and positioning information during acquisition, which are stored along with the image as metadata. The acquired image frames are numbered and cached in chronological order. When the cache reaches a set number of frames or the acquisition of a specified road segment is completed, the consecutively numbered image frames are sequentially stitched together to output an original road surface image sequence. This provides a complete and reliable raw data foundation for subsequent feature analysis and damage identification, ensuring the accuracy and feasibility of the entire identification process.

[0019] Multi-scale feature analysis is performed based on the original road surface image sequence to draw multi-scale feature maps.

[0020] In one embodiment, images are read from the original road surface image sequence at fixed intervals or frame by frame. Each frame is preprocessed, including grayscale conversion and histogram equalization to reduce the impact of illumination differences, and the images are uniformly scaled to a preset input size to obtain a standard image sequence. Subsequently, a multi-scale sliding window is set on each frame image to decompose the same frame into multiple image blocks of different scales. After normalization of each scale image block, it is fed into a backbone network to extract features. This backbone network consists of multiple convolutional stages. Each convolutional stage downsamples the input image blocks level by level through convolution, batch normalization, and nonlinear activation, outputting feature layers of different resolutions, and recording the feature space size parameters corresponding to each feature layer. After obtaining the multi-level original feature layers, channel transformation is performed on the lower-level feature layers to align the number of channels, while deep semantic analysis is performed on the higher-level feature layers and upsampling is used to increase their resolution to be consistent with the lower-level feature layers. Finally, the upsampled high-level features are concatenated or weighted with the corresponding low-level features at the channel dimension to form a fused feature layer that contains both fine-grained texture information and high-level semantic information. The fused feature layers at multiple scales are then combined to form the output, which is a multi-scale feature map for subsequent detection. This multi-scale feature map integrates feature information at different scales and can identify damages of different sizes and shapes in the image in subsequent damage detection, ensuring the comprehensiveness and efficiency of highway pavement detection.

[0021] Furthermore, based on the original road surface image sequence, multi-scale feature analysis is performed to draw multi-scale feature maps. The method includes:

[0022] The original road surface image sequence is traversed to extract multiple image frames, which are then subjected to grayscale equalization processing to generate a standard image sequence. A multi-scale sliding window is set, and the standard image sequence is analyzed at multiple scales through the multi-scale sliding window to determine multiple scale image blocks. A backbone network is constructed, which contains multiple convolutional stages. According to the multiple convolutional stages, the backbone network is activated to downsample the multiple scale image blocks to determine the feature space size parameters. Based on the feature space size parameters, feature transformation is performed to construct multi-level original feature maps. Based on the backbone network, deep semantic analysis is performed to generate high semantic feature maps, which are then upsampled to generate high-level feature maps. The high-level feature maps are then element-wise concatenated and fused with the multi-level original features to construct a multi-scale feature map.

[0023] Preferably, the original road surface image sequence is first traversed according to the image acquisition order, and the road surface image data is read frame by frame. Grayscale processing is performed on each frame, converting the three-channel color image into a single-channel grayscale image. Then, histogram equalization is performed on the grayscale image. This histogram equalization can use an adaptive histogram equalization algorithm (CLAHE), where the block size can be set to 8×8 and the contrast limiting factor can be set to 2.0 to suppress excessively enhanced noise. After the above processing, the images are uniformly scaled to a preset input size, such as 640×640 pixels, to obtain a standard image sequence. Subsequently, based on the size differences of different road surface damage at different spatial scales, multiple sets of sliding window parameters are preset for multi-scale analysis. This multi-scale sliding window includes at least three scales: 32×32 pixels, 64×64 pixels, and 128×128 pixels. The sliding step size can be set to half the corresponding window size to ensure overlapping areas between adjacent image blocks. By sliding a multi-scale sliding window frame-by-frame across a standard image sequence, multiple image patches of different scales are cropped to represent different forms of pavement damage, such as fine cracks, moderate damage, and large-area defects. Next, a backbone network for feature extraction is constructed. This backbone network is a deep convolutional neural network structure composed of multiple cascaded convolutional stages, typically containing 4 to 5 convolutional stages. Each convolutional stage includes at least one convolutional layer, one batch normalization layer, and one non-linear activation layer. Specifically, the kernel size of the first convolutional stage can be set to 3×3, with 64 output channels and a stride of 1; the second convolutional stage can have 128 output channels and a stride of 2; the third convolutional stage can have 256 output channels and a stride of 2; and the fourth convolutional stage can have 512 output channels and a stride of 2, thus achieving progressive downsampling.

[0024] After construction, image patches of multiple scales are input into the backbone network. Feature extraction and downsampling are performed sequentially according to the order of convolution stages, and corresponding feature maps are output at each convolution stage. Taking an input size of 640×640 pixels as an example, the output feature map size after the first convolution stage is 640×640, after the second convolution stage it is 320×320, after the third convolution stage it is 160×160, and after the fourth convolution stage it is 80×80. By recording the spatial size parameters of the output at each stage, the feature space size parameters can be obtained. After obtaining the feature space size parameters of the output at each convolution stage, feature transformation processing is performed on the feature maps of each layer based on the feature space size parameters. This includes unifying the number of channels of different layers of feature maps through 1×1 convolution, for example, unifying them to 256 channels, and forming multi-level original feature maps. The lower-level original feature maps are used to preserve the details of road surface texture, while the higher-level original feature maps are used to express more abstract damage structure features. Then, deep semantic analysis is performed using the high-level output features of the backbone network.

[0025] Specifically, the high-level output features are used as input feature tensors to represent the overall structure and contextual information of the road surface area. Multiple stacked convolutional units then enhance these high-level output features. Each convolutional unit includes at least a 3×3 convolutional layer, a batch normalization layer, and a non-linear activation function, either ReLU or Leaky ReLU. Continuous convolution operations are performed on the high-level output features through these units to expand the receptive field and enhance the response intensity of damaged areas in the high-level semantic space. Next, to enhance the ability of high-level features to distinguish between damaged and undamaged areas of the road surface, a context aggregation operation can be introduced after the convolutional units to integrate a wider range of spatial semantic information. This context aggregation operation is achieved by setting dilated convolutional layers with different dilation rates, such as 2, 4, or 8, to perform parallel convolution processing on the same high-level output features. The parallel output results are then summed or concatenated element-wise, thereby expanding the semantic receptive range without significantly increasing computational cost. After context aggregation, the obtained high-level features undergo channel compression or channel reweighting to highlight semantic channels related to pavement damage. This channel compression is achieved through 1×1 convolution, reducing the number of channels from 512 or 1024 to 256. Alternatively, global average pooling and channel weight calculation are used to redistribute the importance of each channel, giving damage-related channels higher weight responses. After the above multi-level convolution enhancement, context aggregation, and channel reconstruction processes, a high semantic feature map with strong semantic expressive power is obtained. This high semantic feature map retains the overall perception ability of the damaged area in the spatial dimension and can effectively distinguish different types of pavement damage in the semantic dimension.

[0026] After obtaining the high semantic feature map, scale restoration is performed on the high semantic feature map using upsampling. This upsampling can be achieved through bilinear interpolation or deconvolution operations, restoring the spatial resolution of the high semantic feature map to 160×160 or 320×320 pixels, respectively, generating the corresponding high-level feature map. Finally, the high-level feature map and the multi-level original feature map with the same spatial size are element-wise concatenated and fused along the channel dimension. The number of channels in the concatenated feature map can be 512 or 768, and channel compression is performed through 1×1 convolution to obtain the final fused feature layer. The fused feature layers obtained from multiple scales are then combined to form the output, constituting a multi-scale feature map for subsequent region detection. This multi-scale feature map simultaneously takes into account the fine-grained texture information and high-level semantic information of road damage, which can improve the representation ability and detection accuracy of road damage of different scales and morphologies.

[0027] The multi-scale feature map is traversed to perform region detection, generate multiple candidate damage regions for bidirectional analysis, and obtain initial damage identification results.

[0028] In one embodiment, after obtaining the multi-scale feature maps, the multi-scale feature maps are traversed, and the traversed feature maps are input into the region detection process in a hierarchical manner. In this process, multi-scale reference anchor boxes are pre-set with each grid point of each layer of the feature map as the center, such as an area scale of 16² and an aspect ratio of 1:1. A score for each anchor box belonging to the damage foreground is calculated, and the coordinate offset of the anchor box is output simultaneously for regression correction of the bounding box position. Subsequently, all anchor boxes are sorted in descending order of foreground score, and anchor boxes below a preset threshold are removed. Non-maximum suppression (NMS) is performed on the remaining anchor boxes to remove highly overlapping boxes, thereby obtaining multiple candidate damage regions. Then, bidirectional analysis is performed on each candidate damage region. In this process, the first analysis path uses a classification and regression sub-network to align the ROI features of the candidate region and outputs the damage category and the corresponding bounding box refinement results; the second analysis path uses a mask prediction sub-network to perform pixel-level segmentation of the ROI features to obtain the damage pixel distribution within the candidate region. Then, the classification confidence obtained from the first path is fused with the effective mask area ratio obtained from the second path to determine the effective damage region from the candidate regions. The refined bounding box and corresponding binary mask data are then packaged and output to form the initial damage identification result. Through the above process, joint constraints of box-level localization and pixel-level verification are achieved based on candidate selection, thereby improving the accuracy and reliability of the initial damage identification.

[0029] Furthermore, the method for performing region detection by traversing the multi-scale feature map includes:

[0030] The multi-scale feature map is traversed for detection and analysis to generate multiple detection scales; multi-scale spatial regions are divided according to the multiple detection scales, and multi-scale reference anchor frames are preset for the multi-scale spatial regions; damage prediction is performed on the multi-scale feature map based on the multi-scale reference anchor frames to obtain a damage prospect prediction score; damage location analysis is performed on the multi-scale feature map based on the multi-scale reference anchor frames to calculate the damage coordinate offset; the multi-scale reference anchor frames are sorted and filtered according to the damage prospect prediction score and the damage coordinate offset to generate the multiple candidate damage regions.

[0031] Preferably, the multi-scale feature maps are input into the region detection process layer by layer, with each layer of feature map corresponding to a detection scale. For example, the 160×160 feature map in the multi-scale feature map can be used to detect small cracks with a width of less than 20 pixels, the 80×80 feature map can be used to detect medium-sized strip or block damage, and the 40×40 feature map can be used to detect large-area pits or subsidence. Subsequently, for each detection scale, the corresponding feature map is divided into regular grid space regions. Each pixel or each 2×2 pixel region on the feature map is then used as the center point of the anchor frame. At each center point, a multi-scale reference anchor frame is pre-set. The scale and aspect ratio of this multi-scale reference anchor frame can be set according to the morphological characteristics of the road damage. For example, the basic size of the anchor frame is 16×16 pixels on a 160×160 feature map, 32×32 pixels on an 80×80 feature map, and 64×64 pixels on a 40×40 feature map. Three aspect ratios can be set for each basic size: 1:1, 1:2, and 2:1. Thus, nine reference anchor frames are generated for each center point. Subsequently, damage prediction is performed on multi-scale feature maps based on multi-scale reference anchor frames. Specifically, at each detection scale, a convolutional prediction network with shared weights performs convolution operations on the feature map, and outputs a damage prospect prediction score for each anchor frame through corresponding matching calculations. This score represents the probability that the anchor frame region contains pavement damage. Simultaneously, a location regression branch is set in the convolutional prediction network to calculate the corresponding damage coordinate offset for each anchor frame. This coordinate offset includes four parameters: horizontal center offset Δx, vertical center offset Δy, width scale offset Δw, and height scale offset Δh.

[0032] For this location regression branch, a smoothed L1 loss function is used for training and optimization. During the training phase, the target regression offset is first calculated based on the geometric relationship between the true damage bounding box and the reference anchor box. Specifically, the horizontal center offset is typically the ratio of the difference between the x-coordinates of the true damage bounding box center point and the reference anchor box center point coordinates to the width of the reference anchor box; the vertical center offset is typically the ratio of the difference between the y-coordinates of the true damage bounding box center point and the reference anchor box center point coordinates to the height of the reference anchor box; the width-scale offset is typically the logarithm of the ratio of the true damage bounding box width to the reference anchor box width; and the height-scale offset is typically the logarithm of the ratio of the true damage bounding box height to the reference anchor box height. Next, the location regression branch takes multi-scale feature maps as input, outputs the predicted offset corresponding to each reference anchor box through several convolutional layers, and then compares the predicted offset with the target regression offset. A smoothed L1 loss function is used to constrain the regression error, and the location regression branch is trained by minimizing the regression loss function, enabling the network to learn the spatial offset relationship of the damage region relative to the reference anchor box at different scales.

[0033] Then, all anchor boxes are sorted in descending order according to the damage prospect prediction score. Anchor boxes with prediction scores below a preset threshold are removed to reduce the number of invalid candidates. The coordinates of the remaining anchor boxes are then corrected based on the damage coordinate offset to generate corresponding prediction boxes. Non-maximum suppression is then applied to the prediction boxes to retain those with higher prediction scores. Finally, a preset number of prediction boxes are retained at each detection scale, resulting in multiple candidate damage regions. This provides a reliable candidate region basis for subsequent bidirectional analysis and damage identification.

[0034] Furthermore, based on the multi-scale reference anchor boxes, damage prediction is performed on the multi-scale feature maps to obtain a damage prospect prediction score. The method includes:

[0035] A convolutional neural network based on shared weights is used to segment the multi-scale feature map and determine multiple independent spatial regions. The convolutional neural network is then used to match the multi-scale feature map with the multi-scale reference anchor boxes according to the multiple independent spatial regions to generate a target scalar score. Based on the target scalar score, a normalization analysis is performed to calculate the damage prospect probability, and the damage prospect probability is output as the damage prospect prediction score.

[0036] Optionally, a convolutional neural network (CNN) for region segmentation and foreground prediction is first constructed based on shared weights. Multi-scale feature maps are then fed into this CNN as input. This CNN employs the exact same network structure and parameter configuration at each detection scale, consisting of three consecutive convolutional units. Each convolutional unit includes a 3×3 convolutional layer, a batch normalization layer, and a ReLU activation layer. The convolution stride is set to 1, the padding method is "same," and the number of output channels is set to 256, ensuring consistent spatial dimensions between the input and output feature maps. Through the shared weight convolution operation, a set of response feature maps is generated on each multi-scale feature map. Subsequently, each pixel position on the feature map is taken as the center point of an independent spatial region. The size of its corresponding receptive field is determined by the kernel size and network depth. For example, in a three-layer 3×3 convolutional structure, each independent spatial region corresponds to a receptive field of approximately 7×7 pixels in the original image. In this way, the multi-scale feature map can be divided into multiple independent spatial regions, each of which describes the feature response of the corresponding local road surface region. Next, the convolutional neural network is used to match and calculate the multi-scale feature maps with the multi-scale reference anchor boxes according to multiple independent spatial regions. Specifically, at the center of each independent spatial region, a set of multi-scale reference anchor boxes corresponding to that detection scale is pre-set. For example, at a certain detection scale, nine anchor boxes can be pre-set, with a basic size of 32×32 pixels and aspect ratios of 1:1, 1:2, and 2:1, respectively, and three scale ratios are set, such as 0.75, 1.0, and 1.25. For each anchor box, the corresponding target scalar score is calculated at the location of the spatial region using the output features of the convolutional neural network. This target scalar score is output through a 1×1 convolutional layer, and its output channel number is equal to the number of anchor boxes, that is, nine target scalar scores are output for each spatial region to represent the matching strength between each anchor box and the current spatial region features. Then, the target scalar score of the corresponding anchor frame within each spatial region is input into the Sigmoid function for normalization, resulting in nine probability values ​​between 0 and 1. For example, when the target scalar score of an anchor frame is 2.0, the corresponding damage prospect probability after Sigmoid mapping is approximately 0.88, while when the target scalar score is 0, the corresponding damage prospect probability is 0.5. The normalized probability values ​​represent the confidence level that the anchor frame region contains pavement damage. Finally, the damage prospect probability is output as a damage prospect prediction score, and this score is bound to the corresponding multi-scale benchmark anchor frames for subsequent anchor frame sorting and filtering. In summary, through the above-mentioned shared-weight convolutional region segmentation, anchor frame matching, and probability normalization calculation, a unified quantitative evaluation of damage prospects in different spatial regions of multi-scale feature maps can be achieved, improving the comparability and stability of damage prospect prediction scores at different scales and in different spatial regions.

[0037] Furthermore, the method for generating multiple candidate damage regions by sorting and filtering multi-scale benchmark anchor frames based on the damage prospect prediction score and the damage coordinate offset includes:

[0038] The multi-scale benchmark anchor boxes are sorted in descending order according to the damage prospect prediction score to obtain an anchor box sequence; a benchmark score threshold is set, and the damage prospect prediction scores below the benchmark score threshold are filtered out to obtain an optimized damage prospect prediction score; the optimized damage prospect prediction score is matched and extracted with the multi-scale benchmark anchor boxes to determine multiple target benchmark anchor boxes; multiple target benchmark anchor boxes are combined with the damage coordinate offset to perform multi-dimensional adjustment to generate multiple prediction boxes; non-maximum suppression calculation is performed based on the multiple prediction boxes to determine the multiple candidate damage regions.

[0039] Optionally, the system first aggregates the multi-scale benchmark anchor boxes from various detection scales and their corresponding damage prospect prediction scores, and establishes an anchor box index table in memory. For example, in the case of detection scales of 160×160, 80×80, and 40×40, if 9 benchmark anchor boxes are pre-set for each spatial location, approximately 160×160×9≈230400 benchmark anchor boxes can be generated on a single frame feature map. Based on the damage prospect prediction scores, the system sorts all benchmark anchor boxes in descending order, generating an anchor box sequence sorted from highest to lowest score, placing anchor boxes with higher damage probability at the beginning of the sequence. Subsequently, a benchmark score threshold is set to perform the first round of filtering on the anchor box sequence. This benchmark score threshold is set between 0.5 and 0.6, such as 0.55. When the damage prospect prediction score of a benchmark anchor box is less than 0.55, the anchor box is directly removed, and only anchor boxes with scores not lower than the threshold are retained as a candidate set, with their corresponding scores serving as the optimized damage prospect prediction scores. This threshold filtering step rapidly reduces the original hundreds of thousands of anchor frames to approximately 1,000 to 3,000, decreasing subsequent computational complexity while maintaining a high recall rate. Next, based on damage prospect prediction optimization scores, matching and extraction are performed between the optimized scores and multi-scale benchmark anchor frames. Specifically, the threshold-filtered anchor frame set is traversed sequentially according to the score order, with a maximum retention limit set for each detection scale. For example, a maximum of 1,000 anchor frames are retained at the high-resolution detection scale, a maximum of 600 at the medium-resolution scale, and a maximum of 400 at the low-resolution scale. When the number of anchor frames at a certain scale reaches the upper limit, the retention of lower-scoring anchor frames at that scale is stopped, thus obtaining multiple target benchmark anchor frame sets. Then, multiple target reference anchor frames are adjusted in multiple dimensions based on their corresponding damage coordinate offsets to generate multiple prediction boxes. This multidimensional adjustment includes four parameter dimensions: the damage coordinate offset calculated for each anchor frame, and the coordinate offset having four parameters: horizontal center offset Δx, vertical center offset Δy, width scale offset Δw, and height scale offset Δh. Taking a target reference anchor frame as an example, with center point coordinates (x0, y0), width w0, and height h0, the center point coordinates of the corresponding prediction box can be obtained through the inverse operation of the aforementioned offset calculation. The values ​​of Δx and Δy can be limited to [-0.5, 0.5] to avoid excessive offsetting, and the values ​​of Δw and Δh can be limited to [-1, 1] to ensure the stability of the scale adjustment. Finally, non-maximum suppression calculation is performed based on the multiple prediction boxes. That is, the generated prediction boxes are sorted again in descending order according to their corresponding damage prospect prediction scores, and the prediction box with the highest score is selected as the current retained box. The intersection-union ratio (IoU) between the selected box and the remaining prediction boxes is then calculated.When the IoU value between a predicted bounding box and the currently retained bounding box exceeds a preset overlap threshold, the predicted bounding box is discarded. This overlap threshold can be set to 0.5, and can be appropriately reduced to 0.4 in scenarios involving crack-like slender damage. The above steps are repeated until all predicted bounding boxes have been traversed. After non-maximum suppression processing, the remaining predicted bounding boxes are identified as multiple candidate damage regions. This reduces redundant and overlapping detection results while retaining candidate regions with high confidence and good spatial positioning accuracy for subsequent bidirectional analysis and damage identification.

[0040] Furthermore, multiple candidate damage regions are generated for bidirectional analysis to obtain initial damage identification results. The methods include:

[0041] A region of interest (ROI) feature extraction module is constructed, comprising a first analysis path and a second analysis path. The first analysis path includes a classification and regression sub-network. The classification and regression sub-network performs parallel processing on multiple candidate damage regions to generate parallel analysis results. The second analysis path includes a mask prediction sub-network. The mask prediction sub-network performs pixel segmentation on the multiple candidate damage regions to generate binary mask data. Confidence analysis is performed on the parallel analysis results, and the parallel confidence score is fused with the binary mask data. When the binary mask data is greater than a preset mask threshold and the parallel confidence score is higher than the preset confidence threshold, the initial damage identification result is generated.

[0042] Preferably, during bidirectional analysis, a region of interest (ROI) feature extraction module is first constructed. This ROI feature extraction module adopts a dual-path parallel structure, including a first analysis path and a second analysis path. The two paths share the same batch of candidate damage regions as input and are executed in parallel at the same time. The candidate damage regions are the set of predicted bounding boxes obtained from the previous stage, for example, 20 to 100 candidate damage regions in a single frame image. In the input stage, each candidate damage region is mapped back to the corresponding multi-scale feature map, and fixed-size region feature blocks are extracted through ROI alignment. The spatial size of these region feature blocks can be set to 7×7 pixels, and the number of channels is consistent with the input feature map, such as 256 channels, thus obtaining a candidate region feature tensor with a size of 7×7×256. Subsequently, a classification and regression sub-network is constructed in the first analysis path to process the candidate region feature blocks in parallel. This classification and regression sub-network contains two parallel branches: a classification branch and a regression branch. The classification branch consists of two fully connected layers. The output dimension of the first fully connected layer can be set to 1024, and the output dimension of the second fully connected layer is K, where K represents the preset number of damage categories. For example, K=4 represents cracks, pits, network cracks, and repair marks, respectively. The output of the classification branch is normalized using the Softmax function to obtain the probability distribution of each candidate damage region belonging to each damage category. The regression branch has a similar structure to the classification branch, but its final output dimension is 4, corresponding to the bounding box refinement parameters Δx′, Δy′, Δw′, and Δh′, which are used to perform secondary correction on the position and size of the candidate damage regions. Through the above processing, the first analysis path outputs the parallel analysis results for each candidate damage region, including the damage category probability vector and regression correction parameters.

[0043] Next, a mask prediction sub-network is constructed in the second analysis path to perform pixel-level segmentation of candidate damage regions. This mask prediction sub-network consists of four convolutional layers and an upsampling structure. In this sub-network, a 7×7×256 region feature block is first subjected to two 3×3 convolution operations to adjust the number of channels to 256. Then, the feature map is upsampled to 14×14 pixels using deconvolution or bilinear interpolation. After another convolution operation, it is upsampled to 28×28 pixels. Next, a single-channel feature map is output through a 1×1 convolutional layer and normalized using the Sigmoid function to generate a 28×28 damage probability mask. The damage probability mask is then binarized according to a preset pixel threshold (e.g., 0.5) to obtain the corresponding binary mask data, where a pixel value of 1 represents a damaged pixel and a pixel value of 0 represents a non-damaged pixel.

[0044] Then, a confidence analysis is performed on the parallel analysis results output by the first analysis path. Specifically, the maximum probability value is selected from the damage category probability vector output by the classification branch as the parallel confidence of the candidate damage region. For example, when the category probability vector of a certain region is [0.05, 0.10, 0.78, 0.07], its parallel confidence is 0.78. At the same time, the parallel confidence is fine-tuned by combining the bounding box correction magnitude output by the regression branch to reduce the confidence weight of location instability prediction. Next, the parallel confidence score and binary mask data are fused for determination. Specifically, the number of damaged pixels is counted based on the binary mask data, and the proportion of damaged pixels to the total number of pixels is calculated. For example, when the number of damaged pixels in the binary mask is 120 and the total number of pixels is 784, the effective area ratio of the mask is approximately 15.3%. This effective area ratio is then compared with a preset mask threshold, which is preferably set to 8% to 15%, such as 10%. At the same time, the parallel confidence score is compared with the preset confidence threshold to determine all candidate damaged regions that are valid damaged regions. The bounding boxes refined by the regression branch, the corresponding optimal damage category labels, and the binary mask data are packaged and encapsulated to generate the initial damage identification result. This achieves strict screening of candidate damaged regions, thereby improving the identification accuracy while effectively suppressing false detections.

[0045] Furthermore, the multiple candidate damage regions are processed in parallel through the classification and regression sub-network to generate parallel analysis results. The method includes:

[0046] The classification and regression subnetwork includes a classification branch, a regression branch, a first output layer, and a second output layer. The classification branch and the regression branch are parallel, and the first output layer, the second output layer, and the classification branch and the regression branch correspond to each other. Multiple damage feature blocks are extracted based on the multiple candidate damage regions according to a fixed size. The multiple damage feature blocks are flattened to construct a one-dimensional feature vector. The one-dimensional feature vector is then abstracted to obtain abstract features. The abstract features are synchronized to the classification and regression subnetwork for parallel analysis. A damage category vector is output through the first output layer, and a four-dimensional vector is output through the second output layer. The damage category vector and the four-dimensional vector are integrated to construct the parallel analysis result.

[0047] Optionally, the constructed classification and regression sub-networks adopt a parallel structure with shared input and independent branches, including a classification branch, a regression branch, a first output layer, and a second output layer. Both the classification and regression branches are constructed using multiple fully connected layers or a hybrid convolutional-fully connected structure to discriminate and regress the input features. Specifically, the classification branch includes at least one feature transformation layer and one classification output layer. The feature transformation layer performs dimensionality compression and semantic enhancement on the input features, and its output is passed through the first output layer to form a damage category vector. Each element of this damage category vector represents the probability value of a candidate damage region belonging to the corresponding damage category. The regression branch includes at least one feature mapping layer and one location regression output layer. The feature mapping layer extracts spatially relevant feature information, and its output is passed through the second output layer to form a four-dimensional vector for damage location regression, representing the offset of the candidate damage region relative to the center point of the reference position and the adjustment of its width and height. During the training phase, the classification and regression sub-networks are jointly trained using supervised learning. Specifically, the classification branch uses the true damage category of the candidate damage region as the supervision label and employs the cross-entropy loss function to constrain the damage category vector; the regression branch uses the true bounding box parameters of the candidate damage region as the supervision label and employs either smoothed L1 loss or mean squared error loss function to constrain the four-dimensional regression vector. During training, the classification loss and regression loss are weighted and summed according to preset weights as the overall optimization objective of the network. The network parameters of the classification and regression branches are then updated synchronously using the backpropagation algorithm. This ensures that the classification and regression sub-networks simultaneously possess damage category discrimination and fine-grained damage location regression capabilities under the same feature input, providing reliable technical support for subsequent damage identification and localization.

[0048] After constructing the classification and regression sub-networks, each candidate damage region is mapped to its corresponding multi-scale feature map, and region features are extracted using the ROI Align method. The output spatial size of this ROI Align can be set to 7×7, with the number of channels matching the feature map, thus generating a 7×7×256 damage feature block for each candidate damage region. Assuming there are N candidate damage regions in a single frame, N damage feature blocks can be obtained. Subsequently, each 7×7×256 damage feature block is flattened using either channel-first or row-first methods, resulting in a one-dimensional feature vector of length 7×7×256=12544. This one-dimensional feature vector fully preserves the spatial texture and semantic feature information of the candidate damage region. Next, the one-dimensional feature vector is abstracted. This abstraction process is implemented through at least one fully connected layer. For example, the output dimension of the first fully connected layer is set to 1024, and ReLU is used as the activation function to compress the 12544-dimensional one-dimensional feature vector into a 1024-dimensional abstract feature. A Dropout layer can be introduced after this fully connected layer, with a dropout rate of 0.5, to reduce feature redundancy and improve the network's generalization ability. After this step, each candidate lesion region corresponds to a 1024-dimensional abstract feature vector. Then, the abstract features are simultaneously input into the classification and regression branches for parallel analysis. In the classification branch, the abstract features are first processed for category discrimination through one or more fully connected layers. For example, after passing through a 512-dimensional fully connected layer, the abstract features are input into the first output layer. The output dimension of the first output layer is equal to the preset number of lesion categories K. The output result is normalized using the Softmax function to generate a lesion category vector of length K. Each element in this vector represents the probability value of the candidate lesion region belonging to the corresponding lesion category. In the regression branch, abstract features are analyzed for location regression through fully connected layers that are independent of the structural and classification branches. For example, after passing through a 512-dimensional fully connected layer, the data is input into the second output layer. The output dimension of the second output layer is 4, corresponding to the bounding box regression parameters Δx′, Δy′, Δw′, and Δh′, which are used to fine-tune the center point position offset and width-height scaling ratio of the candidate damage region. Finally, using the candidate damage region as the index unit, the damage category vector output by the first output layer is bound and integrated with the four-dimensional regression vector output by the second output layer to form a joint result structure containing damage category probability information and bounding box refinement parameters. This joint result constitutes the parallel analysis result, which is used for subsequent confidence calculation, mask consistency verification, and the final generation of the initial damage identification result. Through the above parallel structure, semantic classification and bounding box refinement can be completed simultaneously on the same candidate region features, improving the accuracy of damage category determination and localization precision.

[0049] Furthermore, a confidence analysis is performed on the parallel analysis results, and the parallel confidence is fused with the binary mask data. When the binary mask data is greater than a preset mask threshold and the parallel confidence is higher than a preset confidence threshold, an initial damage identification result is generated. The method includes:

[0050] Based on the multiple candidate damage regions and the binary mask data, the total number of pixels in the region is calculated to obtain the proportion of damaged pixel area. The proportion of damaged pixel area is compared with the preset mask threshold. The parallel confidence score is compared with the preset confidence threshold. Only when the proportion of damaged pixel area is greater than the preset mask threshold and the parallel confidence score is higher than the preset confidence threshold, the target candidate region is confirmed as a valid damage, and a valid damage region is generated. The valid damage region is dynamically adjusted based on the optimal damage category set by the first analysis path to determine the region bounding box. The region bounding box and the binary mask data are packaged together to construct the initial damage identification result.

[0051] Optionally, for each candidate damage region, the binary mask data output by the mask prediction sub-network is obtained. This binary mask data is a fixed-resolution matrix, where a pixel value of 1 indicates a predicted damaged pixel, and a value of 0 indicates a predicted non-damaged pixel. By traversing this binary mask matrix pixel by pixel, the number of pixels with a value of 1 is counted, denoted as the number of damaged pixels N. d Simultaneously calculate the total number of pixels N in the mask matrix. tBased on the above statistical results, the proportion of damaged pixel area R is calculated by division. For example, when the number of damaged pixels in the binary mask corresponding to a candidate region is 118, its proportion of damaged pixel area R ≈ 118 / 784 ≈ 0.150. Subsequently, the proportion of damaged pixel area R is compared with a preset mask threshold, which is used to limit the minimum effective proportion of damaged pixels in the candidate region. When R ≤ the preset mask threshold, the candidate region is determined to have sparse distribution of damaged pixels, possibly caused by noise or background texture, and is directly marked as invalid. When R > the preset mask threshold, the candidate region is determined to meet the valid damage condition at the pixel level. Simultaneously, the parallel confidence score is compared with a preset confidence threshold. This parallel confidence score originates from the output of the classification branch in the first analysis path, specifically the maximum probability value in the damage category probability vector. For example, if the damage category probability vector of a candidate region is [0.03, 0.12, 0.78, 0.07], then the parallel confidence score C = 0.78. This preset confidence threshold is used to limit the reliability of semantic recognition and can be set between 0.65 and 0.80. For example, if set to 0.70, and C ≤ 0.70, the candidate region is determined to be unreliable in semantic recognition. Only when the proportion of damaged pixel area R is greater than the preset mask threshold and the parallel confidence score C is higher than the preset confidence threshold is the corresponding candidate damage region confirmed as a valid damage region. If either condition is not met, the candidate region is directly eliminated. This dual-condition constraint ensures that the retained regions have high credibility at both the pixel and semantic levels.

[0052] After a candidate region is identified as a valid damage region, the category with the highest probability from the damage category probability vector is selected as the optimal damage category for that valid damage region. Then, based on the morphological features corresponding to this optimal damage category, the bounding box of the original candidate region is adaptively corrected. For example, when the optimal damage category is crack, the minimum bounding rectangle of all damaged pixels in the binary mask is calculated, and the bounding box is expanded vertically or horizontally along the main direction of the mask to make it elongated. When the optimal damage category is pit, the bounding box is proportionally expanded or contracted based on the outer contour of the binary mask to better fit the overall shape of the damage region, thus determining the final region bounding box coordinates. Finally, the region bounding box coordinates, optimal damage category label, parallel confidence score, and binary mask matrix corresponding to each valid damage region are encapsulated into a unified data structure. The same data encapsulation operation is then performed on all valid damage regions in the current image frame, forming an initial damage recognition result set containing multiple damage instances. Through this process, accurate screening of candidate damage regions, adaptive morphological adjustment, and structured output of results are achieved, providing reliable input for subsequent 3D spatial localization and inversion verification.

[0053] Based on the original road surface image sequence, a three-dimensional mapping is performed to construct the original road surface three-dimensional space. The initial damage identification result is synchronized to the original road surface three-dimensional space for localization to determine the spatial location information of the damage.

[0054] In one embodiment, the original road surface image sequence is first traversed according to the image acquisition order. Stable local feature points are extracted from each frame, forming a continuous movement trajectory of these feature points. Then, based on the feature point matching relationship between adjacent image frames, the pose change parameters of the camera during continuous shooting are calculated. Based on these pose parameters and the continuous movement trajectory of the feature points, 3D reconstruction is performed to construct an original 3D road surface space representing the spatial geometry of the actual road surface, serving as the basic coordinate environment for subsequent damage localization. After completing the construction of the original 3D road surface space, for each initial damage identification result, the region bounding box coordinates and corresponding binary mask data in the 2D image are obtained. Then, based on the pixel position of the damaged area in the image and the camera pose parameters corresponding to the image frame, the pixels within the damaged area are back-projected into the 3D space to determine the corresponding 3D point cloud subset. By clustering or statistical analysis of this 3D point cloud subset, the center coordinates, spatial range, and height variation characteristics of the damaged area in the 3D space are calculated. Finally, the calculated three-dimensional coordinate information is output as the damage spatial location information. This damage spatial location information includes at least the center coordinates of the damage in the three-dimensional space of the road surface, spatial size parameters, and relative positional relationship with the road travel direction. In this way, accurate mapping and positioning from two-dimensional damage identification results to the actual three-dimensional space of the highway surface is achieved, providing a reliable spatial data foundation for subsequent inversion verification, damage assessment, and highway pavement condition report generation.

[0055] Furthermore, the method for constructing a three-dimensional space of the original road surface by performing three-dimensional mapping based on the original road surface image sequence includes:

[0056] Feature point detection and matching are performed on the original road surface image sequence to track and obtain the continuous movement trajectory of the feature points; pose detection is performed based on the multiple image frames to obtain platform pose data; the continuous movement trajectory of the feature points is optimized by cluster adjustment according to the platform pose data, and the three-dimensional spatial coordinates of multiple feature points are calculated; multi-view stereo vision matching is performed based on the three-dimensional spatial coordinates of multiple feature points to construct a geometric three-dimensional point cloud of the highway road surface; the geometric three-dimensional point cloud of the highway road surface is meshed to construct the original road surface three-dimensional space.

[0057] Optionally, the original road surface images in the original road surface image sequence are first read frame by frame according to the image acquisition time sequence. Feature point detection is performed on each frame, using a feature detection algorithm with rotation and scale invariance. For example, 1000–3000 corner points or key points are detected in each frame, and a corresponding feature descriptor is calculated for each feature point. Subsequently, a nearest neighbor matching strategy is used to match adjacent frames based on the feature descriptors, and weak matching points are filtered out using a distance ratio threshold (e.g., 0.75). Then, a geometric consistency constraint method is used to further refine the matching results; for example, the fundamental matrix is ​​estimated using a random sampling consistency algorithm, and mismatched points with reprojection errors greater than 2 pixels are removed. By associating and numbering the same feature point in multiple consecutive frames, a continuous movement trajectory of the feature point across multiple frames is formed, with each trajectory containing observation points from at least 3 frames. Subsequently, using the first frame as the initial reference coordinate system, the relative pose change parameters of the camera during continuous shooting are calculated based on the feature point matching relationship between adjacent image frames. These pose change parameters include 3D rotation parameters and 3D translation parameters. The essential matrix can be estimated using the five-point or eight-point method, and the relative pose is recovered by combining it with the camera intrinsic parameter matrix. By accumulating pose changes frame by frame, the platform pose data of each image frame in a unified world coordinate system is obtained. This platform pose data is usually represented in the form of a 4×4 homogeneous transformation matrix, used to describe the position and attitude of the camera in space. Next, the 2D pixel coordinates of the feature points in each image frame, the corresponding platform pose data, and the camera imaging intrinsic parameters are used as inputs. The triangulation method is used to estimate the initial 3D coordinates of the feature points. Then, a bundle adjustment optimization process is introduced to jointly optimize all camera pose parameters and feature point 3D coordinates nonlinearly. The optimization objective is to minimize the reprojection error of all feature points in each observed image. This reprojection error is controlled within 1 pixel. After multiple rounds of iterative optimization, a set of spatially stable 3D feature point coordinates is obtained, typically on the order of tens of thousands.

[0058] Then, using 3D feature points as sparse geometric constraints, multi-view stereo matching calculations are performed between multiple frames of images. Depth estimation is performed on pixel regions not covered by feature points. By fusing depth information from different viewpoints, the point cloud density is gradually increased, generating a dense 3D point cloud covering the entire road surface. In this 3D point cloud, each point contains its 3D coordinates in the world coordinate system, with a point density reaching thousands of points per square meter, used to finely depict road surface undulations and geometric details. Finally, noise point removal and outlier filtering are performed on the point cloud data, eliminating isolated points whose distance from their local neighborhood exceeds a preset threshold. The point cloud is then uniformly sampled to reduce the impact of excessively dense local areas on modeling efficiency. Based on this, a triangular mesh reconstruction algorithm is used to mesh the point cloud, transforming the discrete point cloud data into a continuous 3D surface model, generating a 3D road surface mesh structure composed of numerous triangular facets. This 3D mesh model constitutes the original 3D space of the road surface, realistically reflecting the geometric morphology of the road surface and serving as the basic spatial carrier for subsequent 3D damage localization and inversion verification.

[0059] The damage spatial location information is used for inversion verification. If the verification is successful, a highway pavement condition report is generated.

[0060] In one embodiment, damage spatial location information is obtained based on the output of the 3D mapping stage. This damage spatial location information includes at least the center coordinates of the damage in the original road surface 3D space, the 3D spatial range parameters corresponding to the damage, the set of original image frame numbers associated with the damage, and the platform pose data corresponding to each image frame. Subsequently, for each damage instance, at least one or more original road surface images associated with it are selected, for example, the image frame in which the damage is first detected and one frame before and after it, for a total of 3 frames, and the corresponding platform pose matrix and camera intrinsic parameter matrix are obtained. Based on this pose matrix and camera imaging model, the 3D center coordinates and spatial range of the damage are back-projected onto the image plane to calculate the inverted 2D projection region. This inverted 2D projection region can be represented as a rectangle or polygon, and its pixel coordinate range is determined by the minimum bounding region after the 3D point projection. Then, the consistency between the inverted 2D projection region and the 2D damage region obtained in the initial damage identification stage is verified. The intersection-union ratio (IoU) between the inverted projection region and the initial damage region is calculated and compared with a set inversion consistency threshold, which can be set to 0.4 to 0.6, such as 0.5. When the calculated IoU in a certain image frame is ≥0.5, the damage identification result and the 3D localization result in that frame are considered consistent. If at least a preset proportion of the selected multiple image frames (e.g., 2 / 3) meet the above consistency condition, the damage instance is considered to have passed the inversion verification; otherwise, the damage instance is marked as failing the inversion or a low-confidence damage. Based on the successful inversion verification, the spatial stability of the damage is further verified. That is, the 3D center coordinates of the same damage obtained from inversion in different image frames are statistically analyzed, and their spatial offset is calculated. This spatial offset can be defined as the Euclidean distance between the center coordinates of each frame and its mean coordinates. A spatial stability threshold is then set, such as 5cm or 10cm. When all center coordinate offsets are less than the spatial stability threshold, the damage is determined to be stably located in 3D space, thus confirming it as a spatially valid damage. When both the inversion consistency verification and the spatial stability verification are passed, the damage instance is confirmed as the final valid damage and enters the highway pavement condition report construction stage. For each valid damage instance, its damage category, three-dimensional center coordinates, spatial dimensional parameters, spatial stability index, and corresponding road mileage information are summarized. Simultaneously, the severity of the damage can be graded based on the damage area or length; for example, damage areas greater than 0.05m are classified as... 2 The area was marked as moderate damage, greater than 0.2m. 2Areas marked as severely damaged. Finally, all valid damage instances are organized and statistically analyzed according to road travel direction or mileage order to construct a highway pavement condition report. This report includes statistical results of the number of various types of damage, the spatial distribution of damage, a list of key damage locations, and their corresponding severity levels. Simultaneously, the 3D positioning results are correlated with the original image or 3D pavement model to generate annotation information for visualization. Through the above process, assuming successful inversion verification, the generated highway pavement condition report exhibits high spatial accuracy and reliability, and can be directly used for highway maintenance assessment, repair decisions, and subsequent management analysis, providing a reliable basis for highway maintenance decisions, repair plan formulation, and subsequent inspections.

[0061] In summary, the embodiments of this application have at least the following technical effects:

[0062] First, images are continuously acquired along the road's driving direction to construct an original road surface image sequence. Next, multi-scale feature analysis is performed on the original road surface image sequence to generate a multi-scale feature map. Then, region detection is performed by traversing the multi-scale feature map, generating multiple candidate damage regions for bidirectional analysis to obtain initial damage identification results. Then, three-dimensional mapping is performed based on the original road surface image sequence to construct an original road surface three-dimensional space. The initial damage identification results are synchronized to the original road surface three-dimensional space for localization, determining the spatial location information of the damage. Finally, inversion verification is performed according to the damage spatial location information. If the verification passes, a road surface condition report is generated. This solves the technical problem of existing road surface damage detection methods being susceptible to the effects of illumination, noise, and scale changes in continuous acquisition scenarios, leading to missed or false detections of damage regions and inaccurate localization. It achieves the technical effect of improving the accuracy of damage identification through multi-scale feature analysis and bidirectional verification of candidate regions, and realizing accurate damage localization and automatic assessment through three-dimensional spatial mapping and inversion verification.

[0063] Example 2 is based on the same inventive concept as the deep learning-based self-identification method for road surface damage in the previous examples, such as... Figure 2 As shown, this application provides a deep learning-based self-identification system for highway pavement damage, wherein the system includes:

[0064] Image acquisition unit 11: continuously acquires images along the road travel direction to construct an original road surface image sequence; Feature analysis unit 12: performs multi-scale feature analysis based on the original road surface image sequence to draw a multi-scale feature map; Region detection unit 13: traverses the multi-scale feature map to perform region detection, generates multiple candidate damage regions for bidirectional analysis, and obtains initial damage identification results; Damage localization unit 14: performs three-dimensional mapping based on the original road surface image sequence to construct an original road surface three-dimensional space, synchronizes the initial damage identification results to the original road surface three-dimensional space for localization, and determines the damage spatial location information; Inversion verification unit 15: performs inversion verification according to the damage spatial location information, and when the verification is successful, constructs a road surface condition report.

[0065] Furthermore, the feature analysis unit 12 is used to perform the following method:

[0066] The original road surface image sequence is traversed to extract multiple image frames, which are then subjected to grayscale equalization processing to generate a standard image sequence. A multi-scale sliding window is set, and the standard image sequence is analyzed at multiple scales through the multi-scale sliding window to determine multiple scale image blocks. A backbone network is constructed, which contains multiple convolutional stages. According to the multiple convolutional stages, the backbone network is activated to downsample the multiple scale image blocks to determine the feature space size parameters. Based on the feature space size parameters, feature transformation is performed to construct multi-level original feature maps. Based on the backbone network, deep semantic analysis is performed to generate high semantic feature maps, which are then upsampled to generate high-level feature maps. The high-level feature maps are then element-wise concatenated and fused with the multi-level original features to construct a multi-scale feature map.

[0067] Furthermore, the region detection unit 13 is used to perform the following method:

[0068] The multi-scale feature map is traversed for detection and analysis to generate multiple detection scales; multi-scale spatial regions are divided according to the multiple detection scales, and multi-scale reference anchor frames are preset for the multi-scale spatial regions; damage prediction is performed on the multi-scale feature map based on the multi-scale reference anchor frames to obtain a damage prospect prediction score; damage location analysis is performed on the multi-scale feature map based on the multi-scale reference anchor frames to calculate the damage coordinate offset; the multi-scale reference anchor frames are sorted and filtered according to the damage prospect prediction score and the damage coordinate offset to generate the multiple candidate damage regions.

[0069] Furthermore, the region detection unit 13 is used to perform the following method:

[0070] A convolutional neural network based on shared weights is used to segment the multi-scale feature map and determine multiple independent spatial regions. The convolutional neural network is then used to match the multi-scale feature map with the multi-scale reference anchor boxes according to the multiple independent spatial regions to generate a target scalar score. Based on the target scalar score, a normalization analysis is performed to calculate the damage prospect probability, and the damage prospect probability is output as the damage prospect prediction score.

[0071] Furthermore, the region detection unit 13 is used to perform the following method:

[0072] The multi-scale benchmark anchor boxes are sorted in descending order according to the damage prospect prediction score to obtain an anchor box sequence; a benchmark score threshold is set, and the damage prospect prediction scores below the benchmark score threshold are filtered out to obtain an optimized damage prospect prediction score; the optimized damage prospect prediction score is matched and extracted with the multi-scale benchmark anchor boxes to determine multiple target benchmark anchor boxes; multiple target benchmark anchor boxes are combined with the damage coordinate offset to perform multi-dimensional adjustment to generate multiple prediction boxes; non-maximum suppression calculation is performed based on the multiple prediction boxes to determine the multiple candidate damage regions.

[0073] Furthermore, the region detection unit 13 is used to perform the following method:

[0074] A region of interest (ROI) feature extraction module is constructed, comprising a first analysis path and a second analysis path. The first analysis path includes a classification and regression sub-network. The classification and regression sub-network performs parallel processing on the multiple candidate damage regions to generate parallel analysis results. The second analysis path includes a mask prediction sub-network. The mask prediction sub-network performs pixel segmentation on the multiple candidate damage regions to generate binary mask data. Confidence analysis is performed on the parallel analysis results, and the parallel confidence score is fused with the binary mask data. When the binary mask data is greater than a preset mask threshold and the parallel confidence score is higher than the preset confidence threshold, the initial damage identification result is generated.

[0075] Furthermore, the region detection unit 13 is used to perform the following method:

[0076] The classification and regression subnetwork includes a classification branch, a regression branch, a first output layer, and a second output layer. The classification branch and the regression branch are parallel, and the first output layer, the second output layer, and the classification branch and the regression branch correspond to each other. Multiple damage feature blocks are extracted based on the multiple candidate damage regions according to a fixed size. The multiple damage feature blocks are flattened to construct a one-dimensional feature vector. The one-dimensional feature vector is then abstracted to obtain abstract features. The abstract features are synchronized to the classification and regression subnetwork for parallel analysis. A damage category vector is output through the first output layer, and a four-dimensional vector is output through the second output layer. The damage category vector and the four-dimensional vector are integrated to construct the parallel analysis result.

[0077] Furthermore, the region detection unit 13 is used to perform the following method:

[0078] Based on the multiple candidate damage regions and the binary mask data, the total number of pixels in the region is calculated to obtain the proportion of damaged pixel area. The proportion of damaged pixel area is compared with the preset mask threshold. The parallel confidence score is compared with the preset confidence threshold. Only when the proportion of damaged pixel area is greater than the preset mask threshold and the parallel confidence score is higher than the preset confidence threshold, the target candidate region is confirmed as a valid damage, and a valid damage region is generated. The valid damage region is dynamically adjusted based on the optimal damage category set by the first analysis path to determine the region bounding box. The region bounding box and the binary mask data are packaged together to construct the initial damage identification result.

[0079] Furthermore, the damage localization unit 14 is used to perform the following method:

[0080] Feature point detection and matching are performed on the original road surface image sequence to track and obtain the continuous movement trajectory of the feature points; pose detection is performed based on the multiple image frames to obtain platform pose data; the continuous movement trajectory of the feature points is optimized by cluster adjustment according to the platform pose data, and the three-dimensional spatial coordinates of multiple feature points are calculated; multi-view stereo vision matching is performed based on the three-dimensional spatial coordinates of multiple feature points to construct a geometric three-dimensional point cloud of the highway road surface; the geometric three-dimensional point cloud of the highway road surface is meshed to construct the original road surface three-dimensional space.

[0081] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention in any way. Although the present invention has been disclosed above with reference to preferred embodiments, it is not intended to limit the present invention. Any person skilled in the art can make some modifications or alterations to the above-disclosed technical content to create equivalent embodiments without departing from the scope of the present invention. Any modifications, equivalent changes, and alterations made to the above embodiments based on the technical essence of the present invention without departing from the scope of the present invention shall still fall within the scope of the present invention.

Claims

1. A deep learning-based method for self-identification of road surface damage, characterized in that, The method includes: Images are continuously acquired along the direction of highway travel to construct an original road surface image sequence; Multi-scale feature analysis is performed on the original road surface image sequence to draw multi-scale feature maps; The multi-scale feature map is traversed to perform region detection, generate multiple candidate damage regions for bidirectional analysis, and obtain the initial damage identification result. Based on the original road surface image sequence, a three-dimensional mapping is performed to construct the original road surface three-dimensional space. The initial damage identification result is synchronized to the original road surface three-dimensional space for localization to determine the damage spatial location information. The damage spatial location information is used for inversion verification. If the verification is successful, a highway pavement condition report is generated.

2. The deep learning-based self-identification method for highway pavement damage as described in claim 1, characterized in that, Based on the original road surface image sequence, multi-scale feature analysis is performed to draw multi-scale feature maps. The method includes: The original road surface image sequence is traversed to extract multiple image frames, which are then subjected to grayscale equalization processing to generate a standard image sequence. A multi-scale sliding window is set up, and the standard image sequence is analyzed at multiple scales through the multi-scale sliding window to determine multiple scale image blocks; Construct a backbone network, which includes multiple convolutional stages; According to the multiple convolutional stages, the backbone network is activated to downsample the multiple scale image blocks to determine the feature space size parameters; Based on the feature space size parameters, feature transformation is performed to construct a multi-level original feature map; Deep semantic analysis is performed based on the backbone network to generate a high semantic feature map, which is then upsampled to generate a high-level feature map. The high-level feature map is element-wise concatenated and fused with the multi-level original features to construct a multi-scale feature map.

3. The deep learning-based self-identification method for highway pavement damage as described in claim 1, characterized in that, The method for traversing the multi-scale feature map to perform region detection includes: The multi-scale feature map is traversed for detection and analysis to generate multiple detection scales; Divide the multi-scale spatial region according to the multiple detection scales, and pre-set multi-scale reference anchor frames for the multi-scale spatial region; Damage prediction is performed on the multi-scale feature map based on the multi-scale benchmark anchor frame to obtain a damage prospect prediction score. Based on the multi-scale reference anchor frame, damage location analysis is performed on the multi-scale feature map, and damage coordinate offset is calculated. Based on the damage prospect prediction score and the damage coordinate offset, the multi-scale benchmark anchor frames are sorted and filtered to generate the multiple candidate damage areas.

4. The deep learning-based self-identification method for highway pavement damage as described in claim 3, characterized in that, Based on the multi-scale reference anchor frame, damage prediction is performed on the multi-scale feature map to obtain a damage prospect prediction score. The method includes: Convolutional neural networks based on shared weights are used to segment multi-scale feature maps and identify multiple independent spatial regions. The convolutional neural network performs matching calculations on the multi-scale feature maps and the multi-scale reference anchor boxes according to the multiple independent spatial regions to generate a target scalar score; Normalization analysis is performed based on the target scalar score to calculate the damage prospect probability, and the damage prospect probability is output as the damage prospect prediction score.

5. The deep learning-based self-identification method for highway pavement damage as described in claim 3, characterized in that, The method involves sorting and filtering multi-scale baseline anchor frames based on the damage prospect prediction score and the damage coordinate offset to generate multiple candidate damage regions, including: The multi-scale benchmark anchor frames are sorted in descending order according to the damage prospect prediction score to obtain the anchor frame sequence; A baseline score threshold is set, and damage prospect prediction scores that are lower than the baseline score threshold are filtered out to obtain an optimized damage prospect prediction score. Based on the damage prospect prediction optimization score, multiple target benchmark anchor frames are extracted by matching and extracting the multi-scale benchmark anchor frames. Multiple prediction boxes are generated by multidimensional adjustment based on the multiple target reference anchor frames and the damage coordinate offset; Non-maximum suppression calculation is performed based on the multiple prediction boxes to determine the multiple candidate damage regions.

6. The deep learning-based self-identification method for highway pavement damage as described in claim 1, characterized in that, Multiple candidate damage regions are generated for bidirectional analysis to obtain initial damage identification results. The method includes: A region of interest feature extraction module is constructed, which includes a first analysis path and a second analysis path; The first analysis path includes classification and regression sub-networks; The multiple candidate damage regions are processed in parallel using the classification and regression sub-network to generate parallel analysis results. The second analysis path includes a mask prediction subnetwork; The mask prediction subnetwork is used to perform pixel segmentation on the multiple candidate damage regions to generate binary mask data. The parallel analysis results are subjected to confidence analysis, and the parallel confidence is fused with the binary mask data. When the binary mask data is greater than a preset mask threshold and the parallel confidence is higher than a preset confidence threshold, the initial damage identification result is generated.

7. The deep learning-based self-identification method for highway pavement damage as described in claim 6, characterized in that, The method involves parallel processing of the multiple candidate damage regions using the classification and regression sub-network to generate parallel analysis results, including: The classification and regression subnetwork includes a classification branch, a regression branch, a first output layer, and a second output layer. The classification branch and the regression branch are parallel, and the first output layer, the second output layer, and the classification branch and the regression branch have corresponding relationships. Based on the multiple candidate damage regions, multiple damage feature blocks are extracted according to a fixed size; The multiple damage feature blocks are flattened to construct a one-dimensional feature vector; The one-dimensional feature vector is subjected to feature abstraction to obtain abstract features; The abstract features are synchronized to the classification and regression sub-network for parallel analysis. The damage category vector is output through the first output layer, and a four-dimensional vector is output through the second output layer. The damage category vector is integrated with the four-dimensional vector to construct the parallel analysis result.

8. The deep learning-based self-identification method for highway pavement damage as described in claim 6, characterized in that, The parallel analysis results are subjected to confidence analysis. The parallel confidence score is then fused with the binary mask data. When the binary mask data is greater than a preset mask threshold and the parallel confidence score is higher than a preset confidence threshold, an initial damage identification result is generated. The method includes: Based on the multiple candidate damage regions and the binary mask data, the total number of pixels in the region is calculated to obtain the proportion of damaged pixel area. Compare the percentage of the damaged pixel area with the preset mask threshold; Compare the parallel confidence level with the preset confidence threshold; The target candidate region is confirmed as a valid damage only when the proportion of the damaged pixel area is greater than the preset mask threshold and the parallel confidence level is higher than the preset confidence threshold. Based on the first analysis path, the optimal damage category is set and the effective damage area is dynamically adjusted to determine the region boundary box. The region bounding box and the binary mask data are packaged together to construct the initial damage identification result.

9. The deep learning-based self-identification method for highway pavement damage as described in claim 2, characterized in that, The method for constructing a three-dimensional space of the original road surface based on the original road surface image sequence includes: Feature point detection and matching are performed on the original road surface image sequence to track and obtain the continuous movement trajectory of the feature points; Pose detection is performed based on the multiple image frames to obtain platform pose data; Based on the platform pose data, the continuous movement trajectory of the feature points is bundled and optimized to calculate the three-dimensional spatial coordinates of multiple feature points. Multi-view stereo vision matching is performed based on the three-dimensional spatial coordinates of the multiple feature points to construct a geometric three-dimensional point cloud of the highway surface. The geometric three-dimensional point cloud of the highway surface is meshed to construct the original three-dimensional space of the road surface.

10. A deep learning-based self-identification system for highway pavement damage, characterized in that, The system is used to implement the deep learning-based self-identification method for road surface damage according to any one of claims 1-9, the system comprising: Image acquisition unit: continuously acquires images along the direction of highway travel to construct an original road surface image sequence; Feature analysis unit: Performs multi-scale feature analysis based on the original road surface image sequence and draws multi-scale feature maps; Region detection unit: Traverses the multi-scale feature map to perform region detection, generates multiple candidate damage regions for bidirectional analysis, and obtains initial damage identification results; Damage localization unit: Based on the original road surface image sequence, performs three-dimensional mapping to construct the original road surface three-dimensional space, synchronizes the initial damage identification result to the original road surface three-dimensional space for localization, and determines the spatial location information of the damage; Inversion verification unit: Performs inversion verification according to the damage spatial location information. If the verification is successful, a highway pavement condition report is constructed.

Images