Road defect detection method based on dynamic prototype learning and weakly supervised semantic segmentation

Abstract

The application discloses a road defect detection method based on dynamic learning prototype and weakly supervised semantic segmentation, which obtains accurate recognition results by enhancing the features reflected in the feature map obtained from the FPN, has the accuracy of full supervised semantic segmentation model recognition, and has the advantage of low artificial labeling cost of weakly supervised semantic segmentation model. In addition, the method sets color constancy algorithm and channel-space dual attention algorithm data processing steps, avoids misjudgment of road defect recognition results caused by light interference, feature dispersion in the partition process, and interference of other objects in the image on the defect recognition results by superimposing the binary mask. The method can not only give the distribution range of the road defect, but also obtain the heat map of the defect distribution, so that the detection personnel have a more perfect cognition and more accurate grasp of the defect structure.

Description

Technical Field

[0001] This invention belongs to the field of intelligent road defect detection technology, specifically relating to a road defect detection method based on dynamic prototype learning and weakly supervised semantic segmentation. Background Technology

[0002] Existing intelligent road defect detection technologies that rely on image recognition largely depend on the completeness of the annotation information for accuracy. Traditional fully supervised semantic segmentation models achieve high-precision detection results through comprehensive manual annotation. While these models can achieve high defect recognition accuracy in ideal environments, their strong dependence on pixel-level annotation becomes a significant obstacle for practical applications. The manual cost of image annotation is very high; pixel-level annotation of a standard 648×848 pixel road inspection image requires 25 to 35 minutes of professional manual time. In large-scale applications, such as quarterly inspections of provincial highway networks that require processing over 100,000 images, the manual cost of image annotation becomes extremely high. If the manual cost of intelligent detection is comparable to the traditional method of manually inspecting and confirming road defects, intelligent detection fails to demonstrate its value.

[0003] For this reason, current research on intelligent road defect detection tends to focus on weakly supervised models. The image recognition results of these models are usually only "defect present" and "no defect present", so only simple manual annotation is required, which greatly reduces the labor cost. However, weakly supervised models suffer from three fundamental flaws that significantly limit their value in practical applications: First, there is the problem of feature discreteness during the recognition process. Due to the high heterogeneity of road defects in terms of material reflectivity, damage depth gradient, and ambient lighting conditions, it is difficult to identify the same type of defect using a fixed feature. Second, the blurring of the recognition result boundary caused by geometric distortion is an inherent flaw of weakly supervised models, which may directly lead to the inability to determine the defect type. Third, the complexity of the road structure itself also affects the recognition results. Road defects range from sub-millimeter-level microcracks to meter-level structural cracks, spanning three orders of magnitude. Convolutional neural networks with fixed receptive fields struggle to capture both global structural features and local detail features simultaneously. Furthermore, background interference is equally challenging. Asphalt aggregate textures are very similar to real cracks, and classic threshold segmentation methods can easily misjudge aggregate textures as cracks. While multi-scale fusion partially alleviates this problem, it cannot avoid the deviation of the defect edge after fusion caused by semantic misalignment at different levels of the feature pyramid.

[0004] The aforementioned problems have long trapped intelligent road defect detection in a binary dilemma of "high cost-high accuracy" and "low cost-low accuracy". Therefore, it is necessary to design a new detection method that can overcome the limitations of annotation cost and solve the core problems of feature discretization, geometric distortion and scene adaptability through innovative architecture, so as to provide reliable technical support for highway operation and maintenance. Summary of the Invention

[0005] The purpose of this invention is to provide a road defect detection method based on dynamic prototype learning and weakly supervised semantic segmentation. This method only requires manually circling the area where the road defects are located in the image, and then extracting the dynamic prototype based on the output of multiple convolutional layers in the Feature Pyramid Network (FPN). Based on the extracted data, the number and distribution range of road defects in the image are obtained, and then the characteristics of the road defects are further analyzed to determine the defect type.

[0006] The technical solution adopted in this invention is as follows:

[0007] A road defect detection method based on dynamic learning prototypes and weakly supervised semantic segmentation specifically includes the following steps:

[0008] Step 1. Take an image of the road to be detected. The imaging wavelength of the image is within the visible light wavelength range. The captured image is called the original image.

[0009] Step 2. Preprocess the original image, specifically including the following steps:

[0010] Step 2.1. Number all pixels in the original image and represent the color information of each pixel as a corresponding feature vector; the color of the pixels in the original image is called the original color. The original color is processed using the color constancy algorithm, and the color obtained after processing is called the real color. The image formed after replacing the color of all pixels in the original image with the real color is called the real image.

[0011] Step 2.2. Manually draw a bounding box in the real image. The bounding box is composed of pixels in the real image, and the road defect is located within the area enclosed by the bounding box.

[0012] Step 3. Feature recognition is performed on the region enclosed by the bounding box using the dynamic prototype extraction algorithm. The dynamic prototype extraction algorithm is based on FPN with n convolutional layers. The recognition results of all convolutional layers are output in the form of feature maps, which are denoted as original feature maps. There are a total of n original feature maps.

[0013] Step 4. Perform feature enhancement processing on the original feature map, specifically including the following steps:

[0014] Step 4.1. Take the original feature map as the input feature map F, number all pixels in F, and then process the input feature map F using the channel-spatial dual attention algorithm to generate a single-channel feature map H. The element corresponding to a pixel in H is denoted as the feature value H. i , where i is the number corresponding to that pixel;

[0015] Step 4.2. Use a clustering algorithm to partition all pixels within the bounding box area, with each partition corresponding to a unique number;

[0016] Step 4.3. Derive a similarity feature map based on the similarity between individuals within each region;

[0017] Step 4.4. Process the multiple original feature maps separately as F to obtain the corresponding similarity feature maps, and then combine the multiple similarity feature maps to obtain the fused feature map F. out ;

[0018] Step 5. F out The distribution of median values ​​is presented in the form of a heatmap. Then, a block matrix is ​​determined as the structuring element, and this structuring element is used to perform an erosion operation on the heatmap to determine the edge zone of the defect range. The resulting image after erosion is denoted as F. erode Then, based on the aforementioned structural element, an inverse structural element is determined, and then the inverse structural element is used to apply F. erode Perform a dilation operation to fill the edge bands. The resulting image after dilation is denoted as F. dilate ;

[0019] Step 6. According to F dilate Determine the type of defect; present the partitioning results from Step 4.2 as an image, denoted as the visual defect map H. c Defect type, H c F dilate These three are the final results.

[0020] In existing technologies, fully supervised semantic segmentation models require high manual annotation costs, while weakly supervised semantic segmentation models have low manual annotation costs but limited recognition capabilities. Therefore, the recognition effect of intelligent road defect detection methods depends on the investment in manual annotation costs, making it difficult to realize their value. To address the shortcomings of existing weakly supervised semantic segmentation models, this invention employs a color constancy algorithm to eliminate the influence of road surface material reflectivity and light intensity on the defect recognition process. It also performs pixel-level feature enhancement on road defects, followed by erosion and dilation to determine the boundary contours of the defect portions. In multi-scale fusion, this invention adopts a method of first extracting features and then fusing them, avoiding edge deviations caused by semantic misalignment at different scales. The final output includes three results: defect type, zoning map, and heatmap, providing inspectors with more complete and detailed defect information. Even if the method ultimately fails to determine a reliable defect type, inspectors can make a reliable judgment by combining the zoning map and heatmap.

[0021] Further optimization involves using the color constancy algorithm in Step 2.1 to eliminate the influence of reflected light on the image. Step 2.1 specifically includes the following steps:

[0022] Step 2.1.1. Establish a rectangular coordinate system in the plane containing the original image and determine the rectangular coordinates corresponding to each pixel in the image;

[0023] Step 2.1.2. Select a color space to measure the color of a pixel, and record the coordinates of the pixel in the color space as a vector. The feature vector describing the original color of a pixel is denoted as I(x,y), where x and y are the horizontal and vertical coordinates of the pixel in the Cartesian coordinate system, respectively.

[0024] Step 2.1.3. The color constancy algorithm is expressed as the following formula.

[0025]

[0026] Where R(x,y) is the feature vector describing the true color of the pixel, * indicates convolution calculation, F(x,y) is the Gaussian kernel function, c is a constant, and σ is the kernel function width;

[0027] Step 2.1.4. Take multiple values ​​as candidate solutions for σ, and substitute all candidate solutions into formula (1) one by one. The image formed by R(x,y) calculated when a certain candidate solution is substituted into formula (1) is called the sample image corresponding to the candidate solution.

[0028] Step 2.1.5. First, convert all sample images to grayscale images, with each grayscale image corresponding to a unique number; then, take multiple grayscale values ​​as grayscale samples and calculate the information entropy of the grayscale image using the following formula.

[0029] S=∑ q∈Q [P q ·log2(P q (2)

[0030] Where S is the information entropy, q is the gray value, Q is the set of all gray samples, and P... q This represents the proportion of pixels with grayscale value q in the entire grayscale image.

[0031] Step 2.1.6. Calculate the information entropy of each grayscale image according to formula (2), find the grayscale image with the highest information entropy, and take the sample image corresponding to the grayscale image as the obtained real image.

[0032] The color constancy algorithm used in this invention is an improvement on the Retinex model. The Retinex model assumes that the light reflected from an object remains constant during propagation; the observation result is affected by interference superimposed on the received light. Removing this interference reveals the true image of the object. In this invention, the interference is calculated by convolving the received light with a Gaussian kernel. Since the result of the convolution calculation depends on the parameters of the convolution kernel, the width σ is a decisive parameter in the Gaussian convolution kernel. To make the result more conducive to subsequent recognition, the more information in the convolved image, the better. The amount of information contained in the image is measured by information entropy.

[0033] Further optimization is achieved by the dynamic prototype extraction algorithm in Step 3, which specifically includes the following steps:

[0034] Step 3.1. In the FPN on which the algorithm is based, each convolutional layer, starting from the second convolutional layer, takes the output of the previous convolutional layer as input; the original feature map includes low-level feature maps and high-level feature maps, and the output of the first convolutional layer is denoted as the low-level feature map F. low The output of the second convolutional layer is denoted as the high-level feature map F. high_1 The output of the third convolutional layer is denoted as the high-level feature map F. high_2 The output of the nth convolutional layer is denoted as the high-level feature map F. high_n-1 All of the above feature maps have the same number of channels;

[0035] Step 3.2. Upsample the n-1 high-level feature maps using bilinear interpolation to make the resolution of all high-level feature maps equal to F. low The same applies to F using a 3×3 convolution kernel. low Perform smoothing;

[0036] Step 3.3. Process the F lowPerform element-wise addition with each upsampled high-level feature map respectively. After the addition, F high_1 、F high_2 、…、F high_n-1 correspondingly generate new feature maps P1, P2, …, P n-1 ;

[0037] In the method of the present invention, FPN is improved based on the ResNet-50 network. In FPN, there is one more feature fusion every time a convolutional layer is passed, and the size of the feature map, that is, the resolution, will be correspondingly reduced. Feature maps with higher layers are beneficial for identifying road defects with a larger range, and feature maps with lower layers are beneficial for identifying road defects with a smaller range. In order to avoid too large differences in feature types between high-level feature maps and low-level feature maps, element-wise addition (element-wise addition) needs to be performed between high-level feature maps and low-level feature maps before output.

[0038] For further optimization, the channel-spatial dual attention algorithm in Step 4.1 specifically includes the following steps:

[0039] Step 4.1.1. Take P m as the input feature map F, m is the serial number of the new feature map, m is a positive integer and m ∈ [1, n - 1], perform global average pooling on F to obtain the global feature matrix g;

[0040] Step 4.1.2. Construct the channel weight matrix w c ,w c is expressed as the following formula

[0041] w c =ε[W2·δ(W1·g)] (3)

[0042] where W1 and W2 are the compression weight matrix and the expansion weight matrix respectively. Suppose the number of channels of a certain feature map is C, then after being transformed by W1, the number of channels of this feature map will become C’, C’ < C, and then after being transformed by W2, the number of channels will change back to C again; δ is the ReLU activation function, and ε is the Sigmoid activation function;

[0043] Step 4.1.3. Apply w c to the input feature map F to obtain the intermediate feature map F’, and F’ is expressed as the following formula

[0044]

[0045] ]>where represents multiplication channel by channel; represent the information of pixel i in F’ as the feature vector f’ i ,and then through the softmax function based on f’ iThe weight value of pixel i in F' is obtained, and the weight value is used as the element to construct the spatial weight matrix w. s The weight value of pixel i in w s The position in the middle corresponds to f' i Its position in F';

[0046] Step 4.1.4. Apply w sequentially to F. c With w s The final feature map F is obtained. final F final It is expressed as the following formula

[0047]

[0048] Then, F is processed by a 1x1 convolution kernel. final The number of channels is reduced to 1, and then the F function with 1 channel is applied. final The process is performed to generate a single-channel feature map H.

[0049] This part of the algorithm is divided into two parts: channel attention and spatial attention. The channel attention part adopts the classic "Squeeze-and-Excitation" (SE) structure, while the spatial attention part adopts the self-attention mechanism. After processing by this part of the channel-space dual attention algorithm, the elements in the feature map will have long-distance spatial dependencies.

[0050] Further optimization involves the following steps in the clustering algorithm described in Step 4.2:

[0051] Step 4.2.1. The element corresponding to pixel i in the single-channel feature map H is denoted as the feature value H. i Calculate all eigenvalues ​​H i The mean μH and standard deviation β H And based on μH and β H Calculate the dynamic threshold T p T p It is derived from the following formula

[0052] T p =μ H +0.5β H (6)

[0053] Step 4.2.2. Represent the information of pixel i in the region enclosed by the bounding box as a feature vector f. i and all H i one by one with T p In comparison, if H is satisfied i >T p The condition will then be f iSet F c In the middle, set F c Each f in i As an entity; determine a positive integer as the critical value MinPts for partitioning, and determine the initial radius C according to the type of road to be detected. d ;

[0054] Step 4.2.3. Combine the two eigenvectors f i with f i’ The vector obtained by subtraction is denoted as Δ. i, i', i' is f i’ The corresponding pixel number and i ≠ i', ||Δ i,i’ ||2 is f i with f i’ The distance between them; in set F c Take any one individual as the core individual, and then calculate F one by one. c The distance between other individuals and the core individual, if the distance does not exceed C d The individual is then placed in its neighborhood. After calculating the distances to all other individuals, if the number of individuals in the neighborhood is not less than the critical value MinPts, the core individual is marked as a usable individual; otherwise, it is marked as noise. Once an individual is marked, another unmarked individual is selected as the core individual, and the above process is repeated until F... c Until all individuals are marked;

[0055] Step 4.2.4. Select one available individual as the master individual. Compare the neighborhood of the master individual with the neighborhood of another available individual to see if there are any identical individuals. If so, mark the available individual and the master individual as connectable. After comparing all other available individuals with the master individual, mark the available individual that is the master individual. Then, select an unmarked available individual as the master individual and repeat the above process until all available individuals are marked. Then, connect all two connectable available individuals, group all available individuals in the same topology into the same cluster, and number all clusters.

[0056] Step 4.2.5. Represent the cluster as C k,ts k is the cluster index. All clusters are updated iteratively, specifically including the following steps.

[0057] I. Determine the radius Δ of each cluster k, ts, where ts is the current iteration number, initialized to 1, Δ k, The formula for calculating ts is as follows:

[0058] Where C k,ts-1 For the cluster numbered k generated in the previous iteration, μk,ts-1 C k,ts-1 All individuals in the middle, that is, all f i The average vector, C k,0 For the clusters obtained in Step 4.2.4; compare Δ k, ts and Δ k, If the values ​​of ts-1 and ts-1 are equal, then the iteration ends and C is directly set. k,ts-1 All individual pixels are grouped into the same region, denoted as C. k and C k Output the values; if they are not equal, proceed to step II.

[0059] II. Take C k,ts-1 For each individual, calculate C one by one. k,ts-1 The distance between this individual and other individuals in the group, if the distance exceeds Δ k,ts Then both bodies will be moved from C. k,ts-1 Remove from the middle, and complete this calculation; if the distance never exceeds Δ k, If ts, all individuals are retained to complete the current calculation; after one calculation is completed, another uncalculated individual is taken to start the next calculation, until all individuals in the cluster have completed the calculation; then C k,ts-1 The remaining individuals form a new cluster C. k,ts Then return to step I.

[0060] The clustering algorithm used in this invention is an improvement upon the DBSCAN (Density-Based Spatial Clustering of Applications with Noise) algorithm. Compared to the existing DBSCAN algorithm, this clustering algorithm mainly adds an iterative update step for cluster size after cluster formation. In this invention, the clustering algorithm aims to divide each road defect into a region. However, the initially formed regions may contain noise points, i.e., parts that do not belong to road defects. The presence of noise points will affect the subsequent determination of defect types. Therefore, after the regions are divided, their size needs to be further reduced to obtain a more accurate range of road defects and their boundary morphology.

[0061] Further optimization involves the following steps in Step 4.3 for generating the similarity feature map:

[0062] Step 4.3.1. Calculate the weighted centroid vector p for each region using the following formula. k

[0063]

[0064] Where w i For the eigenvector f i The corresponding weight value;

[0065] Step 4.3.2. Calculate all f i The average vector μ c Then, the deviation feature vector δf corresponding to the feature vector of each pixel within the bounding box region is calculated. i And the deviation centroid vector δp corresponding to the weighted centroid vector of each region. k ,δf i With δp k The calculation formula is as follows:

[0066]

[0067] Step 4.3.3. Calculate the similarity s corresponding to pixel i. c,i s c,i Through δf corresponding to pixel i i and δp corresponding to the region where pixel i is located k The calculation formula is as follows:

[0068]

[0069] Where λ is the scaling factor and exp is the natural exponential function;

[0070] s c,i As elements constituting the similarity feature map S c s c,i In S c The position in the middle corresponds to H i Its position in H.

[0071] Clustering algorithms only obtain the range of the area, that is, the range of the road defect in the image. Similarity feature maps can present all areas in the form of heat maps. The part with the highest heat in the map is the core of the defect. In this way, the structural characteristics of the road defect can be further obtained, providing more basis for the analysis of road defect conditions.

[0072] Further optimization involves generating a fused feature map F in Step 4.4. out The process specifically includes the following steps:

[0073] Step 4.4.1. Place P1, P2, ..., P n-1 Each of these is processed as an input feature map F, P1, P2, ..., P n-1 The corresponding similarity feature maps are denoted as S. c,1 S c,2 S c,n-1 ;

[0074] Step 4.4.2. Calculate the weighted average of all similarity feature maps to obtain the total similarity feature map S. context S context It is expressed as the following formula

[0075]

[0076] Where UP represents the upsampling operation, S c,m For P m When the input feature map F is used as the corresponding generated similarity feature map, all S c,m The feature maps generated after the upsampling operation have the same resolution, α. m For S c,m The corresponding weight value;

[0077] Step 4.4.3. Use the semantic segmentation model U-Net to identify non-road parts in the real image, and output the identification results as a binary mask, denoted as M. road Then combine with M road The spatial model yields the road extent in the real image within M... road The corresponding part, whose mask is denoted as M' road ; M' road With S context Element-wise multiplication yields the enhanced feature map F. p ;

[0078] Step 4.4.4. Place F low The input feature map F is used to derive the corresponding similarity feature map, which is denoted as F. v Then merge F p With F v The fused feature map F is obtained. out F out Obtained through the following formula

[0079]

[0080] Where γ is the adjustment matrix, and J is a matrix with all elements being 1, and J has the same number of rows and columns as γ. This indicates element-wise multiplication.

[0081] The purpose of this section is multi-scale fusion. This section treats high-level feature maps as one category and low-level feature maps as another, because low-level feature maps contain the most information and can supplement the information lost by high-level feature maps during convolution, thus making the fused feature map more comprehensive and complete. Step 4.4.3 involves clearing the elements corresponding to non-road areas in the image. This is because roads often contain debris such as leaves and plastic bottles, which can interfere with the recognition results. If the road image is taken from above, there will inevitably be a portion outside the road area, and this portion also needs to be removed from the image.

[0082] Further optimization is achieved by obtaining S in Step 4.4.2. context The specific process is as follows:

[0083] Determine all α m initial value α m,0 The initial total similarity feature map S is obtained through formula (11). context,0 Then, using a convolutional neural network through S context,0 The distribution of median values ​​is used to determine the spatial dependencies of all pixels within the bounded area of ​​the bounding box, and based on the determination results, each S... c,m Corresponding weight value α m Make adjustments to make S context The spatial dependency of the mid-pixel is strengthened; the adjusted weight value is denoted as α. m,1 Then, the pixels are reweighted and superimposed to obtain S, which has a strong spatial dependency relationship. context,1 , will S context,1 As the final S context .

[0084] Because different convolutional layers extract different features, the defect features described by each original feature map will also be different. Correspondingly, the boundaries of each region and the location of the defect core in each similarity feature map will also be different. The S obtained after stacking these features will be different. context It has a certain degree of randomness, if all α m If the value is fixed, it is difficult to ensure S context It must be meaningful. Therefore, the method of this invention incorporates a step of using a convolutional neural network to determine the spatial dependencies of pixels in the result obtained after superposition, and adjusts α based on the spatial dependencies of pixels. m The value is adjusted to ensure S context It is meaningful.

[0085] Further optimization involves determining the adjustment matrix γ in Step 4.4.4 through the following process:

[0086] The same convolutional neural network is used to identify road defects multiple times within the bounding box region, with random perturbations in each identification process. The identification results for each time are masked with M. t The data is recorded in the form of t, where t is the number of recognitions; then, the number of times pixel i is recognized across all M is calculated. t The variance σ of the corresponding eigenvector i The variance of all pixels within the bounded region needs to be calculated; then, the region enclosed by the bounding box is divided into multiple confidence regions, with each confidence region corresponding to a certain numerical range, based on each σ... i The range of values ​​within which pixel i is located divides it into the corresponding confidence region. Different confidence regions have different confidence values, and the magnitude of the confidence value is positively correlated with the median value of the range of values ​​corresponding to the confidence region. The confidence value of the confidence region where pixel i is located is the confidence value of pixel i. An adjustment matrix γ is constructed using the confidence values ​​of pixels as elements, and the position of the confidence value of pixel i in the adjustment matrix γ corresponds to s. c,i Its position in the similarity feature map.

[0087] Some regions in an image produce stable results after processing by a convolutional neural network, while others exhibit greater randomness. This is an inherent property of images. Regions with higher randomness produce less reliable results after processing by a convolutional neural network, and the more convolutional layers processed, the lower the reliability. For such regions, the fusion result needs to rely more on low-level feature maps to increase the reliability of the fusion result. For regions with stable results, the results from high-level feature maps are retained.

[0088] Further optimization involves determining the defect type in Step 6 as follows:

[0089] Step 6.1. Collect real road defect contour data to establish a database. The contour lines in the database are denoted as α. j j is the outline number, and each α j All are labeled with the corresponding defect type; for F dilate The contour of the defect range is identified, and the identified contour line is denoted as Ω. Then, an α is taken. j , in Ω and α j Take m sampling points from each of the above, number the m sampling points sequentially, and in addition, take a threshold L. threshold Then, the relationship between Ω and α is described using a loss function. j The difference between them is expressed by the loss function as follows:

[0090]

[0091] Where L edge The value is the loss, l is the sampling point number, and n is the sampling point number. l,Ω Let n be the unit vector perpendicular to the contour line at the l-th sampling point on Ω.l,j For α j The unit vector perpendicular to the contour line at the l-th sampling point;

[0092] Step 6.2. Change α multiple times j The sampling point distribution locations are determined, and L is calculated for each location distribution case. edge The minimum value among them is denoted as α. j The corresponding L edge_j ;

[0093] Step 6.3. Obtain an α j The corresponding L edge_j Then take another uncalculated α j Perform calculations until all α values ​​in the database are included. j All yielded the corresponding L edge_j ;

[0094] Step 6.4. Put all L edge_j With L threshold Compare them one by one, if it is L edge_j Less than L threshold Then we consider Ω and α j They belong to the same type of defect.

[0095] The method of this invention determines the type of defect in an image by calculating the degree of overlap between the defect contour in the image and the defect contour in the database. Formula (13) essentially calculates the gradient between two contour lines. After the erosion and dilation operation in Step 5, the thermal value at the edge of the defect part will be significantly different from that of the non-defect part, thus facilitating the determination of the defect contour. Determining the defect type by calculating the degree of overlap not only considers the similarity of the defect shape but also the similarity of the defect size, which is conducive to making a more accurate judgment.

[0096] The beneficial effects of the method of the present invention are as follows:

[0097] 1. The method of the present invention obtains accurate recognition results by enhancing the features reflected in the feature map obtained from FPN. It has both the accuracy of fully supervised semantic segmentation model recognition and the low manual annotation cost of weakly supervised semantic segmentation model.

[0098] 2. The original image is processed by a color constancy algorithm before being used for feature extraction to avoid misjudgment of road defect identification results caused by light interference;

[0099] 3. The feature maps obtained in FPN are processed by the channel-spatial dual attention algorithm before clustering and partitioning, so that the elements in the feature maps have long-distance spatial dependencies, in order to avoid feature discrepancies during the partitioning process;

[0100] 4. The method of the present invention can not only determine the distribution range of road defects through clustering and partitioning, but also obtain a heat map of defect distribution through similarity feature map, thereby enabling inspection personnel to have a more complete understanding and more accurate grasp of defect structure;

[0101] 5. When superimposing similarity feature maps corresponding to high-level feature maps, a convolutional neural network is used to determine the spatial dependencies of pixels in the superimposed result, thereby ensuring the accuracy of the obtained S. context It is meaningful;

[0102] 6. In the fusion of F p With F v It is considered that the accuracy of the results obtained after processing different regions of an image by a convolutional neural network will inevitably differ. Adjusting the matrix can compensate for this difference and improve the accuracy of the final recognition result.

[0103] 7. Using a binary mask M' road With S context Element-wise multiplication removes non-road portions from the feature map, preventing other objects in the image from interfering with the defect recognition results;

[0104] 8. Finally, the method determines the defect type by calculating the contour overlap, which considers not only the similarity of defect shape but also the similarity of defect size, thus facilitating more accurate results. Attached Figure Description

[0105] Figure 1 A schematic diagram of the overall process of the road defect detection method of the present invention;

[0106] Figure 2 A schematic diagram illustrating the process of generating the original feature map using a feature pyramid network. Detailed Implementation

[0107] To make the objectives, technical solutions, and advantages of the present invention clearer, the technical solutions of the present invention will be clearly and completely described below through specific embodiments. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0108] Example 1:

[0109] A road defect detection method based on dynamic learning prototypes and weakly supervised semantic segmentation, the overall process is as follows: Figure 1 As shown, the specific steps include:

[0110] Step 1. Take an image of the road to be inspected. The imaging wavelength of the image is within the visible light wavelength range. The image taken is called the original image. The resolution of the original image is 2448×2048.

[0111] Step 2. Preprocess the original image, specifically including the following steps:

[0112] Step 2.1. Number all pixels in the original image and represent the color information of each pixel as a corresponding feature vector; the color of the pixels in the original image is called the original color. The original color is processed using the color constancy algorithm, and the resulting color is called the true color. The image formed after replacing the colors of all pixels in the original image with the true colors is called the true image. The specific steps include:

[0113] Step 2.1.1. Establish a rectangular coordinate system in the plane where the original image is located. The origin of the coordinate system is the vertex of one corner of the image. The edge with a length of 2448 pixels extends in the x-axis direction, and the edge with a length of 2048 pixels extends in the y-axis direction. Determine the rectangular coordinates corresponding to each pixel in the image.

[0114] Step 2.1.2. Measure the color of a pixel using the RGB color space, and record the R, G, B coordinates of the pixel in the RGB color space as a vector. The feature vector describing the original color of a pixel is denoted as I(x,y), where x and y are the horizontal and vertical coordinates of the pixel in the Cartesian coordinate system, respectively.

[0115] Step 2.1.3. The color constancy algorithm is expressed as the following formula.

[0116]

[0117] Where R(x,y) is the feature vector describing the true color of the pixel, * indicates convolution calculation, F(x,y) is the Gaussian kernel function, c is a constant, and σ is the kernel function width;

[0118] Step 2.1.4. Randomly select 14 values ​​in the range [15,80] as candidate solutions for σ. Substitute all candidate solutions into formula (1) one by one. The image formed by R(x,y) calculated when a candidate solution is substituted into formula (1) is called the sample image corresponding to the candidate solution.

[0119] Step 2.1.5. First, convert all sample images to grayscale images, with each grayscale image corresponding to a unique number; then, take multiple grayscale values ​​as grayscale samples and calculate the information entropy of the grayscale image using the following formula.

[0120] S=∑ q∈Q [P q·log2(P q (2)

[0121] Where S is the information entropy, q is the gray value, Q is the set of all gray samples, and P... q This represents the proportion of pixels with grayscale value q in the entire grayscale image.

[0122] Step 2.1.6. Calculate the information entropy of each grayscale image according to formula (2), find the grayscale image with the highest information entropy, and take the sample image corresponding to the grayscale image as the obtained real image.

[0123] Step 2.2. Manually draw a bounding box in the real image. The bounding box is composed of pixels in the real image, and the road defect is located within the area enclosed by the bounding box.

[0124] Step 3. Feature recognition is performed on the region enclosed by the bounding box using a dynamic prototype extraction algorithm. This algorithm is based on a Feature Pyramid Network (FPN) with four convolutional layers. The recognition results of all convolutional layers are output as feature maps, denoted as the original feature maps. There are four original feature maps in total. The dynamic prototype extraction algorithm specifically includes the following steps:

[0125] Step 3.1. In the FPN on which the algorithm is based, each convolutional layer, starting from the second convolutional layer, takes the output of the previous convolutional layer as input; the original feature map includes low-level feature maps and high-level feature maps, and the output of the first convolutional layer is denoted as the low-level feature map F. low The output of the second convolutional layer is denoted as the high-level feature map F. high_1 The output of the third convolutional layer is denoted as the high-level feature map F. high_2 The output of the 4th convolutional layer is denoted as the high-level feature map F. high_3 All of the above feature maps have the same number of channels; F high_1 F high_2 F high_3 The resolutions are respectively those of the real image

[0126] Step 3.2. Upsample the three high-level feature maps using bilinear interpolation to make the resolution of all high-level feature maps consistent with F. low The same applies to F using a 3×3 convolution kernel. low Perform smoothing processing.

[0127] Step 3.3. Process the F low The sum is then element-wise added to each upsampled high-level feature map, and the result is F. high_1 F high_2 F high_3The corresponding newly generated feature maps are P1, P2, and P3 respectively. The process of the FPN generating the new feature maps is as follows Figure 2 as shown

[0128] Step 4. Perform feature enhancement processing on the original feature map, which specifically includes the following steps

[0129] Step 4.1. Take the original feature map as the input feature map F, number all the pixels in F, and then process the input feature map F through the channel-spatial dual attention algorithm to generate a single-channel feature map H. The element corresponding to a certain pixel in H is denoted as the feature value H i , where i is the number corresponding to this pixel, i ∈ [1, 5013504] and i is an integer; the channel-spatial dual attention algorithm includes the following steps

[0130] Step 4.1.1. Take P m as the input feature map F, where m is the serial number of the new feature map, m is a positive integer and m ∈ {1, 2, 3}. Perform global average pooling on F to obtain the global feature matrix g

[0131] Step 4.1.2. Construct the channel weight matrix w c , w c is expressed as the following formula

[0132] w c = ε[W2·δ(W1·g)] (3)

[0133] where W1 and W2 are the compression weight matrix and the expansion weight matrix respectively. Suppose the number of channels of a certain feature map is C, then after the feature map is transformed by W1, its number of channels will become C’, C’ < C, and then after being transformed by W2, its number of channels will change back to C; δ is the ReLU activation function, and ε is the Sigmoid activation function

[0134] Step 4.1.3. Apply w c to the input feature map F to obtain the intermediate feature map F’. F’ is expressed as the following formula

[0135] [[ID=...]] (omitted for brevity)

[0136] where represents element-wise multiplication; represent the information of pixel i in F’ as the feature vector f’ i , and then obtain the weight value of pixel i in F’ through the softmax function based on f’ i , and construct the spatial weight matrix w with the weight value as an element s , and the position of the weight value of pixel i in w s corresponds to the position of f’ i in F’

[0137] Step 4.1.4. Apply w sequentially to F. c With w s The final feature map F is obtained. final F final It is expressed as the following formula

[0138]

[0139] Then, F is processed by a 1x1 convolution kernel. final The number of channels is reduced to 1, and then the F function with 1 channel is applied. final The process is performed to generate a single-channel feature map H.

[0140] Step 4.2. Use a clustering algorithm to partition all pixels within the bounding box area, with each partition corresponding to a unique number. The clustering algorithm specifically includes the following steps:

[0141] Step 4.2.1. The element corresponding to pixel i in the single-channel feature map H is denoted as the feature value H. i Calculate all eigenvalues ​​H i The average value μ H and standard deviation β H and based on μ H and β H Calculate the dynamic threshold T p T p It is derived from the following formula

[0142] T p =μ H +0.5β H (6) Step 4.2.2. Represent the information of pixel i in the region enclosed by the bounding box as a feature vector f i and all H i one by one with T p In comparison, if H is satisfied i >T p The condition will then be f i Set F c In the middle, set F c Each f in i As an entity; determine a positive integer as the critical value MinPts for partitioning, and determine the initial radius C according to the type of road to be detected. d For asphalt pavement C d =0.3, for cement pavement C d =0.5;

[0143] Step 4.2.3. Combine the two eigenvectors f i with fi’ The vector obtained by subtraction is denoted as Δ. i, i', i' is f i’ The corresponding pixel number and i ≠ i', ||Δ i,i’ ||2 is f i with f i’ The distance between them; in set F c Take any one individual as the core individual, and then calculate F one by one. c The distance between other individuals and the core individual, if the distance does not exceed C d The individual is then placed in its neighborhood. After calculating the distances to all other individuals, if the number of individuals in the neighborhood is not less than the critical value MinPts, the core individual is marked as a usable individual; otherwise, it is marked as noise. Once an individual is marked, another unmarked individual is selected as the core individual, and the above process is repeated until F... c Until all individuals are marked;

[0144] Step 4.2.4. Select one available individual as the master individual. Compare the neighborhood of the master individual with the neighborhood of another available individual to see if there are any identical individuals. If so, mark the available individual and the master individual as connectable. After comparing all other available individuals with the master individual, mark the available individual that is the master individual. Then, select an unmarked available individual as the master individual and repeat the above process until all available individuals are marked. Then, connect all two connectable available individuals, group all available individuals in the same topology into the same cluster, and number all clusters.

[0145] Step 4.2.5. Represent the cluster as C k,ts k is the cluster index. All clusters are updated iteratively, specifically including the following steps.

[0146] I. Determine the radius Δ of each cluster k, ts, where ts is the current iteration number, initialized to 1, Δ k, The formula for calculating ts is as follows:

[0147] Where C k,ts-1 For the cluster numbered k generated in the previous iteration, μ k,ts-1 C k,ts-1 All individuals in the middle, that is, all f i The average vector, C k,0 For the clusters obtained in Step 4.2.4; compare Δ k, ts and Δ k, If the values ​​of ts-1 and ts-1 are equal, then the iteration ends and C is directly set. k,ts-1All individual pixels are grouped into the same region, denoted as C. k and C k Output the values; if they are not equal, proceed to step II.

[0148] II. Take C k,ts-1 For each individual, calculate C one by one. k,ts-1 The distance between this individual and other individuals in the group, if the distance exceeds Δ k,ts Then both bodies will be moved from C. k,ts-1 Remove from the middle, and complete this calculation; if the distance never exceeds Δ k, If ts, all individuals are retained to complete the current calculation; after one calculation is completed, another uncalculated individual is taken to start the next calculation, until all individuals in the cluster have completed the calculation; then C k,ts-1 The remaining individuals form a new cluster C. k,ts Then return to step I.

[0149] Step 4.3. Based on the similarity between individuals within each region, a similarity feature map is generated. The process of generating the similarity feature map includes the following steps:

[0150] Step 4.3.1. Calculate the weighted centroid vector p for each region using the following formula. k

[0151]

[0152] Where w i For the eigenvector f i The corresponding weight value;

[0153] Step 4.3.2. Calculate all f i The average vector μ c Then, the deviation feature vector δf corresponding to the feature vector of each pixel within the bounding box region is calculated. i And the deviation centroid vector δp corresponding to the weighted centroid vector of each region. k ,δf i With δp k The calculation formula is as follows:

[0154]

[0155] Step 4.3.3. Calculate the similarity s corresponding to pixel i. c,i s c,i Through δf corresponding to pixel i i and δp corresponding to the region where pixel i is located k The calculation formula is as follows:

[0156]

[0157] Where λ is the scaling factor and exp is the natural exponential function, and in this embodiment λ = 0.3;

[0158] s c,i As elements constituting the similarity feature map S c s c,i In S c The position in the middle corresponds to H i Its position in H.

[0159] Step 4.4. Process the multiple original feature maps separately as F to obtain the corresponding similarity feature maps, and then combine the multiple similarity feature maps to obtain the fused feature map F. out Generate fused feature map F out The process specifically includes the following steps:

[0160] Step 4.4.1. Process P1, P2, and P3 as input feature maps F, and denote the similarity feature maps corresponding to P1, P2, and P3 as S. c,1 S c,2 S c,3 ;

[0161] Step 4.4.2. Calculate the weighted average of all similarity feature maps to obtain the total similarity feature map S. context S context It is expressed as the following formula

[0162]

[0163] Where UP represents the upsampling operation, S c,m For P m When the input feature map F is used as the corresponding generated similarity feature map, all S c,m The feature maps generated after the upsampling operation have the same resolution, α. m For S c,m The corresponding weight value;

[0164] First determine all α during the calculation. m initial value α m,0 The initial total similarity feature map S is obtained through formula (11). context,0 Then, using the G-CNN network through S context,0 The distribution of median values ​​is used to determine the spatial dependencies of all pixels within the bounded area of ​​the bounding box, and based on the determination results, each S... c,m Corresponding weight value α m Make adjustments to make S context The spatial dependency of the mid-pixel is strengthened; the adjusted weight value is denoted as α. m,1Then, the pixels are reweighted and superimposed to obtain S, which has a strong spatial dependency relationship. context,1 , will S context,1 As the final S context ;

[0165] Step 4.4.3. Use the trained semantic segmentation model U-Net to identify non-road parts in the real image, and output the identification results as a binary mask, denoted as M. road Then combine with M road The spatial model yields the road extent in the real image within M... road The corresponding part, whose mask is denoted as M' road ; M' road With S context Element-wise multiplication yields the enhanced feature map F. p ;

[0166] Step 4.4.4. Place F low The input feature map F is used to derive the corresponding similarity feature map, which is denoted as F. v Then merge F p With F v The fused feature map F is obtained. out F out Obtained through the following formula

[0167]

[0168] Where γ is the adjustment matrix, and J is a matrix with all elements being 1, and J has the same number of rows and columns as γ. This indicates element-wise multiplication;

[0169] During computation, the same pre-trained semantic segmentation model, U-Net, is used to identify road defects multiple times within the bounding box region. Random perturbations are introduced during each identification process. The identification results are then masked with M. t The data is recorded in the form of t, where t is the number of recognitions; then, the number of times pixel i is recognized across all M is calculated. t The variance σ of the corresponding eigenvector i The variance of all pixels within the bounded region needs to be calculated; then, the region enclosed by the bounding box is divided into multiple confidence regions, with each confidence region corresponding to a certain numerical range, based on each σ... i The range of values ​​within which pixel i is located divides it into the corresponding confidence region. Different confidence regions have different confidence values, and the magnitude of the confidence value is positively correlated with the median value of the range of values ​​corresponding to the confidence region. The confidence value of the confidence region where pixel i is located is the confidence value of pixel i. An adjustment matrix γ is constructed using the confidence values ​​of pixels as elements, and the position of the confidence value of pixel i in the adjustment matrix γ corresponds to s. c,iIts position in the similarity feature map.

[0170] Step 5. F out The distribution of the median values ​​is presented in the form of a heatmap. A 3×3 block matrix is ​​then determined as the structuring element, with the central value being 1 and the eight edge values ​​being 0. This structuring element is then used to perform an erosion operation on the heatmap to determine the edge zone of the defect area. The resulting image after erosion is denoted as F. erode Then, based on the structuring element, an inverse structuring element is determined, where the middle value of the inverse structuring element is 0, and the eight values ​​at the edges are all 1; then, the inverse structuring element is used to apply F... erode Perform a dilation operation to fill the edge bands. The resulting image after dilation is denoted as F. dilate ;

[0171] Step 6. According to F dilate The specific process for determining the type of defect is as follows:

[0172] Step 6.1. Collect real road defect contour data to establish a database. Each contour line corresponds to a unique number and a defect type. The database is divided into multiple levels, each level corresponding to a certain area range. Each contour line is matched with the corresponding area range based on the size of the area it encloses, and the contour lines are assigned to the corresponding level according to their area range. The contour lines in the database are denoted as α. j j is the outline number, which includes two parts: the level of the outline and its corresponding sequence number in that level. Each α j All are labeled with the corresponding defect type; for F dilate The contour of the defect range is identified, and the identified contour line is denoted as Ω. Then, the corresponding level is matched according to the area range of Ω, and then an α is selected in the level. j and in Ω and α j Take m sampling points from each of the above, number the m sampling points sequentially, and in addition, take a threshold L. threshold Then, the relationship between Ω and α is described using a loss function. j The difference between them is expressed by the loss function as follows:

[0173]

[0174] Where L edge The value is the loss, l is the sampling point number, and n is the sampling point number. l,Ω Let n be the vector perpendicular to the contour line at the l-th sampling point on Ω. l,j For α j The vector perpendicular to the contour line at the l-th sampling point;

[0175] Step 6.2. Change α multiple timesj The sampling point distribution locations are determined, and L is calculated for each location distribution case. edge The minimum value among them is denoted as α. j The corresponding L edge_j ;

[0176] Step 6.3. Obtain an α j The corresponding L edge_j Then, take another uncalculated α from the hierarchy. j Perform calculations until all α values ​​in the database are included. j All yielded the corresponding L edge_j ;

[0177] Step 6.4. Put all L edge_j With L threshold Compare them one by one, if it is L edge_j Less than L threshold Then we consider Ω and α j They belong to the same type of defect;

[0178] The result obtained by partitioning P3 in Step 4.2 is presented in the form of an image, denoted as the visual defect map H. c Defect type, H c F dilate These three are the final results.

Claims

1. A road defect detection method based on dynamic learning prototypes and weakly supervised semantic segmentation, characterized in that, Specifically, it includes the following steps: Step 1. Take an image of the road to be detected. The imaging wavelength of the image is within the visible light wavelength range. The captured image is called the original image. Step 2. Preprocess the original image, specifically including the following steps: Step 2.

1. Number all pixels in the original image and represent the color information of each pixel as a corresponding feature vector; the color of the pixels in the original image is called the original color. The original color is processed using the color constancy algorithm, and the color obtained after processing is called the real color. The image formed after replacing the color of all pixels in the original image with the real color is called the real image. Step 2.

2. Manually draw a bounding box in the real image. The bounding box is composed of pixels in the real image, and the road defect is located within the area enclosed by the bounding box. Step 3. Feature recognition is performed on the region enclosed by the bounding box using the dynamic prototype extraction algorithm. The dynamic prototype extraction algorithm is based on the Feature Pyramid Network (FPN) with n convolutional layers. The recognition results of all convolutional layers are output in the form of feature maps, denoted as the original feature maps. There are a total of n original feature maps. Step 4. Perform feature enhancement processing on the original feature map, specifically including the following steps: Step 4.

1. Take the original feature map as the input feature map F, number all pixels in F, and then process the input feature map F using the channel-spatial dual attention algorithm to generate a single-channel feature map H. The element corresponding to a pixel in H is denoted as the feature value H. i , where i is the number corresponding to that pixel; Step 4.

2. Use a clustering algorithm to partition all pixels within the bounding box area, with each partition corresponding to a unique number; Step 4.

3. Derive a similarity feature map based on the similarity between individuals within each region; Step 4.

4. Process the multiple original feature maps separately as F to obtain the corresponding similarity feature maps, and then combine the multiple similarity feature maps to obtain the fused feature map F. out ; Step 5. F out The distribution of median values ​​is presented in the form of a heatmap. Then, a block matrix is ​​determined as the structuring element, and this structuring element is used to perform an erosion operation on the heatmap to determine the edge zone of the defect range. The resulting image after erosion is denoted as F. erode Then, based on the aforementioned structural element, an inverse structural element is determined, and then the inverse structural element is used to apply F. erode Perform a dilation operation to fill the edge bands. The resulting image after dilation is denoted as F. dilate ; Step 6. According to F dilate Determine the type of defect; present the partitioning results from Step 4.2 as an image, denoted as the visual defect map H. c Defect type, H c F dilate These three are the final results.

2. The road defect detection method based on dynamic learning prototype and weakly supervised semantic segmentation as described in claim 1, characterized in that, The color constancy algorithm in Step 2.1 is used to eliminate the influence of reflected light on the image. Step 2.1 specifically includes the following steps: Step 2.1.

1. Establish a rectangular coordinate system in the plane containing the original image and determine the rectangular coordinates corresponding to each pixel in the image; Step 2.1.

2. Select a color space to measure the color of a pixel, and record the coordinates of the pixel in the color space as a vector. The feature vector describing the original color of a pixel is denoted as I(x,y), where x and y are the horizontal and vertical coordinates of the pixel in the Cartesian coordinate system, respectively. Step 2.1.

3. The color constancy algorithm is expressed as the following formula. Where R(x,y) is the feature vector describing the true color of the pixel, * indicates convolution calculation, F(x,y) is the Gaussian kernel function, c is a constant, and σ is the kernel function width; Step 2.1.

4. Take multiple values ​​as candidate solutions for σ, and substitute all candidate solutions into formula (1) one by one. The image formed by R(x,y) calculated when a certain candidate solution is substituted into formula (1) is called the sample image corresponding to the candidate solution. Step 2.1.

5. First, convert all sample images to grayscale images, with each grayscale image corresponding to a unique number; then, take multiple grayscale values ​​as grayscale samples and calculate the information entropy of the grayscale image using the following formula. S=∑ q∈Q [P q ·log2(P q )] (2) Where S is the information entropy, q is the gray value, Q is the set of all gray samples, and P... q This represents the proportion of pixels with grayscale value q in the entire grayscale image. Step 2.1.

6. Calculate the information entropy of each grayscale image according to formula (2), find the grayscale image with the highest information entropy, and take the sample image corresponding to the grayscale image as the obtained real image.

3. The road defect detection method based on dynamic learning prototype and weakly supervised semantic segmentation as described in claim 2, characterized in that, The dynamic prototype extraction algorithm in Step 3 specifically includes the following steps: Step 3.

1. In the FPN on which the algorithm is based, each convolutional layer, starting from the second convolutional layer, takes the output of the previous convolutional layer as input; the original feature map includes low-level feature maps and high-level feature maps, and the output of the first convolutional layer is denoted as the low-level feature map F. low The output of the second convolutional layer is denoted as the high-level feature map F. high_1 The output of the third convolutional layer is denoted as the high-level feature map F. high_2 The output of the nth convolutional layer is denoted as the high-level feature map F. high_n-1 All of the above feature maps have the same number of channels; Step 3.

2. Upsample the n-1 high-level feature maps using bilinear interpolation to make the resolution of all high-level feature maps equal to F. low The same applies to F using a 3×3 convolution kernel. low Perform smoothing; Step 3.

3. Process the F low The sum is then element-wise added to each upsampled high-level feature map, and the result is F. high_1 F high_2 F high_n-1 The corresponding newly generated feature maps are P1, P2, ..., P n-1 .

4. The road defect detection method based on dynamic learning prototype and weakly supervised semantic segmentation as described in claim 3, characterized in that, The channel-spatial dual attention algorithm in Step 4.1 specifically includes the following steps: Step 4.1.

1. Place P m As the input feature map F, m is the index of the new feature map, m is a positive integer and m∈[1,n-1]. Global average pooling is performed on F to obtain the global feature matrix g. Step 4.1.

2. Construct the channel weight matrix w c w c Expressed as the following formula w c =ε[W2·δ(W1·g)] (3) Where W1 and W2 are the compression weight matrix and the expansion weight matrix respectively. Suppose the number of channels of a certain feature map is C, then after being transformed by W1, the number of channels of this feature map will become C', C' < C, and then after being transformed by W2, the number of channels will change back to C again; δ is the ReLU activation function, and ε is the Sigmoid activation function; Step 4.1.

3. Apply w to the input feature map F c The intermediate feature map F' is then obtained, and F' is expressed by the following formula. in This represents channel-wise multiplication; the information of pixel i in F' is represented as a feature vector f'. i Then, based on f' using the softmax function. i The weight value of pixel i in F' is obtained, and the weight value is used as the element to construct the spatial weight matrix w. s The weight value of pixel i in w s The position in the middle corresponds to f' i Its position in F'; Step 4.1.

4. Apply w sequentially to F. c With w s The final feature map F is obtained. final F final It is expressed as the following formula Then, F is processed by a 1x1 convolution kernel. final The number of channels is reduced to 1, and then the F function with 1 channel is applied. final The process is performed to generate a single-channel feature map H.

5. The road defect detection method based on dynamic learning prototype and weakly supervised semantic segmentation as described in claim 4, characterized in that, The clustering algorithm in Step 4.2 specifically includes the following steps: Step 4.2.

1. The element corresponding to pixel i in the single-channel feature map H is denoted as the feature value H. i Calculate all eigenvalues ​​H i The average value μ H and standard deviation β H and based on μ H and β H Calculate the dynamic threshold T p T p It is derived from the following formula T p =μ H +0.5b H (6) Step 4.2.

2. Represent the information of pixel i in the region enclosed by the bounding box as a feature vector f. i and all H i one by one with T p In comparison, if H is satisfied i >T p The condition will then be f i Set F c In the middle, set F c Each f in i As an entity; determine a positive integer as the critical value MinPts for the partition, and determine the initial radius C according to the type of road to be detected. d ; Step 4.2.

3. Combine the two eigenvectors f i with f i’ The vector obtained by subtraction is denoted as Δ. i, i', i' is f i’ The corresponding pixel number and i ≠ i', ||Δ i,i’ ||2 is f i with f i’ The distance between them; in set F c Take any one individual as the core individual, and then calculate F one by one. c The distance between other individuals and the core individual, if the distance does not exceed C d The individual is then placed in its neighborhood. After calculating the distances to all other individuals, if the number of individuals in the neighborhood is not less than the critical value MinPts, the core individual is marked as a usable individual; otherwise, it is marked as noise. Once an individual is marked, another unmarked individual is selected as the core individual, and the above process is repeated until F... c Until all individuals are marked; Step 4.2.

4. Select an available individual as the main individual, and compare whether there are the same individuals in the neighborhood of the main individual and the neighborhood of another available individual. If so, mark the available individual and the main individual as connectable. After comparing all other available individuals with the main individual, mark the available individual that serves as the main individual. Then select an unmarked available individual as the main individual and repeat the above process until all available individuals are marked; then connect all pairs of connectable available individuals, group all available individuals in the same topological structure into the same cluster, and number all clusters; Step 4.2.

5. Represent the cluster as C k,ts k is the cluster index. All clusters are updated iteratively, specifically including the following steps. I. Determine the radius Δ of each cluster k, ts, where ts is the current iteration number, initialized to 1, Δ k, The formula for calculating ts is as follows: Where C k,ts-1 For the cluster numbered k generated in the previous iteration, μ k,ts-1 C k,ts-1 All individuals in the middle, that is, all f i The average vector, C k,0 For the clusters obtained in Step 4.2.4; compare Δ k, ts and Δ k, If the values ​​of ts-1 and ts-1 are equal, then the iteration ends and C is directly set. k,ts-1 All individual pixels are grouped into the same region, denoted as C. k and C k Output the values; if they are not equal, proceed to step II. II. Take C k,ts-1 For each individual, calculate C one by one. k,ts-1 The distance between this individual and other individuals in the group, if the distance exceeds Δ k,ts Then both bodies will be moved from C. k,ts-1 Remove from the middle, and complete this calculation; if the distance never exceeds Δ k, If ts, all individuals are retained to complete the current calculation; after one calculation is completed, another uncalculated individual is taken to start the next calculation, until all individuals in the cluster have completed the calculation; then C k,ts-1 The remaining individuals form a new cluster C. k,ts Then return to step I.

6. The road defect detection method based on dynamic learning prototype and weakly supervised semantic segmentation as described in claim 5, characterized in that, The process of generating the similarity feature map in Step 4.3 specifically includes the following steps: Step 4.3.

1. Calculate the weighted centroid vector p for each region using the following formula. k Where w i For the eigenvector f i The corresponding weight value; Step 4.3.

2. Calculate all f i The average vector μ c Then, the deviation feature vector δf corresponding to the feature vector of each pixel within the bounding box region is calculated. i And the deviation centroid vector δp corresponding to the weighted centroid vector of each region. k ,δf i With δp k The calculation formula is as follows: Step 4.3.

3. Calculate the similarity s corresponding to pixel i. c,i s c,i Through δf corresponding to pixel i i and δp corresponding to the region where pixel i is located k The calculation formula is as follows: Where λ is the scaling factor and exp is the natural exponential function; s c,i As elements constituting the similarity feature map S c s c,i In S c The position in the middle corresponds to H i Its position in H.

7. The road defect detection method based on dynamic learning prototype and weakly supervised semantic segmentation as described in claim 6, characterized in that, In Step 4.4, the fused feature map F is generated. out The process specifically includes the following steps: Step 4.4.

1. Place P1, P2, ..., P n-1 Each of these is processed as an input feature map F, P1, P2, ..., P n-1 The corresponding similarity feature maps are denoted as S. c,1 S c,2 S c,n-1 ; Step 4.4.

2. Calculate the weighted average of all similarity feature maps to obtain the total similarity feature map S. context S context It is expressed as the following formula Where UP represents the upsampling operation, S c,m For P m When the input feature map F is used as the corresponding generated similarity feature map, all S c,m The feature maps generated after the upsampling operation have the same resolution, α. m For S c,m The corresponding weight value; Step 4.4.

3. Use the semantic segmentation model U-Net to identify non-road parts in the real image, and output the identification results as a binary mask, denoted as M. road Then combine with M road The spatial model yields the road extent in the real image within M... road The corresponding part, whose mask is denoted as M' road ; M' road With S context Element-wise multiplication yields the enhanced feature map F. p ; Step 4.4.

4. Place F low The input feature map F is used to derive the corresponding similarity feature map, which is denoted as F. v Then merge F p With F v The fused feature map F is obtained. out F out Obtained through the following formula Where γ is the adjustment matrix, J is a matrix with all elements being 1, the number of rows and columns of J is the same as that of γ, and ⊕ represents element-wise multiplication.

8. The road defect detection method based on dynamic learning prototype and weakly supervised semantic segmentation as described in claim 7, characterized in that, In Step 4.4.2, S is obtained. context The specific process is as follows: Determine all α m initial value α m,0 The initial total similarity feature map S is obtained through formula (11). context,0 Then, using a convolutional neural network through S context,0 The distribution of median values ​​is used to determine the spatial dependencies of all pixels within the bounded area of ​​the bounding box, and based on the determination results, each S... c,m Corresponding weight value α m Make adjustments to make S context The spatial dependency of the mid-pixel is strengthened; the adjusted weight value is denoted as α. m,1 Then, the pixels are reweighted and superimposed to obtain S, which has a strong spatial dependency relationship. context,1 , will S context,1 As the final S context .

9. The road defect detection method based on dynamic learning prototype and weakly supervised semantic segmentation as described in claim 8, characterized in that, The adjustment matrix γ in Step 4.4.4 is specifically determined through the following process: The same convolutional neural network is used to identify road defects multiple times within the bounding box region, with random perturbations in each identification process. The identification results for each time are masked with M. t The data is recorded in the form of t, where t is the number of recognitions; then, the number of times pixel i is recognized across all M is calculated. t The variance σ of the corresponding eigenvector i The variance of all pixels within the bounded region needs to be calculated; then, the region enclosed by the bounding box is divided into multiple confidence regions, with each confidence region corresponding to a certain numerical range, based on each σ... i The range of values ​​within which pixel i is located divides it into the corresponding confidence region. Different confidence regions have different confidence values, and the magnitude of the confidence value is positively correlated with the median value of the range of values ​​corresponding to the confidence region. The confidence value of the confidence region where pixel i is located is the confidence value of pixel i. An adjustment matrix γ is constructed using the confidence values ​​of pixels as elements, and the position of the confidence value of pixel i in the adjustment matrix γ corresponds to s. c,i Its position in the similarity feature map.

10. The road defect detection method based on dynamic learning prototype and weakly supervised semantic segmentation as described in claim 9, characterized in that, The specific process of determining the type of defect in Step 6 is: Step 6.

1. Collect real road defect contour data to establish a database. The contour lines in the database are denoted as α. j j is the outline number, and each α j All are labeled with the corresponding defect type; for F dilate The contour of the defect range is identified, and the identified contour line is denoted as Ω. Then, an α is taken. j , in Ω and α j Take m sampling points from each of the above, number the m sampling points sequentially, and in addition, take a threshold L. threshold Then, the relationship between Ω and α is described using a loss function. j The difference between them is expressed by the loss function as follows: Where L edge The value is the loss, l is the sampling point number, and n is the sampling point number. l,Ω Let n be the unit vector perpendicular to the contour line at the l-th sampling point on Ω. l,j For α j The unit vector perpendicular to the contour line at the l-th sampling point; Step 6.

2. Change α multiple times j The sampling point distribution locations are determined, and L is calculated for each location distribution case. edge The minimum value among them is denoted as α. j The corresponding L edge_j ; Step 6.

3. Obtain an α j The corresponding L edge_j Then take another uncalculated α. j Perform calculations until all α values ​​in the database are included. j All yielded the corresponding L edge_j ; Step 6.

4. Put all L edge_j With L threshold Compare them one by one, if it is L edge_j Less than L threshold Then we consider Ω and α j They belong to the same type of defect.

Images