Unmanned aerial vehicle road property inspection image recognition method and system based on semantic segmentation

Abstract

The invention discloses an unmanned aerial vehicle road property inspection image recognition method and system based on semantic segmentation, and the method comprises the steps: carrying out the spatial partitioning of a single-source road property inspection image, which is aerially photographed by an unmanned aerial vehicle, according to a size label of a road property target, after obtaining a plurality of road property inspection sub-images containing complete local road property areas, inputting the road property inspection sub-images into a feature pyramid semantic segmentation model, adjusting the robustness loss weight of the model based on shooting angle deviation and illumination non-uniform features, and outputting abnormal shooting correction sub-images; adjusting the feature extraction weight of the small target attention mechanism module, extracting the features of the small target road property region, and splicing and integrating the features to obtain a semantic segmentation result; extracting a shape texture feature set according to a semantic segmentation result, matching the shape texture feature set with a preset road property target classification library to obtain classification information, extracting state features, and performing evolution analysis to determine an abnormal state; and performing spatial geographical coupling analysis by using the classification information, the abnormal state and the unmanned aerial vehicle flight log, and generating an inspection report carrying road property position marks.

Description

Technical Field

[0001] This invention relates to the field of image recognition technology, and in particular to a method and system for image recognition of unmanned aerial vehicle (UAV) roadside inspection based on semantic segmentation. Background Technology

[0002] With the continuous increase in highway mileage, the quantity and types of road assets (such as guardrails, signs, and streetlights) are becoming increasingly complex. Traditional manual inspection methods suffer from low efficiency, high cost, and limited coverage, making them insufficient to meet the needs of modern highway asset management. Against this backdrop, drone aerial photography technology, with its advantages of high flexibility, wide coverage, and high inspection efficiency, is gradually being applied to the field of road asset inspection. Currently, the technology of using drone aerial images for road asset identification has achieved some development, with related technologies mainly focusing on image segmentation and target recognition. For example, existing technologies often use semantic segmentation models to process road property inspection images captured by drones to identify road property targets in the images. Meanwhile, to improve the model's adaptability, some technologies adjust the model based on shooting conditions (such as angle and lighting), but these adjustments are mostly fixed parameter adjustments and lack specificity. Furthermore, for the classification and status assessment of road property targets, existing technologies mostly rely on matching single features, making it difficult to achieve accurate classification and in-depth status analysis. Regarding the correlation between road property information and geospatial data, existing technologies mostly simply record the geographical location of road property, lacking deep coupling analysis with drone flight logs. Therefore, improving the accuracy and efficiency of road property identification is a technical problem that needs to be solved. Summary of the Invention

[0003] This invention provides a method and system for image recognition of roadside inspection by unmanned aerial vehicles (UAVs) based on semantic segmentation.

[0004] One embodiment of the present invention provides a semantic segmentation-based image recognition method for UAV roadside inspection, applied to a UAV roadside inspection image recognition system, the method comprising: Based on the size label of the road property target, the single-source road property inspection image taken by the UAV is spatially segmented to obtain multiple road property inspection sub-images, each of which contains a complete local road property area. Each road property inspection sub-image is input into the feature pyramid semantic segmentation model. The robustness loss weight of the feature pyramid semantic segmentation model is adjusted based on the shooting angle deviation and uneven lighting features of each road property inspection sub-image. The abnormal shooting correction sub-image corresponding to each road property inspection sub-image is output through the feature pyramid semantic segmentation model. The small target attention mechanism module of the feature pyramid semantic segmentation model is adjusted to optimize the feature weighting of each abnormal shooting correction sub-image. Based on the feature weighting, the small target road property area features of each abnormal shooting correction image are extracted. The small target road property area features of all abnormal shooting correction images are spliced ​​and integrated according to their spatial positions to obtain the semantic segmentation result of the single-source road property inspection image. Based on the semantic segmentation results, shape and texture feature sets of road property targets are extracted. The shape and texture feature sets are matched with category features in a preset road property target classification library to obtain road property target classification information. The road property target classification information is used to extract state features of road property targets from the semantic segmentation results. Based on the state features, state evolution analysis is performed to determine abnormal states of road property targets. Using the road property target classification information and the abnormal status of the road property targets, a spatial geographic coupling analysis of the UAV's flight logs is performed to generate an inspection report with road property location markings.

[0005] One embodiment of the present invention provides an image recognition system for unmanned aerial vehicle (UAV) roadside inspection, comprising: A processor; a storage device having a computer program stored thereon; a network interface for providing network communication functions; when the computer program is executed by the processor, the processor enables the processor to implement any of the aforementioned semantic segmentation-based UAV roadside inspection image recognition methods.

[0006] One embodiment of the present invention provides a readable storage medium on which a program or instruction is stored. When the program or instruction is executed by a processor, it implements the steps of the image recognition method for UAV roadside inspection based on semantic segmentation.

[0007] This invention achieves end-to-end optimization of UAV road property inspection image recognition through a collaborative logic of spatial segmentation, robustness correction, small target feature enhancement, semantic segmentation, classification and state evolution analysis, and spatial geographic coupling. First, spatial segmentation based on road property target size labels ensures that each road property inspection sub-image contains a complete local road property area. Second, the robustness loss weights of the feature pyramid semantic segmentation model are dynamically adjusted to address shooting angle deviations and uneven lighting characteristics. Combined with feature enhancement weight adjustments from the small target attention mechanism module, this solves the problem of small target road property area features being easily obscured under complex shooting conditions, improving the accuracy and completeness of semantic segmentation results. Third, classification information is obtained by matching shape and texture feature sets with a preset road property target classification library. Anomalies are determined by combining state feature evolution analysis, achieving in-depth mining of road property targets from "identification" to "state assessment." Finally, spatial geographic coupling analysis of road property target classification information, anomalies, and UAV flight logs is used to generate inspection reports with location annotations, achieving precise correlation between road property information and geographic space. The embodiments of the present invention overcome the technical bottlenecks in traditional road property inspection image recognition, such as non-targeted segmentation, insufficient model robustness, difficulty in extracting small target features, isolated state assessment, and lack of geographical association, thereby improving the intelligence level, accuracy, and efficiency of road property inspection. Attached Figure Description

[0008] To more clearly illustrate the technical solutions in the embodiments of the present invention or related technologies, the drawings used in the description of the embodiments or related technologies will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0009] Figure 1 This is a flowchart of an image recognition method for UAV roadside inspection based on semantic segmentation, provided in an embodiment of the present invention.

[0010] Figure 2 This is a schematic diagram of the basic structure of an image recognition system for unmanned aerial vehicle (UAV) roadside inspection provided in an embodiment of the present invention.

[0011] Figure 3 This is a functional module block diagram of an image recognition device for unmanned aerial vehicle (UAV) roadside inspection provided in an embodiment of the present invention. Detailed Implementation

[0012] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0013] Please see Figure 1 , Figure 1 This is a flowchart of a UAV road property inspection image recognition method based on semantic segmentation provided by an embodiment of the present invention. The method can be executed by a UAV road property inspection image recognition system, or it can be executed jointly by the UAV road property inspection image recognition system and a server. The method may include steps 110-150.

[0014] Step 110: Based on the size label of the road property target, perform spatial block processing on the single-source road property inspection image taken by the UAV to obtain multiple road property inspection sub-images. Each road property inspection sub-image contains a complete local road property area.

[0015] In this embodiment of the invention, a UAV road property inspection image recognition system acquires single-source road property inspection images taken by a UAV. These images contain road property targets, such as guardrails, signs, and streetlights. Each road property target has a corresponding size label, which contains pre-labeled actual size information of the road property target. The system first reads the size labels of the road property targets to determine the size range of different road property targets. For example, the length of a guardrail ranges from L1 to L2, and the width of a sign ranges from W1 to W2.

[0016] Next, the system determines the size of the spatial blocks based on the size labels of the road property targets, so that each road property inspection sub-image after the block can contain the complete local road property area. For example, a larger block size is set for guardrails with a larger length, and a relatively smaller block size is set for signs with a smaller size.

[0017] Then, the system performs spatial block processing on the single-source road property inspection image according to the determined block size, dividing the single-source road property inspection image into multiple road property inspection sub-images. Each road property inspection sub-image contains at least one complete local road property area. For example, the sub-image Img1 obtained after division contains a complete guardrail, and the sub-image Img2 contains a complete sign.

[0018] Step 120: Input each road property inspection sub-image into the feature pyramid semantic segmentation model, adjust the robustness loss weight of the feature pyramid semantic segmentation model based on the shooting angle deviation and uneven lighting features of each road property inspection sub-image, and output the abnormal shooting correction sub-image corresponding to each road property inspection sub-image through the feature pyramid semantic segmentation model.

[0019] In this embodiment of the invention, the UAV road property inspection image recognition system sequentially inputs each road property inspection sub-image obtained in step 110 into the feature pyramid semantic segmentation model. First, the system extracts the shooting angle deviation feature and the illumination unevenness feature of each road property inspection sub-image. The shooting angle deviation feature is determined by analyzing the degree of perspective distortion of the road property target in the sub-image. For example, the perspective distortion angle of the road property target guardrail in sub-image Img1 is A1. The illumination unevenness feature is determined by analyzing the gray value distribution of the sub-image. For example, the gray value variance of sub-image Img1 is V1.

[0020] Next, the system adjusts the robustness loss weights of the feature pyramid semantic segmentation model based on the shooting angle deviation and uneven lighting characteristics. For example, when the shooting angle deviation is large, the loss weights related to angle correction in the model are increased; when the degree of uneven lighting is high, the loss weights related to lighting compensation in the model are increased.

[0021] Then, the system applies the adjusted loss weights to the feature pyramid semantic segmentation model. The model processes the road inspection sub-images, corrects the shooting angle deviation and uneven lighting, and outputs the abnormal shooting correction sub-image corresponding to each road inspection sub-image. For example, the abnormal shooting correction sub-image corresponding to sub-image Img1 is CorrImg1, and the abnormal shooting correction sub-image corresponding to sub-image Img2 is CorrImg2.

[0022] Step 130: Adjust the feature weighting weight of the small target attention mechanism module of the feature pyramid semantic segmentation model for each abnormal shooting correction sub-image, extract the small target road property area features of each abnormal shooting correction sub-image according to the feature weighting weight, and stitch and integrate the small target road property area features of all abnormal shooting correction sub-images according to their spatial positions to obtain the semantic segmentation result of the single source road property inspection image.

[0023] Step 131: Obtain the resolution information and the proportion information of the small target road property area of ​​each abnormal shooting correction sub-image, determine the pixel coverage range of the small target road property area based on the resolution information, and determine the distribution density of the small target road property area in the sub-image based on the proportion information.

[0024] In this embodiment of the invention, the UAV road property inspection image recognition system first acquires the resolution information of each abnormal shooting correction sub-image. The resolution information includes the number of pixels in width and height of the sub-image. For example, the resolution of sub-image CorrImg1 is Width1×Height1, and the resolution of sub-image CorrImg2 is Width2×Height2. Next, the system extracts the proportion information of small target road property areas in each abnormal shooting correction sub-image. The proportion information is the ratio of the number of pixels in the small target road property area to the total number of pixels in the sub-image. For example, in sub-image CorrImg1, the number of pixels in the small target road property area is P1, and the total number of pixels is TotalP1, so the proportion information is P1 / TotalP1; in sub-image CorrImg2, the number of pixels in the small target road property area is P2, and the total number of pixels is TotalP2, so the proportion information is P2 / TotalP2.

[0025] Then, the system determines the pixel coverage area of ​​the small target road property region based on the resolution information. For example, for a sub-image CorrImg1 with a resolution of Width1×Height1, if the number of pixels in the small target road property region is P1, then the pixel coverage area is a rectangular area with a width of Wp1 and a height of Hp1, starting from the top-left pixel coordinates of the small target road property region. Wp1 and Hp1 are calculated through the relationship between P1 and the resolution. At the same time, the system determines the distribution density of the small target road property region in the sub-image based on the proportion information. The distribution density is the ratio of the proportion information of the small target road property region to the area of ​​the sub-image. For example, if the area of ​​the sub-image CorrImg1 is Width1×Height1, the distribution density is (P1 / TotalP1) / (Width1×Height1).

[0026] Step 132: Normalize the pixel coverage area to obtain the normalized pixel coverage area, normalize the distribution density to obtain the normalized distribution density, and set the initial value of the feature weighting weight based on the normalized pixel coverage area and the normalized distribution density. The smaller the normalized pixel coverage area, the larger the initial value of the feature weighting weight; the higher the normalized distribution density, the larger the initial value of the feature weighting weight.

[0027] In this embodiment of the invention, the UAV roadside inspection image recognition system first normalizes the pixel coverage area obtained in step 131, converting the pixel coverage area value to the range of 0 to 1. For example, the pixel coverage area of ​​sub-image CorrImg1 is Area1, the maximum possible pixel coverage area is MaxArea1, and the normalized pixel coverage area is Area1 / MaxArea1; the pixel coverage area of ​​sub-image CorrImg2 is Area2, the maximum possible pixel coverage area is MaxArea2, and the normalized pixel coverage area is Area2 / MaxArea2. Next, the system normalizes the distribution density, converting the distribution density value to the range of 0 to 1. For example, the distribution density of sub-image CorrImg1 is Density1, the maximum possible distribution density is MaxDensity1, and the normalized distribution density is Density1 / MaxDensity1; the distribution density of sub-image CorrImg2 is Density2, the maximum possible distribution density is MaxDensity2, and the normalized distribution density is Density2 / MaxDensity2. Then, the system sets the initial value of the feature weighting weight based on the normalized pixel coverage and normalized distribution density. The setting rule is that the smaller the normalized pixel coverage, the larger the initial value of the feature weighting weight; the higher the normalized distribution density, the larger the initial value of the feature weighting weight. For example, the normalized pixel coverage of sub-image CorrImg1 is 0.3, the normalized distribution density is 0.8, and the initial value of the feature weighting weight is set to 0.7; the normalized pixel coverage of sub-image CorrImg2 is 0.5, the normalized distribution density is 0.6, and the initial value of the feature weighting weight is set to 0.5.

[0028] Step 133: Input the initial value of the feature enhancement weight into the small target attention mechanism module. The small target attention mechanism module performs weighted processing on the channel dimension features of the abnormal shooting correction sub-image to enhance the feature response of the channel corresponding to the small target road area and suppress the feature response of the channel corresponding to the non-small target area.

[0029] In this embodiment of the invention, the UAV road property inspection image recognition system inputs the initial value of the feature enhancement weights obtained in step 132 into the small target attention mechanism module of the feature pyramid semantic segmentation model. The small target attention mechanism module first acquires the channel dimension features of the abnormal shooting correction sub-image. The channel dimension features are feature maps of different channels extracted by the model. For example, the channel dimension features of sub-image CorrImg1 include feature maps of channels C1, C2, ..., Cn. Next, the small target attention mechanism module performs weighted processing on the channel dimension features according to the initial value of the feature enhancement weights. For the feature map of the channel corresponding to the small target road property area, its weight is increased to enhance the feature response. For example, the feature map of channel C3 corresponding to the small target road property area in sub-image CorrImg1 has its weight increased to the initial weight multiplied by the initial value of the feature enhancement weights, 0.7. For the feature map of the channel corresponding to the non-small target area, its weight is decreased to suppress the feature response. For example, the feature map of channel C5 corresponding to the non-small target area has its weight decreased to the initial weight multiplied by (1-0.7).

[0030] Step 134: Perform spatial dimension feature aggregation on the weighted channel dimension features. Traverse each pixel of the abnormal shooting correction sub-image through a sliding window, aggregate the feature values ​​within the sliding window to obtain the local feature values ​​of the pixel, and extract the region where the local feature value exceeds the preset feature value as the small target road property region feature.

[0031] In this embodiment of the invention, the UAV road property inspection image recognition system first performs spatial dimension feature aggregation on the weighted channel dimension features obtained in step 133. The system sets the size of the sliding window, for example, the width of the sliding window is Sw1 and the height is Sw2. Next, the system traverses each pixel of the abnormal capture correction sub-image through the sliding window. For each pixel, the feature values ​​within the area covered by the sliding window are aggregated. The aggregation method can be summation, averaging, etc. For example, for pixel (x, y), the feature values ​​within the area covered by the sliding window are F1, F2, ..., Fm, and the aggregated local feature value is (F1 + F2 + ... + Fm) / m. Then, the system sets a preset feature value, for example, Th1. Finally, the system extracts regions where the local feature value exceeds the preset feature value Th1 as small target road property area features. For example, in sub-image CorrImg1, the region where the local feature value exceeds Th1 is the small target road property area feature Feat1, and in sub-image CorrImg2, the region where the local feature value exceeds Th1 is the small target road property area feature Feat2.

[0032] Step 135: Record the spatial location index of each abnormal shooting correction sub-image, wherein the spatial location index contains the row and column number of the abnormal shooting correction sub-image in the single-source road inspection image.

[0033] In this embodiment of the invention, when the UAV road property inspection image recognition system performs spatial block processing on a single-source road property inspection image to obtain road property inspection sub-images, it simultaneously records the spatial location index of each road property inspection sub-image within the single-source road property inspection image. After the road property inspection sub-image is processed by the feature pyramid semantic segmentation model to obtain anomaly correction sub-images, the system assigns the corresponding spatial location index to the anomaly correction sub-image. The spatial location index includes the row and column numbers of the anomaly correction sub-image within the single-source road property inspection image. For example, sub-image CorrImg1 has row number R1 and column number C1 within the single-source road property inspection image, and its spatial location index is (R1, C1); sub-image CorrImg2 has row number R2 and column number C2 within the single-source road property inspection image, and its spatial location index is (R2, C2).

[0034] Step 136: According to the order of the spatial location index, spatially contiguously stitch together the small target road property area features of all abnormal shooting correction sub-images, and map the stitched small target road property area features into the global features of the single-source road property inspection image to generate a semantic segmentation result containing all small target road property area features.

[0035] In this embodiment of the invention, the UAV road property inspection image recognition system first obtains the spatial location index of each abnormal shooting correction sub-image. Then, it performs spatial continuity stitching on the small target road property area features of all abnormal shooting correction sub-images according to the order of the spatial location index. For example, it stitches the small target road property area features Feat1 of sub-image CorrImg1, Feat2 of sub-image CorrImg2, etc., sequentially according to the order of row number from smallest to largest and column number from smallest to largest. Next, the system maps and converts the stitched small target road property area features into global features of the single-source road property inspection image. The mapping and conversion method is to place the stitched features into the corresponding positions in the single-source road property inspection image according to the spatial location index. Finally, the system generates a semantic segmentation result containing all small target road property area features. The semantic segmentation result clearly marks the position and feature information of each small target road property area in the single-source road property inspection image.

[0036] Step 140: Extract the shape and texture feature set of the road property target based on the semantic segmentation result, match the shape and texture feature set with the category features in the preset road property target classification library to obtain road property target classification information, use the road property target classification information to extract the state features of the road property target from the semantic segmentation result, and perform state evolution analysis based on the state features to determine the abnormal state of the road property target.

[0037] Step 141: Extract the contour information and texture information of each road property target from the semantic segmentation result. The contour information includes the boundary point sequence of the road property target, and the texture information includes the gray value distribution sequence of the road property target.

[0038] In this embodiment of the invention, the UAV road property inspection image recognition system first extracts the contour information of each road property target from the semantic segmentation result obtained in step 136. The contour information is the boundary contour of the road property target, including the boundary point sequence of the road property target. The boundary point sequence is the pixel coordinates on the boundary of the road property target arranged in order, for example, the boundary point sequence of a guardrail is (x1, y1), (x2, y2), ..., (xn, yn). Next, the system extracts the texture information of each road property target. The texture information is the texture feature of the surface of the road property target, including the gray value distribution sequence of the road property target. The gray value distribution sequence is the gray value of the pixels arranged in order within the area of ​​the road property target, for example, the gray value distribution sequence of a sign is G1, G2, ..., Gm.

[0039] Step 142: Traverse the boundary point sequence, calculate the direction change value of adjacent boundary points and fit the continuous trend curve of the direction change value, extract the curve feature parameters of the continuous trend curve; calculate the curvature value of each boundary point, determine the concavity and convexity attributes of the boundary points based on the curvature value and statistically analyze the distribution pattern of the concavity and convexity attributes; determine the shape descriptor based on the curve feature parameters and the distribution pattern.

[0040] In this embodiment of the invention, the UAV roadside inspection image recognition system first traverses the boundary point sequence obtained in step 141. For every two adjacent boundary points, their direction vectors are calculated. For example, the direction vectors of adjacent boundary points (xi, yi) and (xi+1, yi+1) are (xi+1-xi, yi+1-yi). Next, the angle between adjacent direction vectors is calculated to obtain the direction change value of adjacent boundary points. For example, if the angle between direction vectors V1 and V2 is θ, then the direction change value is θ. Then, the system arranges the direction change values ​​of all adjacent boundary points in order and fits them to obtain a continuous trend curve. For example, a polynomial fitting method is used to fit the sequence of direction change values ​​to obtain a continuous trend curve. Next, the system extracts the curve feature parameters of the continuous trend curve, including the slope, intercept, and number of extreme points. For example, the slope of the continuous trend curve is K1, the intercept is B1, and the number of extreme points is N1. Simultaneously, the system calculates the curvature value for each boundary point. The curvature value is calculated by taking the curve's curvature at that point's coordinates; for example, the curvature value for boundary point (xi, yi) is Curvaturei. Then, the system determines the concavity / convexity attribute of the boundary point based on the curvature value. A positive curvature value indicates a convex boundary point, while a negative curvature value indicates a concave boundary point. For example, a positive Curvaturei indicates a convex boundary point (xi, yi), and a negative Curvaturei indicates a concave boundary point (xi, yi). Next, the system statistically analyzes the distribution patterns of the concavity / convexity attributes. These patterns include the number of convex boundary points, the number of concave boundary points, and their location. For example, the number of convex boundary points is M1, the number of concave boundary points is M2, and convex boundary points are mainly distributed at the two ends of the boundary. Finally, the system determines the shape descriptor based on the curve's feature parameters and the distribution patterns. The shape descriptor is a feature vector containing the curve's feature parameters and the distribution patterns; for example, the shape descriptor is (K1, B1, N1, M1, M2).

[0041] Step 1421: Traverse each pair of adjacent boundary points in the boundary point sequence, calculate the vector direction from the previous boundary point to the next boundary point, and subtract the current vector direction from the vector direction of the previous pair of adjacent boundary points to obtain the direction change value.

[0042] In this embodiment of the invention, the UAV roadside inspection image recognition system traverses each pair of adjacent boundary points in the boundary point sequence obtained in step 141. For example, if the boundary point sequence is (x1, y1), (x2, y2), (x3, y3), ..., (xn, yn), then the adjacent boundary point pairs are ((x1, y1), (x2, y2)), ((x2, y2), (x3, y3)), ..., ((xn-1, yn-1), (xn, yn)). For each pair of adjacent boundary points ((xi, yi), (xi+1, yi+1)), the system calculates the vector direction from the previous boundary point (xi, yi) to the next boundary point (xi+1, yi+1). The vector direction is obtained by calculating the angle between the vector and the positive x-axis. For example, if the angle between the vector (xi+1-xi, yi+1-yi) and the positive x-axis is αi, then the vector direction is αi. Next, the system subtracts the current vector direction αi from the vector direction αi-1 of the previous pair of adjacent boundary points to obtain the direction change value Δαi=αi-αi-1.

[0043] Step 1422: Arrange the direction change values ​​of all adjacent boundary points in order, and fit the arranged direction change values ​​to obtain a continuous trend curve.

[0044] In this embodiment of the invention, the UAV roadside inspection image recognition system arranges the direction change values ​​of all adjacent boundary points obtained in step 1421 in sequence, for example, the direction change value sequence is Δα1, Δα2, ..., Δαn-1. Next, the system fits the arranged direction change values. The fitting method can be linear fitting, polynomial fitting, etc. For example, a quadratic polynomial is used to fit the direction change value sequence to obtain a continuous trend curve with the equation Δα=a*t²+b*t+c, where t is the index of the adjacent boundary point pair, and a, b, and c are the fitting coefficients.

[0045] Step 1423: Extract the number of peak points, the number of valley points, and the overall slope of the continuous trend curve as curve feature parameters.

[0046] In this embodiment of the invention, the UAV roadside inspection image recognition system first analyzes the continuous trend curve obtained in step 1422 to find the peak points and valley points of the curve. Peak points are the points corresponding to the local maximum values ​​in the curve, and valley points are the points corresponding to the local minimum values ​​in the curve. Next, the system counts the number of peak points and valley points; for example, the number of peak points is Pn, and the number of valley points is Vn. Then, the system calculates the overall slope characteristic of the curve, which is the average slope of the curve. For example, if the starting point of the curve is (t1, Δα1) and the ending point is (tn-1, Δαn-1), then the overall slope is (Δαn-1-Δα1) / (tn-1-t1). Finally, the system uses the number of peak points, the number of valley points, and the overall slope characteristic as curve feature parameters.

[0047] Step 1424: Calculate the curvature value at each boundary point based on the coordinates of the boundary point and the preset number of adjacent boundary points before and after it, and determine the concavity or convexity attribute of the boundary point according to the sign of the curvature value; where a positive curvature value corresponds to a convex attribute and a negative curvature value corresponds to a concave attribute.

[0048] In this embodiment of the invention, the UAV roadside inspection image recognition system first sets a preset number, for example, k. Then, for each boundary point (xi, yi), the system acquires the coordinates of that boundary point and its k adjacent boundary points before and after it, i.e., (xi-k, yi-k), ..., (xi-1, yi-1), (xi, yi), (xi+1, yi+1), ..., (xi+k, yi+k). Next, the system calculates the curvature value at the boundary point (xi, yi) based on these coordinates. The calculation method is to fit a circle containing these points; the curvature of the circle is the curvature value of the boundary point. For example, if the radius of the fitted circle is R, then the curvature value is 1 / R. Then, the system determines the concavity / convexity attribute of the boundary point based on the sign of the curvature value; a positive curvature value indicates a convex boundary point, and a negative curvature value indicates a concave boundary point.

[0049] Step 1425: Calculate the proportion of convex attribute points, the proportion of concave attribute points, and the distribution of the continuous occurrence length of concave and convex attribute points among all boundary points to obtain the distribution pattern of concave and convex attributes.

[0050] In this embodiment of the invention, the UAV roadside inspection image recognition system first counts the number of convex and concave attribute points among all boundary points, for example, the number of convex attribute points is Cp, and the number of concave attribute points is Cc. Next, the system calculates the proportion of convex attribute points as Cp / (Cp+Cc) and the proportion of concave attribute points as Cc / (Cp+Cc). Then, the system statistically analyzes the distribution of consecutive occurrence lengths of convex and concave attribute points. This distribution represents the frequency distribution of consecutive occurrence lengths of convex and concave attribute points; for example, the frequency of consecutive occurrence lengths of convex attribute points being 1 is F1, and the frequency of consecutive occurrence lengths of 2 is F2, etc.; similarly, the frequency of consecutive occurrence lengths of concave attribute points being 1 is G1, and the frequency of consecutive occurrence lengths of 2 is G2, etc. Finally, the system uses the proportion of convex attribute points, the proportion of concave attribute points, and the distribution of consecutive occurrence lengths of convex and concave attribute points as the distribution patterns of convex and concave attributes.

[0051] Step 1426: Normalize the number of peak points, the number of valley points, and the overall slope feature in the curve feature parameters to obtain normalized curve feature parameters. Normalize the proportion of convex attributes, the proportion of concave attributes, and the distribution of consecutive occurrence lengths in the distribution pattern to obtain a normalized distribution pattern. Concatenate the number of peak points, the number of valley points, and the overall slope feature in the normalized curve feature parameters with the proportion of convex attributes, the proportion of concave attributes, and the distribution of consecutive occurrence lengths in the normalized distribution pattern in sequence to achieve feature combination and obtain a shape descriptor.

[0052] In this embodiment of the invention, the UAV roadside inspection image recognition system first normalizes the number of peak points, the number of valley points, and the overall slope features in the curve feature parameters obtained in step 1423, converting their values ​​to the range of 0 to 1. For example, the maximum value of the number of peak points is Pmax, and the normalized number of peak points is Pn / Pmax; the maximum value of the number of valley points is Vmax, and the normalized number of valley points is Vn / Vmax; the maximum value of the overall slope is Smx, and the minimum value is Smin, and the normalized overall slope is (S-Smin) / (Smx-Smin). Next, the system normalizes the proportion of convex attributes, the proportion of concave attributes, and the distribution of consecutive occurrence lengths in the distribution pattern obtained in step 1425. The proportion of convex attributes and the proportion of concave attributes are already in the range of 0 to 1 and do not need to be normalized; the normalization method for the distribution of consecutive occurrence lengths is to divide the frequency of each consecutive occurrence length by the maximum frequency. For example, the frequency of consecutive occurrence length of 1 is F1, the maximum frequency is Fmax, and the normalized frequency is F1 / Fmax. Then, the system concatenates the peak number, valley number, and overall slope features in the normalized curve characteristic parameters with the proportion of convex attributes, the proportion of concave attributes, and the continuous occurrence length distribution in the normalized distribution law in sequence. For example, the concatenation order is: normalized peak number, normalized valley number, normalized overall slope, proportion of convex attributes, proportion of concave attributes, and normalized continuous occurrence length distribution, to obtain the shape descriptor.

[0053] Step 143: Determine the multi-directional calculation rules, perform gray-level co-occurrence matrix calculation on the gray-level value distribution sequence based on the multi-directional calculation rules, and obtain the matrix feature parameters of the gray-level co-occurrence matrix; determine the window size of the local binary mode according to the texture density of the road property target area, calculate the gray-level value distribution sequence based on the window size, and obtain the mode features; combine the matrix feature parameters and the mode features to determine the texture descriptor.

[0054] In this embodiment of the invention, the UAV roadside inspection image recognition system first determines multi-directional calculation rules, which define multiple directions, such as 0°, 45°, 90°, and 135°. Next, based on these multi-directional calculation rules, the system calculates a gray-level co-occurrence matrix (GLCM) on the gray-level value distribution sequence obtained in step 141. The GLCM describes the distribution pattern of gray-level values ​​in the image at different directions and distances. The calculation method involves statistically analyzing the frequency of gray-level value combinations between two pixels at a distance *d* in the gray-level value distribution sequence for each direction, generating the GLCM. Then, the system extracts the matrix feature parameters of the GLCM, including contrast, correlation, and energy. For example, contrast is the variance of the elements in the GLCM, correlation is the degree of linear correlation between the elements, and energy is the sum of squares of the elements. Simultaneously, the system determines the window size of the local binary pattern based on the texture density of the target area. Texture density refers to the density of texture within the target area; the higher the texture density, the smaller the window size. For example, a texture density of D1 corresponds to a window size of W1. Next, the system calculates the pattern features based on the grayscale value distribution sequence using the window size. The calculation method for the local binary pattern involves comparing the grayscale value of each pixel with that of its neighboring pixels within the window to obtain the binary pattern. The frequency of occurrence of the binary pattern across all pixels is then counted as the pattern feature. Finally, the system combines the matrix feature parameters and the pattern features to determine the texture descriptor, which is a feature vector containing both the matrix feature parameters and the pattern features.

[0055] Step 1431: Determine the multi-directional calculation rules, which include a preset number of directional angles, each directional angle corresponding to the adjacent direction of a pixel within the target area of ​​the road property.

[0056] In this embodiment of the invention, the UAV road property inspection image recognition system determines multi-directional calculation rules. The preset number of directional angles can be four, namely 0°, 45°, 90°, and 135°. Each directional angle corresponds to the adjacent direction of a pixel in the road property target area. The 0° direction is horizontal to the right, the 45° direction is upper right, the 90° direction is vertical upward, and the 135° direction is upper left.

[0057] Step 1432: Based on the multi-directional calculation rule, for each pixel in the gray value distribution sequence, find its neighboring pixels at each directional angle, count the frequency of gray value combinations of neighboring pixel pairs, and generate a gray co-occurrence matrix corresponding to each directional angle.

[0058] In this embodiment of the invention, the UAV roadside inspection image recognition system first selects a distance d, for example, d=1. Then, for each pixel (x, y) in the grayscale value distribution sequence, it finds its neighboring pixels at each directional angle. For example, at 0°, the neighboring pixels are (x+1, y); at 45°, they are (x+1, y+1); at 90°, they are (x, y+1); and at 135°, they are (x-1, y+1). Next, the system counts the frequency of grayscale value combinations between neighboring pixel pairs, for example, the number of times the grayscale value combination (g1, g2) appears is Count, generating a grayscale co-occurrence matrix corresponding to each directional angle. The rows and columns of the grayscale co-occurrence matrix are grayscale values, and the matrix elements are the frequencies of the corresponding grayscale value combinations.

[0059] Step 1433: Extract the contrast, correlation and energy features of each gray-level co-occurrence matrix as matrix feature parameters, and average the matrix feature parameters corresponding to all directions and angles to obtain the comprehensive matrix feature parameters.

[0060] In this embodiment of the invention, the UAV roadside inspection image recognition system first extracts the contrast, correlation, and energy features of each gray-level co-occurrence matrix. Contrast is calculated by multiplying the square of (ij) of each element (i, j) in the gray-level co-occurrence matrix by the frequency of that element, and then summing the results. Correlation is calculated by calculating the mean and variance of the elements in the gray-level co-occurrence matrix, and then calculating the degree of linear correlation according to a formula. Energy is calculated as the sum of squares of the elements in the gray-level co-occurrence matrix. Next, the system averages the matrix feature parameters corresponding to all directional angles. For example, for contrast, the system calculates the average contrast across four directional angles to obtain the comprehensive contrast; for correlation, it calculates the average correlation across four directional angles to obtain the comprehensive correlation; and for energy, it calculates the average energy across four directional angles to obtain the comprehensive energy. Finally, the system uses the comprehensive contrast, comprehensive correlation, and comprehensive energy as comprehensive matrix feature parameters.

[0061] Step 1434: Calculate the texture density of the target area based on the gray value change frequency of pixels within the area, and determine the window size of the local binary mode according to the texture density. The larger the texture density, the smaller the corresponding window size.

[0062] In this embodiment of the invention, the UAV road property inspection image recognition system first calculates the grayscale value change frequency of pixels within the road property target area. The grayscale value change frequency is the ratio of the number of adjacent pixels with different grayscale values ​​to the total number of adjacent pixels. For example, if there are N pixels in the road property target area, M adjacent pixels, and K adjacent pixels with different grayscale values, then the grayscale value change frequency is K / M. Next, the system determines the texture density based on the grayscale value change frequency. The texture density is directly proportional to the grayscale value change frequency; for example, the texture density D = K / M * 100. Then, the system determines the window size of the local binary mode based on the texture density and sets a threshold for the texture density. For example, when the texture density is greater than T1, the window size is 3×3; when the texture density is between T2 and T1, the window size is 5×5; and when the texture density is less than T2, the window size is 7×7, where T1 > T2.

[0063] Step 1435: Based on the window size, for each pixel in the gray value distribution sequence, calculate the gray value difference between all its adjacent pixels and the center pixel within the window, and obtain the binary mode value according to the sign of the difference.

[0064] In this embodiment of the invention, the UAV roadside inspection image recognition system first sets the window size, for example, 3×3. Then, for each pixel (x, y) in the grayscale value distribution sequence, it is used as the center pixel, and adjacent pixels within the window are obtained. For example, the pixels within the window are (x-1, y-1), (x-1, y), (x-1, y+1), (x, y-1), (x, y+1), (x+1, y-1), (x+1, y), (x+1, y+1). Next, the system calculates the grayscale value difference between each adjacent pixel and the center pixel. If the difference is greater than 0, it is recorded as 1; if the difference is less than or equal to 0, it is recorded as 0. These binary numbers are arranged in order to obtain a binary pattern value, for example, 10101010.

[0065] Step 1436: Count the frequency of occurrence of binary pattern values ​​of all pixels to obtain pattern features; normalize the contrast, correlation and energy features in the comprehensive matrix feature parameters to obtain normalized comprehensive matrix feature parameters; normalize the frequency of occurrence of binary pattern values ​​in the pattern features to obtain normalized pattern features; concatenate the contrast, correlation and energy features in the normalized comprehensive matrix feature parameters with the frequency of occurrence of binary pattern values ​​in the normalized pattern features in order to achieve feature fusion and obtain texture descriptor.

[0066] In this embodiment of the invention, the UAV roadside inspection image recognition system first counts the frequency of occurrence of binary pattern values ​​for all pixels. For example, the number of times the binary pattern value 10101010 appears is F1, and the number of times the binary pattern value 11110000 appears is F2, etc., to obtain pattern features. Next, the system normalizes the contrast, correlation, and energy features in the comprehensive matrix feature parameters obtained in step 1433, converting their values ​​to the range of 0 to 1. For example, the maximum value of contrast is Cmax, and the minimum value is Cmin, so the normalized contrast is (C-Cmin) / (Cmax-Cmin); the maximum value of correlation is Rmax, and the minimum value is Rmin, so the normalized correlation is (R-Rmin) / (Rmax-Rmin); the maximum value of energy is Emax, and the minimum value is Emin, so the normalized energy is (E-Emin) / (Emax-Emin). Then, the system normalizes the frequency of binary pattern values ​​in the pattern features by dividing the frequency of each binary pattern value by the maximum frequency. For example, the frequency of the binary pattern value 10101010 is F1, the maximum frequency is Fmax, and the normalized frequency is F1 / Fmax. Finally, the system concatenates the contrast, correlation, and energy features in the normalized composite matrix with the frequency of binary pattern values ​​in the normalized pattern features in a sequential order, such as normalized contrast, normalized correlation, normalized energy, and normalized binary pattern value frequency, to obtain the texture descriptor.

[0067] Step 144: Combine the shape descriptor with the texture descriptor to form a shape texture feature set.

[0068] In this embodiment of the invention, the UAV road inspection image recognition system merges the shape descriptor obtained in step 1426 with the texture descriptor obtained in step 1436. The merging method is to concatenate the shape descriptor and texture descriptor in sequence to obtain a shape and texture feature set. For example, the shape descriptor is (S1, S2, S3, S4, S5, S6), the texture descriptor is (T1, T2, T3, T4), and the merged shape and texture feature set is (S1, S2, S3, S4, S5, S6, T1, T2, T3, T4).

[0069] Step 145: Obtain all category features in the preset road product target classification library. Each category feature contains the standard shape descriptor and standard texture descriptor of the corresponding category.

[0070] In this embodiment of the invention, the UAV road property inspection image recognition system obtains all category features from a preset road property target classification library. This library stores category features for different types of road property targets, such as guardrail category features, sign category features, and street light category features. Each category feature includes a standard shape descriptor and a standard texture descriptor for the corresponding category. For example, the standard shape descriptor for the guardrail category feature is (SS1, SS2, SS3, SS4, SS5, SS6), and the standard texture descriptor is (ST1, ST2, ST3, ST4); the standard shape descriptor for the sign category feature is (SS1', SS2', SS3', SS4', SS5', SS6'), and the standard texture descriptor is (ST1', ST2', ST3', ST4').

[0071] Step 146: Calculate the similarity between the shape and texture feature set and each category feature based on the matching degree of the shape descriptor and the matching degree of the texture descriptor, and select the category corresponding to the category feature with the highest similarity as the road product target classification information.

[0072] In this embodiment of the invention, the UAV roadside inspection image recognition system first calculates the matching degree between the shape descriptor and the standard shape descriptor for each category feature. The matching degree is calculated by the Euclidean distance between the shape descriptor and the standard shape descriptor. The smaller the Euclidean distance, the higher the matching degree. For example, if the shape descriptor is S and the standard shape descriptor is SS, the Euclidean distance is sqrt((S1-SS1)²+(S2-SS2)²+…+(S6-SS6)²). Next, the system calculates the matching degree between the texture descriptor and the standard texture descriptor for each category feature. The matching degree is calculated by the Euclidean distance between the texture descriptor and the standard texture descriptor. The smaller the Euclidean distance, the higher the matching degree. For example, if the texture descriptor is T and the standard texture descriptor is ST, the Euclidean distance is sqrt((T1-ST1)²+(T2-ST2)²+…+(T4-ST4)²). Then, the system calculates the similarity between the shape / texture feature set and each category feature based on the matching degree of the shape descriptor and the matching degree of the texture descriptor. The similarity is calculated as a weighted sum of the shape descriptor matching degree and the texture descriptor matching degree. For example, similarity Sim = w1 * ShapeMatch + w2 * TextureMatch, where w1 and w2 are weights, and w1 + w2 = 1. Finally, the system selects the category corresponding to the feature with the highest similarity as the road property target classification information. For example, if the shape / texture feature set has the highest similarity to the guardrail category feature, then the road property target classification information is guardrail.

[0073] Step 147: Based on the road property target classification information, extract the status information of the corresponding category of road property targets from the semantic segmentation results; perform time-series analysis on the status information in conjunction with historical status information in historical inspection data to obtain the change pattern of the status information, and determine the abnormal status of road property targets based on the change pattern.

[0074] In this embodiment of the invention, the UAV road property inspection image recognition system first extracts the state information of the road property targets corresponding to the categories obtained in step 136 from the semantic segmentation results obtained in step 146, based on the road property target classification information obtained in step 146. The state information refers to the current state parameters of the road property targets. For example, the state information of guardrails includes the degree of deformation and the degree of damage; the state information of signs includes the area of ​​stains and the tilt angle. Next, the system acquires historical state information from historical inspection data. The historical state information refers to the state information of the road property target during previous inspections. For example, the degree of deformation of guardrails in historical inspections is D1, D2, ..., Dn. Then, the system performs time-series analysis on the current state information in conjunction with the historical state information to analyze the changing trend of the state information. For example, it calculates the rate of change of the state information. If the rate of change of the degree of deformation gradually increases, the change pattern is intensified deformation. Finally, the system determines the abnormal state of the road property targets based on the change pattern. For example, when the change pattern is intensified deformation, it determines that the guardrail is in an abnormal state.

[0075] Step 150: Using the road property target classification information and the abnormal status of the road property targets, perform spatial geographic coupling analysis on the UAV's flight logs to generate an inspection report with road property location markings.

[0076] In this embodiment of the invention, the UAV road property inspection image recognition system first acquires the UAV's flight log, which includes the UAV's flight location information, shooting time information, etc. Next, the system performs spatial-geographic coupling analysis on the flight log using road property target classification information and abnormal road property target states. This spatial-geographic coupling analysis associates the road property target's classification information, abnormal state, and spatial location information in the flight log to determine the road property target's geographical location. Then, the system generates an inspection report with road property location annotations. The inspection report includes information such as the road property target's classification information, abnormal state, and geographical location. For example, the inspection report might record a guardrail as a road property target, in a deformed abnormal state, with its geographical location as latitude and longitude (Lat1, Lon1).

[0077] Step 151: Extract spatiotemporal correlation data from the UAV flight log. The spatiotemporal correlation data includes the shooting time information and corresponding geographic coordinate information of the road property inspection images. Based on the correspondence between the shooting time information and the road property target information, bind the spatiotemporal correlation data with the road property target classification information and the road property target abnormal status.

[0078] In this embodiment of the invention, the UAV road property inspection image recognition system first extracts spatiotemporal correlation data from the UAV flight log. The flight log records the UAV's flight position at different times, i.e., geographic coordinate information, and the shooting time information of the road property inspection image. For example, the shooting time information is Time1, and the corresponding geographic coordinate information is (Lat1, Lon1); the shooting time information is Time2, and the corresponding geographic coordinate information is (Lat2, Lon2). Next, based on the correspondence between the shooting time information and the road property target information, the system binds the spatiotemporal correlation data with the road property target classification information and the road property target abnormal state. For example, if the road property target classification information in the road property inspection image corresponding to the shooting time information Time1 is guardrail, and the road property target abnormal state is deformation, then the spatiotemporal correlation data (Time1, (Lat1, Lon1)) is bound with the guardrail classification information and the deformation abnormal state.

[0079] Step 152: Construct a spatial geographic topology network for road property targets, map each road property target as a network node, use the spatial adjacency relationship between road property targets as network edges, and set the weight of the edge as the classification correlation degree of the road property target.

[0080] Step 1521: Extract the unique identifier, geographic coordinate information and corresponding classification information of each road asset target from the bound information; assign a unique node identifier to each road asset target, and bind the geographic coordinate information of the road asset target with the node identifier.

[0081] In this embodiment of the invention, the UAV road property inspection image recognition system first extracts the unique identifier, geographic coordinate information, and corresponding classification information of each road property target from the information bound in step 151. The unique identifier is the road property target's number; for example, the unique identifier of road property target 1 is ID1, its geographic coordinate information is (Lat1, Lon1), and its classification information is guardrail; the unique identifier of road property target 2 is ID2, its geographic coordinate information is (Lat2, Lon2), and its classification information is sign. Next, the system assigns a unique node identifier to each road property target. The node identifier can be a numerical number; for example, the node identifier of road property target 1 is Node1, and the node identifier of road property target 2 is Node2. Then, the system binds the geographic coordinate information of the road property target with the node identifier, for example, binding (Lat1, Lon1) to Node1, and (Lat2, Lon2) to Node2.

[0082] Step 1522: Taking each node as the center, traverse all other nodes in the spatial geographic topology network and calculate the spatial distance between the current node and other nodes; determine whether the nodes satisfy the spatial adjacency relationship based on the spatial distance. The adjacency condition of the spatial adjacency relationship is that the spatial distance is less than the preset adjacency threshold. Establish network edges for node pairs that satisfy the adjacency condition and record the identifiers of the two nodes of the network edge.

[0083] In this embodiment of the invention, the UAV roadside inspection image recognition system first traverses all other nodes in the spatial geographic topology network, using each node as the center. For example, using node Node1 as the center, it traverses nodes Node2, Node3, etc. Next, the system calculates the spatial distance between the current node and other nodes. The spatial distance is calculated based on geographic coordinates. For example, the geographic coordinates of node Node1 are (Lat1, Lon1), and the geographic coordinates of node Node2 are (Lat2, Lon2), with a spatial distance of sqrt((Lat2-Lat1)²+(Lon2-Lon1)²). Then, the system determines whether the nodes satisfy a spatial adjacency relationship based on the spatial distance. A preset adjacency threshold is Dth. If the spatial distance is less than Dth, the spatial adjacency relationship is satisfied. A network edge is established for node pairs that satisfy the adjacency condition, and the identifiers of the two nodes on the network edge are recorded. For example, if nodes Node1 and Node2 satisfy a spatial adjacency relationship, a network edge is established and recorded as (Node1, Node2).

[0084] Step 1523: Calculate the classification correlation degree of the two road property targets corresponding to each network edge based on the matching degree of the classification information of the two road property targets, and set the classification correlation degree as the weight of the corresponding network edge; generate an initial spatial geographic topology network containing all node identifiers, node geographic coordinates, network edges and edge weights.

[0085] In this embodiment of the invention, the UAV road property inspection image recognition system first calculates the classification correlation degree between two road property targets corresponding to each network edge based on the matching degree of their classification information. The classification correlation degree is calculated as follows: if the two road property targets have the same classification information, the classification correlation degree is 1; if the classification information is different but belongs to the same major category, such as both being traffic facilities, the classification correlation degree is 0.7; if the classification information is completely different, the classification correlation degree is 0.3. Next, the system sets the classification correlation degree as the weight of the corresponding network edge. For example, if the two road property targets corresponding to network edge (Node1, Node2) have the same classification information, the weight is set to 1; if the two road property targets corresponding to network edge (Node1, Node3) have the same classification information, the weight is set to 0.7. Then, the system generates an initial spatial geographic topology network containing all node identifiers, node geographic coordinates, network edges, and edge weights. The initial spatial geographic topology network is a graph structure, where nodes are road property targets, edges represent spatial adjacency relationships, and edge weights represent classification correlation degrees.

[0086] Step 1524: Based on whether the weight of an edge is lower than a preset weight threshold, perform redundant edge removal processing on the initial network, check the node connectivity of the processed network, and save the spatial geographic topology network that passes the connectivity check; traverse each node in the network and update the neighbor node list of the node, the neighbor node list containing the neighbor node identifier and the corresponding edge weight; sort the updated neighbor node list according to the size of the edge weight and save the sorted neighbor node list.

[0087] In this embodiment of the invention, the UAV roadside inspection image recognition system first sets a preset weight threshold, for example, 0.5. Next, the system performs redundant edge removal processing on the initial spatial geographic topology network based on whether the edge weight is lower than the preset weight threshold. If the edge weight is lower than the preset weight threshold, the edge is removed. For example, the weight of network edge (Node1, Node4) is 0.4, which is lower than 0.5, so the edge is removed. Then, the system performs a node connectivity check on the processed network. The node connectivity check checks whether all nodes in the network can be connected to each other through edges. If there are isolated nodes, the connectivity check fails, and the preset weight threshold needs to be adjusted or the network edges need to be re-established. If all nodes are connected, the connectivity check passes, and the spatial geographic topology network that passes the connectivity check is saved. Next, the system traverses each node in the network and updates the node's neighbor node list. The neighbor node list contains the neighbor node identifier and the corresponding edge weight. For example, the neighbor node list of node Node1 is (Node2, 1) and (Node3, 0.7). Then, the system sorts the updated list of adjacent nodes according to the edge weights. For example, sorting by edge weight from largest to smallest, the list of adjacent nodes of node Node1 becomes (Node2, 1) and (Node3, 0.7). Finally, the system saves the sorted list of adjacent nodes.

[0088] Step 153: Perform coupling analysis on the spatial geographic topology network, map the abnormal state of road assets to the corresponding nodes, identify the topological propagation path of the abnormal nodes and the spatial clustering characteristics of classification information, and obtain the coupling analysis results; combine the shooting time information to identify the abnormal state evolution trend of the same node at different times and the abnormal state synchronization of adjacent nodes, and add the abnormal state evolution trend and the abnormal state synchronization to the coupling analysis results.

[0089] In this embodiment of the invention, the UAV road property inspection image recognition system first performs coupling analysis on the spatial geographic topology network. This coupling analysis analyzes the propagation and clustering of abnormal states of road property targets within the spatial geographic topology network. Next, the system maps the abnormal states of road property targets to corresponding nodes. For example, if road property target 1 is in an abnormal state, node Node1 is marked as an abnormal node. Then, the system identifies the topological propagation path of the abnormal node. The topological propagation path is the path from the abnormal node to other nodes. For example, if the adjacent nodes of abnormal node Node1 are Node2 and Node3, and the adjacent node of Node2 is Node4, then the topological propagation path is Node1→Node2→Node4. Simultaneously, the system identifies the spatial clustering characteristics of classification information. Spatial clustering characteristics refer to the clustering of road property targets of the same classification information within the spatial geographic topology network. For example, road property targets of the guardrail category mainly cluster in a certain part of the network. Then, the system combines the shooting time information to identify the evolution trend of the abnormal state of the same node at different times. For example, node Node1 is in a slightly abnormal state at Time1 and in a severely abnormal state at Time2, with the evolution trend being an intensification of the abnormality. Simultaneously, the system identifies the synchronicity of abnormal states of adjacent nodes. This synchronicity refers to whether the changes in abnormal states of adjacent nodes are synchronized. For example, if nodes Node1 and Node2 both change from a normal state to an abnormal state at the same time, their synchronicity is high. Finally, the system adds the abnormal state evolution trend and abnormal state synchronicity to the coupling analysis results.

[0090] Step 1531: Extract the abnormal status information of each road asset target from the bound information, map the abnormal status information to the corresponding node of the spatial geographic topology network, and mark the abnormal status node as a key node.

[0091] In this embodiment of the invention, the UAV road property inspection image recognition system first extracts the abnormal state information of each road property target from the information bound in step 151. For example, the abnormal state information of road property target 1 is deformation abnormality, and the abnormal state information of road property target 2 is normal. Next, the system maps the abnormal state information to the corresponding nodes in the spatial geographic topology network. For example, road property target 1 corresponds to node Node1, which is marked as an abnormal state node; road property target 2 corresponds to node Node2, which is marked as a normal state node. Then, the system marks the abnormal state nodes as key nodes, for example, Node1 as a key node.

[0092] Step 1532: Traverse each key node and extract the identifiers of all its neighboring nodes and the weights of the corresponding network edges; normalize the weights of all neighboring edges of the key node to obtain normalized weights; calculate the propagation probability from the key node to each neighboring node based on the normalized weights; sort the neighboring nodes according to the propagation probability to determine the direct propagation nodes of the key node.

[0093] In this embodiment of the invention, the UAV roadside inspection image recognition system first traverses each key node, such as key node Node1, and extracts the identifiers of all its neighboring nodes and the weights of their corresponding network edges. The neighboring nodes are identified as Node2 and Node3, and their corresponding network edge weights are 1 and 0.7, respectively. Next, the system normalizes the weights of all neighboring edges of the key node. The normalized weights are calculated by dividing the weight of each neighboring edge by the sum of all neighboring edge weights. For example, the sum of neighboring edge weights is 1 + 0.7, so the normalized weight for Node2 is 1 / 1.7, and the normalized weight for Node3 is 0.7 / 1.7. Then, the system calculates the propagation probability from the key node to each neighboring node based on the normalized weights. The propagation probability is equal to the normalized weight; for example, the propagation probability from Node1 to Node2 is 0.588, and the propagation probability to Node3 is 0.412. Finally, the system sorts the neighboring nodes according to the propagation probability, for example, sorting them from largest to smallest, with the neighboring nodes in the order of Node2 and Node3. Finally, the system determines the direct propagation nodes of the key nodes. Direct propagation nodes are adjacent nodes whose propagation probability is greater than a preset propagation threshold. For example, if the preset propagation threshold is 0.5, the propagation probability of Node2 is 0.588, which is greater than 0.5, so it is determined to be a direct propagation node; the propagation probability of Node3 is 0.412, which is less than 0.5, so it is not determined to be a direct propagation node.

[0094] Step 1533: For each direct propagation node, repeat the steps of extracting its neighboring nodes and calculating the propagation probability to determine the indirect propagation nodes; integrate the direct and indirect propagation nodes to form the topological propagation path of the abnormal state.

[0095] In this embodiment of the invention, the UAV roadside inspection image recognition system first repeats step 1532 for each directly propagating node, such as Node2, to extract the identifiers of its neighboring nodes and the weights of their corresponding network edges. The neighboring nodes are identified as Node1 and Node4, and their corresponding network edge weights are 1 and 0.8, respectively. Next, the system normalizes the weights of all neighboring edges of Node2, resulting in a sum of 1 + 0.8. The normalized weight for Node1 is 1 / 1.8, and the normalized weight for Node4 is 0.8 / 1.8. Then, based on the normalized weights, the system calculates the propagation probability from Node2 to each neighboring node. The propagation probability from Node2 to Node1 is 0.556, and the propagation probability to Node4 is 0.444. Finally, the system sorts the neighboring nodes according to their propagation probabilities, in the order of Node1 and Node4. Then, the system identifies the direct propagation node for Node2, with a preset propagation threshold of 0.5. Node1's propagation probability is 0.556, which is greater than 0.5, so it is identified as a direct propagation node. However, Node1 is the original critical node and is not counted again. Node4's propagation probability is 0.444, which is less than 0.5, so it is uncertain whether it is a direct propagation node. Next, the system identifies the indirect propagation nodes, which are the direct propagation nodes of the direct propagation nodes (excluding the original critical nodes). There are no indirect propagation nodes here. Finally, the system integrates the direct and indirect propagation nodes to form the topology propagation path for the abnormal state, for example, the topology propagation path is Node1→Node2.

[0096] Step 1534: Traverse all nodes in the spatial geographic topology network, count the number and distribution of nodes with the same classification information; determine the spatial clustering area of ​​nodes with the same classification information based on the spatial distance and adjacency relationship between nodes, calculate the proportion of key nodes in the clustering area, determine the correlation strength between classification information and abnormal state, integrate the propagation path of all key nodes and the clustering area of ​​all classification information to obtain the coupling analysis result; traverse each clustering area in the coupling analysis result, record the node identifier and corresponding abnormal state information in the clustering area.

[0097] In this embodiment of the invention, the UAV roadside inspection image recognition system first traverses all nodes in the spatial geographic topology network, counting the number and distribution of nodes with the same category of information. For example, the number of nodes in the guardrail category is N1, distributed in the eastern part of the network; the number of nodes in the sign category is N2, distributed in the western part of the network. Next, the system determines spatial clustering areas of nodes with the same category of information based on the spatial distance and adjacency relationship between nodes. A spatial clustering area is a region where the spatial distance between nodes with the same category of information is less than a preset clustering threshold and they are interconnected through adjacency relationships. For example, if the preset clustering threshold is Dth2, the spatial distance between nodes Node1, Node2, and Node3 of the guardrail category is less than Dth2, and they are adjacent to each other, thus forming a spatial clustering area. Then, the system calculates the proportion of key nodes within the clustering area. The proportion is the ratio of the number of key nodes in the clustering area to the total number of nodes in the clustering area. For example, if there are M nodes in the clustering area and K key nodes, the proportion is K / M. Next, the system determines the correlation strength between classification information and abnormal states. The correlation strength is directly proportional to the proportion of numbers; the higher the proportion, the stronger the correlation. For example, if the proportion is 0.6, the correlation strength is 0.6. Then, the system integrates the propagation paths of all key nodes and the aggregation regions of all classification information to obtain the coupling analysis results. The coupling analysis results include information such as the propagation paths of key nodes, the spatial aggregation regions of classification information, and the correlation strength between classification information and abnormal states. Finally, the system traverses each aggregation region in the coupling analysis results, recording the node identifiers and corresponding abnormal state information within the aggregation region. For example, the node identifiers in aggregation region 1 are Node1, Node2, and Node3, and the corresponding abnormal state information is deformed abnormal, normal, and deformed abnormal.

[0098] Step 154: Based on the coupling analysis results, convert the geographic coordinate information into a location description in the inspection report and mark the location of each road property target; integrate the road property target classification information, abnormal status information and location marking information to generate an inspection report with road property location markings.

[0099] In this embodiment of the invention, the UAV road property inspection image recognition system first converts the geographic coordinate information into a location description in the inspection report based on the coupling analysis results obtained in step 153. The location description converts latitude and longitude coordinates into an actual geographic location description, for example, latitude and longitude (Lat1, Lon1) is converted into "left side of XX Expressway K100+500". Next, the system marks the location of each road property target and associates the location description with the road property target. Then, the system integrates road property target classification information, abnormal status information, and location annotation information to generate an inspection report carrying road property location annotations. Each record in the inspection report contains information such as road property target classification, abnormal status, and location description, for example, "guardrail, abnormal deformation, left side of XX Expressway K100+500".

[0100] Step 1541: Extract the geographic coordinate information and corresponding classification information and abnormal status information of each road property target from the coupling analysis results; based on the matching relationship between geographic coordinates and preset road segment information, the location description includes the road segment name, relative mileage position and the identifier of adjacent road property targets, and convert the geographic coordinate information of each road property target into a location description in the inspection report.

[0101] In this embodiment of the invention, the UAV road property inspection image recognition system first extracts the geographic coordinate information, corresponding classification information, and abnormal status information of each road property target from the coupling analysis results obtained in step 153. For example, the geographic coordinate information of road property target 1 is (Lat1, Lon1), the classification information is guardrail, and the abnormal status information is deformation anomaly; the geographic coordinate information of road property target 2 is (Lat2, Lon2), the classification information is signboard, and the abnormal status information is stain anomaly. Next, the system obtains preset road segment information, which includes the geographic coordinate range, road segment name, mileage information, etc. of different road segments. For example, the geographic coordinate range of XX Expressway from K100+000 to K101+000 is Lat range LatA to LatB, and Lon range LonA to LonB. Then, based on the matching relationship between geographic coordinates and preset road segment information, the system converts the geographic coordinate information of each road property target into a location description in the inspection report. The location description includes the road segment name, relative mileage position, and the identifier of adjacent road property targets. For example, the geographic coordinates (Lat1, Lon1) of road property target 1 belong to the range of K100+000 to K101+000 of XX Expressway, the relative mileage position is K100+500, and the identifier of the adjacent road property target is road property target 3. Then the location description is "XX Expressway K100+500, adjacent to road property target 3".

[0102] Step 1542: By comparing the consistency between the location description and the original geographic coordinate information, the accuracy of each location description is verified. The verified location descriptions are bound with the corresponding classification information and abnormal status information to form a complete information entry for each road asset target. All complete information entries are sorted by road segment name and relative mileage position. The sorted information entries are integrated to generate an inspection report containing a list of road asset targets, abnormal status statistics, and location annotations. Each information entry in the inspection report is traversed to supplement the abnormal status association information corresponding to the information entry. The association information includes the propagation path and clustering area of ​​the abnormal status.

[0103] In this embodiment of the invention, the UAV road property inspection image recognition system first verifies the accuracy of each location description by comparing its consistency with the original geographic coordinate information. For example, if the location description is "XX Expressway K100+500", and the corresponding original geographic coordinate information is (Lat1, Lon1), the system checks whether this geographic coordinate does indeed belong to the range of XX Expressway K100+500. If it does, the verification passes; otherwise, the verification fails, and the location description needs to be reconstructed. Next, the system binds the verified location description with the corresponding classification information and abnormal status information to form a complete information entry for each road property target. For example, the complete information entry is "Guardrail, Deformation Anomaly, XX Expressway K100+500, Adjacent Road Property Target 3". Then, the system sorts all complete information entries by road segment name and relative mileage position, for example, first sorting by road segment name, and then sorting within the same road segment by relative mileage position from smallest to largest. Next, the system integrates the sorted information entries and generates an inspection report containing a list of road property targets, anomaly statistics, and location annotations. The road property target list consists of the complete sorted information entries; the anomaly statistics count the number of road property targets with different anomalies, such as 5 road property targets with deformation anomalies and 3 road property targets with stain anomalies; the location annotations describe the location of the road property targets. Finally, the system iterates through each information entry in the inspection report and supplements the corresponding anomaly status association information. The association information includes the propagation path and clustering area of ​​the anomaly status. For example, the association information for the information entry "Guardrail, Deformation Anomaly, XX Expressway K100+500" is the propagation path Node1→Node2, and the clustering area is the guardrail clustering area from K100+000 to K100+600 on the XX Expressway.

[0104] In an optional embodiment, the method further includes: Step 210: Extract the shooting condition feature set corresponding to the abnormal state of road property from the inspection report with road property location markings. The shooting condition feature set includes the angle deviation type and the uneven lighting type of the abnormal road property sub-image. Map the shooting condition feature set to the loss weight adjustment rule library of the feature pyramid semantic segmentation model. The loss weight adjustment rule library stores the correspondence between shooting condition features and loss weight adjustment direction. The correspondence is based on the logical association of the impact of shooting conditions on model feature extraction. Determine the loss weight adjustment amount of the initial adjustment layer based on the mapping result. The initial adjustment layer is the feature extraction layer in the model that is directly related to the shooting condition features.

[0105] In this embodiment of the invention, the UAV road property inspection image recognition system first extracts the shooting condition feature set corresponding to the abnormal road property status from the inspection report carrying road property location markings. For example, if the abnormal road property status is deformation abnormality, the corresponding abnormal road property sub-image's angle deviation type is pitch angle deviation, and the illumination unevenness type is backlighting. Next, the system maps the shooting condition feature set to the loss weight adjustment rule library of the feature pyramid semantic segmentation model. The loss weight adjustment rule library stores the correspondence between shooting condition features and loss weight adjustment directions. For example, the loss weight adjustment direction corresponding to pitch angle deviation is to increase angle-related loss weight, and the loss weight adjustment direction corresponding to backlighting is to increase illumination-related loss weight. Then, based on the mapping result, the system determines the loss weight adjustment amount of the initial adjustment layer. The initial adjustment layer is the feature extraction layer in the model directly related to the shooting condition features. For example, the initial adjustment layer corresponding to the angle deviation type is the model's angle feature extraction layer, and the initial adjustment layer corresponding to the illumination unevenness type is the model's illumination feature extraction layer. The loss weight adjustment amount for the angle feature extraction layer is determined to be ΔW1, and the loss weight adjustment amount for the illumination feature extraction layer is determined to be ΔW2.

[0106] Step 220: The adjustment amount of the initial adjustment layer is used as input and passed to the adjacent upper-level feature fusion layer of the feature pyramid semantic segmentation model. The upper-level adjustment amount is determined according to the inter-layer feature transfer logic, which is the dependency relationship between the upper-level feature fusion layer and the lower-level feature extraction layer. The adjustment amount is repeatedly passed to higher layers of the model until it is passed to the output layer of the feature pyramid semantic segmentation model, forming a chain adjustment path covering the entire model. The adjustment amounts of each layer in the chain adjustment path are integrated to generate the global robustness loss weight parameter set of the feature pyramid semantic segmentation model. The global robustness loss weight parameter set contains the loss weight values ​​of each layer of the feature pyramid semantic segmentation model. The global robustness loss weight parameter set is applied to the next training process of the feature pyramid semantic segmentation model to output the adjusted model parameters.

[0107] In this embodiment of the invention, the UAV roadside inspection image recognition system first uses the adjustment amount of the initial adjustment layer as input and passes it to the adjacent upper-layer feature fusion layer of the feature pyramid semantic segmentation model. For example, the adjustment amount ΔW1 of the angle feature extraction layer is passed to the adjacent upper-layer feature fusion layer 1, and the adjustment amount ΔW2 of the illumination feature extraction layer is passed to the adjacent upper-layer feature fusion layer 2. Next, the system determines the upper-layer adjustment amount according to the inter-layer feature transfer logic, which is the dependency relationship between the upper-layer feature fusion layer and the lower-layer feature extraction layer. For example, the dependency coefficient of feature fusion layer 1 on the angle feature extraction layer is α1, and the upper-layer adjustment amount is ΔW1*α1; the dependency coefficient of feature fusion layer 2 on the illumination feature extraction layer is α2, and the upper-layer adjustment amount is ΔW2*α2. Then, the system repeatedly propagates the adjustment amounts to higher layers of the model. For example, the adjustment amount of feature fusion layer 1 is propagated to feature fusion layer 3, where the adjustment amount is determined as (ΔW1*α1)*α3 based on the inter-layer dependency coefficient α3; the adjustment amount of feature fusion layer 2 is propagated to feature fusion layer 4, where the adjustment amount is determined as (ΔW2*α2)*α4 based on the inter-layer dependency coefficient α4, and so on, until it reaches the output layer of the feature pyramid semantic segmentation model. Next, the system forms a chain-like adjustment path covering all layers of the model, recording the adjustment amount propagation process for each layer. Then, the system integrates the adjustment amounts from each layer in the chain-like adjustment path to generate a global robustness loss weight parameter set for the feature pyramid semantic segmentation model. This global robustness loss weight parameter set contains the loss weight values ​​for each layer of the model; for example, the loss weight value for the angle feature extraction layer is the original weight value plus ΔW1, and the loss weight value for feature fusion layer 1 is the original weight value plus ΔW1*α1, etc. Finally, the system applies the global robust loss weight parameter set to the next training process of the feature pyramid semantic segmentation model, trains the model using the new loss weight parameters, and outputs the adjusted model parameters.

[0108] In an optional embodiment, the method further includes: Step 310: Extract the multi-time-point state semantic sequence of the same road property target from the inspection report carrying road property location markings. The state semantic sequence includes the anomaly type and degree of the road property target. Convert the state semantic sequence into a semantic node sequence. Each semantic node corresponds to the state semantics of a time point. The semantic node contains state type and state duration information. Generate the evolutionary association relationship between semantic nodes based on the similarity of semantic nodes at adjacent time points, and determine the connection logic between nodes through the evolutionary association relationship. The similarity of semantic nodes at adjacent time points is determined based on the change logic of state type.

[0109] In this embodiment of the invention, the UAV road property inspection image recognition system first extracts a multi-time-point state semantic sequence of the same road property target from the inspection report carrying road property location markings. For example, the state semantic of road property target 1 at Time 1 is "abnormal deformation, slight", the state semantic of Time 2 is "abnormal deformation, moderate", and the state semantic of Time 3 is "abnormal deformation, severe". Next, the system converts the state semantic sequence into a semantic node sequence. Each semantic node corresponds to the state semantic of a time point and contains information on the state type and duration. For example, the semantic node corresponding to Time 1 is (abnormal deformation, slight, duration T1), the semantic node corresponding to Time 2 is (abnormal deformation, moderate, duration T2), and the semantic node corresponding to Time 3 is (abnormal deformation, severe, duration T3). Then, the system generates an evolutionary association relationship between semantic nodes based on the similarity of semantic nodes at adjacent time points. The similarity is determined based on the logic of state type changes. For example, if the state types of adjacent semantic nodes are both abnormal deformation, the similarity is high, and an evolutionary association relationship is generated; if the state type changes, the similarity is low, and no evolutionary association relationship is generated. Next, the system determines the connection logic between nodes through evolutionary association relationships. The connection logic is the evolutionary direction between semantic nodes. For example, the connection logic from (deformation abnormality, slight) to (deformation abnormality, moderate) is the aberration degree intensified.

[0110] Step 320: Construct a state semantic evolution path based on the connection logic. The state semantic evolution path includes semantic nodes and the evolution direction between nodes. The evolution direction is the change trend of state semantics. Determine the evolution trend type of the state semantic evolution path through the change pattern of semantic node types in the evolution path. Predict the subsequent state semantic nodes of the road asset target based on the evolution trend type. The subsequent nodes include the predicted state type and the predicted time point. Add the predicted subsequent state semantic nodes to the inspection report and output the updated inspection report. The updated inspection report contains the subsequent state prediction information.

[0111] In this embodiment of the invention, the UAV road property inspection image recognition system first constructs a state semantic evolution path based on the connection logic determined in step 220. The state semantic evolution path includes semantic nodes and the evolution direction between nodes. For example, the state semantic evolution path is (abnormal deformation, slight) → (abnormal deformation, moderate) → (abnormal deformation, severe), with the evolution direction being an increase in the degree of abnormality. Next, the system determines the evolution trend type of the state semantic evolution path through the changing patterns of the semantic node types, for example, the evolution trend type is continuously aggravated. Then, the system predicts the subsequent state semantic nodes of the road property target based on the evolution trend type. The subsequent nodes include the predicted state type and the predicted time point. For example, the predicted subsequent state semantic node is (abnormal deformation, extremely severe), and the predicted time point is Time4. Finally, the system adds the predicted subsequent state semantic nodes to the inspection report and outputs an updated inspection report. The updated inspection report includes subsequent state prediction information, for example, adding "When Time4 is predicted, the state of road property target 1 is abnormal deformation, extremely severe."

[0112] This invention achieves end-to-end optimization of UAV road property inspection image recognition through a collaborative logic of spatial segmentation, robustness correction, small target feature enhancement, semantic segmentation, classification and state evolution analysis, and spatial geographic coupling. First, spatial segmentation based on road property target size labels ensures that each road property inspection sub-image contains a complete local road property area. Second, the robustness loss weights of the feature pyramid semantic segmentation model are dynamically adjusted to address shooting angle deviations and uneven lighting characteristics. Combined with feature enhancement weight adjustments from the small target attention mechanism module, this solves the problem of small target road property area features being easily obscured under complex shooting conditions, improving the accuracy and completeness of semantic segmentation results. Third, classification information is obtained by matching shape and texture feature sets with a preset road property target classification library. Anomalies are determined by combining state feature evolution analysis, achieving in-depth mining of road property targets from "identification" to "state assessment." Finally, spatial geographic coupling analysis of road property target classification information, anomalies, and UAV flight logs is used to generate inspection reports with location annotations, achieving precise correlation between road property information and geographic space. The embodiments of the present invention overcome the technical bottlenecks in traditional road property inspection image recognition, such as non-targeted segmentation, insufficient model robustness, difficulty in extracting small target features, isolated state assessment, and lack of geographical association, thereby improving the intelligence level, accuracy, and efficiency of road property inspection.

[0113] Please see Figure 2The figure is a schematic diagram of the basic structure of a UAV road property inspection image recognition system 200 provided in an embodiment of the present invention. The UAV road property inspection image recognition system 200 includes: a processor 201; a storage device 202 on which a computer program 2020 is stored; and a network interface 203 for providing network communication functions. When the computer program 2020 is executed by the processor 201, the processor 201 implements any of the UAV road property inspection image recognition methods based on semantic segmentation.

[0114] Please see Figure 3 The present invention provides a functional module block diagram of a UAV road property inspection image recognition device, which includes: The spatial segmentation processing module is used to perform spatial segmentation processing on the single-source road property inspection images taken by UAVs based on the size labels of the road property targets, to obtain multiple road property inspection sub-images, each of which contains a complete local road property area. The robust loss adjustment module is used to input each road property inspection sub-image into the feature pyramid semantic segmentation model, adjust the robustness loss weight of the feature pyramid semantic segmentation model based on the shooting angle deviation and uneven lighting features of each road property inspection sub-image, and output the abnormal shooting correction sub-image corresponding to each road property inspection sub-image through the feature pyramid semantic segmentation model. The semantic segmentation output module is used to adjust the feature weighting weight of the small target attention mechanism module of the feature pyramid semantic segmentation model for each abnormal shooting correction sub-image, extract the small target road property area features of each abnormal shooting correction image according to the feature weighting weight, and stitch and integrate the small target road property area features of all abnormal shooting correction images according to their spatial positions to obtain the semantic segmentation result of the single source road property inspection image. The semantic segmentation analysis module is used to extract the shape and texture feature set of the road property target based on the semantic segmentation result, match the shape and texture feature set with the category features in the preset road property target classification library to obtain road property target classification information, use the road property target classification information to extract the state features of the road property target from the semantic segmentation result, and perform state evolution analysis based on the state features to determine the abnormal state of the road property target. The inspection report generation module is used to perform spatial-geographic coupling analysis of road property targets using the road property target classification information and the abnormal status of the road property targets on the UAV's flight logs, and generate an inspection report with road property location markings.

[0115] Based on the above, a readable storage medium is provided, on which a program or instructions are stored, and when the program or instructions are executed by a processor, the steps of the above method are implemented.

[0116] Furthermore, it should be noted that this embodiment of the invention also provides a computer program product, which may include a computer program that can be stored in a computer-readable storage medium. The processor of the UAV roadside inspection image recognition system reads the computer program from the computer-readable storage medium, and the processor can execute the computer program, causing the UAV roadside inspection image recognition system to perform the aforementioned... Figure 1 The methods described in the corresponding embodiments are already known, and therefore will not be repeated here. Furthermore, the beneficial effects of using the same method will also not be repeated. For technical details not disclosed in the computer program product embodiments related to this invention, please refer to the description of the method embodiments of this invention.

[0117] It should be noted that the various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the systems or apparatus disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the descriptions are relatively simple, and relevant parts can be referred to the method section.

Claims

1. A method for image recognition of UAV roadside inspection based on semantic segmentation, characterized in that, The method includes: Based on the size label of the road property target, the single-source road property inspection image taken by the UAV is spatially segmented to obtain multiple road property inspection sub-images, each of which contains a complete local road property area. Each road property inspection sub-image is input into the feature pyramid semantic segmentation model. The robustness loss weight of the feature pyramid semantic segmentation model is adjusted based on the shooting angle deviation and uneven lighting features of each road property inspection sub-image. The abnormal shooting correction sub-image corresponding to each road property inspection sub-image is output through the feature pyramid semantic segmentation model. The small target attention mechanism module of the feature pyramid semantic segmentation model is adjusted to optimize the feature weighting of each abnormal shooting correction sub-image. Based on the feature weighting, the small target road property area features of each abnormal shooting correction image are extracted. The small target road property area features of all abnormal shooting correction images are spliced ​​and integrated according to their spatial positions to obtain the semantic segmentation result of the single-source road property inspection image. Based on the semantic segmentation results, shape and texture feature sets of road property targets are extracted. The shape and texture feature sets are matched with category features in a preset road property target classification library to obtain road property target classification information. The road property target classification information is used to extract state features of road property targets from the semantic segmentation results. Based on the state features, state evolution analysis is performed to determine abnormal states of road property targets. Using the road property target classification information and the abnormal status of the road property targets, a spatial geographic coupling analysis of the UAV's flight logs is performed to generate an inspection report with road property location markings.

2. The method as described in claim 1, characterized in that, The small target attention mechanism module of the feature pyramid semantic segmentation model adjusts the feature weighting weights of each abnormal shooting correction sub-image, extracts the small target road property area features of each abnormal shooting correction image according to the feature weighting weights, and concatenates and integrates the small target road property area features of all abnormal shooting correction images according to their spatial positions to obtain the semantic segmentation result of the single-source road property inspection image, including: The resolution information and the proportion information of small target road property areas of each abnormal shooting correction sub-image are obtained. The pixel coverage range of the small target road property area is determined based on the resolution information, and the distribution density of the small target road property area in the sub-image is determined based on the proportion information. The pixel coverage is normalized to obtain the normalized pixel coverage, and the distribution density is normalized to obtain the normalized distribution density. The initial value of the feature weighting weight is set according to the normalized pixel coverage and the normalized distribution density. The smaller the normalized pixel coverage, the larger the initial value of the feature weighting weight. The higher the normalized distribution density, the larger the initial value of the feature weighting weight. The initial value of the feature enhancement weight is input into the small target attention mechanism module. The small target attention mechanism module performs weighted processing on the channel dimension features of the abnormal shooting correction sub-image, thereby enhancing the feature response of the channel corresponding to the small target road area and suppressing the feature response of the channel corresponding to the non-small target area. Spatial feature aggregation is performed on the weighted channel dimension features. Each pixel of the abnormal shooting correction sub-image is traversed through a sliding window. The feature values ​​within the sliding window are aggregated to obtain the local feature values ​​of the pixel. The region with local feature values ​​exceeding the preset feature value is extracted as the small target road property region feature. Record the spatial location index of each abnormal shooting correction sub-image, the spatial location index containing the row and column numbers of the abnormal shooting correction image in the single-source road property inspection image; According to the spatial location index, the small target road property area features of all abnormal shooting correction sub-images are spatially continuous and stitched together. The stitched small target road property area features are then mapped and converted into global features of the single-source road property inspection image, generating a semantic segmentation result containing all small target road property area features.

3. The method as described in claim 1, characterized in that, The process of extracting shape and texture feature sets of road property targets based on the semantic segmentation results, matching the shape and texture feature sets with category features in a preset road property target classification library to obtain road property target classification information, extracting state features of road property targets from the semantic segmentation results using the road property target classification information, and performing state evolution analysis based on the state features to determine abnormal states of road property targets includes: The contour information and texture information of each road property target are extracted from the semantic segmentation result. The contour information includes the boundary point sequence of the road property target, and the texture information includes the gray value distribution sequence of the road property target. Traverse the boundary point sequence, calculate the direction change value of adjacent boundary points and fit a continuous trend curve of the direction change value, extract the curve feature parameters of the continuous trend curve; calculate the curvature value of each boundary point, determine the concavity / convexity attribute of the boundary point based on the curvature value and statistically analyze the distribution pattern of the concavity / convexity attribute; determine the shape descriptor based on the curve feature parameters and the distribution pattern. A multi-directional calculation rule is determined, and a gray-level co-occurrence matrix is ​​calculated on the gray-level value distribution sequence based on the multi-directional calculation rule to obtain the matrix feature parameters of the gray-level co-occurrence matrix; the window size of the local binary mode is determined according to the texture density of the road property target area, and the gray-level value distribution sequence is calculated based on the window size to obtain the mode features; the texture descriptor is determined by combining the matrix feature parameters and the mode features. The shape descriptor and the texture descriptor are matched and combined to form a shape-texture feature set; Obtain all category features from the preset road property target classification library. Each category feature contains the standard shape descriptor and standard texture descriptor for the corresponding category. The similarity between the shape and texture feature set and each category feature is calculated based on the matching degree of the shape descriptor and the matching degree of the texture descriptor. The category corresponding to the category feature with the highest similarity is selected as the road property target classification information. Based on the road property target classification information, extract the state information of the corresponding category of road property targets from the semantic segmentation results; By combining historical status information from historical inspection data with time-series analysis of the status information, the change pattern of the status information is obtained, and the abnormal status of road property targets is determined based on the change pattern.

4. The method as described in claim 3, characterized in that, The process involves traversing the boundary point sequence, calculating the direction change value of adjacent boundary points, fitting a continuous trend curve of the direction change value, and extracting the curve feature parameters of the continuous trend curve. Calculate the curvature value of each boundary point, determine the concavity / convexity attribute of the boundary point based on the curvature value, and statistically analyze the distribution pattern of the concavity / convexity attribute; determine the shape descriptor based on the curve feature parameters and the distribution pattern, including: Traverse each pair of adjacent boundary points in the boundary point sequence, calculate the vector direction from the previous boundary point to the next boundary point, and subtract the current vector direction from the vector direction of the previous pair of adjacent boundary points to obtain the direction change value. Arrange the direction change values ​​of all adjacent boundary points in order, and fit the arranged direction change values ​​to obtain a continuous trend curve; The number of peak points, the number of valley points, and the overall slope of the continuous trend curve are extracted as curve feature parameters. The curvature value at each boundary point is calculated based on the coordinates of the boundary point and a preset number of adjacent boundary points before and after it. The concavity or convexity attribute of the boundary point is determined according to the sign of the curvature value; where a positive curvature value corresponds to a convex attribute and a negative curvature value corresponds to a concave attribute. By statistically analyzing the proportion of convex attribute points, the proportion of concave attribute points, and the distribution of the consecutive occurrence length of concave and convex attribute points among all boundary points, the distribution pattern of concave and convex attributes can be obtained. The number of peak points, the number of valley points, and the overall slope feature in the curve feature parameters are normalized to obtain normalized curve feature parameters. The proportion of convex attributes, the proportion of concave attributes, and the distribution of consecutive occurrence lengths in the distribution pattern are normalized to obtain normalized distribution patterns. The number of peak points, the number of valley points, and the overall slope feature in the normalized curve feature parameters are sequentially concatenated with the proportion of convex attributes, the proportion of concave attributes, and the distribution of consecutive occurrence lengths in the normalized distribution patterns to achieve feature combination and obtain shape descriptors.

5. The method as described in claim 3, characterized in that, The process involves determining multi-directional calculation rules, and then performing gray-level co-occurrence matrix calculation on the gray-level value distribution sequence based on these rules to obtain the matrix feature parameters of the gray-level co-occurrence matrix. The window size of the local binary pattern is determined based on the texture density of the target area of ​​the road property, and the gray value distribution sequence is calculated based on the window size to obtain the pattern features; By combining the matrix feature parameters and the pattern features, a texture descriptor is determined, including: Determine multi-directional calculation rules, which include a preset number of directional angles, each directional angle corresponding to the adjacent direction of a pixel within the target area of ​​the road property; Based on the multi-directional calculation rules, for each pixel in the gray value distribution sequence, its neighboring pixels are found at each directional angle, the frequency of gray value combinations of neighboring pixel pairs is counted, and a gray co-occurrence matrix corresponding to each directional angle is generated. The contrast, correlation, and energy features of each gray-level co-occurrence matrix are extracted as matrix feature parameters. The matrix feature parameters corresponding to all directions and angles are averaged to obtain the comprehensive matrix feature parameters. The texture density of the target area is calculated based on the frequency of grayscale value changes of pixels within the area. The window size of the local binary mode is determined according to the texture density. The larger the texture density, the smaller the corresponding window size. Based on the window size, for each pixel in the gray value distribution sequence, calculate the difference in gray value between all its adjacent pixels and the center pixel within the window, and obtain the binary mode value according to the sign of the difference; The frequency of occurrence of binary pattern values ​​for all pixels is counted to obtain pattern features; The contrast, correlation, and energy features in the comprehensive matrix feature parameters are normalized to obtain normalized comprehensive matrix feature parameters. The frequency of occurrence of binary pattern values ​​in the pattern features is normalized to obtain normalized pattern features. The contrast, correlation, and energy features in the normalized comprehensive matrix feature parameters are concatenated with the frequency of occurrence of binary pattern values ​​in the normalized pattern features in order to achieve feature fusion and obtain the texture descriptor.

6. The method as described in claim 1, characterized in that, The step of performing spatial-geographic coupling analysis of the UAV's flight logs using the road property target classification information and the abnormal status of the road property targets to generate an inspection report with road property location markings includes: Extract spatiotemporal correlation data from the UAV flight log, the spatiotemporal correlation data including the shooting time information and corresponding geographic coordinate information of the road inspection images; Based on the correspondence between shooting time information and road property target information, the spatiotemporal correlation data is bound to the road property target classification information and the road property target abnormal status. Construct a spatial geographic topology network for road property targets, mapping each road property target to a network node, using the spatial adjacency relationship between road property targets as network edges, and setting the weight of the edges as the classification correlation degree of the road property targets; A coupling analysis is performed on the spatial geographic topology network to map the abnormal state of road assets to the corresponding nodes, identify the topological propagation path of the abnormal nodes and the spatial clustering characteristics of the classification information, and obtain the coupling analysis results. By combining the shooting time information, the abnormal state evolution trend of the same node at different times and the abnormal state synchronization of adjacent nodes are identified, and the abnormal state evolution trend and the abnormal state synchronization are added to the coupling analysis results. Based on the coupling analysis results, the geographic coordinate information is converted into a location description in the inspection report, and the location of each road property target is marked. Based on the road property target classification information, abnormal status information and location marking information, an inspection report with road property location marking is generated.

7. The method as described in claim 6, characterized in that, The construction of the spatial geographic topology network for road property targets maps each road property target to a network node, uses the spatial adjacency relationships between road property targets as network edges, and sets the weight of the edges as the classification correlation degree of the road property targets, including: Extract the unique identifier, geographic coordinates, and corresponding classification information for each road asset target from the bound information; Assign a unique node identifier to each road asset target and bind the geographical coordinate information of the road asset target to the node identifier; Using each node as the center, traverse all other nodes in the spatial geographic topology network and calculate the spatial distance between the current node and other nodes. The spatial distance is used to determine whether the nodes satisfy the spatial adjacency relationship. The adjacency condition of the spatial adjacency relationship is that the spatial distance is less than the preset adjacency threshold. For the node pairs that satisfy the adjacency condition, a network edge is established, and the identifiers of the two nodes of the network edge are recorded. The classification correlation degree between the two road property targets corresponding to each network edge is calculated based on the matching degree of the classification information of the two road property targets, and the classification correlation degree is set as the weight of the corresponding network edge. Generate an initial spatial geographic topology network containing all node identifiers, node geographic coordinates, network edges and edge weights; Based on whether the weight of an edge is lower than a preset weight threshold, redundant edges are removed from the initial network, node connectivity is checked on the processed network, and the spatial geographic topology network that passes the connectivity check is saved. Traverse each node in the network and update the node's neighbor list, which includes neighbor node identifiers and corresponding edge weights; sort the updated neighbor list according to the edge weights and save the sorted neighbor list.

8. The method as described in claim 6, characterized in that, The coupling analysis performed on the spatial geographic topology network maps the abnormal state of road assets to corresponding nodes, identifies the topological propagation path of abnormal nodes and the spatial clustering characteristics of classification information, and obtains the coupling analysis results, including: Extract the abnormal status information of each road asset target from the bound information, map the abnormal status information to the corresponding node of the spatial geographic topology network, and mark the abnormal status node as a key node. Traverse each key node and extract the identifiers of all its adjacent nodes and the weights of the corresponding network edges; The weights of all adjacent edges of the key node are normalized to obtain normalized weights. The propagation probability from the key node to each adjacent node is calculated based on the normalized weights. The adjacent nodes are sorted according to the propagation probability to determine the direct propagation nodes of the key node. For each directly propagating node, repeat the steps of extracting its adjacent nodes and calculating the propagation probability to determine the indirect propagating nodes; integrate the directly propagating nodes and the indirect propagating nodes to form the topological propagation path of the abnormal state; Traverse all nodes in the spatial geographic topology network and count the number and distribution of nodes with the same category of information. Based on the spatial distance and adjacency relationship between nodes, the spatial clustering area of ​​nodes of the same category is determined, the proportion of key nodes in the clustering area is calculated, the correlation strength between the category information and the abnormal state is determined, and the propagation path of all key nodes and the clustering area of ​​all category information are integrated to obtain the coupling analysis results. Traverse each clustered region in the coupling analysis results and record the node identifiers and corresponding abnormal state information within the clustered region; Based on the coupling analysis results, the geographic coordinate information is converted into location descriptions in the inspection report, and the location of each road asset target is marked; the road asset target classification information, abnormal status information, and location marking information are integrated to generate an inspection report with road asset location markings, including: The geographic coordinates, classification information, and abnormal status information of each road asset target are extracted from the coupling analysis results. Based on the matching relationship between geographic coordinates and preset road segment information, the location description includes the road segment name, relative mileage position and the identifier of adjacent road property targets, and converts the geographic coordinate information of each road property target into a location description in the inspection report; By comparing the consistency between the location description and the original geographic coordinate information, the accuracy of each location description is verified. The verified location descriptions are then bound to the corresponding classification information and abnormal status information to form a complete information entry for each road asset target. All complete information entries are sorted by road segment name and relative mileage location. The sorted information entries are integrated to generate an inspection report that includes a list of road property targets, abnormal status statistics and location markings. Iterate through each information entry in the inspection report and supplement the abnormal status association information corresponding to the information entry. The association information includes the propagation path and aggregation area of ​​the abnormal status.

9. A drone-based roadside inspection image recognition system, characterized in that, include: processor; Storage device on which computer programs are stored; network interface for providing network communication functions; When the computer program is executed by the processor, the processor implements the image recognition method for UAV roadside inspection based on semantic segmentation as described in any one of claims 1-8.

10. A readable storage medium, characterized in that, The readable storage medium stores a program or instructions, which, when executed by a processor, implement the image recognition method for UAV roadside inspection based on semantic segmentation as described in any one of claims 1-8.

Images