Method for inspecting road infrastructure by cooperating unmanned aerial vehicle with high-resolution remote sensing

Abstract

The present application relates to a method for inspecting road infrastructure by cooperating unmanned aerial vehicles with high-resolution remote sensing, and relates to the field of intelligent road inspection, comprising: obtaining multiple road shoulder recognition accuracies according to the discrimination degrees of multiple road shoulder inspection areas; performing extraction processing on road shoulder detection images according to the road shoulder recognition accuracies to obtain multiple first road shoulder detection image sets, and recognizing multiple road shoulder damage degrees; performing extraction processing according to the multiple road shoulder damage degrees and the multiple road shoulder recognition accuracies to obtain multiple second road shoulder detection image sets, performing road shoulder drainage anomaly recognition to obtain multiple drainage anomaly degrees, and calculating multiple road shoulder anomaly degrees as road shoulder inspection results in combination with the multiple road shoulder damage degrees. The present application can solve the technical problems of traditional methods lacking a dynamic adjustment mechanism, being unable to set a matching inspection strategy according to the road shoulder state, and having low inspection efficiency and insufficient accuracy; and can realize efficient and accurate road shoulder inspection, and significantly improve the inspection efficiency, accuracy and inspection quality.

Description

Technical Field

[0001] This invention relates to the field of intelligent road inspection, and more particularly to a method for inspecting road infrastructure using a combination of unmanned aerial vehicles (UAVs) and high-resolution remote sensing. Background Technology

[0002] Road infrastructure inspection is a regular inspection and maintenance process of roads and their ancillary facilities (such as shoulders, bridges, drainage systems, lighting facilities, etc.). The purpose of the inspection is to ensure the safety, stability and functionality of road facilities, to identify potential problems in a timely manner, to prevent accidents, to reduce maintenance costs, and to improve the quality of road services.

[0003] Currently, traditional shoulder inspection methods lack real-time feedback and dynamic adjustment mechanisms. Once the inspection plan and route are determined, they are often not adjusted to adapt to the actual situation. For example, the damage to the shoulder may change with seasonal changes, traffic flow changes, or weather conditions. However, traditional inspection methods cannot flexibly adjust the inspection focus and precision according to these changes, which makes the inspection inefficient and unable to effectively focus on high-risk areas, thus failing to detect and deal with damaged areas in a timely manner. Summary of the Invention

[0004] This invention addresses the technical problems of traditional road shoulder inspection methods, which lack a dynamic adjustment mechanism and cannot set matching inspection strategies according to the road shoulder status, resulting in low inspection efficiency and insufficient accuracy. It provides a road infrastructure inspection method that combines UAVs and high-resolution remote sensing to solve these problems.

[0005] The technical solution of the present invention to solve the above-mentioned technical problems is as follows:

[0006] In a first aspect, the present invention provides a method for road infrastructure inspection using a combination of unmanned aerial vehicles (UAVs) and high-resolution remote sensing, comprising: acquiring high-resolution remote sensing images of a target road using high-resolution remote sensing; identifying and dividing shoulder images to obtain shoulder images; performing discriminability analysis on the shoulder images to obtain a shoulder discriminability map; filtering to obtain multiple shoulder inspection areas; and determining multiple shoulder recognition accuracies based on the discriminability of the multiple shoulder inspection areas; controlling the UAV to acquire video of the multiple shoulder inspection areas to obtain multiple shoulder detection videos; extracting shoulder detection images from the multiple shoulder detection videos according to the multiple shoulder recognition accuracies to obtain multiple first shoulder detection image sets; identifying shoulder damage to obtain multiple shoulder damage degrees; extracting shoulder detection images from the multiple shoulder detection videos according to the multiple shoulder damage degrees and multiple shoulder recognition accuracies to obtain multiple second shoulder detection image sets; identifying shoulder drainage anomalies to obtain multiple drainage anomaly degrees; and calculating multiple shoulder anomaly degrees based on the multiple shoulder damage degrees, which serve as the shoulder inspection results.

[0007] Secondly, the present invention also provides an electronic device, comprising:

[0008] At least one processor; a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor to enable the at least one processor to perform the steps of the method described in any one of the first aspects above.

[0009] Thirdly, a computer-readable storage medium storing a computer program that, when executed, implements the steps of the method described in any one of the first aspects above.

[0010] The beneficial effects of this invention are as follows: High-resolution remote sensing images of the target road are acquired, and shoulder images are identified and segmented to obtain shoulder images. Then, the shoulder images are analyzed for discriminability to obtain a shoulder discriminability map. Multiple shoulder inspection areas are selected, and based on the discriminability of these multiple shoulder inspection areas, a decision is made regarding the identification accuracy of multiple shoulders. Next, a drone is controlled to acquire video of the multiple shoulder inspection areas, obtaining multiple shoulder detection videos. Shoulder detection images are extracted from these videos according to the identified accuracy, resulting in multiple first shoulder detection image sets. Shoulder damage is then identified, and the results are obtained. Multiple shoulder damage levels are assessed. Finally, based on these multiple shoulder damage levels and identification accuracy, shoulder detection images are extracted from the multiple shoulder detection videos to obtain multiple second shoulder detection image sets. Shoulder drainage anomaly identification is then performed to obtain multiple drainage anomaly levels. These are combined with the multiple shoulder damage levels to calculate multiple shoulder anomaly scores, which serve as the shoulder inspection results. This method allows for dynamic adjustment of inspection accuracy based on shoulder status, quickly focusing on high-risk areas of shoulder damage. This optimizes resource utilization, promptly identifies potential road damage and drainage problems, and achieves efficient and accurate shoulder inspection, significantly improving inspection efficiency, accuracy, and quality. Attached Figure Description

[0011] Figure 1 A flowchart illustrating the road infrastructure inspection method using a combination of unmanned aerial vehicles (UAVs) and high-resolution remote sensing provided by this invention.

[0012] Figure 2 This is a schematic diagram of the structure of the electronic device provided by the present invention;

[0013] Figure 3 This is a schematic diagram of the structure of a computer-readable storage medium provided by the present invention.

[0014] The components represented by each number in the attached diagram are explained below:

[0015] Electronic device 500, memory 510, processor 520, first computer program 511, computer-readable storage medium 600, second computer program 611. Detailed Implementation

[0016] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0017] In the description of this invention, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the stated features. In the description of this invention, "a plurality of" means two or more, unless otherwise explicitly specified.

[0018] In the description of this invention, the term "for example" is used to mean "used as an example, illustration, or description." Any embodiment described as "for example" in this invention is not necessarily to be construed as being more preferred or advantageous than other embodiments. The following description is provided to enable any person skilled in the art to make and use the invention. Details are set forth in the following description for purposes of explanation. It should be understood that those skilled in the art will recognize that the invention can be made without using these specific details. In other instances, well-known structures and processes will not be described in detail to avoid obscuring the description of the invention with unnecessary detail. Therefore, the invention is not intended to be limited to the embodiments shown, but is consistent with the broadest scope of the principles and features disclosed herein.

[0019] Example 1, as Figure 1 As shown in the figure, this invention provides a method for road infrastructure inspection using a combination of unmanned aerial vehicles (UAVs) and high-resolution remote sensing, specifically including the following steps:

[0020] S100: High-resolution remote sensing images of the target road are acquired through high-resolution remote sensing, and shoulder images are identified and divided to obtain shoulder images.

[0021] Furthermore, step S100 of the present invention further includes:

[0022] S110: High-resolution remote sensing images of the target road are acquired through high-resolution remote sensing.

[0023] Specifically, firstly, based on the size and complexity of the area where the target road is located, as well as the required image resolution, an appropriate remote sensing platform (such as satellite remote sensing) is selected, and a sensor (optical sensor, lidar, etc.) is chosen according to the required image resolution and application needs. Next, the target road is photographed and scanned using the remote sensing platform (such as satellite remote sensing) to obtain high-resolution remote sensing images of the target road. By utilizing high-resolution remote sensing to acquire high-resolution images of the target road, large-scale image data of the target road area can be quickly obtained, providing data support for the next step of shoulder area identification.

[0024] S120: Pre-trained shoulder area recognizer, which performs shoulder area recognition on the high-resolution remote sensing image to obtain the shoulder area.

[0025] Furthermore, step S120 of the present invention further includes:

[0026] S121: Based on historical data of road shoulder detection, collect a set of sample high-resolution remote sensing images, and identify the shoulder area within each sample high-resolution remote sensing image to obtain a set of sample shoulder areas; S122: Based on a convolutional neural network, construct the network structure of the shoulder area recognizer, and train the shoulder area recognizer using the set of sample high-resolution remote sensing images and the set of sample shoulder areas until convergence; S123: Input the high-resolution remote sensing images into the shoulder area recognizer and output the obtained shoulder areas.

[0027] Specifically, the process involves acquiring historical shoulder detection data for the target road, such as collecting past shoulder detection data from historical inspection reports, road maintenance records, sensor data, or manual inspection data. Then, multiple high-resolution remote sensing images of different scenarios are collected from the historical shoulder detection data to obtain a set of sample high-resolution remote sensing images. Next, the shoulder areas within each sample high-resolution remote sensing image are identified (e.g., by experts manually identifying the shoulder areas in the images), resulting in a set of sample shoulder areas, where there is a one-to-one correspondence between the sample high-resolution remote sensing images and the sample shoulder areas.

[0028] The network structure of the road shoulder region identifier is based on a convolutional neural network. The road shoulder region identifier includes an input layer, a convolutional layer, a pooling layer, a fully connected layer, and an output layer. The input data of the input layer is high-resolution remote sensing image. The convolutional layer is used to extract local features of the image through filters (convolutional kernels). The pooling layer is used to reduce the spatial size of the image, reduce the amount of computation, and retain important spatial feature information. The fully connected layer is used to map high-dimensional features to the output layer for classification or regression tasks. The output layer is used to output the road shoulder region.

[0029] Next, using high-resolution remote sensing images as input and roadside areas as output, the roadside area recognizer is trained under supervised supervision using the set of high-resolution remote sensing images and roadside areas as training data. First, the input high-resolution remote sensing images are fed into the network, processed layer by layer through convolutional and pooling layers, ultimately producing a predicted output (predicted roadside areas). Then, a loss function is used to measure the difference between the model output and the ground truth annotations, calculating the loss value. Further, the network weights are updated by calculating the gradient of the loss function relative to each network parameter. Backpropagation utilizes the chain rule, propagating the error layer by layer to calculate the gradient of each parameter. An optimization algorithm (such as Adam) is used to adjust the network weights and biases based on the calculated gradients. Then, iterative training is performed using the sample training data, gradually optimizing the network weights so that the model can better fit the training data. As training progresses, the loss function value gradually decreases, and the network gradually optimizes until the loss function converges, meaning it no longer decreases significantly. This indicates that the model has learned effective features, resulting in a successfully trained roadside area recognizer.

[0030] Finally, the high-resolution remote sensing image is input into the shoulder area recognizer for identification, and the shoulder area is output. By constructing a shoulder area recognizer based on a convolutional neural network, the intelligence and automation of shoulder area recognition can be improved, while also increasing recognition accuracy and efficiency.

[0031] S130: Cropping the high-resolution remote sensing image within the shoulder area to obtain the shoulder image.

[0032] Specifically, the process involves acquiring labeled data for the shoulder area, which includes the precise location of the shoulder area—that is, which parts of the image belong to the shoulder area. Then, based on the labeled shoulder area coordinates, the corresponding portion is cropped from the high-resolution remote sensing image, extracting the identified shoulder area from the entire image to obtain the shoulder image. Obtaining the shoulder image supports subsequent shoulder area inspection and analysis.

[0033] S200: Perform a discrimination analysis on the shoulder image to obtain a shoulder discrimination map, filter out multiple shoulder inspection areas, and determine the recognition accuracy of multiple shoulders based on the discrimination of the multiple shoulder inspection areas.

[0034] Furthermore, step S200 of the present invention further includes:

[0035] S210: Perform grayscale processing on the shoulder image, select a first shoulder pixel in the grayscale shoulder image, and extract Q neighboring grayscale values ​​from Q neighboring pixels near the first shoulder pixel, where Q is an integer greater than 1; S220: Randomly extract Q random grayscale values ​​from Q random pixels in the grayscale shoulder image, and calculate the deviation ratio between the mean of the Q neighboring grayscale values ​​and the mean of the Q random grayscale values, as the first distinguishability; S230: Continue to calculate and obtain the total distinguishability of all shoulder pixels to obtain a shoulder distinguishability map; S240: Filter shoulder pixels with a distinguishability greater than a preset distinguishability threshold to obtain multiple shoulder inspection areas.

[0036] Specifically, firstly, the road shoulder image is converted to grayscale to obtain a grayscale road shoulder image. Grayscale conversion refers to the process of converting a color image (such as an RGB image) into a grayscale image by converting RGB values ​​into brightness values. Each pixel value in a grayscale image typically ranges from 0 to 255, representing different grayscale levels from black to white. Brighter areas have higher grayscale values, and darker areas have lower grayscale values. Next, an arbitrary road shoulder pixel is randomly selected within the grayscale road shoulder image and designated as the first road shoulder pixel. The grayscale values ​​of Q neighboring pixels surrounding the first road shoulder pixel are then extracted. The selection of neighboring pixels is generally based on a local region (such as a rectangular or circular neighborhood), and Q is an integer greater than 1, representing the number of neighboring pixels selected. The value of Q can be set according to the actual scene; for example, if the local region is set to a 3*3 pixel area, then Q is 8.

[0037] Then, Q random gray values ​​are randomly selected from Q random pixels in the grayscale shoulder image, and the average of the Q neighboring gray values ​​is calculated to obtain the neighboring gray average. The average of the Q random gray values ​​is then calculated to obtain the random gray average. Then, the deviation ratio between the neighboring gray average and the random gray average is calculated, that is, the ratio of the gray difference between the neighboring gray average and the random gray average to the random gray average is calculated to obtain the first distinguishability. For example, assuming the neighboring gray average is 60 and the random gray average is 50, the first distinguishability is (60-50) / 50 = 20%. Here, the distinguishability reflects the degree of fluctuation of the gray values ​​of the first shoulder pixels. A higher first distinguishability means that the gray value difference is large, which may be a damaged or abnormal area of ​​the shoulder.

[0038] Using the same method as calculating the first discriminant value, the discriminant values ​​of other shoulder pixels within the grayscale shoulder image are calculated to obtain the total discriminant value of all shoulder pixels. The discriminant values ​​of all shoulder pixels are then aggregated into a single image to construct a shoulder discriminant map. A preset discriminant threshold is configured, which can be set according to task requirements. This threshold is used to filter out shoulder pixels with high discriminant values. This threshold can be obtained through experiments or based on historical data analysis, typically by selecting areas with significant texture changes or damage. Next, the multiple shoulder inspection areas are filtered according to the preset discriminant threshold. Shoulder pixels with a discriminant value greater than the preset discriminant threshold are marked as inspection pixels, resulting in multiple inspection pixels. Then, connectivity analysis is performed on the selected pixels that meet the criteria, grouping adjacent pixels into the same inspection area, resulting in multiple shoulder inspection areas.

[0039] By selecting road shoulder pixels with a difference greater than a preset threshold and aggregating these pixels into multiple road shoulder inspection areas, resources can be effectively concentrated on areas with more severe road shoulder damage or abnormalities. This not only improves the efficiency of inspections but also ensures the accuracy and comprehensiveness of the inspection results.

[0040] S250: Based on the distinguishability of the multiple shoulder inspection areas, determine the multiple shoulder recognition accuracy.

[0041] Furthermore, step S250 of the present invention further includes:

[0042] S251: Obtain the distinguishability of all shoulder pixels within the multiple shoulder inspection areas, calculate the average value of each, and obtain multiple area distinguishability; S252: Define the shoulder recognition accuracy corresponding to the largest area distinguishability as the maximum shoulder recognition accuracy, wherein the maximum shoulder recognition accuracy includes the maximum image extraction ratio; S253: Calculate the ratio of the other multiple area distinguishability to the largest area distinguishability, multiply it by the maximum shoulder recognition accuracy, and obtain multiple shoulder recognition accuracies.

[0043] Specifically, firstly, the distinguishability of all shoulder pixels within the multiple shoulder inspection areas is obtained. Then, for each shoulder inspection area, the average distinguishability of all shoulder pixels within the area is calculated, and the average result is used as the region distinguishability, resulting in multiple region distinguishability values. Next, the shoulder recognition accuracy corresponding to the region with the highest distinguishability value among the multiple region distinguishability values ​​is calibrated as the maximum shoulder recognition accuracy. The maximum shoulder recognition accuracy includes the maximum image extraction ratio, which can be set according to the maximum distinguishability value (e.g., 50% or 60%).

[0044] Further, the ratios of the discrimination scores of multiple other regions to the highest discrimination score of the region are calculated to obtain multiple discrimination ratios. These ratios are then multiplied by the highest shoulder recognition accuracy to obtain multiple shoulder recognition accuracies. Each shoulder recognition accuracy corresponds one-to-one with a shoulder inspection area; regions with higher discrimination scores have higher recognition accuracy and a larger proportion of extracted image areas, while regions with lower discrimination scores have relatively lower recognition accuracy and a smaller proportion of extracted image areas. By dynamically adjusting the recognition accuracy based on the discrimination score of each shoulder inspection area, more resources (i.e., a higher image extraction ratio) can be allocated to different areas (such as severely damaged or textured areas), effectively improving the accuracy and efficiency of inspection tasks, focusing on high-risk areas, and reducing resource waste in low-risk areas.

[0045] S300: Control the drone to collect video of the multiple shoulder inspection areas, obtain multiple shoulder detection videos, extract shoulder detection images from the multiple shoulder detection videos according to the multiple shoulder recognition accuracy, obtain multiple first shoulder detection image sets, perform shoulder damage recognition, and obtain multiple shoulder damage degrees.

[0046] Furthermore, step S300 of the present invention also includes:

[0047] S310: Extract shoulder detection images from the multiple shoulder detection videos according to the image extraction ratio within the multiple shoulder recognition accuracy range to obtain multiple first shoulder detection image sets; S320: Pre-train a shoulder damage recognizer, wherein sample shoulder detection image sets and sample shoulder damage degrees are collected, and the shoulder damage recognizer is trained based on a convolutional neural network, wherein the sample shoulder damage degree includes the marked damage percentage; S330: Input the multiple first shoulder detection image sets into the shoulder damage recognizer respectively, and recognize and output multiple shoulder damage degrees.

[0048] Specifically, firstly, the coordinates of multiple shoulder inspection areas are acquired, and the drone's flight path and data collection route are planned based on these pre-set coordinates to ensure coverage of all critical shoulder areas. Next, the drone is controlled to collect video data from these multiple shoulder inspection areas. Rainy weather often exposes water accumulation caused by poor drainage or damage, as rainwater may accumulate in cracks and subsidence areas, making drainage anomalies more easily apparent. Therefore, image acquisition under rainy conditions is prioritized. Rules can be set to automatically trigger drone flight and begin image acquisition when rainfall exceeds a certain threshold. Then, shoulder detection images are extracted from multiple shoulder inspection areas within the multiple shoulder detection videos according to the image extraction ratio within the specified shoulder recognition accuracy. Specifically, images are extracted from the shoulder detection video corresponding to each shoulder inspection area according to the set image extraction ratio, resulting in multiple first shoulder detection image sets, each corresponding to one shoulder inspection area.

[0049] Next, a road shoulder damage identifier is constructed based on a convolutional neural network. This identifier is used to identify the degree of damage based on road shoulder detection images. It includes an input layer, a convolutional layer, a pooling layer, a fully connected layer, and an output layer. The input data of the input layer is the road shoulder detection image, and the output data of the output layer is the road shoulder damage degree. Then, sample road shoulder detection image sets are collected, and the damage degree of different sample road shoulder detection image sets is labeled to obtain sample road shoulder damage degrees. The sample road shoulder damage degree includes the labeled percentage of damage (e.g., 20%, which can be labeled according to the degree of damage; the greater the degree of damage, the larger the percentage of damage). Multiple sample road shoulder detection image sets and multiple sample road shoulder damage degrees are obtained.

[0050] Further, using sample roadside detection image sets as input and sample roadside damage levels as output, the roadside damage recognizer is trained under supervised supervision using the multiple sample roadside detection image sets and multiple sample roadside damage levels as training data. First, each input image is forward-propagated through the network to generate a predicted damage level. Then, the mean squared error (MSE) is used as the loss function to calculate the difference between the predicted damage level and the actual damage level. Next, the gradient of the loss function with respect to the network parameters (weights and biases) is calculated using the backpropagation algorithm, and these parameters are adjusted using the gradient descent method. Iterative training is performed until the loss function converges, resulting in a trained roadside damage recognizer.

[0051] Finally, the multiple first shoulder detection image sets are input into the shoulder damage identifier for analysis, and multiple shoulder damage degrees are output. By constructing a shoulder damage identifier for damage degree identification, the deep learning capabilities of convolutional neural networks can be utilized to quickly identify subtle damage features in the images, thereby improving the accuracy and efficiency of shoulder damage degree analysis, and significantly enhancing the efficiency, accuracy, and intelligence level of road infrastructure inspection.

[0052] S400: Based on multiple shoulder damage degrees and multiple shoulder recognition accuracies, the multiple shoulder detection videos are processed to extract shoulder detection images to obtain multiple second shoulder detection image sets. Shoulder drainage anomaly recognition is performed to obtain multiple drainage anomaly degrees. Combined with the multiple shoulder damage degrees, multiple shoulder anomaly degrees are calculated as the shoulder inspection results.

[0053] Furthermore, step S400 of the present invention further includes:

[0054] S410: Extract the minimum shoulder damage degree among the multiple shoulder damage degrees; S420: Calculate the ratio of the multiple shoulder damage degrees to the minimum shoulder damage degree to obtain multiple drainage identification accuracy correction coefficients; S430: Multiply the multiple drainage identification accuracy correction coefficients by the corresponding multiple shoulder identification accuracies to obtain multiple corrected shoulder identification accuracies; S440: Extract shoulder detection images from the multiple shoulder detection videos according to the image extraction ratio within the multiple corrected shoulder identification accuracies to obtain multiple second shoulder detection image sets.

[0055] Specifically, firstly, the minimum shoulder damage degree among the multiple shoulder damage degrees is extracted; then, the ratio of each of the multiple shoulder damage degrees to the minimum shoulder damage degree is calculated, and this ratio is set as a drainage identification accuracy correction coefficient, where the larger the damage degree, the larger the correction coefficient; then, each of the multiple drainage identification accuracy correction coefficients is multiplied by the corresponding multiple shoulder identification accuracies, and the product of the two is used as the corrected shoulder identification accuracy to obtain multiple corrected shoulder identification accuracies. Finally, multiple image extraction ratios corresponding to the multiple corrected shoulder identification accuracies are obtained, and shoulder detection images are extracted from the multiple shoulder detection videos according to the multiple image extraction ratios, that is, shoulder detection images corresponding to the shoulder detection areas are extracted according to the corrected image extraction ratios to obtain multiple second shoulder detection image sets.

[0056] By correcting the shoulder identification accuracy and obtaining the corrected image extraction ratio, image extraction can be performed to ensure that the identification accuracy matches the actual damage situation, quickly focusing on high-risk areas of shoulder damage, thereby improving the efficiency and accuracy of drainage anomaly analysis.

[0057] Furthermore, step S400 of the present invention further includes:

[0058] S450: Pre-trained drainage anomaly detector, wherein sample second shoulder detection image sets and sample drainage anomaly degrees are collected, and the drainage anomaly detector is trained based on a convolutional neural network, wherein the sample drainage anomaly degree includes the marked drainage anomaly percentage; S460: The multiple second shoulder detection image sets are respectively input into the drainage anomaly detector, and multiple drainage anomaly degrees are obtained by recognition output; S470: Multiple shoulder anomaly degrees are calculated based on the multiple shoulder damage degrees and multiple drainage anomaly degrees, and are used as the shoulder inspection results.

[0059] Specifically, firstly, a drainage anomaly detector is constructed based on a convolutional neural network. This detector identifies the degree of drainage anomaly based on road shoulder detection images. It includes an input layer, a convolutional layer, a pooling layer, a fully connected layer, and an output layer. The input data for the input layer is the second road shoulder detection image, and the output data for the output layer is the drainage anomaly degree, which includes an identified percentage of drainage anomaly (e.g., 30%, used to indicate the degree of drainage anomaly; the higher the degree of anomaly, the higher the percentage). Next, multiple sets of sample second road shoulder detection images are collected, and the percentage of drainage anomaly is assigned to different sets of sample second road shoulder detection images to obtain multiple sample drainage anomaly degrees.

[0060] Then, using the sample second shoulder detection image set as input and the sample drainage anomaly degree as supervision, the drainage anomaly recognizer is trained under supervision using multiple sample second shoulder detection image sets and multiple sample drainage anomaly degrees. First, each sample (second shoulder detection image) is input into the model, and the predicted value (i.e., drainage anomaly degree) is calculated. Forward propagation extracts image features and makes predictions through convolution, pooling, and fully connected operations in each layer. Next, the loss between the predicted value and the true label is calculated, and then the gradient of the loss with respect to each network parameter is calculated through the backpropagation algorithm. The model parameters are updated using the gradient to minimize the loss. Iterative training continues until the loss function converges, resulting in the trained drainage anomaly recognizer.

[0061] The multiple second shoulder detection image sets are then input into the drainage anomaly identifier for identification, outputting multiple drainage anomaly scores. Weights are then configured for shoulder damage scores and drainage anomaly scores, which can be set according to the influence of the indicators on the inspection results. For example, if the inspection results indicate a stronger tendency towards shoulder damage, the weight corresponding to the shoulder damage score is larger; conversely, if the inspection results indicate a stronger tendency towards drainage anomalies, the weight corresponding to the drainage anomaly score is larger, and the sum of the shoulder damage weight and the drainage anomaly weight is 1. Furthermore, based on the shoulder damage weight and the drainage anomaly weight, the multiple shoulder damage scores and the multiple drainage anomaly scores are weighted and calculated, and the weighted calculation result is used as the shoulder anomaly score to obtain multiple shoulder anomaly scores. Finally, the multiple shoulder inspection areas and the multiple shoulder anomaly scores are integrated as the shoulder inspection result.

[0062] The road infrastructure inspection method combining UAVs and high-resolution remote sensing provided in this invention has at least the following technical effects:

[0063] High-resolution remote sensing images of the target road are acquired, and shoulder images are identified and segmented to obtain shoulder images. Next, discriminability analysis is performed on the shoulder images to obtain a shoulder discriminability map. Multiple shoulder inspection areas are selected, and based on the discriminability of these areas, a decision is made regarding the identification accuracy of multiple shoulders. Then, a drone is controlled to capture video of these shoulder inspection areas, obtaining multiple shoulder detection videos. Shoulder detection images are extracted from these videos according to the identified accuracy, resulting in multiple first shoulder detection image sets. Shoulder damage is then identified, and multiple shoulder damage images are obtained. The method involves several steps: First, based on multiple shoulder damage levels and multiple shoulder recognition accuracies, shoulder detection images are extracted from the multiple shoulder detection videos to obtain multiple second shoulder detection image sets. Shoulder drainage anomaly identification is then performed to obtain multiple drainage anomaly levels. These are combined with the multiple shoulder damage levels to calculate multiple shoulder anomaly scores, which serve as the shoulder inspection results. This method allows for dynamic adjustment of inspection accuracy based on shoulder conditions, quickly focusing on high-risk areas of shoulder damage. This optimizes resource utilization, promptly identifies potential road damage and drainage problems, and achieves efficient and accurate shoulder inspection, significantly improving inspection efficiency, accuracy, and quality.

[0064] Example 2, please refer to Figure 2 , Figure 2 This is a schematic diagram illustrating an embodiment of the electronic device provided in this invention. For example... Figure 2As shown, this embodiment of the invention provides an electronic device 500, including a memory 510, a processor 520, and a first computer program 511 stored in the memory 510 and executable on the processor 520. When the processor 520 executes the first computer program 511, it performs the following steps: acquiring high-resolution remote sensing images of a target road using high-resolution remote sensing; performing shoulder image recognition and segmentation to obtain shoulder images; performing discrimination analysis on the shoulder images to obtain a shoulder discrimination map; filtering to obtain multiple shoulder inspection areas; and determining the recognition accuracy of multiple shoulders based on the discrimination of the multiple shoulder inspection areas; controlling a drone to... Video is captured for the multiple shoulder inspection areas to obtain multiple shoulder detection videos. Shoulder detection images are extracted from these videos according to the specified shoulder recognition accuracy to obtain multiple first shoulder detection image sets. Shoulder damage is then identified to obtain multiple shoulder damage degrees. Based on the multiple shoulder damage degrees and the multiple shoulder recognition accuracy, shoulder detection images are extracted from the multiple shoulder detection videos to obtain multiple second shoulder detection image sets. Shoulder drainage anomalies are then identified to obtain multiple drainage anomaly degrees. Combining the multiple shoulder damage degrees, multiple shoulder anomaly degrees are calculated as the shoulder inspection results.

[0065] Example 3, please refer to Figure 3 , Figure 3 This is a schematic diagram illustrating an embodiment of a computer-readable storage medium provided by an embodiment of the present invention. For example... Figure 3 As shown, this embodiment provides a computer-readable storage medium 600, on which a second computer program 611 is stored. When the second computer program 611 is executed by a processor, it performs the following steps: acquiring high-resolution remote sensing images of the target road through high-resolution remote sensing, performing shoulder image recognition and segmentation to obtain shoulder images; performing discrimination analysis on the shoulder images to obtain a shoulder discrimination map, filtering to obtain multiple shoulder inspection areas, and determining the recognition accuracy of multiple shoulders based on the discrimination of the multiple shoulder inspection areas; controlling a drone to perform video acquisition on the multiple shoulder inspection areas. Multiple shoulder detection videos are obtained. Shoulder detection images are extracted from these videos according to the specified shoulder recognition accuracy to obtain multiple first shoulder detection image sets. Shoulder damage is then identified to obtain multiple shoulder damage degrees. Based on the multiple shoulder damage degrees and the multiple shoulder recognition accuracy, shoulder detection images are extracted from the multiple shoulder detection videos to obtain multiple second shoulder detection image sets. Shoulder drainage anomaly is then identified to obtain multiple drainage anomaly degrees. Combining these multiple shoulder damage degrees, multiple shoulder anomaly degrees are calculated as the shoulder inspection results.

[0066] It should be noted that the descriptions of each embodiment in the above embodiments have different focuses. For parts that are not described in detail in a certain embodiment, please refer to the relevant descriptions in other embodiments.

[0067] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0068] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded computer, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0069] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0070] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0071] Although preferred embodiments of the invention have been described, those skilled in the art, once they have learned the basic inventive concept, can make other changes and modifications to these embodiments.

[0072] Obviously, those skilled in the art can make various modifications and variations to this invention without departing from its spirit and scope. Therefore, if these modifications and variations fall within the scope of this invention and its equivalents, this invention also intends to include these modifications and variations.

Claims

1. A method for road infrastructure inspection using a combination of unmanned aerial vehicles (UAVs) and high-resolution remote sensing, characterized in that the method... include: High-resolution remote sensing images of the target road are acquired by high-resolution remote sensing, and shoulder images are identified and divided to obtain shoulder images. The road shoulder images are analyzed for discriminability to obtain a road shoulder discriminability map. Multiple road shoulder inspection areas are selected and determined based on the discriminability of the multiple road shoulder inspection areas. The drone is controlled to collect video of the multiple shoulder inspection areas, and multiple shoulder detection videos are obtained. The shoulder detection images are extracted from the multiple shoulder detection videos according to the multiple shoulder recognition accuracy to obtain multiple first shoulder detection image sets. Shoulder damage is identified to obtain multiple shoulder damage degrees. Based on multiple shoulder damage degrees and multiple shoulder recognition accuracies, the multiple shoulder detection videos are processed to extract shoulder detection images to obtain multiple second shoulder detection image sets. Shoulder drainage anomaly identification is performed to obtain multiple drainage anomaly degrees. Combined with the multiple shoulder damage degrees, multiple shoulder anomaly degrees are calculated as the shoulder inspection results. The road shoulder images are subjected to discrimination analysis to obtain a road shoulder discrimination map. Multiple road shoulder inspection areas are selected, and based on the discrimination of the multiple road shoulder inspection areas, a decision is made on the recognition accuracy of multiple road shoulders, including: The road shoulder image is converted to grayscale. A first road shoulder pixel is selected in the grayscale road shoulder image, and Q neighboring grayscale values ​​of Q neighboring pixels are extracted from the first road shoulder pixel, where Q is an integer greater than 1. Within the grayscale shoulder image, Q random pixels are randomly selected, and Q random grayscale values ​​are calculated. The deviation ratio between the mean of the Q neighboring grayscale values ​​and the mean of the Q random grayscale values ​​is used as the first distinguishability. Continue calculating the total discriminant values ​​of all road shoulder pixels to obtain a road shoulder discriminant map; Filter out road shoulder pixels with a difference greater than a preset difference threshold to obtain multiple road shoulder inspection areas; Based on the distinguishability of the multiple shoulder inspection areas, a decision is made to obtain the multiple shoulder recognition accuracies; Based on the distinguishability of the multiple shoulder inspection areas, a decision is made to obtain the recognition accuracy of multiple shoulders, including: Obtain the distinguishability of all shoulder pixels within the multiple shoulder inspection areas, calculate the mean value for each, and obtain the distinguishability of multiple areas. The road shoulder recognition accuracy corresponding to the highest regional differentiation is calibrated as the maximum road shoulder recognition accuracy, wherein the maximum road shoulder recognition accuracy includes the maximum image extraction ratio; Calculate the ratio of the other multiple region discrimination scores to the largest region discrimination score, multiply it by the largest shoulder recognition accuracy, and obtain multiple shoulder recognition accuracies.

2. The method for road infrastructure inspection using a combination of unmanned aerial vehicles (UAVs) and high-resolution remote sensing as described in claim 1, characterized in that, High-resolution remote sensing images of the target road are acquired through high-resolution remote sensing, and shoulder image identification and segmentation are performed to obtain shoulder images, including: High-resolution remote sensing images of the target road were acquired using high-resolution remote sensing technology. A pre-trained shoulder area recognizer is used to identify shoulder areas in the high-resolution remote sensing image to obtain the shoulder areas; The high-resolution remote sensing image within the shoulder area is cropped to obtain the shoulder image.

3. The method for road infrastructure inspection using a combination of unmanned aerial vehicles (UAVs) and high-resolution remote sensing as described in claim 2, characterized in that, A pre-trained shoulder area recognizer identifies shoulder areas in the high-resolution remote sensing image, including: Based on historical data of road shoulder detection, a set of sample high-resolution remote sensing images is collected, and the shoulder area in each sample high-resolution remote sensing image is marked to obtain a set of sample shoulder areas; Based on a convolutional neural network, the network structure of the shoulder area identifier is constructed. The shoulder area identifier is trained using the sample high-resolution remote sensing image set and the sample shoulder area set until convergence. The high-resolution remote sensing image is input into the shoulder area identifier, and the shoulder area is output as a result.

4. The method for road infrastructure inspection using a combination of unmanned aerial vehicles (UAVs) and high-resolution remote sensing as described in claim 1, characterized in that, According to the multiple shoulder recognition accuracies, shoulder detection images are extracted from the multiple shoulder detection videos to obtain multiple first shoulder detection image sets. Shoulder damage is then identified to obtain multiple shoulder damage degrees, including: According to the image extraction ratio within the multiple shoulder recognition accuracy ranges, shoulder detection images are extracted from the multiple shoulder detection videos to obtain multiple first shoulder detection image sets. A pre-trained road shoulder damage identifier is provided, wherein a set of sample road shoulder detection images and sample road shoulder damage degrees are collected, and the road shoulder damage identifier is trained based on a convolutional neural network, wherein the sample road shoulder damage degree includes the identified percentage of damage; The multiple first shoulder detection image sets are respectively input into the shoulder damage identifier, and the identification output obtains multiple shoulder damage degrees.

5. The method for road infrastructure inspection using a combination of unmanned aerial vehicles (UAVs) and high-resolution remote sensing as described in claim 1, characterized in that, Based on multiple shoulder damage levels and multiple shoulder recognition accuracies, the multiple shoulder detection videos are processed to extract shoulder detection images, resulting in multiple second shoulder detection image sets, including: Extract the minimum shoulder damage degree among the multiple shoulder damage degrees; Calculate the ratio of the damage degree of each of the multiple road shoulders to the minimum road shoulder damage degree to obtain multiple drainage identification accuracy correction coefficients; The multiple drainage identification accuracy correction coefficients are respectively multiplied by the corresponding multiple road shoulder identification accuracies to obtain multiple corrected road shoulder identification accuracies; According to the image extraction ratio within the multiple corrected shoulder recognition accuracy ranges, shoulder detection images are extracted from the multiple shoulder detection videos to obtain multiple second shoulder detection image sets.

6. The method for road infrastructure inspection using a combination of unmanned aerial vehicles (UAVs) and high-resolution remote sensing as described in claim 1, characterized in that, Shoulder drainage anomaly identification is performed to obtain multiple drainage anomaly degrees. These multiple shoulder damage degrees are then combined to calculate the shoulder inspection results, including: A pre-trained drainage anomaly detector is prepared by acquiring a set of sample second shoulder detection images and sample drainage anomaly degrees, and training the drainage anomaly detector based on a convolutional neural network, wherein the sample drainage anomaly degree includes the identified drainage anomaly percentage. The multiple sets of second shoulder detection images are respectively input into the drainage anomaly identifier, and multiple drainage anomaly degrees are obtained by the identifier output. Based on the multiple shoulder damage levels and multiple drainage anomaly levels, multiple shoulder anomaly levels are calculated and used as the shoulder inspection results.

7. An electronic device, characterized in that, include: Memory, used to store computer software programs; A processor is used to read and execute the computer software program, thereby implementing the steps of the road infrastructure inspection method of UAV and high-resolution remote sensing collaboration as described in any one of claims 1 to 6.

8. A non-transitory computer-readable storage medium, characterized in that, The storage medium stores a computer software program, which, when executed by a processor, implements the steps of the road infrastructure inspection method that combines UAVs and high-resolution remote sensing as described in any one of claims 1 to 6.

Images