Road infrastructure inspection method based on cooperation of unmanned aerial vehicle and high-resolution remote sensing

Abstract

The invention relates to a road infrastructure inspection method based on cooperation of an unmanned aerial vehicle and high-resolution remote sensing, and relates to the field of road intelligent inspection, and the method comprises the steps: obtaining the recognition precision of a plurality of road shoulders according to the discrimination decision of a plurality of road shoulder inspection areas; extracting road shoulder detection images according to the road shoulder recognition precision to obtain a plurality of first road shoulder detection image sets, and recognizing to obtain a plurality of road shoulder damage degrees; and performing extraction processing according to the plurality of road shoulder damage degrees and the plurality of road shoulder recognition accuracies to obtain a plurality of second road shoulder detection image sets, performing road shoulder drainage anomaly recognition to obtain a plurality of drainage anomaly degrees, and performing calculation by combining the plurality of road shoulder damage degrees to obtain a plurality of road shoulder anomaly degrees as road shoulder inspection results. The technical problems that a traditional method lacks a dynamic adjustment mechanism, a matched inspection strategy cannot be set according to the road shoulder state, the inspection efficiency is low, and the accuracy is insufficient can be solved; efficient and accurate road shoulder inspection can be achieved, and the inspection efficiency, accuracy and inspection quality are remarkably improved.

Description

Technical Field

[0001] The present invention relates to the field of intelligent road inspection, and in particular to a road infrastructure inspection method coordinated by unmanned aerial vehicles and high-resolution remote sensing. Background Art

[0002] Road infrastructure inspection is a regular inspection and maintenance process of roads and their ancillary facilities (such as shoulders, bridges, drainage systems, lighting facilities, etc.). The purpose of the inspection is to ensure the safety, stability and functionality of road facilities, and to detect potential problems in a timely manner, prevent accidents, reduce maintenance costs, and improve road service quality.

[0003] At present, traditional road shoulder inspection methods lack real-time feedback and dynamic adjustment mechanisms. Once the inspection plan and route are determined, they are often no longer adaptively adjusted according to actual conditions. For example, with seasonal changes, traffic flow changes or weather influences, the damage to the road shoulder may change. However, traditional inspection methods cannot flexibly adjust the inspection focus and accuracy according to these changes, which makes the inspection efficiency low and cannot effectively focus on high-risk areas, failing to detect and deal with damaged areas in a timely manner. Summary of the invention

[0004] The present invention aims to solve the technical problems that traditional shoulder inspection methods lack a dynamic adjustment mechanism, cannot set a matching inspection strategy according to the shoulder status, and have low inspection efficiency and insufficient accuracy. A road infrastructure inspection method that cooperates with drones and high-resolution remote sensing is provided to solve the problem.

[0005] The technical solution of the present invention to solve the above technical problems is as follows:

[0006] In a first aspect, the present invention provides a road infrastructure inspection method in which an unmanned aerial vehicle (UAV) and high-resolution remote sensing are coordinated, including: collecting high-resolution remote sensing images of a target road through high-resolution remote sensing, identifying and dividing shoulder images, and obtaining shoulder images; performing a discrimination analysis on the shoulder images to obtain a shoulder discrimination map, screening and obtaining a plurality of shoulder inspection areas, and deciding to obtain a plurality of shoulder recognition accuracies based on the discriminations of the plurality of shoulder inspection areas; controlling the unmanned aerial vehicle to perform video acquisition on the plurality of shoulder inspection areas to obtain a plurality of shoulder detection videos, extracting and processing shoulder detection images on the plurality of shoulder detection videos according to the plurality of shoulder recognition accuracies, obtaining a plurality of first shoulder detection image sets, performing shoulder damage identification, and obtaining a plurality of shoulder damage degrees; extracting and processing shoulder detection images on the plurality of shoulder detection videos according to the plurality of shoulder damage degrees and the plurality of shoulder recognition accuracies to obtain a plurality of second shoulder detection image sets, performing shoulder drainage anomaly identification, and obtaining a plurality of drainage anomaly degrees, and calculating and obtaining a plurality of shoulder anomaly degrees in combination with the plurality of shoulder damage degrees as shoulder inspection results.

[0007] In a second aspect, the present invention further provides an electronic device, comprising:

[0008] At least one processor; a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor so that the at least one processor can perform the steps of any one of the methods described in the first aspect above.

[0009] In a third aspect, a computer-readable storage medium is provided, wherein a computer program is stored on the computer-readable storage medium, and when the computer program is executed, the steps of the method described in any one of the first aspects are implemented.

[0010] The beneficial effects of the present invention are as follows: high-resolution remote sensing images of the target road are collected by high-resolution remote sensing, shoulder image recognition and division are performed, and shoulder images are obtained; then, the shoulder images are analyzed for discrimination, a shoulder discrimination map is obtained, a plurality of shoulder inspection areas are screened, and a plurality of shoulder recognition accuracies are obtained according to the discrimination of the plurality of shoulder inspection areas; then, a drone is controlled to collect videos of the plurality of shoulder inspection areas to obtain a plurality of shoulder detection videos, shoulder detection images are extracted and processed in the plurality of shoulder detection videos according to the plurality of shoulder recognition accuracies, a plurality of first shoulder detection image sets are obtained, shoulder damage recognition is performed, and a plurality of shoulder detection image sets are obtained. multiple shoulder damage degrees; finally, according to the multiple shoulder damage degrees and the multiple shoulder recognition accuracies, the multiple shoulder detection videos are subjected to shoulder detection image extraction processing to obtain multiple second shoulder detection image sets, shoulder drainage anomaly recognition is performed to obtain multiple drainage anomaly degrees, and multiple shoulder anomaly degrees are calculated and obtained in combination with the multiple shoulder damage degrees as shoulder inspection results; the above method can dynamically adjust the inspection accuracy according to the shoulder status, quickly focus on the high-risk areas of shoulder damage, thereby optimizing resource utilization, timely discovering potential road damage and drainage problems, realizing efficient and accurate shoulder inspections, and significantly improving inspection efficiency, accuracy and quality. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] Figure 1 A schematic diagram of the process flow of the road infrastructure inspection method using a drone and high-resolution remote sensing collaboration provided by the present invention;

[0012] Figure 2 A schematic diagram of the structure of an electronic device provided by the present invention;

[0013] Figure 3 A schematic diagram of the structure of a computer-readable storage medium provided by the present invention.

[0014] In the accompanying drawings, the components represented by the reference numerals are described as follows:

[0015] Electronic device 500 , memory 510 , processor 520 , first computer program 511 , computer-readable storage medium 600 , second computer program 611 . DETAILED DESCRIPTION

[0016] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.

[0017] In the description of the present invention, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Thus, the features defined as "first" and "second" may explicitly or implicitly include one or more of the features. In the description of the present invention, the meaning of "plurality" is two or more, unless otherwise clearly and specifically defined.

[0018] In the description of the present invention, the term "for example" is used to mean "used as an example, illustration or explanation". Any embodiment described as "for example" in the present invention is not necessarily to be construed as being more preferred or advantageous than other embodiments. The following description is given to enable any technician in the field to implement and use the present invention. In the following description, details are listed for the purpose of explanation. It should be understood that a person of ordinary skill in the art can recognize that the present invention can be implemented without using these specific details. In other examples, well-known structures and processes will not be elaborated in detail to avoid obscuring the description of the present invention with unnecessary details. Therefore, the present invention is not intended to be limited to the embodiments shown, but is consistent with the widest scope consistent with the principles and features disclosed in the present invention.

[0019] Embodiment 1, as Figure 1 As shown, the embodiment of the present invention provides a road infrastructure inspection method coordinated by a drone and high-resolution remote sensing, which specifically includes the following steps:

[0020] S100: collecting high-resolution remote sensing images of the target road through high-resolution remote sensing, performing shoulder image recognition and segmentation, and obtaining a shoulder image.

[0021] Furthermore, step S100 of the present invention further includes:

[0022] S110: Collect high-resolution remote sensing images of the target road through high-resolution remote sensing.

[0023] Specifically, first, according to the size, complexity and required image resolution of the target road area, select an appropriate remote sensing platform (such as satellite remote sensing), and select sensors (optical sensors, lidar, etc.) according to the required image resolution and application requirements. Then, the target road is photographed and scanned through a remote sensing platform (such as satellite remote sensing) to obtain a high-resolution remote sensing image of the target road. By using high-resolution remote sensing to collect high-resolution remote sensing images of the target road, large-scale image data of the target road area can be quickly obtained, providing data support for the next step of shoulder area identification.

[0024] S120: Pre-training a road shoulder area identifier to perform road shoulder area identification on the high-resolution remote sensing image to obtain a road shoulder area.

[0025] Furthermore, step S120 of the present invention further includes:

[0026] S121: Based on the historical data of shoulder detection of the road, a set of sample high-resolution remote sensing images is collected, and the shoulder area in each sample high-resolution remote sensing image is marked to obtain a set of sample shoulder areas; S122: Based on a convolutional neural network, a network structure of the shoulder area identifier is constructed, and the sample high-resolution remote sensing image set and the sample shoulder area set are used to train the shoulder area identifier until convergence; S123: The high-resolution remote sensing image is input into the shoulder area identifier, and the shoulder area is obtained as output.

[0027] Specifically, the target shoulder detection historical data is obtained, such as collecting the previous shoulder detection data from historical inspection reports, road maintenance records, sensor data or manual inspection data; then, sample high-resolution remote sensing images under multiple different scenes are collected from the shoulder detection historical data to obtain a sample high-resolution remote sensing image set. Then, the shoulder area in each sample high-resolution remote sensing image is marked (such as manually marking the shoulder area in the image by an expert), and a sample shoulder area set is obtained, wherein the sample high-resolution remote sensing image and the sample shoulder area correspond one to one.

[0028] A network structure of a shoulder area identifier is constructed based on a convolutional neural network. The shoulder area identifier includes an input layer, a convolution layer, a pooling layer, a fully connected layer and an output layer. The input data of the input layer is a high-resolution remote sensing image; the convolution layer is used to extract local features of the image through a filter (convolution kernel); the pooling layer is used to reduce the spatial size of the image and reduce the amount of calculation while maintaining important spatial feature information; the fully connected layer is used to map high-dimensional features to the output layer for classification or regression tasks; and the output layer is used to output the shoulder area.

[0029] Then, the sample high-resolution remote sensing image is used as input, the sample shoulder area is used as output, and the sample high-resolution remote sensing image set and the sample shoulder area set are used as training data to supervise the shoulder area identifier. First, the input sample high-resolution remote sensing image is input into the network, and is processed layer by layer such as convolution layer and pooling layer to finally generate a predicted output (predicted shoulder area); then, the gap between the model output and the true annotation is measured by the loss function, and the loss value is calculated; further, the network weight is updated by calculating the gradient of the loss function relative to each network parameter. The back propagation process uses the chain rule to propagate the error layer by layer to calculate the gradient of each parameter, and uses an optimization algorithm (such as Adam) to adjust the network weight and bias according to the calculated gradient; then, iterative training is performed using the sample training data to gradually optimize the network weights so that the model can better fit the training data. As the training proceeds, the loss function value will gradually decrease, and the network will be gradually optimized until the loss function converges, that is, the loss function no longer decreases significantly, indicating that the model has learned effective features, and a trained shoulder area identifier is obtained.

[0030] Finally, the high-resolution remote sensing image is input into the shoulder area identifier for identification, and the shoulder area is output. By constructing a shoulder area identifier based on a convolutional neural network to identify the shoulder area, the intelligence and automation of shoulder area identification can be improved, while improving the accuracy and efficiency of identification.

[0031] S130: Crop the high-resolution remote sensing image in the road shoulder area to obtain a road shoulder image.

[0032] Specifically, the annotation data of the shoulder area is obtained, which includes the precise location of the shoulder area, that is, which parts of the image belong to the shoulder area; then, according to the coordinates of the annotated shoulder area, the corresponding part is cropped from the high-resolution remote sensing image, that is, the marked shoulder area is extracted from the entire high-resolution remote sensing image to obtain the shoulder image. Obtaining the shoulder image provides support for subsequent inspection and analysis of the shoulder area.

[0033] S200: Performing a discrimination analysis on the shoulder image to obtain a shoulder discrimination map, screening to obtain a plurality of shoulder inspection areas, and determining a plurality of shoulder recognition accuracies based on the discrimination of the plurality of shoulder inspection areas.

[0034] Furthermore, step S200 of the present invention further includes:

[0035] S210: grayscale the shoulder image, select the first shoulder pixel point in the grayscale shoulder image, extract Q neighboring grayscale values ​​of Q neighboring pixel points near the first shoulder pixel point, where Q is an integer greater than 1; S220: randomly extract Q random grayscale values ​​of Q random pixel points in the grayscale shoulder image, calculate the deviation ratio between the mean of the Q neighboring grayscale values ​​and the mean of the Q random grayscale values, as the first discrimination; S230: continue to calculate to obtain all the discriminations of all shoulder pixel points, and obtain a shoulder discrimination map; S240: screen shoulder pixel points whose discriminations are greater than a preset discrimination threshold, and obtain multiple shoulder inspection areas.

[0036] Specifically, first, the shoulder image is grayed to obtain a gray shoulder image, wherein graying refers to the process of converting a color image (such as an RGB image) into a gray image, which is achieved by converting RGB values ​​into brightness values. Each pixel value in the gray image usually ranges from 0 to 255, representing different gray levels from black to white, with brighter areas having higher gray values ​​and darker areas having lower gray values. Then, any shoulder pixel point is randomly selected in the gray shoulder image as the first shoulder pixel point, and the gray values ​​of Q neighboring pixels around the first shoulder pixel point are extracted. The selection of neighboring pixels is generally based on a local area (such as a rectangular or circular neighborhood), and Q is an integer greater than 1, indicating the number of selected neighboring pixels. The value of Q can be set according to the actual scene, such as setting the local area to a 3*3 pixel area, then Q is 8.

[0037] Then, Q random grayscale values ​​of Q random pixel points are randomly selected in the grayscale shoulder image, and the mean of the Q adjacent grayscale values ​​is calculated to obtain the adjacent grayscale mean, and the mean of the Q random grayscale values ​​is calculated to obtain the random grayscale mean; then, the deviation ratio of the adjacent grayscale mean and the random grayscale mean is calculated, that is, the ratio of the grayscale difference between the adjacent grayscale mean and the random grayscale mean to the random grayscale mean is calculated to obtain a first discrimination degree. For example, assuming that the adjacent grayscale mean is 60 and the random grayscale mean is 50, the first discrimination degree is (60-50) / 50=20%; wherein the discrimination degree reflects the degree of fluctuation of the grayscale value of the first shoulder pixel point. A higher first discrimination degree means a larger difference in grayscale values, which may be a damaged shoulder or an abnormal area.

[0038] Using the same method for calculating the first discrimination, continue to calculate the discrimination of other shoulder pixels in the grayscale shoulder image to obtain the total discrimination of all shoulder pixels, and summarize the discrimination values ​​of all shoulder pixels into an image to construct a shoulder discrimination map. Configure a preset discrimination threshold, which can be set according to task requirements and is used to screen out shoulder pixels with greater discrimination. The threshold can be obtained through experiments or based on historical data analysis, and is usually set by selecting more obvious texture changes or damaged areas. Then, the multiple shoulder inspection areas are screened according to the preset discrimination threshold, and shoulder pixels with discrimination greater than the preset discrimination threshold are selected and marked as inspection pixels to obtain multiple inspection pixels; then, connectivity analysis is performed on the screened pixels that meet the conditions, and adjacent pixels are classified as the same inspection area to obtain multiple shoulder inspection areas.

[0039] By screening out shoulder pixels with a distinction greater than a preset threshold and aggregating these pixels into multiple shoulder inspection areas, resources can be effectively concentrated on areas where shoulder damage or abnormalities are more serious. This not only improves the efficiency of inspections, but also ensures the accuracy and comprehensiveness of detection results.

[0040] S250: Determine and obtain multiple road shoulder recognition accuracies based on the distinctions of the multiple road shoulder inspection areas.

[0041] Furthermore, step S250 of the present invention further includes:

[0042] S251: Obtain the distinction of all shoulder pixel points in the multiple shoulder inspection areas, calculate the average respectively, and obtain multiple regional distinctions; S252: Calibrate the shoulder recognition accuracy corresponding to the maximum regional distinction as the maximum shoulder recognition accuracy, wherein the maximum shoulder recognition accuracy includes the maximum image extraction ratio; S253: Calculate the ratio of other multiple regional distinctions to the maximum regional distinction, multiply by the maximum shoulder recognition accuracy, and obtain multiple shoulder recognition accuracies.

[0043] Specifically, first, the discrimination of all shoulder pixels in the plurality of shoulder inspection areas is obtained, and then for each shoulder inspection area, the discrimination of all shoulder pixels in the area is averaged, and the averaged result is used as the regional discrimination to obtain a plurality of regional discriminations. Then, the shoulder recognition accuracy corresponding to the largest regional discrimination among the plurality of regional discriminations is calibrated as the maximum shoulder recognition accuracy, wherein the maximum shoulder recognition accuracy includes the maximum image extraction ratio, and the corresponding maximum image extraction ratio (such as 50% or 60%) can be set according to the maximum value of the discrimination.

[0044] Further, the ratios of the discriminations of the other multiple regions to the maximum regional discrimination are calculated to obtain multiple discrimination ratios; and the multiple discrimination ratios are multiplied by the maximum shoulder recognition accuracy to obtain multiple shoulder recognition accuracy, wherein the shoulder recognition accuracy corresponds to the shoulder inspection area one by one, and the recognition accuracy corresponding to the area with a larger discrimination will be higher, and the ratio of the extracted image area will also be larger; while the recognition accuracy of the area with a smaller discrimination is relatively low, and the ratio of the extracted image area is smaller. By dynamically adjusting the recognition accuracy according to the discrimination of each shoulder inspection area, more resources (i.e., a higher image extraction ratio) can be allocated to different areas (such as areas with severe damage or obvious textures), which can effectively improve the accuracy and efficiency of the inspection task, focus on high-risk areas, and reduce resource waste in low-risk areas.

[0045] S300: Control the UAV to collect videos of the multiple shoulder inspection areas to obtain multiple shoulder detection videos, extract shoulder detection images from the multiple shoulder detection videos according to the multiple shoulder recognition accuracies, obtain multiple first shoulder detection image sets, perform shoulder damage recognition, and obtain multiple shoulder damage degrees.

[0046] Furthermore, step S300 of the present invention further includes:

[0047] S310: Extract shoulder detection images from the multiple shoulder detection videos according to the image extraction ratio within the multiple shoulder recognition accuracy, and obtain multiple first shoulder detection image sets; S320: Pre-train a shoulder damage identifier, wherein sample shoulder detection image sets and sample shoulder damage degrees are collected, and the shoulder damage identifier is trained based on a convolutional neural network, and the sample shoulder damage degrees include the marked damage percentage; S330: Input the multiple first shoulder detection image sets into the shoulder damage identifier respectively, and obtain multiple shoulder damage degrees through recognition output.

[0048] Specifically, first, the regional coordinates of multiple shoulder inspection areas are obtained, and the flight route and collection path of the drone are planned according to the coordinates of the pre-set shoulder inspection areas to ensure that the drone can cover all key shoulder areas. Then, the drone is controlled to collect videos of the multiple shoulder inspection areas. In rainy days, water accumulation caused by poor drainage or damage can often be exposed. Rainwater may accumulate in cracks and settlement areas, making it easier to expose drainage anomalies. Therefore, it is preferred to collect images under rainy conditions. Rules can be set. When it is detected that the precipitation exceeds a certain threshold, the drone flight is automatically triggered and image collection begins. Then, according to the image extraction ratio within the multiple shoulder recognition accuracy, the shoulder detection images are extracted in the multiple shoulder inspection areas in the multiple shoulder detection videos, that is, in the shoulder detection video corresponding to each shoulder inspection area, the image is extracted according to the set image extraction ratio to obtain multiple first shoulder detection image sets, wherein each first shoulder detection image set corresponds to a shoulder inspection area.

[0049] Next, a shoulder damage identifier is constructed based on a convolutional neural network. The shoulder damage identifier is used to identify the degree of damage based on the shoulder detection image, and includes an input layer, a convolution layer, a pooling layer, a fully connected layer, and an output layer, wherein the input data of the input layer is the shoulder detection image, and the output data of the output layer is the shoulder damage degree. Then, a sample shoulder detection image set is collected, and the damage degree of different sample shoulder detection image sets is marked to obtain a sample shoulder damage degree, wherein the sample shoulder damage degree includes a marked damage percentage (such as 20%, which can be marked according to the degree of damage, the greater the degree of damage, the greater the damage percentage), and multiple sample shoulder detection image sets and multiple sample shoulder damage degrees are obtained.

[0050] Further, taking a sample shoulder detection image set as input and a sample shoulder damage degree as output, the multiple sample shoulder detection image sets and multiple sample shoulder damage degrees are used as training data to perform supervised training on the shoulder damage identifier. First, each input image is forward propagated through the network to generate a predicted value of the damage degree; then, the mean square error (MSE) is used as the loss function to calculate the difference between the predicted damage degree and the actual damage degree; then, the gradient of the loss function with respect to the network parameters (weights and biases) is calculated through the back-propagation algorithm, and these parameters are adjusted through the gradient descent method; iterative training is performed until the loss function converges to obtain a trained shoulder damage identifier.

[0051] Finally, the plurality of first shoulder detection image sets are respectively input into the shoulder damage identifier for analysis, and a plurality of shoulder damage degrees are output. By constructing a shoulder damage identifier for damage degree identification, the deep learning capability of the convolutional neural network can be used to quickly identify subtle damage features in the image, thereby improving the accuracy and efficiency of shoulder damage degree analysis, thereby significantly improving the efficiency, accuracy and intelligence level of road infrastructure inspection.

[0052] S400: Based on multiple shoulder damage degrees and multiple shoulder recognition accuracies, the multiple shoulder detection videos are subjected to shoulder detection image extraction processing to obtain multiple second shoulder detection image sets, shoulder drainage anomaly recognition is performed to obtain multiple drainage anomaly degrees, and multiple shoulder anomaly degrees are calculated and obtained in combination with the multiple shoulder damage degrees as shoulder inspection results.

[0053] Furthermore, step S400 of the present invention further includes:

[0054] S410: Extract the minimum shoulder damage degree among the multiple shoulder damage degrees; S420: Calculate the ratios of the multiple shoulder damage degrees to the minimum shoulder damage degree respectively to obtain multiple drainage recognition accuracy correction coefficients; S430: Use the multiple drainage recognition accuracy correction coefficients respectively, multiply by the corresponding multiple shoulder recognition accuracies, to obtain multiple corrected shoulder recognition accuracies; S440: Extract shoulder detection images from the multiple shoulder detection videos according to the image extraction ratios within the multiple corrected shoulder recognition accuracies, to obtain multiple second shoulder detection image sets.

[0055] Specifically, first, extract the minimum shoulder damage degree among the multiple shoulder damage degrees; then calculate the ratio of the multiple shoulder damage degrees to the minimum shoulder damage degree respectively, and set the ratio as the drainage recognition accuracy correction coefficient, wherein the greater the damage degree, the greater the correction coefficient; then respectively use the multiple drainage recognition accuracy correction coefficients, multiply the corresponding multiple shoulder recognition accuracies, and use the product of the two as the corrected shoulder recognition accuracy to obtain multiple corrected shoulder recognition accuracies. Finally, obtain multiple image extraction ratios corresponding to the multiple corrected shoulder recognition accuracies, and extract shoulder detection images from the multiple shoulder detection videos according to the multiple image extraction ratios, that is, extract the shoulder detection images corresponding to the shoulder detection area according to the corrected image extraction ratios to obtain multiple second shoulder detection image sets.

[0056] By correcting the shoulder recognition accuracy and obtaining the corrected image extraction ratio for image extraction, it can be ensured that the recognition accuracy matches the actual damage situation, and the focus can be quickly placed on high-risk areas of shoulder damage, thereby improving the efficiency and accuracy of drainage anomaly analysis.

[0057] Furthermore, step S400 of the present invention further includes:

[0058] S450: Pre-training a drainage anomaly identifier, wherein a sample second shoulder detection image set and a sample drainage anomaly degree are collected, and the drainage anomaly identifier is trained based on a convolutional neural network, wherein the sample drainage anomaly degree includes an identified drainage anomaly percentage; S460: respectively inputting the multiple second shoulder detection image sets into the drainage anomaly identifier, and identifying and outputting multiple drainage anomaly degrees; S470: calculating and obtaining multiple shoulder anomaly degrees based on the multiple shoulder damage degrees and the multiple drainage anomaly degrees as shoulder inspection results.

[0059] Specifically, first, a drainage anomaly identifier is constructed based on a convolutional neural network. The drainage anomaly identifier is used to identify the degree of drainage anomaly based on the shoulder detection image, including an input layer, a convolution layer, a pooling layer, a fully connected layer, and an output layer. The input data of the input layer is the second shoulder detection image, and the output data of the output layer is the drainage anomaly degree. The drainage anomaly degree includes a marked drainage anomaly percentage (such as 30%, which is used to indicate the degree of drainage anomaly. The greater the degree of anomaly, the greater the anomaly percentage). Next, multiple sample second shoulder detection image sets are collected, and drainage anomaly percentages of different sample second shoulder detection image sets are marked to obtain multiple sample drainage anomaly degrees.

[0060] Then, the sample second shoulder detection image set is taken as input and the sample drainage anomaly degree is taken as supervision. Multiple sample second shoulder detection image sets and multiple sample drainage anomaly degrees are used to carry out supervised training on the drainage anomaly identifier. First, each sample (second shoulder detection image) is input into the model, and the predicted value (i.e., drainage anomaly degree) is calculated. The forward propagation extracts image features and makes predictions through convolution, pooling, full connection and other operations of each layer. Then, the loss between the predicted value and the true label is calculated, and then the gradient of the loss to each network parameter is calculated through the back-propagation algorithm. The model parameters are updated using the gradient to minimize the loss. The iterative training is continued until the loss function converges to obtain the trained drainage anomaly identifier.

[0061] Further, the plurality of second shoulder detection image sets are respectively input into the drainage anomaly identifier for identification, and a plurality of drainage anomaly degrees are output. Then, weights are configured for the shoulder damage degree and the drainage anomaly degree, respectively, wherein the weights can be set according to the degree of influence of the indicator on the inspection result. For example, if the inspection result is more inclined to shoulder damage, the weight corresponding to the shoulder damage degree is greater; on the contrary, if the inspection result is more inclined to drainage anomaly, the weight corresponding to the drainage anomaly degree is greater, and the sum of the shoulder damage weight and the drainage anomaly weight is 1. Further, according to the shoulder damage weight and the drainage anomaly weight, the plurality of shoulder damage degrees and the plurality of drainage anomaly degrees are weightedly calculated, and the weighted calculation results are used as the shoulder anomaly degree to obtain a plurality of shoulder anomaly degrees. Finally, the plurality of shoulder inspection areas and the plurality of shoulder anomaly degrees are data integrated as the shoulder inspection result.

[0062] The road infrastructure inspection method using drones and high-resolution remote sensing in collaboration provided by the embodiment of the present invention has at least the following technical effects:

[0063] The high-resolution remote sensing image of the target road is collected by high-resolution remote sensing, and shoulder image recognition and division are performed to obtain a shoulder image; then, the shoulder image is analyzed for discrimination to obtain a shoulder discrimination map, and a plurality of shoulder inspection areas are screened and obtained, and a plurality of shoulder recognition accuracies are obtained according to the discrimination of the plurality of shoulder inspection areas; then, the drone is controlled to collect videos of the plurality of shoulder inspection areas to obtain a plurality of shoulder detection videos, and shoulder detection images are extracted and processed in the plurality of shoulder detection videos according to the plurality of shoulder recognition accuracies to obtain a plurality of first shoulder detection image sets, and shoulder damage recognition is performed to obtain a plurality of shoulder damage detection images. damage degree; finally, according to the multiple shoulder damage degrees and the multiple shoulder recognition accuracies, the multiple shoulder detection videos are subjected to shoulder detection image extraction processing to obtain multiple second shoulder detection image sets, shoulder drainage anomaly recognition is performed to obtain multiple drainage anomaly degrees, and multiple shoulder anomaly degrees are calculated and obtained in combination with the multiple shoulder damage degrees as shoulder inspection results; the above method can dynamically adjust the inspection accuracy according to the shoulder status, quickly focus on the high-risk areas of shoulder damage, thereby optimizing resource utilization, timely discovering potential road damage and drainage problems, realizing efficient and accurate shoulder inspection, and significantly improving inspection efficiency, accuracy and inspection quality.

[0064] For example 2, please refer to Figure 2 , Figure 2 Schematic diagram of an electronic device provided by an embodiment of the present invention. Figure 2As shown, an embodiment of the present invention provides an electronic device 500, including a memory 510, a processor 520, and a first computer program 511 stored in the memory 510 and executable on the processor 520. When the processor 520 executes the first computer program 511, the following steps are implemented: high-resolution remote sensing images of the target road are collected by high-resolution remote sensing, shoulder image recognition and segmentation are performed, and shoulder images are obtained; a distinction degree analysis is performed on the shoulder image to obtain a shoulder distinction degree map, multiple shoulder inspection areas are obtained by screening, and multiple shoulder recognition accuracies are obtained according to the distinction degrees of the multiple shoulder inspection areas; a drone is controlled to Video capture is performed on the multiple shoulder inspection areas to obtain multiple shoulder detection videos, shoulder detection images are extracted and processed on the multiple shoulder detection videos according to the multiple shoulder recognition accuracies to obtain multiple first shoulder detection image sets, shoulder damage recognition is performed to obtain multiple shoulder damage degrees; shoulder detection images are extracted and processed on the multiple shoulder detection videos according to the multiple shoulder damage degrees and the multiple shoulder recognition accuracies to obtain multiple second shoulder detection image sets, shoulder drainage anomaly recognition is performed to obtain multiple drainage anomaly degrees, and multiple shoulder anomaly degrees are calculated in combination with the multiple shoulder damage degrees as shoulder inspection results.

[0065] For example 3, please refer to Figure 3 , Figure 3 A schematic diagram of an embodiment of a computer-readable storage medium provided in an embodiment of the present invention. Figure 3 As shown, this embodiment provides a computer-readable storage medium 600, on which a second computer program 611 is stored. When the second computer program 611 is executed by a processor, the following steps are implemented: high-resolution remote sensing images of the target road are collected by high-resolution remote sensing, shoulder image recognition and segmentation are performed, and shoulder images are obtained; the shoulder image is analyzed for discrimination, a shoulder discrimination map is obtained, a plurality of shoulder inspection areas are obtained by screening, and a plurality of shoulder recognition accuracies are obtained according to the discrimination of the plurality of shoulder inspection areas; a drone is controlled to collect video of the plurality of shoulder inspection areas , obtain multiple shoulder detection videos, extract and process shoulder detection images from the multiple shoulder detection videos according to the multiple shoulder recognition accuracies, obtain multiple first shoulder detection image sets, perform shoulder damage recognition, and obtain multiple shoulder damage degrees; according to the multiple shoulder damage degrees and the multiple shoulder recognition accuracies, extract and process shoulder detection images from the multiple shoulder detection videos, obtain multiple second shoulder detection image sets, perform shoulder drainage anomaly recognition, obtain multiple drainage anomaly degrees, and combine the multiple shoulder damage degrees to calculate and obtain multiple shoulder anomaly degrees as the shoulder inspection result.

[0066] It should be noted that in the above embodiments, the description of each embodiment has its own emphasis, and for parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0067] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0068] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded computer, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0069] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0070] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0071] Although preferred embodiments of the present invention have been described, additional changes and modifications may occur to these embodiments once those skilled in the art understand the basic inventive concepts.

[0072] Obviously, those skilled in the art can make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention belong to the scope of the present invention and its equivalent technologies, the present invention is also intended to include these changes and variations.

Claims

1. A road infrastructure inspection method using drones and high-resolution remote sensing, characterized in that: include: The high-resolution remote sensing image of the target road is collected through high-resolution remote sensing, and the shoulder image is recognized and divided to obtain the shoulder image; Performing a discrimination analysis on the shoulder image to obtain a shoulder discrimination map, screening to obtain a plurality of shoulder inspection areas, and determining to obtain a plurality of shoulder recognition accuracies based on the discrimination of the plurality of shoulder inspection areas; Controlling the UAV to collect videos of the multiple shoulder inspection areas to obtain multiple shoulder detection videos, extracting shoulder detection images from the multiple shoulder detection videos according to the multiple shoulder recognition accuracies to obtain multiple first shoulder detection image sets, performing shoulder damage recognition, and obtaining multiple shoulder damage degrees; According to multiple shoulder damage degrees and multiple shoulder recognition accuracies, the multiple shoulder detection videos are subjected to shoulder detection image extraction processing to obtain multiple second shoulder detection image sets, shoulder drainage anomaly identification is performed to obtain multiple drainage anomaly degrees, and multiple shoulder anomaly degrees are calculated and obtained in combination with the multiple shoulder damage degrees as shoulder inspection results.

2. The road infrastructure inspection method using drones and high-resolution remote sensing as claimed in claim 1 is characterized in that: The high-resolution remote sensing image of the target road is collected through high-resolution remote sensing, and shoulder image recognition and segmentation are performed to obtain shoulder images, including: Collect high-resolution remote sensing images of the target road through high-resolution remote sensing; Pre-training a shoulder area identifier to perform shoulder area identification on the high-resolution remote sensing image to obtain a shoulder area; The high-resolution remote sensing image in the road shoulder area is cropped to obtain a road shoulder image.

3. The road infrastructure inspection method using drones and high-resolution remote sensing as claimed in claim 2 is characterized in that: Pre-training a road shoulder area identifier to perform road shoulder area identification on the high-resolution remote sensing image, including: According to the historical data of road shoulder detection, a sample high-resolution remote sensing image set is collected, and the shoulder area in each sample high-resolution remote sensing image is marked to obtain a sample shoulder area set; Based on a convolutional neural network, a network structure of the shoulder area identifier is constructed, and the shoulder area identifier is trained using the sample high-resolution remote sensing image set and the sample shoulder area set until convergence; The high-resolution remote sensing image is input into the road shoulder area identifier, and the road shoulder area is obtained as output.

4. The road infrastructure inspection method using drones and high-resolution remote sensing as claimed in claim 1 is characterized in that: Performing a discrimination analysis on the shoulder image to obtain a shoulder discrimination map, screening to obtain a plurality of shoulder inspection areas, and deciding to obtain a plurality of shoulder recognition accuracies based on the discrimination of the plurality of shoulder inspection areas, including: Grayscale processing is performed on the shoulder image, a first shoulder pixel point is selected in the grayscale shoulder image, and Q neighboring grayscale values ​​of Q neighboring pixel points near the first shoulder pixel point are extracted, where Q is an integer greater than 1; Randomly extract Q random grayscale values ​​of Q random pixel points in the grayscale shoulder image, and calculate the deviation ratio between the mean of the Q adjacent grayscale values ​​and the mean of the Q random grayscale values ​​as the first discrimination degree; Continue to calculate and obtain all the distinctiveness of all shoulder pixels to obtain a shoulder distinctiveness map; Screening shoulder pixel points whose distinctiveness is greater than a preset distinctiveness threshold to obtain multiple shoulder inspection areas; According to the distinctions of the plurality of road shoulder inspection areas, a plurality of road shoulder recognition accuracies are determined.

5. The road infrastructure inspection method using drones and high-resolution remote sensing as claimed in claim 4 is characterized in that: According to the distinctions of the plurality of shoulder inspection areas, a plurality of shoulder recognition accuracies are determined, including: Obtain the distinctions of all shoulder pixels in the plurality of shoulder inspection areas, calculate the averages respectively, and obtain the distinctions of a plurality of areas; The road shoulder recognition accuracy corresponding to the maximum regional distinction is calibrated as the maximum road shoulder recognition accuracy, wherein the maximum road shoulder recognition accuracy includes the maximum image extraction ratio; The ratio of the other multiple regional distinctions to the maximum regional distinction is calculated, and the ratio is multiplied by the maximum road shoulder recognition accuracy to obtain multiple road shoulder recognition accuracy.

6. The road infrastructure inspection method using drones and high-resolution remote sensing as claimed in claim 1 is characterized in that: Extracting shoulder detection images from the plurality of shoulder detection videos according to the plurality of shoulder recognition accuracies respectively to obtain a plurality of first shoulder detection image sets, performing shoulder damage recognition to obtain a plurality of shoulder damage degrees, including: Extracting shoulder detection images from the plurality of shoulder detection videos according to image extraction ratios within the plurality of shoulder recognition accuracies, respectively, to obtain a plurality of first shoulder detection image sets; Pre-training a shoulder damage identifier, wherein a sample shoulder detection image set and a sample shoulder damage degree are collected, and the shoulder damage identifier is trained based on a convolutional neural network, wherein the sample shoulder damage degree includes a marked damage percentage; The plurality of first shoulder detection image sets are respectively input into the shoulder damage identifier, and a plurality of shoulder damage degrees are obtained by identification output.

7. The road infrastructure inspection method using drones and high-resolution remote sensing as claimed in claim 1 is characterized in that: According to multiple shoulder damage degrees and multiple shoulder recognition accuracies, the multiple shoulder detection videos are subjected to shoulder detection image extraction processing to obtain multiple second shoulder detection image sets, including: extracting a minimum shoulder damage degree among the plurality of shoulder damage degrees; Calculating the ratios of the plurality of shoulder damage degrees to the minimum shoulder damage degree respectively to obtain a plurality of drainage identification accuracy correction coefficients; The plurality of drainage recognition accuracy correction coefficients are respectively used to multiply the corresponding plurality of road shoulder recognition accuracies to obtain a plurality of corrected road shoulder recognition accuracies; The shoulder detection images are extracted from the plurality of shoulder detection videos according to the image extraction ratios within the plurality of corrected shoulder recognition accuracies, respectively, to obtain a plurality of second shoulder detection image sets.

8. The road infrastructure inspection method using drones and high-resolution remote sensing as claimed in claim 1 is characterized in that: Perform shoulder drainage anomaly identification to obtain multiple drainage anomaly degrees, and calculate shoulder inspection results based on the multiple shoulder damage degrees, including: Pre-training a drainage anomaly identifier, wherein a sample second shoulder detection image set and a sample drainage anomaly degree are collected, and the drainage anomaly identifier is trained based on a convolutional neural network, wherein the sample drainage anomaly degree includes a marked drainage anomaly percentage; Inputting the plurality of second shoulder detection image sets into the drainage anomaly identifier respectively, and obtaining a plurality of drainage anomaly degrees through identification output; According to the plurality of shoulder damage degrees and the plurality of drainage abnormality degrees, a plurality of shoulder abnormality degrees are calculated and obtained as shoulder inspection results.

9. An electronic device, characterized in that: include: Memory for storing computer software programs; A processor is used to read and execute the computer software program, thereby implementing the steps of the road infrastructure inspection method using drones and high-resolution remote sensing as described in any one of claims 1 to 8.

10. A non-transitory computer-readable storage medium, characterized in that: The storage medium stores a computer software program, which, when executed by a processor, implements the steps of the road infrastructure inspection method in which a drone and high-resolution remote sensing are coordinated as described in any one of claims 1 to 8.

Images