Bridge handrail damage detection and repair opportunity determination method based on unmanned aerial vehicle

Abstract

The invention discloses a bridge handrail damage detection and repair opportunity determination method based on an unmanned aerial vehicle, and belongs to the technical field of bridge disease detection. The problems that in the prior art, a traditional bridge disease detection method does not consider bridge handrail damage detection and the handrail damage repair time is difficult to determine are solved. The shortest flight path of the unmanned aerial vehicle is planned according to parameter information and operation information of the unmanned aerial vehicle; acquiring bridge handrail apparent images to generate an image matrix, and performing image quality enhancement and image pixel updating; establishing a basic model, an auxiliary model and a strengthening model, constructing an interlayer loss function, training to obtain an optimal model, and outputting a final railing damage identification result; and establishing a change rule of the number of the damaged areas and the area of the damaged areas along with the operation time of the bridge handrail, calculating to obtain the minimum value of the total repair cost, and determining the optimal repair opportunity of the bridge handrail. The method avoids early maintenance or delayed repair of the bridge, and can be applied to determining the repair time of the bridge.

Description

Technical Field

[0001] The present invention relates to a method for detecting damage of a bridge railing and determining a repairing time, and in particular to a method for detecting damage of a bridge railing and determining a repairing time based on an unmanned aerial vehicle, and belongs to the technical field of bridge disease detection. Background Art

[0002] Bridge railings may become damaged during their service life, which is mainly affected by the following factors: natural factors (such as weathering, corrosion, freeze-thaw cycles and ultraviolet radiation, which gradually erode the railing material and cause railing damage), human factors (including mechanical impact, negligent maintenance and heavy load pressure, which cause railing damage). In addition, errors in design and construction can also lead to early damage to the railings. Railing damage may have a negative impact on the safety and functionality of the bridge. First, damaged railings cannot effectively protect pedestrians and vehicles, increasing the risk of accidents. Second, damaged railings affect the overall appearance of the bridge and may lead to a negative public perception of bridge safety. Finally, damage that is not repaired in time may further deteriorate, increasing maintenance costs and complexity. Therefore, rapid and intelligent identification of bridge railing damage can monitor the condition of the railing in real time, detect and mark damaged areas in a timely manner, and promote rapid and effective repairs, which can not only improve the safety of the bridge, but also help reduce the long-term maintenance costs that may increase due to delayed repairs.

[0003] On the other hand, during the process of drone inspection of damaged railings, there is a lack of research on drone flight paths, which leads to low operational efficiency, increased mission execution time and energy consumption. The use of unoptimized flight paths to perform flight missions cannot maximize the coverage of all inspection areas, thereby missing key damaged areas and affecting the accuracy and completeness of the inspection. Research on determining the shortest flight path for drones can improve the efficiency and economy of inspection operations, ensuring that the inspection task is completed in the shortest time using the least energy. At the same time, the optimized flight path can improve the coverage and quality of data collection, ensure the comprehensiveness and accuracy of disease detection, and improve the maintenance and safety management of bridges.

[0004] At the same time, if the timing of repairing damaged bridge railings is not determined, it will lead to a variety of problems, such as increased safety hazards, higher maintenance costs, reduced traffic efficiency and shortened bridge life. Delayed repairs will not only aggravate the degree of damage, but also lead to higher repair costs and frequent maintenance needs.

[0005] Therefore, a method for detecting bridge railing damage and determining the timing of repair based on drones is needed. Summary of the invention

[0006] A brief overview of the present invention is provided below in order to provide a basic understanding of certain aspects of the present invention. It should be understood that this overview is not an exhaustive overview of the present invention. It is not intended to identify key or important parts of the present invention, nor is it intended to limit the scope of the present invention. Its purpose is merely to present certain concepts in a simplified form as a prelude to a more detailed description discussed later.

[0007] In view of this, in order to solve the problem that the traditional bridge disease detection method in the prior art does not take into account the detection of bridge railing damage and it is difficult to determine the timing of repairing the damaged railing, the present invention provides a method for detecting bridge railing damage and determining the timing of repair based on a drone.

[0008] The technical solution is as follows: A method for detecting bridge railing damage and determining the timing of repair based on a drone includes the following steps: S1. Plan the shortest flight path of the drone based on the drone parameter information and operation information; S2. Collect the apparent image of the bridge railing along the shortest flight path of the UAV to generate an image matrix, and enhance the image quality to obtain a new image matrix; S3. Equally divide the image in the new image matrix and update the image pixels to obtain an image after pixel update; S4. construct a data set based on the new image matrix combined with disease information, input the data set into the established basic model, auxiliary model and enhanced model for training, fuse the loss functions of the three models, and obtain the fused loss function; S5. Establish an inter-layer loss function from the enhanced model to the basic model, combine the fused loss function to obtain the total loss function, train the three models, use the trained basic model as the final model, and output the final railing damage recognition result; S6. Based on the final railing damage identification results, establish the changing pattern of the number and area of ​​damaged areas with the operation time of the bridge railing, calculate the minimum total repair cost, and determine the best time to repair the bridge railing.

[0009] Furthermore, the S1 specifically includes the following steps: S11. Determine the take-off and landing points of the drone and select the observation position of the operator; S12. Determine the flight altitude AVh according to the focal length and sensor size of the camera carried by the drone, obtain the image coverage and resolution, and set the drone flight speed to AVv; S13. Develop the shortest flight path based on the take-off and landing points of the drone, and determine the coordinate information of the railing and the location information of the drone shooting point; In S13, the take-off point coordinates are AVo (AVoX, AVoY, AVoZ). On this basis, all drone shooting points are determined. The number of any drone shooting point is p, and its corresponding drone shooting point coordinates are AVp (AVpX, AVpY, AVpZ). The number set of drone shooting points is expressed as {1, 2, 3, ..., XN}, and the corresponding drone shooting point coordinate set is expressed as {AV1 (AV1X, AV1Y, AV1Z), AV2 (AV2X, AV2Y, AV2Z), ..., AVXN (AVXNX, AVXNY, AVXNZ)}, where XN is the number of drone shooting points. The adjacent distance between any drone shooting point AVp and its adjacent drone shooting point AV (p+1) is calculated. ; Taking the take-off point AVo as the starting point, according to the distance between any drone shooting point and the take-off point , calculate the distance between all drone shooting points and the take-off point, select the three drone shooting points corresponding to the minimum distance, and randomly select them to obtain the first drone shooting point that the drone arrives at after departing from the take-off point; After determining the first shooting point that the drone arrives at, calculate the distance between all shooting points and the first shooting point that is arrived at, screen out the three drone shooting points corresponding to the smallest distance, and randomly select them to obtain the second shooting point that the drone arrives at after departing from the first shooting point it arrives at. Repeat the above steps until all the shooting points that the drone arrives at are obtained. According to the different random selections each time, calculate the total distance TDis1 of the drone's flight under the first flight mode. Repeat the above steps, the number of repetitions is CN, and integrate to obtain the total distance set of the drone's flight under different flight modes {TDis1, TDis2, TDis3, TDis4, …, TDisCN}, arrange them in order from small to large, and select the total distances and corresponding flight routes ranked in the top TN, TN is the preset value, count the intersection information of TN flight routes, and select the flight route with the largest number of intersections to obtain the shortest flight path of the drone.

[0010] Furthermore, the step S2 specifically includes the following steps: S21. Using the high-definition camera on the drone, collect the bridge railing surface image ph along the shortest flight path of the drone, and the corresponding image matrix is ​​PH; S22. Construct a 5th-order denoising kernel function N, normalize all elements in the kernel function, and obtain the normalized denoising kernel function ; S23. According to the normalized denoising kernel function , perform denoising kernel function processing on the bridge railing apparent image ph to obtain a new image matrix P to achieve image quality enhancement; In S22, the fifth-order denoising kernel function N is expressed as: ; ; Where x=1, 2, 3, 4 or 5, x is the row number of the matrix, and y=1, 2, 3, 4 or 5, y is the column number of the matrix. is the standard deviation of all elements in the kernel function, is the base of natural logarithms; Normalized denoising kernel function It is expressed as: ; In S23, the new image matrix P is expressed as: ; Among them, PH is the image matrix corresponding to the apparent image ph of the bridge railing collected along the shortest flight path of the UAV.

[0011] Furthermore, the step S3 specifically includes the following steps: S31. According to the new image matrix P, the pixel grayscale value is divided into 16 grayscale levels, and the image in the new image matrix P is divided into n equal parts to obtain the divided image; S32. Count the pixel frequencies corresponding to each gray level in the equally divided image , calculate the grayscale frequency cumulative value corresponding to each grayscale level in the equally divided image ; S33.According to the gray frequency cumulative value , construct a pixel value update method and calculate the pixel update value corresponding to the gray level , get the image after pixel update; In S31, the pixel gray value is 0-255, and the gray level is , =1, 2, 3, …, 16, is the grayscale number, and the image after equal division is Pnum, num=1, 2, ..., n; In S32, each gray level in the equally divided image Pnum The corresponding grayscale frequency cumulative value It is expressed as: ; in, is less than or equal to The gray level number, Indicates the gray level number is Gray level; In S33, the pixel update value corresponding to the gray level It is expressed as: ; in, is the minimum non-zero value among all pixel frequency accumulation values, is the total number of pixels corresponding to the equally divided image Pnum, is the maximum value of the gray level.

[0012] Furthermore, the step S4 specifically includes the following steps: S41. Based on the new image matrix, select all images containing damaged railings, cracks, weathering, and peeling coatings as input images; S42. Through manual annotation, the damaged area of ​​the railing and other three diseases are selected in the input image, and a data set is constructed. The data set is divided into a training set and a test set according to a ratio of 8:2; S43. According to the data set, a basic model, an auxiliary model and an enhanced model are constructed, and the loss functions of the three models are calculated and integrated to obtain a fused loss function; In S42, the damaged area of ​​the railing is marked as Damage, and the three types of damage, namely cracks, weathering and coating peeling, are marked as other1, other2 and other3 respectively; In S43, a network structure of a basic model, i.e., a deep learning model A, is constructed, wherein the deep learning model A consists of 1 input layer, 5 convolutional layers, 5 activation layers, 4 pooling layers, 1 global average pooling layer and 1 output layer; Input the data set into the deep learning model A, calculate the weights and biases of the neural network units of the deep learning model A through the error back propagation algorithm, complete the training of the deep learning model A, and obtain the trained deep learning model A and its loss function ; The output value of the output unit node corresponding to the deep learning model A is , respectively, are the output values ​​of the first output unit node , the output value of the second output unit node , the output value of the third output unit node and the output value of the fourth output unit node , according to the exponential function First, the output values ​​of the four output unit nodes corresponding to the deep learning model A are initially converted into the probability values ​​of the first preliminary output value and the first preliminary output value. , and then further optimize and transform it to obtain the first optimized output value and the enhanced probability value of the first optimized output value ; The enhanced probability value using the first optimized output value And the data set, combined with the error chain derivative rule, continue to train the deep learning model A. After the training is completed, the output value of the first disease is obtained The actual disease annotations corresponding to the nodes in the output layer of the deep learning model A , the output value of the first disease The output values ​​of the first railing damage are , the output value of the first crack , the first weathering output value and first coating peeling output value ; Construct the first auxiliary model, namely deep learning model B, which consists of 1 input layer, 10 convolutional layers, 10 activation layers, 5 pooling layers, 1 global average pooling layer and 1 output layer; Input the data set into the deep learning model B, calculate the weights and biases of the neural network units of the deep learning model B through the error back propagation algorithm, complete the training of the deep learning model B, and obtain the trained deep learning model B and its loss function ; The output value of the output unit node corresponding to the deep learning model B is , respectively, are the output values ​​of the fifth output unit node , the output value of the sixth output unit node , the output value of the seventh output unit node and the output value of the eighth output unit node , according to the exponential function First, the output values ​​of the four output unit nodes corresponding to the deep learning model B are initially converted into the second preliminary output value and the probability value of the second preliminary output value , and then further optimize and transform it to obtain the second optimized output value and the enhanced probability value of the second optimized output value ; The enhanced probability value using the second optimized output value And the data set, combined with the error chain derivative rule, continue to train the deep learning model B. After the training is completed, the output value of the second disease is obtained The actual disease annotations corresponding to the nodes in the output layer of the deep learning model B , the output value of the second disease They are the output values ​​of the second railing damage respectively , the output value of the second crack , Second weathering output value and second coating peeling output value ; Construct the second auxiliary model, namely the deep learning model C, which consists of 1 input layer, 15 convolutional layers, 15 activation layers, 5 pooling layers, 1 global average pooling layer and 1 output layer; Input the data set into the deep learning model C, calculate the weights and biases of the neural network units of the deep learning model C through the error back propagation algorithm, complete the training of the deep learning model C, and obtain the trained deep learning model C and its loss function ; The output value of the output unit node corresponding to the deep learning model C is , respectively, are the output values ​​of the ninth output unit node , the output value of the tenth output unit node , the output value of the eleventh output unit node and the output value of the twelfth output unit node , according to the exponential function First, the output values ​​of the four output unit nodes corresponding to the deep learning model C are initially converted into the third preliminary output value and the probability value of the third preliminary output value , and then further optimize and transform it to obtain the third optimized output value and the enhanced probability value of the third optimized output value ; The enhanced probability value using the third optimized output value And the data set, combined with the error chain derivative rule, continue to train the deep learning model C. After the training is completed, the output value of the third disease is obtained The actual disease annotations corresponding to the nodes in the output layer of the deep learning model B , the third disease output value They are the damaged output values ​​of the third railing respectively , the output value of the third crack , the third weathering output value and the third coating peeling output value ; According to the second disease output value is and the enhanced probability value of the first optimized output value , construct the loss function between deep learning model B and deep learning model A , get the corresponding weight coefficient; Probability value based on the first preliminary output value The actual disease annotations corresponding to the nodes in the output layer of the deep learning model A , construct the probability value considering the first preliminary output value The loss function of the deep learning model A is , get the corresponding weight coefficient; Based on the loss function between deep learning model B and deep learning model A and the probability value considering the first preliminary output value The loss function of the deep learning model A is , establish the relationship loss function between the output layer of deep learning model B and deep learning model A ; According to the above operations, the relationship loss function between the output layers of deep learning model C and deep learning model B is established. The relationship loss function between the output layer of deep learning model C and deep learning model A .

[0013] Furthermore, the S5 specifically includes the following steps: S51. Establish an inter-layer loss function from the enhanced model to the basic model, and combine the fused loss function to obtain the total loss function; S52. Based on the chain derivation rule, the three models are trained by error back propagation to obtain the trained deep learning model A, i.e., the optimal model; In the S51, the selected layers of the deep learning model A are connected to the selected layers of the deep learning model C, the selected layers in the deep learning model A include the second layer: convolution layer 1, the third layer: activation layer 1, the fourth layer: pooling layer 1 and the seventeenth layer: output layer, and the selected layers of the deep learning model C include the fourth layer: convolution layer 2, the sixth layer: pooling layer 1, the tenth layer: activation layer 4 and the thirty-eighth layer: output layer; The selected layers of the deep learning model A are , the corresponding size is ,in, and 17, is the selected number of layers of the deep learning model A, is the selected layer width of the deep learning model A, is the selected layer length of the deep learning model A, is the number of channels of the selected layer of deep learning model A, and the selected layer of deep learning model C is , the corresponding size is ,in, and 38, is the selected number of layers of the deep learning model C, is the selected layer width of the deep learning model C, is the selected layer length of the deep learning model C, is the number of selected layer channels of the deep learning model C; when When , a convolution operation is added to the selected layer output of the deep learning model A, with a convolution kernel size of 1×1 and a number of convolution kernels. , get the selected layer output of deep learning model A , and the corresponding output value of each channel is , =1, 2, 3, ..., , Output channel number for the selected layer of deep learning model A and the selected layer of deep learning model C The corresponding output values ​​of each channel are , =1, 2, 3, ..., , Output channel number for the selected layer of deep learning model C, and establish the loss function between the selected layer of deep learning model A and the selected layer of deep learning model C ; when When , a convolution operation is added to the selected layer output of the deep learning model C, with a convolution kernel size of 1×1 and a number of convolution kernels. , the selected layer output of the deep learning model C is , and the corresponding output value of each channel is , the selected layers of the deep learning model C The corresponding output values ​​of each channel are , establish the loss function between the selected layer of deep learning model A and the selected layer of deep learning model C ; According to the above calculation, the loss function between the second layer of deep learning model A and the fourth layer of deep learning model C is obtained , the loss function between the third layer of deep learning model A and the sixth layer of deep learning model C And the loss function between the fourth layer of deep learning model A and the tenth layer of deep learning model C , combining the relationship loss function between the output layers of deep learning model B and deep learning model A , the relationship loss function between the output layers of deep learning model C and deep learning model B The relationship loss function between the output layer of deep learning model C and deep learning model A , define the total loss function .

[0014] Furthermore, in S6, the final railing damage identification result is reviewed on site, and the number of damaged areas at different times is counted. and the area of ​​damaged area , establish the number of damaged areas and the area of ​​damaged area With bridge railing operation time The law of change; Number of damaged areas Over time The changing law of is expressed as: ; in, is the number of damaged areas of the railing at the initial moment, is the square term of time in the number of damaged areas The coefficient of The time in the number of damaged areas The coefficient of is a constant in the number of damaged areas; Damaged area Over time The changing law of is expressed as: ; in, is the area of ​​the damaged railing at the initial moment, is the square term of time in the damaged area The coefficient of The time in the damaged area The coefficient of is a constant in the area of ​​the damaged region; The relationship between the repair cost and the number and area of ​​damage before and after the railing repair is constructed to obtain the cost before the railing repair and the cost of railing repair ; Before the railings are repaired, the relationship between the repair cost and the number and area of ​​damage is expressed as: ; in, is the coefficient of the number of damaged areas before the railing is repaired, The coefficient of the damaged area of ​​the railing before repair; After the railings are repaired, the relationship between the repair cost and the number and area of ​​damage is expressed as: ; in, is the coefficient of the number of damaged areas after the railing is repaired, is the coefficient of the damaged area after the railing is repaired, Time for railing repairs; According to the total time , further establish the total cost of restoration Running time with bridge railings The relationship is expressed as: ; The method of solving the minimum value of the quadratic function by the matching method is used to solve the minimum value of the total repair cost. , the corresponding railing operation time is the best time to repair .

[0015] The beneficial effects of the present invention are as follows: when the present invention adopts a drone to detect damaged bridge railings, the shortest flight path helps to improve the detection efficiency, reduce the required time and battery power consumption, thereby increasing the operating endurance of the drone, reducing the operating cost, and being able to obtain the integrity information of key structures in the shortest time, ensuring the safe operation of the bridge structure; the present invention updates the pixels of the image of the damaged railing, and adjusts the pixel value to make the image illumination uniform, which can reduce the error caused by the change in illumination and make the true state of the railing easier to accurately identify; the present invention realizes the rapid and intelligent identification of damaged bridge railings, and introduces additional supervision signals at different levels by establishing a basic model, an auxiliary model and an enhanced model, which helps to improve the generalization ability of the model and reduce overfitting; the present invention can effectively determine the timing of repairing damaged bridge railings, avoids the high cost caused by premature repair or delayed repair, optimizes the use of resources, provides data support for subsequent repairs, and reduces the overall maintenance cost. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] The drawings described herein are used to provide a further understanding of the present invention and constitute a part of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings: Figure 1 Schematic diagram of the flow chart of the method for detecting bridge railing damage and determining the repair timing based on drones. DETAILED DESCRIPTION

[0017] In order to make the technical solutions and advantages of the embodiments of the present invention more clearly understood, the exemplary embodiments of the present invention are further described in detail below in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than an exhaustive list of all the embodiments. It should be noted that the embodiments of the present invention and the features in the embodiments can be combined with each other without conflict.

[0018] refer to Figure 1 The present embodiment is described in detail. The method for detecting damage of bridge railings and determining the repair timing based on a drone specifically includes the following steps: S1. Plan the shortest flight path of the drone based on the drone parameter information and operation information; S2. Collect the apparent image of the bridge railing along the shortest flight path of the UAV to generate an image matrix, and enhance the image quality to obtain a new image matrix; S3. Equally divide the image in the new image matrix and update the image pixels to obtain an image after pixel update; S4. construct a data set based on the new image matrix combined with disease information, input the data set into the established basic model, auxiliary model and enhanced model for training, fuse the loss functions of the three models, and obtain the fused loss function; S5. Establish an inter-layer loss function from the enhanced model to the basic model, combine the fused loss function to obtain the total loss function, train the three models, use the trained basic model as the final model, and output the final railing damage recognition result; S6. Based on the final railing damage identification results, establish the changing pattern of the number and area of ​​damaged areas with the operation time of the bridge railing, calculate the minimum total repair cost, and determine the best time to repair the bridge railing.

[0019] Furthermore, the S1 specifically includes the following steps: S11. Determine the take-off and landing points of the drone and select the observation position of the operator; Specifically, before the drone takes off, it is necessary to choose a location where the drone can take off and land safely, away from people and traffic, and avoid high-voltage wires and large metal structures that may interfere with the drone's navigation system. At the same time, choose an observation position with a good field of view so that the operator can maintain a good line of sight and facilitate operation; S12. Determine the appropriate flight altitude AVh according to the focal length and sensor size of the camera carried by the drone, obtain sufficient image coverage and resolution, and set the drone flight speed to AVv; S13. Develop a flight path based on the take-off and landing points of the drone. Based on the drawing parameters of the bridge, combined with the field of view and clarity of the camera carried by the drone, determine the coordinate information of the railing and the shooting point information of the drone, ensure that the images of all point information can fully and clearly cover the railing, and design the shortest flight path of the drone; In S13, the take-off point coordinates are AVo (AVoX, AVoY, AVoZ). On this basis, all drone shooting points are determined. The number of any drone shooting point is p, and its corresponding drone shooting point coordinates are AVp (AVpX, AVpY, AVpZ). The number set of drone shooting points is expressed as {1, 2, 3, ..., XN}, and the corresponding drone shooting point coordinate set is expressed as {AV1 (AV1X, AV1Y, AV1Z), AV2 (AV2X, AV2Y, AV2Z), ..., AVXN (AVXNX, AVXNY, AVXNZ)}, where XN is the number of drone shooting points. The adjacent distance between any drone shooting point AVp and its adjacent drone shooting point AV (p+1) is calculated. ; Adjacent distance It is expressed as: ; Among them, (AVpX, AVpY, AVpZ) are the coordinates of any drone shooting point, and (AV(p+1)X, AV(p+1)Y, AV(p+1)Z) are the coordinates of the drone shooting point adjacent to any drone shooting point; Taking the take-off point AVo as the starting point, according to the distance between any drone shooting point and the take-off point , calculate the distance between all drone shooting points and the take-off point, select the three drone shooting points corresponding to the minimum distance, and randomly select them to obtain the first drone shooting point that the drone arrives at after departing from the take-off point; distance It is expressed as: ; After determining the first shooting point that the drone arrives at, calculate the distance between all shooting points and the first shooting point that is arrived at, screen out the three drone shooting points corresponding to the smallest distance, and randomly select them to obtain the second shooting point that the drone arrives at after departing from the first shooting point that it arrives at. It should be noted that the route of the drone will not return, that is, the drone will not fly to the same shooting point twice. Repeat the above steps until all the shooting points arrived at by the drone are obtained. According to the difference in each random selection, calculate the total flight distance TDis1 of the drone under the first flight mode, repeat the above steps, the number of repetitions is CN, and integrate to obtain the total flight distance set of the drone under different flight modes {TDis1, TDis2, TDis3, TDis4, …, TDisCN}, arrange them in order from small to large, and select the total distance and corresponding flight route ranked in the top TN, TN is a preset value, count the intersection information of TN flight routes, and select the flight route with the largest number of intersections to obtain the shortest flight path of the drone.

[0020] Furthermore, the step S2 specifically includes the following steps: S21. Using the high-definition camera on the drone, collect the bridge railing surface image ph along the shortest flight path of the drone, and the corresponding image matrix is ​​PH; S22. Construct a 5th-order denoising kernel function N, normalize all elements in the kernel function, and obtain the normalized denoising kernel function ; S23. According to the normalized denoising kernel function , perform denoising kernel function processing on the image to obtain a new image matrix P to achieve image quality enhancement; In S22, the fifth-order denoising kernel function N is expressed as: ; ; Where x=1, 2, 3, 4 or 5, x is the row number of the matrix, and y=1, 2, 3, 4 or 5, y is the column number of the matrix. is the standard deviation of all elements in the kernel function, is the base of natural logarithms; Normalized denoising kernel function It is expressed as: ; In S23, the new image matrix P is expressed as: ; Among them, PH is the image matrix corresponding to the apparent image ph of the bridge railing collected along the shortest flight path of the UAV; Specifically, when using drones to collect surface images of bridge railings, due to the vibration and stability problems in the drone flight environment, the imaging quality of the drone camera, adverse weather conditions (such as fog, rain, strong winds), and uneven lighting conditions, etc., there will be noise in the collected images, which will have a negative impact on the image quality; in addition, electronic interference and compression loss during image transmission may also increase noise. Noise will mask or blur the key features of the railings, making it difficult for the automatic recognition algorithm to accurately detect damaged areas. Noise will cause the algorithm to mistakenly identify normal areas as damaged, or ignore actual damage, increasing the number of false alarms and missed alarms. At the same time, noise will increase the computational complexity of the image preprocessing and feature extraction stages, reducing the efficiency and speed of the recognition process.

[0021] Furthermore, the step S3 specifically includes the following steps: S31. According to the new image matrix P, the pixel grayscale value is divided into 16 grayscale levels, and the image in the new image matrix P is divided into n equal parts to obtain the divided image; S32. Count the pixel frequencies corresponding to each gray level in the equally divided image , calculate the grayscale frequency cumulative value corresponding to each grayscale level in the equally divided image ; S33.According to the gray frequency cumulative value , construct a pixel value update method and calculate the pixel update value corresponding to the gray level , get the image after pixel update; In S31, the pixel gray value is 0-255, and the gray level is , =1, 2, 3, …, 16, is the grayscale number, and the image after equal division is Pnum, num=1, 2, ..., n; In S32, each gray level in the equally divided image Pnum The corresponding grayscale frequency cumulative value It is expressed as: ; in, is less than or equal to The gray level number, Indicates the gray level number is Gray level; In S33, the pixel update value corresponding to the gray level It is expressed as: ; in, is the minimum non-zero value among all pixel frequency accumulation values, is the total number of pixels corresponding to the equally divided image Pnum, is the maximum value of the gray level.

[0022] Specifically, due to the influence of the environment, light and dark areas will appear in the apparent image of the bridge railing, which will affect the quality of features extracted by the image processing algorithm. Therefore, the pixels of the apparent image of the bridge railing are updated to facilitate the subsequent model to accurately identify the damage of the railing.

[0023] Furthermore, the step S4 specifically includes the following steps: S41. Based on the new image matrix, select all images containing damaged railings, cracks, weathering, and peeling coatings as input images; S42. Through manual annotation, the damaged area of ​​the railing and other three diseases are selected in the input image, and a data set is constructed. The data set is divided into a training set and a test set according to a ratio of 8:2; S43. According to the data set, a basic model, an auxiliary model and an enhanced model are constructed, and the loss functions of the three models are calculated and integrated to obtain a fused loss function; In S42, the damaged area of ​​the railing is marked as Damage, and the three types of damage, namely cracks, weathering and coating peeling, are marked as other1, other2 and other3 respectively; In S43, a network structure of a basic model, i.e., a deep learning model A, is constructed, wherein the deep learning model A consists of 1 input layer, 5 convolutional layers, 5 activation layers, 4 pooling layers, 1 global average pooling layer and 1 output layer; The layers in the network structure of deep learning model A are connected in the following order: Input layer: image size 227×227×3 (width×height×channel); Convolution layer 1: kernel size 3×3, number of kernels 32, Stride: 1, Padding: Same; Activation layer 1: ReLU; Pooling layer 1: Type: Max pooling; Pooling size: 2×2; Stride: 2; Convolutional layer 2: kernel size 3×3, number of kernels 64, Stride: 1, Padding: Same; Activation layer 2: ReLU; Pooling layer 2: Type: max pooling, pooling size: 2×2, stride: 2; Convolution layer 3: convolution kernel size 3×3, number of convolution kernels 128, stride: 1, padding: same; Activation layer 3: ReLU; Pooling layer 3: Type: Max pooling, pooling size: 2×2, Stride: 2; Convolutional layer 4: convolution kernel size 3×3, number of convolution kernels 256, stride: 1, padding: same; Activation layer 4: ReLU; Convolution layer 5: kernel size 3×3, number of kernels 256, stride: 1, padding: same; Activation layer 5: ReLU; Pooling layer 4: Type: Max pooling, pooling size: 2×2, Stride: 2; Global average pooling layer 1; Output layer: Type: Softmax, Number of neurons: 4; Input the data set into the deep learning model A, calculate the weights and biases of the neural network units of the deep learning model A through the error back propagation algorithm, complete the training of the deep learning model A, and obtain the trained deep learning model A and its loss function ; Loss function of the trained deep learning model A It is expressed as: ; in, is the total number of data samples in the dataset of deep learning model A, For the The true labels of samples, For deep learning model A, predict The probability that a sample is of category 1, is the natural logarithm; The output value of the output unit node corresponding to the deep learning model A is , respectively, are the output values ​​of the first output unit node , the output value of the second output unit node , the output value of the third output unit node and the output value of the fourth output unit node , according to the exponential function First, the output values ​​of the four output unit nodes corresponding to the deep learning model A are initially converted into the probability values ​​of the first preliminary output value and the first preliminary output value. , and then further optimize and transform it to obtain the first optimized output value and the enhanced probability value of the first optimized output value ; Probability value of the first preliminary output value It is expressed as: ; The enhanced probability value of the first optimized output value It is expressed as: ; in, G is the enhancement coefficient; The enhanced probability value using the first optimized output value And the data set, combined with the error chain derivative rule, continue to train the deep learning model A. After the training is completed, the output value of the first disease is obtained The actual disease annotations corresponding to the nodes in the output layer of the deep learning model A , the output value of the first disease The output values ​​of the first railing damage are , the output value of the first crack , the first weathering output value and first coating peeling output value ; Construct the first auxiliary model, namely deep learning model B, which consists of 1 input layer, 10 convolutional layers, 10 activation layers, 5 pooling layers, 1 global average pooling layer and 1 output layer; The layers in the network structure of deep learning model B are connected in the following order: Input layer: image size 227×227×3 (width×height×channel); Convolution layer 1: kernel size 3×3, number of kernels 32, Stride: 1, Padding: Same; Activation layer 1: ReLU; Convolutional layer 2: kernel size 3×3, number of kernels 64, Stride: 1, Padding: Same; Activation layer 2: ReLU; Pooling layer 1: Type: Max pooling, pooling size: 2×2, Stride: 2; Convolution layer 3: convolution kernel size 3×3, number of convolution kernels 64, stride: 1, padding: same; Activation layer 3: ReLU; Convolutional layer 4: kernel size 3×3, number of kernels 64, Stride: 1, Padding: Same; Activation layer 4: ReLU; Pooling layer 2: Type: max pooling, pooling size: 2×2, stride: 2; Convolution layer 5: kernel size 3×3, number of kernels 128, Stride: 1, Padding: Same; Activation layer 5: ReLU; Convolution layer 6: kernel size 3×3, number of kernels 128, Stride: 1, Padding: Same; Activation layer 6: ReLU; Pooling layer 3: Type: Max pooling, pooling size: 2×2, Stride: 2; Convolution layer 7: kernel size 3×3, number of kernels 256, Stride: 1, Padding: Same; Activation layer 7: ReLU; Convolution layer 8: kernel size 3×3, number of kernels 256, Stride: 1, Padding: Same; Activation layer 8: ReLU; Convolution layer 9: kernel size 3×3, number of kernels 256, Stride: 1, Padding: Same; Activation layer 9: ReLU; Pooling layer 4: Type: Max pooling, pooling size: 2×2, Stride: 2; Convolution layer 10: kernel size 3×3, number of kernels 512, Stride: 1, Padding: Same; Activation layer 10: ReLU; Pooling layer 5: Type: Max pooling, pooling size: 2×2, Stride: 2; Global average pooling layer 1; Output layer: Type: Softmax, Number of neurons: 4; Input the data set into the deep learning model B, calculate the weights and biases of the neural network units of the deep learning model B through the error back propagation algorithm, complete the training of the deep learning model B, and obtain the trained deep learning model B and its loss function ; Loss function of the trained deep learning model B It is expressed as: ; in, is the total number of data samples in the dataset of deep learning model B, For the The true labels of samples, For deep learning model B to predict The probability that a sample is class 1; The output value of the output unit node corresponding to the deep learning model B is , respectively, are the output values ​​of the fifth output unit node , the output value of the sixth output unit node , the output value of the seventh output unit node and the output value of the eighth output unit node , according to the exponential function First, the output values ​​of the four output unit nodes corresponding to the deep learning model B are initially converted into the second preliminary output value and the probability value of the second preliminary output value , and then further optimize and transform it to obtain the second optimized output value and the enhanced probability value of the second optimized output value ; Probability value of the second preliminary output value It is expressed as: ; The enhanced probability value of the second optimized output value It is expressed as: ; The enhanced probability value using the second optimized output value And the data set, combined with the error chain derivative rule, continue to train the deep learning model B. After the training is completed, the output value of the second disease is obtained The actual disease annotations corresponding to the nodes in the output layer of the deep learning model B , the output value of the second disease They are the output values ​​of the second railing damage respectively , the output value of the second crack , Second weathering output value and second coating peeling output value ; Construct the second auxiliary model, namely the deep learning model C, which consists of 1 input layer, 15 convolutional layers, 15 activation layers, 5 pooling layers, 1 global average pooling layer and 1 output layer; The layers in the network structure of the deep learning model C are connected in the following order: Input layer: image size 227×227×3 (width×height×channel); Convolution layer 1: kernel size 3×3, number of kernels 64, Stride: 1, Padding: Same; Activation layer 1: ReLU; Convolutional layer 2: kernel size 3×3, number of kernels 64, Stride: 1, Padding: Same; Activation layer 2: ReLU; Pooling layer 1: Type: Max pooling, pooling size: 2×2, Stride: 2; Convolution layer 3: convolution kernel size 3×3, number of convolution kernels 128, stride: 1, padding: same; Activation layer 3: ReLU; Convolutional layer 4: kernel size 3×3, number of kernels 128, Stride: 1, Padding: Same; Activation layer 4: ReLU; Pooling layer 2: Type: max pooling, pooling size: 2×2, stride: 2; Convolution layer 5: kernel size 3×3, number of kernels 256, stride: 1, padding: same; Activation layer 5: ReLU; Convolution layer 6: kernel size 3×3, number of kernels 256, Stride: 1, Padding: Same; Activation layer 6: ReLU; Convolution layer 7: kernel size 3×3, number of kernels 256, Stride: 1, Padding: Same; Activation layer 7: ReLU; Pooling layer 3: Type: Max pooling, pooling size: 2×2, Stride: 2; Convolutional layer 8: kernel size 3×3, number of kernels 512, Stride: 1, Padding: Same; Activation layer 8: ReLU; Convolution layer 9: kernel size 3×3, number of kernels 512, Stride: 1, Padding: Same; Activation layer 9: ReLU; Convolution layer 10: kernel size 3×3, number of kernels 512, Stride: 1, Padding: Same; Activation layer 10: ReLU; Pooling layer 4: Type: Max pooling, pooling size: 2×2, Stride: 2; Convolution layer 11: kernel size 3×3, number of kernels 512, Stride: 1, Padding: Same; Activation layer 11: ReLU; Convolution layer 12: kernel size 3×3, number of kernels 512, Stride: 1, Padding: Same; Activation layer 12: ReLU; Convolution layer 13: kernel size 3×3, number of kernels 512, Stride: 1, Padding: Same; Activation layer 13: ReLU; Convolution layer 14: kernel size 3×3, number of kernels 512, stride: 1, padding: same; Activation layer 14: ReLU; Convolution layer 15: kernel size 3×3, number of kernels 512, Stride: 1, Padding: Same; Activation layer 15: ReLU; Pooling layer 5: Type: Max pooling, pooling size: 2×2, Stride: 2; Global average pooling layer 1; Output layer: Type: Softmax, Number of neurons: 4; Input the data set into the deep learning model C, calculate the weights and biases of the neural network units of the deep learning model C through the error back propagation algorithm, complete the training of the deep learning model C, and obtain the trained deep learning model C and its loss function ; Loss function of the trained deep learning model C It is expressed as: ; in, is the total number of data samples in the dataset of the deep learning model C, For the The true labels of samples, For the deep learning model C to predict The probability that a sample is class 1; The output value of the output unit node corresponding to the deep learning model C is , respectively, are the output values ​​of the ninth output unit node , the output value of the tenth output unit node , the output value of the eleventh output unit node and the output value of the twelfth output unit node , according to the exponential function First, the output values ​​of the four output unit nodes corresponding to the deep learning model C are initially converted into the third preliminary output value and the probability value of the third preliminary output value , and then further optimize and transform it to obtain the third optimized output value and the enhanced probability value of the third optimized output value ; The probability value of the third preliminary output value It is expressed as: ; The enhanced probability value of the third optimized output value It is expressed as: ; The enhanced probability value using the third optimized output value And the data set, combined with the error chain derivative rule, continue to train the deep learning model C. After the training is completed, the output value of the third disease is obtained The actual disease annotations corresponding to the nodes in the output layer of the deep learning model B , the third disease output value They are the damaged output values ​​of the third railing respectively , the output value of the third crack , the third weathering output value and the third coating peeling output value ; According to the second disease output value is and the enhanced probability value of the first optimized output value , construct the loss function between deep learning model B and deep learning model A , get the corresponding weight coefficient; Loss function between deep learning model B and deep learning model A It is expressed as: ; Probability value based on the first preliminary output value The actual disease annotations corresponding to the nodes in the output layer of the deep learning model A , construct the probability value considering the first preliminary output value The loss function of the deep learning model A is , get the corresponding weight coefficient; Consider the probability value of the first preliminary output value The loss function of the deep learning model A is It is expressed as: ; Based on the loss function between deep learning model B and deep learning model A and the probability value considering the first preliminary output value The loss function of the deep learning model A is , establish the relationship loss function between the output layer of deep learning model B and deep learning model A ; The relationship loss function between the output layer of deep learning model B and deep learning model A It is expressed as: ; in, is the loss function between deep learning model B and deep learning model A The weight coefficient of is the probability value considering the first preliminary output value The loss function of the deep learning model A is The weight coefficient of According to the above operations, the relationship loss function between the output layers of deep learning model C and deep learning model B is established. The relationship loss function between the output layer of deep learning model C and deep learning model A ; Loss function between deep learning model C and deep learning model B It is expressed as: ; Consider the probability value of the second preliminary output value The loss function of the deep learning model B is It is expressed as: ; The relationship loss function between the output layers of deep learning model C and deep learning model B It is expressed as: ; in, is the loss function between deep learning model C and deep learning model B The weight coefficient of is the probability value considering the second preliminary output value The loss function of the deep learning model B is The weight coefficient of Loss function between deep learning model C and deep learning model A It is expressed as: ; The relationship loss function between the output layer of deep learning model C and deep learning model A It is expressed as: , in, is the loss function between deep learning model C and deep learning model A The weight coefficient of Specifically, the characteristics of bridge railing damage are cracks, fractures, deformations or missing parts. The above features are not very obvious in the image and are easily confused with background noise. In order to accurately identify the characteristics of the damaged area of ​​the railing, the designed model structure includes input layer, convolution layer, activation layer, pooling layer, global average pooling layer, and output layer. The functions of each structural layer are as follows: Input layer: accepts the raw image data collected by the drone as the starting point for model processing. The image contains railings and complex backgrounds such as water surfaces, vegetation, roads, and other structures; Convolution layer: extracts image features by applying a series of trainable filters. Each convolution layer can identify different features in the image, from simple edges and textures to more complex patterns, such as the specific shape of the railing and the characteristics of damaged bridge railings; Activation layer: can introduce nonlinearity so that the model can learn and represent more complex feature patterns. For the detection of damaged railings, the model needs to be able to better distinguish between the normal and damaged parts of the railings; Pooling layer: reduces the amount of calculation by reducing the spatial resolution of the feature map, and increases the model's sensitivity to small objects. The robustness of changes and deviations enables the model to better extract key features from the entire image and reduce sensitivity to noise and small displacements in the image; Global average pooling layer: In the deep layer of the network, the global average pooling layer takes the average value of each feature map as output, which helps to reduce the number of parameters of the model, reduces the risk of overfitting, and enables the model to focus on the key features of the damaged railings rather than their exact locations; Output layer: That is, the fully connected layer with an activation function, which converts the features extracted by the previous layers into the final classification decision output. For bridge railing damage detection, the output layer can predict the probability of railing damage in the image and provide more detailed information, such as the type or severity of damage.

[0024] Furthermore, the S5 specifically includes the following steps: S51. Establish an inter-layer loss function from the enhanced model to the basic model, and combine the fused loss function to obtain the total loss function; S52. Based on the chain derivation rule, the three models are trained by error back propagation to obtain the trained deep learning model A, i.e., the optimal model; In the S51, the selected layers of the deep learning model A are connected to the selected layers of the deep learning model C, the selected layers in the deep learning model A include the second layer: convolution layer 1, the third layer: activation layer 1, the fourth layer: pooling layer 1 and the seventeenth layer: output layer, and the selected layers of the deep learning model C include the fourth layer: convolution layer 2, the sixth layer: pooling layer 1, the tenth layer: activation layer 4 and the thirty-eighth layer: output layer; The selected layers of the deep learning model A are , the corresponding size is ,in, and 17, is the selected number of layers of the deep learning model A, is the selected layer width of the deep learning model A, is the selected layer length of the deep learning model A, is the number of channels of the selected layer of deep learning model A, and the selected layer of deep learning model C is , the corresponding size is ,in, and 38, is the selected number of layers of the deep learning model C, is the selected layer width of the deep learning model C, is the selected layer length of the deep learning model C, is the number of selected layer channels of the deep learning model C; when When , a convolution operation is added to the selected layer output of the deep learning model A, with a convolution kernel size of 1×1 and a number of convolution kernels. , get the selected layer output of deep learning model A , and the corresponding output value of each channel is , =1, 2, 3, …, , Output channel number for the selected layer of deep learning model A and the selected layer of deep learning model C The corresponding output values ​​of each channel are , =1, 2, 3, …, , Output channel number for the selected layer of deep learning model C, and establish the loss function between the selected layer of deep learning model A and the selected layer of deep learning model C ; when When , the loss function between the selected layer of deep learning model A and the selected layer of deep learning model C is It is expressed as: ; when When , a convolution operation is added to the selected layer output of the deep learning model C, with a convolution kernel size of 1×1 and a number of convolution kernels. , the selected layer output of the deep learning model C is , and the corresponding output value of each channel is , the selected layers of the deep learning model C The corresponding output values ​​of each channel are , establish the loss function between the selected layer of deep learning model A and the selected layer of deep learning model C ; when When , the loss function between the selected layer of deep learning model A and the selected layer of deep learning model C is It is expressed as: ; According to the above calculation, the loss function between the second layer of deep learning model A and the fourth layer of deep learning model C is obtained , the loss function between the third layer of deep learning model A and the sixth layer of deep learning model C And the loss function between the fourth layer of deep learning model A and the tenth layer of deep learning model C , combining the relationship loss function between the output layers of deep learning model B and deep learning model A , the relationship loss function between the output layers of deep learning model C and deep learning model B The relationship loss function between the output layer of deep learning model C and deep learning model A , define the total loss function ; Total loss function It is expressed as: ; in, is the loss function between the third layer of deep learning model A and the sixth layer of deep learning model C The corresponding weight coefficient is, is the loss function between the third layer of deep learning model A and the sixth layer of deep learning model C The corresponding weight coefficient is, is the loss function between the fourth layer of deep learning model A and the tenth layer of deep learning model C The corresponding weight coefficient.

[0025] In S6, the final railing damage identification result is reviewed on site, and the number of damaged areas at different times is counted. and the area of ​​damaged area , establish the number of damaged areas and the area of ​​damaged area With bridge railing operation time The law of change; Number of damaged areas Over time The changing law of is expressed as: ; in, is the number of damaged areas of the railing at the initial moment, is the square term of time in the number of damaged areas The coefficient of The time in the number of damaged areas The coefficient of is a constant in the number of damaged areas; Damaged area Over time The changing law of is expressed as: ; in, is the area of ​​the damaged railing at the initial moment, is the square term of time in the damaged area The coefficient of The time in the damaged area The coefficient of is a constant in the area of ​​the damaged region; The relationship between the repair cost and the number and area of ​​damage before and after the railing repair is constructed to obtain the cost before the railing repair and the cost of railing repair ; Before the railings are repaired, the relationship between the repair cost and the number and area of ​​damage is expressed as: ; in, is the coefficient of the number of damaged areas before the railing is repaired, The coefficient of the damaged area of ​​the railing before repair; After the railings are repaired, the relationship between the repair cost and the number and area of ​​damage is expressed as: ; in, is the coefficient of the number of damaged areas after the railing is repaired, is the coefficient of the damaged area after the railing is repaired, Time for railing repairs; According to the total time , further establish the total cost of restoration Running time with bridge railings relationship; Total cost of repair Running time with bridge railings The relationship is expressed as: ; The method of solving the minimum value of the quadratic function by the matching method is used to solve the minimum value of the total repair cost. , the corresponding railing operation time is the best time to repair ; Specifically, the repair time and repair cost of bridge railings before and after damage are different. Early repair is usually less expensive and can control the scope of damage; delayed repair may lead to the expansion of damage, requiring more complex and costly intervention measures; therefore, the relationship between repair costs and the number and area of ​​damage is calculated in stages; usually, the total time needs to be found. The best time to repair At this point, the total cost of repair within this time range can be established. Running time with bridge railings relationship.

[0026] Although the present invention has been described according to a limited number of embodiments, it will be apparent to those skilled in the art, with the benefit of the above description, that other embodiments may be envisioned within the scope of the invention thus described. In addition, it should be noted that the language used in this specification is selected primarily for readability and teaching purposes, rather than for explaining or defining the subject matter of the present invention. Therefore, many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the appended claims. The disclosure of the present invention is illustrative, not restrictive, with respect to the scope of the present invention, which is defined by the appended claims.

Claims

1. A method for detecting bridge railing damage and determining the repair timing based on drones, characterized in that: The following steps are involved: S1. Plan the shortest flight path of the drone based on the drone parameter information and operation information; S2. Collect the apparent image of the bridge railing along the shortest flight path of the UAV to generate an image matrix, and enhance the image quality to obtain a new image matrix; S3. Equally divide the image in the new image matrix and update the image pixels to obtain an image after pixel update; S4. construct a data set based on the new image matrix combined with the disease information, input the data set into the established basic model, auxiliary model and enhanced model for training, fuse the loss functions of the three models, and obtain the fused loss function; S5. Establish an inter-layer loss function from the enhanced model to the basic model, combine the fused loss function to obtain the total loss function, train the three models, use the trained basic model as the final model, and output the final railing damage recognition result; S6. Based on the final railing damage identification results, establish the change pattern of the number and area of ​​damaged areas with the operation time of the bridge railing, calculate the minimum total repair cost, and determine the best time to repair the bridge railing.

2. The method for detecting damage of bridge railings and determining the timing of repair based on drones according to claim 1 is characterized in that: The S1 specifically includes the following steps: S11. Determine the take-off and landing points of the drone and select the observation position of the operator; S12. Determine the flight altitude AVh according to the focal length and sensor size of the camera carried by the drone, obtain the image coverage and resolution, and set the drone flight speed to AVv; S13. Develop the shortest flight path based on the take-off and landing points of the drone, and determine the coordinate information of the railing and the location information of the drone shooting point; In S13, the take-off point coordinates are AVo (AVoX, AVoY, AVoZ). On this basis, all drone shooting points are determined. The number of any drone shooting point is p, and its corresponding drone shooting point coordinates are AVp (AVpX, AVpY, AVpZ). The number set of drone shooting points is expressed as {1, 2, 3, ..., XN}, and the corresponding drone shooting point coordinate set is expressed as {AV1 (AV1X, AV1Y, AV1Z), AV2 (AV2X, AV2Y, AV2Z), ..., AVXN (AVXNX, AVXNY, AVXNZ)}, where XN is the number of drone shooting points. The adjacent distance between any drone shooting point AVp and its adjacent drone shooting point AV (p+1) is calculated. ; Taking the take-off point AVo as the starting point, according to the distance between any drone shooting point and the take-off point , calculate the distance between all drone shooting points and the take-off point, select the three drone shooting points corresponding to the minimum distance, and randomly select them to obtain the first drone shooting point that the drone arrives at after departing from the take-off point; After determining the first shooting point that the drone arrives at, calculate the distance between all shooting points and the first shooting point that is arrived at, screen out the three drone shooting points corresponding to the smallest distance, and randomly select them to obtain the second shooting point that the drone arrives at after departing from the first shooting point it arrives at. Repeat the above steps until all the shooting points that the drone arrives at are obtained. According to the different random selections each time, calculate the total distance TDis1 of the drone's flight under the first flight mode. Repeat the above steps, the number of repetitions is CN, and integrate to obtain the total distance set of the drone's flight under different flight modes {TDis1, TDis2, TDis3, TDis4, …, TDisCN}, arrange them in order from small to large, and select the total distances and corresponding flight routes ranked in the top TN, TN is the preset value, count the intersection information of TN flight routes, and select the flight route with the largest number of intersections to obtain the shortest flight path of the drone.

3. The method for detecting damage of bridge railings and determining the timing of repair based on drone according to claim 2 is characterized in that: The S2 specifically includes the following steps: S21. Using the high-definition camera on the drone, collect the bridge railing surface image ph along the shortest flight path of the drone, and the corresponding image matrix is ​​PH; S22. Construct a 5th-order denoising kernel function N, normalize all elements in the kernel function, and obtain the normalized denoising kernel function ; S23. According to the normalized denoising kernel function , perform denoising kernel function processing on the bridge railing apparent image ph to obtain a new image matrix P to achieve image quality enhancement; In S22, the fifth-order denoising kernel function N is expressed as: ； ； Where x=1, 2, 3, 4 or 5, x is the row number of the matrix, and y=1, 2, 3, 4 or 5, y is the column number of the matrix. is the standard deviation of all elements in the kernel function, is the base of natural logarithms; Normalized denoising kernel function It is expressed as: ； In S23, the new image matrix P is expressed as: ； Among them, PH is the image matrix corresponding to the apparent image ph of the bridge railing collected along the shortest flight path of the UAV.

4. The method for detecting damage of bridge railings and determining the timing of repair based on drones according to claim 3 is characterized in that: The S3 specifically includes the following steps: S31. According to the new image matrix P, the pixel grayscale value is divided into 16 grayscale levels, and the image in the new image matrix P is divided into n equal parts to obtain the divided image; S32. Count the pixel frequencies corresponding to each gray level in the equally divided image , calculate the grayscale frequency cumulative value corresponding to each grayscale level in the equally divided image ; S33.According to the gray frequency cumulative value , construct a pixel value update method and calculate the pixel update value corresponding to the gray level , get the image after pixel update; In S31, the pixel gray value is 0-255, and the gray level is , =1, 2, 3, …, 16, is the grayscale number, and the image after equal division is Pnum, num=1, 2, ..., n; In S32, each gray level in the equally divided image Pnum The corresponding gray frequency cumulative value It is expressed as: ； in, is less than or equal to The gray level number, Indicates the gray level number is Gray level; In S33, the pixel update value corresponding to the gray level It is expressed as: ； in, is the minimum non-zero value among all pixel frequency accumulation values, is the total number of pixels corresponding to the equally divided image Pnum, is the maximum value of the gray level.

5. The method for detecting damage of bridge railings and determining the timing of repair based on drones according to claim 4 is characterized in that: The S4 specifically includes the following steps: S41. Based on the new image matrix, select all images containing damaged railings, cracks, weathering, and peeling coatings as input images; S42. Through manual annotation, the damaged area of ​​the railing and other three diseases are selected in the input image, and a data set is constructed. The data set is divided into a training set and a test set according to a ratio of 8:2; S43. According to the data set, a basic model, an auxiliary model and an enhanced model are constructed, and the loss functions of the three models are calculated and integrated to obtain a fused loss function; In S42, the damaged area of ​​the railing is marked as Damage, and the three types of damage, namely cracks, weathering and coating peeling, are marked as other1, other2 and other3 respectively; In S43, a network structure of a basic model, i.e., a deep learning model A, is constructed, wherein the deep learning model A consists of 1 input layer, 5 convolutional layers, 5 activation layers, 4 pooling layers, 1 global average pooling layer and 1 output layer; Input the data set into the deep learning model A, calculate the weights and biases of the neural network units of the deep learning model A through the error back propagation algorithm, complete the training of the deep learning model A, and obtain the trained deep learning model A and its loss function ; The output value of the output unit node corresponding to the deep learning model A is , respectively, are the output values ​​of the first output unit node , the output value of the second output unit node , the output value of the third output unit node and the output value of the fourth output unit node , according to the exponential function First, the output values ​​of the four output unit nodes corresponding to the deep learning model A are initially converted into the probability values ​​of the first preliminary output value and the first preliminary output value. , and then further optimize and transform it to obtain the first optimized output value and the enhanced probability value of the first optimized output value ; The enhanced probability value using the first optimized output value And the data set, combined with the error chain derivative rule, continue to train the deep learning model A. After the training is completed, the output value of the first disease is obtained The actual disease annotations corresponding to the nodes in the output layer of the deep learning model A , the output value of the first disease The first railing damage output value , the output value of the first crack , the first weathering output value and first coating peeling output value ; Construct the first auxiliary model, namely deep learning model B, which consists of 1 input layer, 10 convolutional layers, 10 activation layers, 5 pooling layers, 1 global average pooling layer and 1 output layer; Input the data set into the deep learning model B, calculate the weights and biases of the neural network units of the deep learning model B through the error back propagation algorithm, complete the training of the deep learning model B, and obtain the trained deep learning model B and its loss function ; The output value of the output unit node corresponding to the deep learning model B is , respectively, are the output values ​​of the fifth output unit node , the output value of the sixth output unit node , the output value of the seventh output unit node and the output value of the eighth output unit node , according to the exponential function First, the output values ​​of the four output unit nodes corresponding to the deep learning model B are initially converted into the second preliminary output value and the probability value of the second preliminary output value , and then further optimize and transform it to obtain the second optimized output value and the enhanced probability value of the second optimized output value ; The enhanced probability value using the second optimized output value And the data set, combined with the error chain derivative rule, continue to train the deep learning model B. After the training is completed, the output value of the second disease is obtained The actual disease annotations corresponding to the nodes in the output layer of the deep learning model B , the output value of the second disease They are the output values ​​of the second railing damage respectively , the output value of the second crack , Second weathering output value and second coating peeling output value ; Construct the second auxiliary model, namely the deep learning model C, which consists of 1 input layer, 15 convolutional layers, 15 activation layers, 5 pooling layers, 1 global average pooling layer and 1 output layer; Input the data set into the deep learning model C, calculate the weights and biases of the neural network units of the deep learning model C through the error back propagation algorithm, complete the training of the deep learning model C, and obtain the trained deep learning model C and its loss function ; The output value of the output unit node corresponding to the deep learning model C is , respectively, are the output values ​​of the ninth output unit node , the output value of the tenth output unit node , the output value of the eleventh output unit node and the output value of the twelfth output unit node , according to the exponential function First, the output values ​​of the four output unit nodes corresponding to the deep learning model C are initially converted into the third preliminary output value and the probability value of the third preliminary output value , and then further optimize and transform it to obtain the third optimized output value and the enhanced probability value of the third optimized output value ; The enhanced probability value using the third optimized output value And the data set, combined with the error chain derivative rule, continue to train the deep learning model C. After the training is completed, the output value of the third disease is obtained The actual disease annotations corresponding to the nodes in the output layer of the deep learning model B , the third disease output value They are the damaged output values ​​of the third railing respectively , the output value of the third crack , the third weathering output value and the third coating peeling output value ; According to the second disease output value is and the enhanced probability value of the first optimized output value , construct the loss function between deep learning model B and deep learning model A , get the corresponding weight coefficient; Probability value based on the first preliminary output value The actual disease annotations corresponding to the nodes in the output layer of the deep learning model A , construct the probability value considering the first preliminary output value The loss function of the deep learning model A is , get the corresponding weight coefficient; Based on the loss function between deep learning model B and deep learning model A and the probability value considering the first preliminary output value The loss function of the deep learning model A is , establish the relationship loss function between the output layer of deep learning model B and deep learning model A ; According to the above operations, the relationship loss function between the output layers of deep learning model C and deep learning model B is established. The relationship loss function between the output layer of deep learning model C and deep learning model A .

6. The method for detecting damage of bridge railings and determining the timing of repair based on drones according to claim 5 is characterized in that: The S5 specifically includes the following steps: S51. Establish an inter-layer loss function from the enhanced model to the basic model, and combine the fused loss function to obtain the total loss function; S52. Based on the chain derivation rule, the three models are trained by error back propagation to obtain the trained deep learning model A, i.e., the optimal model; In the S51, the selected layers of the deep learning model A are connected to the selected layers of the deep learning model C, the selected layers in the deep learning model A include the second layer: convolution layer 1, the third layer: activation layer 1, the fourth layer: pooling layer 1 and the seventeenth layer: output layer, and the selected layers of the deep learning model C include the fourth layer: convolution layer 2, the sixth layer: pooling layer 1, the tenth layer: activation layer 4 and the thirty-eighth layer: output layer; The selected layers of the deep learning model A are , the corresponding size is ,in, and 17, is the selected number of layers of the deep learning model A, is the selected layer width of the deep learning model A, is the selected layer length of the deep learning model A, is the number of channels of the selected layer of deep learning model A, and the selected layer of deep learning model C is , the corresponding size is ,in, and 38, is the selected number of layers of the deep learning model C, is the selected layer width of the deep learning model C, is the selected layer length of the deep learning model C, is the number of selected layer channels of the deep learning model C; when When , a convolution operation is added to the selected layer output of the deep learning model A, with a convolution kernel size of 1×1 and a number of convolution kernels. , get the selected layer output of deep learning model A , and the corresponding output value of each channel is , =1, 2, 3, ..., , Output channel number for the selected layer of deep learning model A and the selected layer of deep learning model C The corresponding output values ​​of each channel are , =1, 2, 3, ..., , Output channel number for the selected layer of deep learning model C, and establish the loss function between the selected layer of deep learning model A and the selected layer of deep learning model C ; when When , a convolution operation is added to the selected layer output of the deep learning model C, with a convolution kernel size of 1×1 and a number of convolution kernels. , the selected layer output of the deep learning model C is , and the corresponding output value of each channel is , the selected layers of the deep learning model C The corresponding output values ​​of each channel are , establish the loss function between the selected layer of deep learning model A and the selected layer of deep learning model C ; According to the above calculation, the loss function between the second layer of deep learning model A and the fourth layer of deep learning model C is obtained , the loss function between the third layer of deep learning model A and the sixth layer of deep learning model C And the loss function between the fourth layer of deep learning model A and the tenth layer of deep learning model C , combining the relationship loss function between the output layers of deep learning model B and deep learning model A , the relationship loss function between the output layers of deep learning model C and deep learning model B The relationship loss function between the output layer of deep learning model C and deep learning model A , define the total loss function .

7. The method for detecting damage of bridge railings and determining the repair timing based on drones according to claim 6 is characterized in that: In S6, the final railing damage identification result is reviewed on site, and the number of damaged areas at different times is counted. and the area of ​​damaged area , establish the number of damaged areas and the area of ​​damaged area With bridge railing operation time The law of change; Number of damaged areas Over time The changing law of is expressed as: ； in, is the number of damaged areas of the railing at the initial moment, is the square term of time in the number of damaged areas The coefficient of The time in the number of damaged areas The coefficient of is a constant in the number of damaged areas; Damaged area Over time The changing law of is expressed as: ； in, is the area of ​​the damaged railing at the initial moment, is the square term of time in the damaged area The coefficient of The time in the damaged area The coefficient of is a constant in the area of ​​the damaged region; The relationship between the repair cost and the number and area of ​​damage before and after the railing repair is constructed to obtain the cost before the railing repair and the cost of railing repair ; Before the railings are repaired, the relationship between the repair cost and the number and area of ​​damage is expressed as: ； in, is the coefficient of the number of damaged areas before the railing is repaired, The coefficient of the damaged area of ​​the railing before repair; After the railings are repaired, the relationship between the repair cost and the number and area of ​​damage is expressed as: ； in, is the coefficient of the number of damaged areas after the railing is repaired, is the coefficient of the damaged area after the railing is repaired, Time for railing repairs; According to the total time , further establish the total cost of restoration Running time with bridge railings relationship; Total cost of repair Running time with bridge railings The relationship is expressed as: ； The method of solving the minimum value of the quadratic function by the matching method is used to solve the minimum value of the total repair cost. , the corresponding railing operation time is the best time to repair .

Images