Stay cable safety detection, early warning and broken wire identification method based on unmanned aerial vehicle
By planning the drone's flight path and image pixel update, combined with multi-model loss function training, the problems of cable-stayed bridge cable-stayed cable wire break recognition and drone safety detection are solved, and efficient and safe wire break recognition and early warning are achieved. , reduce maintenance costs and security risks.
Patent Information
- Application Number
- CN202411594982.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-11-11
- Publication Date
- 2024-12-06
- Estimated Expiration
- Not applicable · inactive patent
AI Technical Summary
The existing technology has failed to effectively combine the rapid identification of cable-stayed bridge cable-stayed cable breaks and safety issues in drone flight detection, resulting in delays in bridge maintenance, increased safety risks and high maintenance costs.
By planning the shortest flight path of the drone, collecting location, temperature, wind and humidity information for early warning, combining image pixel update and multi-model loss function training, the identification of cable-stayed cable wire breaks and drone safety detection is realized.
It improves the accuracy and safety of cable-stayed cable wire break detection, reduces energy consumption, ensures the safety of the detection process and the accuracy of the results, and enhances the recognition and generalization capabilities of the model.
Smart Images

Figure CN119091338A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to a safety detection, early warning and broken wire identification method, and in particular to a safety detection, early warning and broken wire identification method for a stay cable based on an unmanned aerial vehicle (UAV), belonging to the technical field of stay cable detection and UAV early warning. Background Art
[0002] During the service of the cables of cable-stayed bridges, major defects such as cable wire breakage may occur. The main causes of the defects include long-term exposure to alternating stress and fatigue, corrosion (such as rust), environmental factors (such as temperature difference, humidity, and chemical erosion), and construction or design defects. The above factors may cause the steel wires in the cables to break over time, that is, wire breakage. Cable wire breakage will have a serious impact on the safety and stability of the bridge. Wire breakage reduces the bearing capacity of the cables, which may cause uneven deformation of the bridge and even cause structural failure in extreme cases. Cable wire breakage will also accelerate the fatigue process of steel wires in other cables, increasing the risk of the overall structure. Rapid and intelligent identification of cable wire breakage can timely discover potential structural problems, avoid safety accidents, ensure the normal operation of the bridge and the safety of users, and at the same time help reduce maintenance costs and extend the service life of the bridge.
[0003] On the other hand, when drones are inspecting cable-stayed cables, even if the flight path is set, safety issues during the drone flight still need to be considered. This is due to the complexity and uncertainty of the flight environment. Changes in wind and climate, GPS signal interference, complex bridge structures, technical failures of drones, and operational errors may all cause the drone to deviate from the predetermined path. Research on drone flight safety warnings can improve flight accuracy and safety, reduce the accident rate, reduce damage and maintenance costs caused by collisions, and avoid greater economic losses caused by accidents.
[0004] Prior art 1, a patent document with publication (announcement) number CN109682460A discloses a method for identifying broken wires in cables based on vibration signals. The method collects lateral vibration signals by installing acceleration sensors in cables or hangers, and uses the pulse vibration amplitude of the cable body generated when the wire is broken, combined with signal energy or power as characteristic indicators, and then calculates the energy ratio as a judgment parameter for broken wires to distinguish the difference between broken wire signals and conventional vibrations. The above method helps to achieve long-term monitoring of cable damage; Prior art 2, a patent document with publication (announcement) number CN114595733A discloses a method and system for identifying broken wire signals of bridge cables based on long short-term memory networks. The method comprehensively utilizes time domain, frequency domain and time-frequency analysis to extract signal features, and constructs a comprehensive feature vector to characterize the acoustic emission signal, and then uses an LSTM network to establish a high-performance broken wire signal recognition model. Compared with traditional machine learning algorithms, this model can more accurately distinguish between broken wire signals and non-broken wire signals, showing better recognition effects; however, the above method does not take into account the ability to quickly identify broken wires in the construction of cable-stayed bridges. If the broken wires cannot be identified in time, the best time for bridge maintenance will be delayed, increasing the risk of structural failure, affecting the safe operation of the bridge, and leading to higher maintenance costs and potential safety accidents. At the same time, safety issues during drone flight inspections have not yet been considered.
[0005] Therefore, a UAV-based stay cable safety detection, early warning and broken wire identification method is needed. Summary of the invention
[0006] A brief overview of the present invention is provided below in order to provide a basic understanding of certain aspects of the present invention. It should be understood that this overview is not an exhaustive overview of the present invention. It is not intended to identify key or important parts of the present invention, nor is it intended to limit the scope of the present invention. Its purpose is merely to present certain concepts in a simplified form as a prelude to a more detailed description discussed later.
[0007] In view of this, in order to solve the problem that the traditional cable safety detection method of the cable-stayed bridge in the prior art does not take drone detection and early warning into consideration, the present invention provides a cable safety detection, early warning and broken wire identification method based on drones.
[0008] The technical solution is as follows: A method for safety detection, early warning and broken wire identification of a stay cable based on a drone includes the following steps: S1. Plan the shortest flight path of the drone based on the drone parameter information and operation information; S2. Collect the location information, temperature information, wind information and humidity information of the drone during flight along the shortest flight path of the drone, set corresponding thresholds, conduct comprehensive warning of drone detection and grade the warning degree; S3. Collect the cable image along the shortest flight path of the UAV, divide it into equal parts and update the image pixels to obtain an image with updated pixels; S4. construct a data set based on the pixel updated image combined with the disease information, input the data set into the established basic model, auxiliary model and enhanced model for training, fuse the loss functions of the three models, and obtain the fused loss function; S5. Establish the inter-layer loss function from the enhanced model to the basic model, combine the fused loss function to obtain the total loss function, train the three models, use the trained basic model as the final model, and output the final broken wire recognition result.
[0009] Furthermore, the S1 specifically includes the following steps: S11. Determine the take-off and landing points of the drone and select the observation position of the operator; S12. Determine the flight altitude AVh according to the focal length and sensor size of the camera carried by the drone, obtain the image coverage and resolution, and set the drone flight speed to AVv; S13. Develop the shortest flight path based on the take-off and landing points of the UAV, and determine the coordinate information of the cable and the location information of the UAV shooting points; In S13, the take-off point coordinates are AVo (AVoX, AVoY, AVoZ). On this basis, all drone shooting points are determined. The number of any drone shooting point is p, and its corresponding drone shooting point coordinates are AVp (AVpX, AVpY, AVpZ). The number set of drone shooting points is expressed as {1, 2, 3, ..., XN}, and the corresponding drone shooting point coordinate set is expressed as {AV1 (AV1X, AV1Y, AV1Z), AV2 (AV2X, AV2Y, AV2Z), ..., AVXN (AVXNX, AVXNY, AVXNZ)}, where XN is the number of drone shooting points. The adjacent distance between any drone shooting point AVp and its adjacent drone shooting point AV (p+1) is calculated. ; Taking the take-off point AVo as the starting point, according to the distance between any drone shooting point and the take-off point , calculate the distance between all drone shooting points and the take-off point, select the three drone shooting points corresponding to the minimum distance, and randomly select them to obtain the first drone shooting point that the drone arrives at after departing from the take-off point; After determining the first shooting point that the drone arrives at, calculate the distance between all shooting points and the first shooting point that is arrived at, screen out the three drone shooting points corresponding to the smallest distance, and randomly select them to obtain the second shooting point that the drone arrives at after departing from the first shooting point it arrives at. Repeat the above steps until all the shooting points that the drone arrives at are obtained. According to the different random selections each time, calculate the total distance TDis1 of the drone's flight under the first flight mode. Repeat the above steps, the number of repetitions is CN, and integrate to obtain the total distance set of the drone's flight under different flight modes {TDis1, TDis2, TDis3, TDis4, …, TDisCN}, arrange them in order from small to large, and select the total distances and corresponding flight routes ranked in the top TN, TN is the preset value, count the intersection information of TN flight routes, and select the flight route with the largest number of intersections to obtain the shortest flight path of the drone.
[0010] Furthermore, the step S2 specifically includes the following steps: S21. Select the key warning points of the cable-stayed bridge, collect the position information of the drone during flight, and design the drone collision warning; S22. Set a safe temperature threshold, collect temperature information during the flight of the drone, and design a temperature warning for the drone; S23. Set a safe wind threshold, collect wind information during the flight of the drone, and design a wind warning for the drone; S24. Set a safe humidity threshold, collect humidity information during the flight of the drone, and design a humidity warning for the drone; S25. Considering drone collision warning, drone temperature warning, drone wind warning and drone humidity warning, conduct comprehensive drone detection warning and grade the warning degree; In S21, the key warning points of the cable-stayed bridge are GJD1, GJD2, …, GJDgn respectively, where gn is the number of key warning points. Through the on-board positioning system of the drone and in combination with the shortest flight path of the drone, the position information WRD during the flight of the drone is located in real time. A safety distance threshold SJLtr is set, and the distance SJLim between the position information during the flight of the drone and all the key warning points of the cable-stayed bridge is calculated. SJLim = |GJDim - WRD|, where GJDim is the key warning point of the cable-stayed bridge corresponding to the position information during the current flight of the drone, and im is the number of the key warning point, im = 1, 2, …, gn. According to the distance SJLim, a distance matrix SJL is constructed, SJL = [SJL1, SJL2, SJL3, …, SJLgn]. The minimum element value SJLmin in the distance matrix is calculated, SJLmin = min[SJL1, SJL2, SJL3, …, SJLgn], where min is the minimum value. The relationship between the minimum element value SJLmin in the distance matrix and the safety distance threshold SJLtr is compared. When SJLmin ≥ SJLtr, it is considered that the position of the drone is safe and no warning is required. When SJLmin < SJLtr, it is considered that the drone has a risk of colliding with the bridge and a warning is required; In S22, the safety temperature threshold is set to WDtr. Through the on-board temperature sensor of the drone, the temperature information WWD during the flight of the drone is collected in real time. The relationship between it and the safety temperature threshold WDtr is compared. When WWD ≥ WDtr, it is considered that the current temperature is suitable for the drone to detect and no warning is required. When WWD < WDtr, it is considered that the current temperature is not suitable for the drone to detect and a warning is required; In S23, the safety wind force threshold is set to FDtr. Through the on-board wind speed sensor of the drone, the wind force information WFL during the flight of the drone is collected in real time. The relationship between it and the safety wind force threshold FDtr is compared. When WFL ≥ FDtr, it is considered that the current wind force is suitable for the drone to detect and no warning is required. When WFL < FDtr, it is considered that the wind force is not suitable for the drone to detect and a warning is required; In S24, the safety humidity threshold is set to SDtr. Through the on-board humidity sensor of the drone, the humidity information WSD during the flight of the drone is collected in real time. The relationship between it and the safety humidity threshold SDtr is compared. When WSD ≥ SDtr, it is considered that the current humidity is suitable for the drone to detect and no warning is required. When WSD < SDtr, it is considered that the current humidity is not suitable for the drone to detect and a warning is required; In S25, the grade value YJL of the warning level classification is expressed as: YJL = YJpz + YJwd + YJfl + YJsd Among them, YJpz is the warning value corresponding to the UAV collision warning. When SJLmin ≥ SJLtr, YJpz = 0; when SJLmin < SJLtr, YJpz = 1. YJwd is the warning value corresponding to the temperature warning. When WWD ≥ WDtr, YJwd = 0; when WWD < WDtr, YJwd = 1. YJfl is the warning value corresponding to the wind force warning. When WFL ≥ FDtr, YJfl = 0; when WFL < FDtr, YJfl = 1. YJsd is the warning value corresponding to the humidity warning. When WSD ≥ SDtr, YJsd = 0; when WSD < SDtr, YJsd = 1. When YJL > 0, it is necessary to give a warning to the UAV.
[0011] Further, in step S3, it specifically includes the following steps: S31. Use an on-board high-definition camera of the UAV to collect the image of the stay cables of the cable-stayed bridge along the determined shortest flight path. S32. Divide the pixel gray values of the stay cable image into 16 gray levels equally, and divide the stay cable image into n equal parts to obtain the divided image. S33. Statistically calculate the pixel frequency corresponding to each gray level in the divided image , and calculate the cumulative value of the gray frequency corresponding to each gray level in the divided image ; S34. According to the cumulative value of the gray frequency , construct a pixel value update method, calculate the pixel update value corresponding to the gray level , and obtain the image after pixel update. In step S32, the pixel gray value is 0 - 255, and the gray level is , = 1, 2, 3, …, 16, is the gray number, and the divided image is Pnum, where num = 1, 2, …, n; In step S33, the cumulative value of the gray frequency corresponding to each gray level in the divided image Pnum is expressed as: ; Among them, is the gray level number less than or equal to , represents the gray level with the gray level number ; In step S34, the pixel update value corresponding to the gray level is expressed as: ; Among them, is the minimum non-zero value among all pixel frequency accumulation values, is the total number of pixels corresponding to the equally divided image Pnum, is the maximum value of the gray level.
[0012] Furthermore, the step S4 specifically includes the following steps: S41. Based on the pixel updated image, select all images containing broken wires, wear, outer layer damage and corrosion of the inclined cable as input images; S42. Through manual annotation, the cable broken wire area and other three types of defects are selected in the input image, and a data set is constructed. The data set is divided into a training set and a test set according to a ratio of 8:2; S43. According to the data set, a basic model, an auxiliary model and an enhanced model are constructed, and the loss functions of the three models are calculated and integrated to obtain a fused loss function; In S42, the broken wire area is marked as Wire, and the three types of diseases, namely wear, outer layer damage and rust, are marked as other1, other2 and other3 respectively; In the S43, a network structure of a basic model, i.e., a deep learning model A, is constructed, and the deep learning model A consists of 1 input layer, 9 convolutional layers, 9 activation layers, 3 pooling layers, 1 merging layer, 1 upsampling layer, 1 PANet layer, 1 pooling splicing layer, 1 SPP layer, and 1 output layer; Input the data set into the deep learning model A, calculate the weights and biases of the neural network units of the deep learning model A through the error back propagation algorithm, complete the training of the deep learning model A, and obtain the trained deep learning model A and its loss function ; The output value of the output unit node corresponding to the deep learning model A is , respectively, are the output values of the first output unit node , the output value of the second output unit node , the output value of the third output unit node and the output value of the fourth output unit node , according to the exponential function First, the output values of the four output unit nodes corresponding to the deep learning model A are initially converted into the probability values of the first preliminary output value and the first preliminary output value. , and then further optimize and transform it to obtain the first optimized output value and the enhanced probability value of the first optimized output value ; The enhanced probability value using the first optimized output value And the data set, combined with the error chain derivative rule, continue to train the deep learning model A. After the training is completed, the output value of the first disease is obtained The actual disease annotations corresponding to the nodes in the output layer of the deep learning model A , the output value of the first disease They are the output values of the first inclined cable broken wire respectively , the output value of the first wear , the first outer layer damage output value and the first rust output value ; Construct an auxiliary model, namely deep learning model B, which consists of 1 input layer, 9 convolutional layers, 9 activation layers, 5 pooling layers, 1 merging layer, 1 upsampling layer, 1 PANet layer, 1 pooling splicing layer, 1 SPP layer and 1 output layer; Input the data set into the deep learning model B, calculate the weights and biases of the neural network units of the deep learning model B through the error back propagation algorithm, complete the training of the deep learning model B, and obtain the trained deep learning model B and its loss function ; The output value of the output unit node corresponding to the deep learning model B is , respectively, are the output values of the fifth output unit node , the output value of the sixth output unit node , the output value of the seventh output unit node and the output value of the eighth output unit node , according to the exponential function First, the output values of the four output unit nodes corresponding to the deep learning model B are initially converted into the second preliminary output value and the probability value of the second preliminary output value , and then further optimize and transform it to obtain the second optimized output value and the enhanced probability value of the second optimized output value ; The enhanced probability value using the second optimized output value And the data set, combined with the error chain derivative rule, continue to train the deep learning model B. After the training is completed, the output value of the second disease is obtained The actual disease annotations corresponding to the nodes in the output layer of the deep learning model B , the output value of the second disease They are the output values of the second inclined cable broken wire respectively , the output value of the second wear , the second outer layer damage output value and the second rust output value ; Construct a reinforcement model, namely deep learning model C. The deep learning model C consists of 1 input layer, 12 convolution layers, 12 activation layers, 5 pooling layers, 1 merging layer, 1 upsampling layer, 1 PANet layer, 1 pooling splicing layer, 1 SPP layer and 1 output layer; Input the data set into the deep learning model C, calculate the weights and biases of the neural network units of the deep learning model C through the error back propagation algorithm, complete the training of the deep learning model C, and obtain the trained deep learning model C and its loss function ; The output value of the output unit node corresponding to the deep learning model C is , respectively, are the output values of the ninth output unit node , the output value of the tenth output unit node , the output value of the eleventh output unit node and the output value of the twelfth output unit node , according to the exponential function First, the output values of the four output unit nodes corresponding to the deep learning model C are initially converted into the third preliminary output value and the probability value of the third preliminary output value , and then further optimize and transform it to obtain the third optimized output value and the enhanced probability value of the third optimized output value ; The enhanced probability value using the third optimized output value And the data set, combined with the error chain derivative rule, continue to train the deep learning model C. After the training is completed, the output value of the third disease is obtained The actual disease annotations corresponding to the nodes in the output layer of the deep learning model B , the third disease output value They are the output values of the third inclined cable broken wire respectively. , Output value of the third wear , the third outer layer damage output value and the third rust output value ; According to the second disease output value is and the enhanced probability value of the first optimized output value , construct the loss function between deep learning model B and deep learning model A , get the corresponding weight coefficient; Probability value based on the first preliminary output value The actual disease annotations corresponding to the nodes in the output layer of the deep learning model A , construct the probability value considering the first preliminary output value The loss function of the deep learning model A is , get the corresponding weight coefficient; Based on the loss function between deep learning model B and deep learning model A and the probability value considering the first preliminary output value The loss function of the deep learning model A is , establish the relationship loss function between the output layer of deep learning model B and deep learning model A ; According to the above operations, the relationship loss function between the output layers of deep learning model C and deep learning model B is established. The relationship loss function between the output layer of deep learning model C and deep learning model A .
[0013] Furthermore, the S5 specifically includes the following steps: S51. Establish an inter-layer loss function from the enhanced model to the basic model, and combine the fused loss function to obtain the total loss function; S52. Based on the chain derivation rule, the three models are trained by error back propagation to obtain the trained deep learning model A, i.e., the optimal model; In the S51, the selected layers of the deep learning model A are connected to the selected layers of the deep learning model C, the selected layers in the deep learning model A include the second layer: convolution layer 1, the third layer: activation layer 1, the fourth layer: convolution layer 2 and the twenty-eighth layer: output layer, and the selected layers of the deep learning model C include the fourth layer: convolution layer 2, the sixth layer: convolution layer 3, the tenth layer: convolution layer 5 and the thirty-sixth layer: output layer; The selected layers of the deep learning model A are , the corresponding size is ,in, =2, 3, 4 and 28, is the selected number of layers of the deep learning model A, is the selected layer width of the deep learning model A, is the selected layer length of the deep learning model A, is the number of channels of the selected layer of deep learning model A, and the selected layer of deep learning model C is , the corresponding size is ,in, =4, 6, 10 and 36, is the selected number of layers of the deep learning model C, is the selected layer width of the deep learning model C, is the selected layer length of the deep learning model C, is the number of selected layer channels of the deep learning model C; when When , a convolution operation is added to the selected layer output of the deep learning model A, with a convolution kernel size of 1×1 and a number of convolution kernels. , get the selected layer output of deep learning model A , and the corresponding output value of each channel is , =1, 2, 3, ..., , Output channel number for the selected layer of deep learning model A and the selected layer of deep learning model C The corresponding output values of each channel are , =1, 2, 3, …, , Output channel number for the selected layer of deep learning model C, and establish the loss function between the selected layer of deep learning model A and the selected layer of deep learning model C ; when When , a convolution operation is added to the selected layer output of the deep learning model C, with a convolution kernel size of 1×1 and a number of convolution kernels. , the selected layer output of the deep learning model C is , and the corresponding output value of each channel is , the selected layers of the deep learning model C The corresponding output values of each channel are , establish the loss function between the selected layer of deep learning model A and the selected layer of deep learning model C ; According to the above calculation, the loss function between the second layer of deep learning model A and the fourth layer of deep learning model C is obtained , the loss function between the third layer of deep learning model A and the sixth layer of deep learning model C And the loss function between the fourth layer of deep learning model A and the tenth layer of deep learning model C , combining the relationship loss function between the output layers of deep learning model B and deep learning model A , the relationship loss function between the output layers of deep learning model C and deep learning model B The relationship loss function between the output layer of deep learning model C and deep learning model A , define the total loss function ; Total loss function It is expressed as: ; in, is the loss function between the second layer of deep learning model A and the fourth layer of deep learning model C The corresponding weight coefficient is, is the loss function between the third layer of deep learning model A and the sixth layer of deep learning model C The corresponding weight coefficient is, is the loss function between the fourth layer of deep learning model A and the tenth layer of deep learning model C The corresponding weight coefficient.
[0014] Furthermore, in S43, the layers in the network structure of the deep learning model A are connected in the following order: Input layer: image size 227×227×3; Convolutional layer 1: kernel size 3×3, number of kernels 32, stride: 1, batch normalization; Activation layer 1: LeakyReLU; Convolutional layer 2: kernel size 3×3, number of kernels 64, stride: 1, batch normalization; Activation layer 2: LeakyReLU; Convolution layer 3: convolution kernel size 3×3, number of convolution kernels 128, stride: 1, batch normalization; Activation layer 3: LeakyReLU; Convolutional layer 4: kernel size 3×3, number of kernels 256, stride: 1, batch normalization; Activation layer 4: LeakyReLU; SPP layer: extracts features of different scales and fuses them to increase the receptive field of the model; Pooling layer 1: Type: global average pooling, pooling size: 5×5, stride: 1; Pooling layer 2: Type: global average pooling, pooling size: 9×9, Stride: 1; Pooling layer 3: Type: global average pooling, pooling size: 13×13, Stride: 1; Pooling concatenation layer: connect the outputs of the three pooling layers; PANet layer: aggregates features at different levels and implements feature pyramids through upsampling and downsampling operations; Convolutional layer 5: kernel size 3×3, number of kernels 768, stride: 2, batch normalization; Activation layer 5: LeakyReLU; Convolutional layer 6: kernel size 3×3, number of kernels 768, stride: 2, batch normalization; Activation layer 6: LeakyReLU; Upsampling layer: uses the nearest neighbor interpolation method to upsample to the same size as the pooling splicing layer; Merging layer: The output feature map of the downsampling layer is merged with the output feature map of the upsampling layer; Convolutional layer 7: kernel size 3×3, number of kernels 256, stride: 1, batch normalization; Activation layer 7: LeakyReLU; Convolutional layer 8: kernel size 3×3, number of kernels 128, stride: 1, batch normalization; Activation layer 8: LeakyReLU; Convolutional layer 9: kernel size 3×3, number of kernels 64, stride: 1, batch normalization; Activation layer 9: LeakyReLU; Output layer: Type: Softmax, Number of neurons: 4.
[0015] Furthermore, in S43, the layers in the network structure of the deep learning model B are connected in the following order: Input layer: image size 227×227×3; Convolutional layer 1: kernel size 3×3, number of kernels 32, stride: 1, batch normalization; Activation layer 1: LeakyReLU; Convolutional layer 2: kernel size 3×3, number of kernels 64, stride: 1, batch normalization; Activation layer 2: LeakyReLU; Convolution layer 3: convolution kernel size 3×3, number of convolution kernels 128, stride: 1, batch normalization; Activation layer 3: LeakyReLU; Convolutional layer 4: kernel size 3×3, number of kernels 256, stride: 1, batch normalization; Activation layer 4: LeakyReLU; SPP layer: extracts features of different scales and fuses them to increase the receptive field of the model; Pooling layer 1: Type: global average pooling, pooling size: 5×5, stride: 1; Pooling layer 2: Type: global average pooling, pooling size: 7×7, Stride: 1; Pooling layer 3: Type: global average pooling, pooling size: 9×9, Stride: 1; Pooling layer 4: Type: global average pooling, pooling size: 11×11, Stride: 1; Pooling layer 5: Type: global average pooling, pooling size: 13×13, stride: 1; Pooling concatenation layer: connect the outputs of the five pooling layers; PANet layer: aggregates features at different levels and implements feature pyramids through upsampling and downsampling operations; Convolution layer 5: kernel size 3×3, number of kernels 1280, stride: 2, batch normalization; Activation layer 5: LeakyReLU; Convolutional layer 6: kernel size 3×3, number of kernels 1280, stride: 2, batch normalization; Activation layer 6: LeakyReLU; Upsampling layer: uses the nearest neighbor interpolation method to upsample to the same size as the pooling splicing layer; Merging layer: The output feature map of the downsampling layer is merged with the output feature map of the upsampling layer; Convolutional layer 7: kernel size 3×3, number of kernels 512, stride: 1, batch normalization; Activation layer 7: LeakyReLU; Convolutional layer 7: kernel size 3×3, number of kernels 256, stride: 1, batch normalization; Activation layer 7: LeakyReLU; Convolutional layer 8: kernel size 3×3, number of kernels 128, stride: 1, batch normalization; Activation layer 8: LeakyReLU; Convolutional layer 9: kernel size 3×3, number of kernels 64, stride: 1, batch normalization; Activation layer 9: LeakyReLU; Output layer: Type: Softmax, Number of neurons: 4.
[0016] Furthermore, in S43, the layers in the network structure of the deep learning model C are connected in the following order: Input layer: image size 227×227×3; Convolutional layer 1: kernel size 3×3, number of kernels 32, stride: 1, batch normalization; Activation layer 1: LeakyReLU; Convolutional layer 2: kernel size 3×3, number of kernels 64, stride: 1, batch normalization; Activation layer 2: LeakyReLU; Convolution layer 3: convolution kernel size 3×3, number of convolution kernels 128, stride: 1, batch normalization; Activation layer 3: LeakyReLU; Convolutional layer 4: convolution kernel size 3×3, number of convolution kernels 128, stride: 1, batch normalization; Activation layer 4: LeakyReLU; Convolutional layer 5: convolution kernel size 3×3, number of convolution kernels 256, stride: 1, batch normalization; Activation layer 5: LeakyReLU; Convolutional layer 6: kernel size 3×3, number of kernels 256, stride: 1, batch normalization; Activation layer 6: LeakyReLU; SPP layer: extracts features of different scales and fuses them to increase the receptive field of the model; Pooling layer 1: Type: global average pooling, pooling size: 5×5, stride: 1; Pooling layer 2: Type: global average pooling, pooling size: 7×7, Stride: 1; Pooling layer 3: Type: global average pooling, pooling size: 9×9, Stride: 1; Pooling layer 4: Type: global average pooling, pooling size: 11×11, Stride: 1; Pooling layer 5: Type: global average pooling, pooling size: 13×13, stride: 1; Pooling concatenation layer: connect the outputs of the three pooling layers; PANet layer: aggregates features at different levels and implements feature pyramids through upsampling and downsampling operations; Convolutional layer 7: kernel size 3×3, number of kernels 1280, stride: 2, batch normalization; Activation layer 7: LeakyReLU; Convolutional layer 8: kernel size 3×3, number of kernels 1280, stride: 2, batch normalization; Activation layer 8: LeakyReLU; Upsampling layer: uses the nearest neighbor interpolation method to upsample to the same size as the pooling splicing layer; Merging layer: The output feature map of the downsampling layer is merged with the output feature map of the upsampling layer; Convolutional layer 9: kernel size 3×3, number of kernels 512, stride: 1, batch normalization; Activation layer 9: LeakyReLU; Convolution layer 10: kernel size 3×3, number of kernels 256, stride: 1, batch normalization; Activation layer 10: LeakyReLU; Convolutional layer 11: kernel size 3×3, number of kernels 128, stride: 1, batch normalization; Activation layer 11: LeakyReLU; Convolutional layer 12: kernel size 3×3, number of kernels 64, stride: 1, batch normalization; Activation layer 12: LeakyReLU; Output layer: Type: Softmax, Number of neurons: 4.
[0017] The beneficial effects of the present invention are as follows: the present invention constructs a method for determining the shortest flight path of an unmanned aerial vehicle, improves the subsequent detection effect of broken wires of inclined cables, reduces energy consumption, and ensures the safety of the detection process; the present invention designs an early warning method for safely detecting the inclined cables of a cable-stayed bridge by an unmanned aerial vehicle based on the position information, temperature information, wind information and humidity information during the flight of the unmanned aerial vehicle, which can issue an early warning when the unmanned aerial vehicle is in an unsafe state to ensure the safety of the detection process; the present invention updates the image pixels of the collected inclined cable images, enhances the visibility of the broken wire areas in the images, and supports the accuracy of subsequent output results; the present invention constructs the output layer loss functions of the basic model, the auxiliary model and the enhanced model, so that the basic model can learn complex data faster during the training process; the present invention jointly optimizes the three models based on the multi-model loss function to obtain the optimal model, realizes the identification of broken wires in the inclined cables of cable-stayed bridges, and at the same time, further strengthens the connection between the three models, ensures that the neural networks of multiple levels of the basic model can receive enough feature information, and improves the recognition ability and generalization ability of the model. BRIEF DESCRIPTION OF THE DRAWINGS
[0018] The drawings described herein are used to provide a further understanding of the present invention and constitute a part of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings: Figure 1 The figure is a flow chart of the UAV-based stay cable safety detection, early warning and broken wire identification method. DETAILED DESCRIPTION
[0019] In order to make the technical solutions and advantages of the embodiments of the present invention more clearly understood, the exemplary embodiments of the present invention are further described in detail below in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than an exhaustive list of all the embodiments. It should be noted that the embodiments of the present invention and the features in the embodiments can be combined with each other without conflict.
[0020] refer to Figure 1 The present embodiment is described in detail. The method for safety detection, early warning and broken wire identification of the inclined cable based on the drone specifically includes the following steps: S1. Plan the shortest flight path of the drone based on the drone parameter information and operation information; S2. Collect the location information, temperature information, wind information and humidity information of the drone during flight along the shortest flight path of the drone, set corresponding thresholds, conduct comprehensive warning of drone detection and grade the warning degree; S3. Collect the cable image along the shortest flight path of the UAV, divide it into equal parts and update the image pixels to obtain an image with updated pixels; S4. construct a data set based on the pixel updated image combined with the disease information, input the data set into the established basic model, auxiliary model and enhanced model for training, fuse the loss functions of the three models, and obtain the fused loss function; S5. Establish the inter-layer loss function from the enhanced model to the basic model, combine the fused loss function to obtain the total loss function, train the three models, use the trained basic model as the final model, and output the final broken wire recognition result.
[0021] Furthermore, the S1 specifically includes the following steps: S11. Determine the take-off and landing points of the drone and select the observation position of the operator; Specifically, before the drone takes off, it is necessary to choose a location where the drone can take off and land safely, away from people and traffic, and avoid high-voltage wires and large metal structures that may interfere with the drone's navigation system. At the same time, choose an observation position with a good field of view so that the operator can maintain a good line of sight and facilitate operation; S12. Determine the appropriate flight altitude AVh according to the focal length and sensor size of the camera carried by the drone, obtain sufficient image coverage and resolution, and set the drone flight speed to AVv; S13. Develop a flight path based on the take-off and landing points of the UAV. Based on the drawing parameters of the cable-stayed bridge and the field of view and clarity of the camera carried by the UAV, determine the coordinate information of the cable-stayed cables and the location information of the UAV shooting points. Ensure that the images of all location information can fully and clearly cover the cable-stayed cables, and design the shortest flight path for the UAV. In S13, the take-off point coordinates are AVo (AVoX, AVoY, AVoZ). On this basis, all drone shooting points are determined. The number of any drone shooting point is p, and its corresponding drone shooting point coordinates are AVp (AVpX, AVpY, AVpZ). The number set of drone shooting points is expressed as {1, 2, 3, ..., XN}, and the corresponding drone shooting point coordinate set is expressed as {AV1 (AV1X, AV1Y, AV1Z), AV2 (AV2X, AV2Y, AV2Z), ..., AVXN (AVXNX, AVXNY, AVXNZ)}, where XN is the number of drone shooting points. The adjacent distance between any drone shooting point AVp and its adjacent drone shooting point AV (p+1) is calculated. ; Adjacent distance It is expressed as: ; Among them, (AVpX, AVpY, AVpZ) are the coordinates of any drone shooting point, and (AV(p+1)X, AV(p+1)Y, AV(p+1)Z) are the coordinates of the drone shooting point adjacent to any drone shooting point; Taking the take-off point AVo as the starting point, according to the distance between any drone shooting point and the take-off point , calculate the distance between all drone shooting points and the take-off point, select the three drone shooting points corresponding to the minimum distance, and randomly select them to obtain the first drone shooting point that the drone arrives at after departing from the take-off point; distance It is expressed as: ; After determining the first shooting point that the drone arrives at, calculate the distance between all shooting points and the first shooting point that is arrived at, screen out the three drone shooting points corresponding to the smallest distance, and randomly select them to obtain the second shooting point that the drone arrives at after departing from the first shooting point that it arrives at. It should be noted that the drone's route will not return, that is, the drone will not fly to the same shooting point twice. Repeat the above steps until all the shooting points arrived at by the drone are obtained. According to the difference in each random selection, calculate the total flight distance TDis1 of the drone under the first flight mode. Repeat the above steps, the number of repetitions is CN, and integrate to obtain the total flight distance set of the drone under different flight modes {TDis1, TDis2, TDis3, TDis4, ..., TDisCN}, arrange them in order from small to large, and select the total distance and corresponding flight route ranked in the top TN, TN is a preset value, count the intersection information of TN flight routes, and select the flight route with the largest number of intersections to obtain the shortest flight path of the drone.
[0022] Furthermore, the step S2 specifically includes the following steps: S21. Select the key warning points of the cable-stayed bridge, collect the position information of the drone during flight, and design the drone collision warning; S22. Set a safe temperature threshold, collect temperature information during the flight of the drone, and design a temperature warning for the drone; S23. Set a safe wind threshold, collect wind information during the flight of the drone, and design a wind warning for the drone; S24. Set a safe humidity threshold, collect humidity information during the flight of the drone, and design a humidity warning for the drone; S25. Considering drone collision warning, drone temperature warning, drone wind warning and drone humidity warning, conduct comprehensive drone detection warning and grade the warning degree; In S21, the key warning points of the cable-stayed bridge are GJD1, GJD2, …, GJDgn respectively, where gn is the number of key warning points. Through the UAV on-board positioning system, combined with the shortest flight path of the UAV, the position information WRD during the UAV flight is located in real time. The safety distance threshold SJLtr is set, and the distance SJLim between the position information during the UAV flight and all the key warning points of the cable-stayed bridge is calculated. SJLim = |GJDim - WRD|, where GJDim is the key warning point of the cable-stayed bridge corresponding to the current position information during the UAV flight, and im is the key warning point number, im = 1, 2, …, gn. According to the distance SJLim, the distance matrix SJL is constructed. SJL = [SJL1, SJL2, SJL3, …, SJLgn]. The minimum element value SJLmin in the distance matrix is calculated. SJLmin = min[SJL1, SJL2, SJL3, …, SJLgn], where min is the minimum value. The relationship between the minimum element value SJLmin in the distance matrix and the safety distance threshold SJLtr is compared. When SJLmin ≥ SJLtr, it is considered that the UAV position is safe and no warning is required. When SJLmin < SJLtr, it is considered that the UAV has a risk of colliding with the bridge and a warning is required; In S22, the safety temperature threshold is set to WDtr. Through the UAV on-board temperature sensor, the temperature information WWD during the UAV flight is collected in real time. The relationship between it and the safety temperature threshold WDtr is compared. When WWD ≥ WDtr, it is considered that the current temperature is suitable for UAV detection and no warning is required. When WWD < WDtr, it is considered that the current temperature is not suitable for UAV detection and a warning is required; In S23, the safety wind force threshold is set to FDtr. Through the UAV on-board wind speed sensor, the wind force information WFL during the UAV flight is collected in real time. The relationship between it and the safety wind force threshold FDtr is compared. When WFL ≥ FDtr, it is considered that the current wind force is suitable for UAV detection and no warning is required. When WFL < FDtr, it is considered that the wind force is not suitable for UAV detection and a warning is required; In S24, the safety humidity threshold is set to SDtr. Through the UAV on-board humidity sensor, the humidity information WSD during the UAV flight is collected in real time. The relationship between it and the safety humidity threshold SDtr is compared. When WSD ≥ SDtr, it is considered that the current humidity is suitable for UAV detection and no warning is required. When WSD < SDtr, it is considered that the current humidity is not suitable for UAV detection and a warning is required; In S25, the grade value YJL of the warning level classification is expressed as: YJL = YJpz + YJwd + YJfl + YJsd Among them, YJpz is the warning value corresponding to the UAV collision warning. When SJLmin ≥ SJLtr, YJpz = 0; when SJLmin < SJLtr, YJpz = 1. YJwd is the warning value corresponding to the temperature warning. When WWD ≥ WDtr, YJwd = 0; when WWD < WDtr, YJwd = 1. YJfl is the warning value corresponding to the wind warning. When WFL ≥ FDtr, YJfl = 0; when WFL < FDtr, YJfl = 1. YJsd is the warning value corresponding to the humidity warning. When WSD ≥ SDtr, YJsd = 0; when WSD < SDtr, YJsd = 1; When YJL > 0, it is necessary to give a warning to the UAV, and the warning level value is the value of YJL; Specifically, when performing the cable-stayed bridge inspection, the UAV needs to fly in a complex bridge structure, which includes crossbeams, columns, stay cables and other details. The structural complexity increases the risk of collision. Collision not only causes damage to the UAV, but may also damage the bridge inspection equipment and affect the accuracy of data. Temperature conditions will also affect the UAV inspection. Extreme high or low temperatures will affect the battery performance of the UAV and the stability of electronic components, thus shortening the flight time or causing system failures. Wind is another important consideration for UAV flight inspection, especially in open areas such as bridges. Strong winds will interfere with the flight stability and heading control of the UAV, making it difficult for the UAV to stay on the predetermined inspection path. Humidity will affect the UAV operation. High humidity will cause the electronic components of the UAV to get damp, increasing the risk of short circuits. In addition, moisture will form condensation on the camera or other sensors, thus reducing the image quality and the effectiveness of the sensors. Therefore, the present invention conducts UAV inspection and warning from the perspectives of collision warning, temperature warning, wind warning, humidity warning, etc.
[0023] Further, in S2, it specifically includes the following steps: S21. Use an on-board high-definition camera of the UAV to collect the stay cable images of the cable-stayed bridge along the determined shortest flight path; S22. Divide the pixel gray values of the stay cable images into 16 gray levels equally, and divide the stay cable images into n equal parts to obtain the divided images; S23. Count the pixel frequencies corresponding to each gray level in the divided images and calculate the cumulative gray frequency values corresponding to each gray level in the divided images ; S24. According to the cumulative gray frequency values construct a pixel value update method, calculate the pixel update values corresponding to the gray levels and obtain the images after pixel update; In S22, the pixel gray value is 0-255, and the gray level is , =1, 2, 3, …, 16, is the grayscale number, and the image after equal division is Pnum, num=1, 2, ..., n; In S23, each gray level in the equally divided image Pnum The corresponding grayscale frequency cumulative value It is expressed as: ; in, is less than or equal to The gray level number, Indicates the gray level number is Gray level; In S24, the pixel update value corresponding to the gray level It is expressed as: ; in, is the minimum non-zero value among all pixel frequency accumulation values, is the total number of pixels corresponding to the equally divided image Pnum, is the maximum value of gray level; Specifically, in the captured images of the cables of the cable-stayed bridge, due to the influence of environmental factors and acquisition equipment, the images may have low contrast, uneven lighting, and changes in color and brightness, making it difficult to accurately distinguish the broken wire area from the background. Therefore, the image of the broken wire of the cable-stayed cable is updated with pixels.
[0024] Furthermore, the step S4 specifically includes the following steps: S41. Based on the pixel updated image, select all images containing broken wires, wear, outer layer damage and corrosion of the inclined cable as input images; S42. Through manual annotation, the cable broken wire area and other three types of defects are selected in the input image, and a data set is constructed. The data set is divided into a training set and a test set according to a ratio of 8:2; S43. According to the data set, a basic model, an auxiliary model and an enhanced model are constructed, and the loss functions of the three models are calculated and integrated to obtain a fused loss function; In S42, the broken wire area is marked as Wire, and the three types of diseases, namely wear, outer layer damage and rust, are marked as other1, other2 and other3 respectively; In the S43, a network structure of a basic model, i.e., a deep learning model A, is constructed, and the deep learning model A consists of 1 input layer, 9 convolutional layers, 9 activation layers, 3 pooling layers, 1 merging layer, 1 upsampling layer, 1 PANet layer, 1 pooling splicing layer, 1 SPP layer, and 1 output layer; The data set is input into the deep learning model A, and the weights and biases of the neural network units of the deep learning model A are calculated through the error back propagation algorithm, and the training of the deep learning model A is completed to obtain the trained deep learning model A; Loss function of the trained deep learning model A It is expressed as: ; in, is the total number of data samples in the dataset of deep learning model A, For the The true labels of samples, For deep learning model A, predict The probability that a sample is of category 1, is the natural logarithm; The output value of the output unit node corresponding to the deep learning model A is , respectively, are the output values of the first output unit node , the output value of the second output unit node , the output value of the third output unit node and the output value of the fourth output unit node In order to further clarify the differences between the output values, the model output is converted into a probability distribution to improve the model's ability to distinguish. According to the exponential function , firstly, the output values of the four output unit nodes corresponding to the deep learning model A are initially converted into the first preliminary output value and its probability value, and then further optimized and converted to obtain the first optimized output value and its enhanced probability value in order to amplify the probability difference between the output values and enhance the distinguishing ability of the model; Probability value of the first preliminary output value It is expressed as: ; The enhanced probability value of the first optimized output value It is expressed as: ; in, G is the enhancement coefficient; The enhanced probability value using the first optimized output value And the data set, combined with the error chain derivative rule, continue to train the deep learning model A. After the training is completed, the output value of the first disease is obtained The actual disease annotations corresponding to the nodes in the output layer of the deep learning model A , the output value of the first disease They are the output values of the first inclined cable broken wire respectively , the output value of the first wear , the first outer layer damage output value and the first rust output value ; Construct an auxiliary model, namely deep learning model B, which consists of 1 input layer, 9 convolutional layers, 9 activation layers, 5 pooling layers, 1 merging layer, 1 upsampling layer, 1 PANet layer, 1 pooling splicing layer, 1 SPP layer and 1 output layer; The data set is input into the deep learning model B, and the weights and biases of the neural network units of the deep learning model B are calculated through the error back propagation algorithm, and the training of the deep learning model B is completed to obtain the trained deep learning model B; Loss function of the trained deep learning model B It is expressed as: ; in, is the total number of data samples in the dataset of deep learning model B, For the The true labels of samples, For deep learning model B to predict The probability that a sample is class 1; The output value of the output unit node corresponding to the deep learning model B is , respectively, are the output values of the fifth output unit node , the output value of the sixth output unit node , the output value of the seventh output unit node and the output value of the eighth output unit node In order to further clarify the differences between the output values, the model output is converted into a probability distribution to improve the model's ability to distinguish. According to the exponential function , firstly, the output values of the four output unit nodes corresponding to the deep learning model B are initially converted into the second preliminary output values and their probability values, and then further optimized and converted to obtain the second optimized output values and their enhanced probability values in order to amplify the probability difference between the output values and enhance the distinguishing ability of the model; Probability value of the second preliminary output value It is expressed as: ; The enhanced probability value of the second optimized output value It is expressed as: ; The enhanced probability value using the second optimized output value And the data set, combined with the error chain derivative rule, continue to train the deep learning model B. After the training is completed, the output value of the second disease is obtained The actual disease annotations corresponding to the nodes in the output layer of the deep learning model B , the output value of the second disease They are the output values of the second inclined cable broken wire respectively , the output value of the second wear , the second outer layer damage output value and the second rust output value ; Construct a reinforcement model, namely deep learning model C. The deep learning model C consists of 1 input layer, 12 convolution layers, 12 activation layers, 5 pooling layers, 1 merging layer, 1 upsampling layer, 1 PANet layer, 1 pooling splicing layer, 1 SPP layer and 1 output layer; The data set is input into the deep learning model C, and the weights and biases of the neural network units of the deep learning model C are calculated through the error back propagation algorithm, and the training of the deep learning model C is completed to obtain the trained deep learning model C; Loss function of the trained deep learning model C It is expressed as: ; in, is the total number of data samples in the dataset of the deep learning model C, For the The true labels of samples, For the deep learning model C to predict The probability that a sample is class 1; The output value of the output unit node corresponding to the deep learning model C is , respectively, are the output values of the ninth output unit node , the output value of the tenth output unit node , the output value of the eleventh output unit node and the output value of the twelfth output unit node In order to further clarify the differences between the output values, the model output is converted into a probability distribution to improve the model's ability to distinguish. According to the exponential function , firstly, the output values of the four output unit nodes corresponding to the deep learning model C are initially converted into the third preliminary output value and its probability value, and then further optimized and converted to obtain the third optimized output value and its enhanced probability value in order to amplify the probability difference between the output values and enhance the distinguishing ability of the model; The probability value of the third preliminary output value It is expressed as: ; The enhanced probability value of the third optimized output value It is expressed as: ; The enhanced probability value using the third optimized output value And the data set, combined with the error chain derivative rule, continue to train the deep learning model C. After the training is completed, the output value of the third disease is obtained The actual disease annotations corresponding to the nodes in the output layer of the deep learning model B , the third disease output value They are the output values of the third inclined cable broken wire respectively. , Output value of the third wear , the third outer layer damage output value and the third rust output value ; According to the second disease output value is and the enhanced probability value of the first optimized output value , construct the loss function between deep learning model B and deep learning model A , get the corresponding weight coefficient; Loss function between deep learning model B and deep learning model A It is expressed as: ; Probability value based on the first preliminary output value The actual disease annotations corresponding to the nodes in the output layer of the deep learning model A , construct the probability value considering the first preliminary output value The loss function of the deep learning model A is , get the corresponding weight coefficient; Consider the probability value of the first preliminary output value The loss function of the deep learning model A is It is expressed as: ; Based on the loss function between deep learning model B and deep learning model A and the probability value considering the first preliminary output value The loss function of the deep learning model A is , establish the relationship loss function between the output layer of deep learning model B and deep learning model A ; The relationship loss function between the output layer of deep learning model B and deep learning model A It is expressed as: ; in, is the loss function between deep learning model B and deep learning model A The weight coefficient of is the probability value considering the first preliminary output value The loss function of the deep learning model A is The weight coefficient of Output value according to the third disease and the enhanced probability value of the second optimized output value , construct the loss function between deep learning model C and deep learning model B , get the corresponding weight coefficient; Loss function between deep learning model C and deep learning model B It is expressed as: ; Probability value based on the second preliminary output value And the second actual disease mark , construct the probability value considering the second preliminary output value The loss function of the deep learning model B , get the corresponding weight coefficient; Consider the probability value of the second preliminary output value The loss function of the deep learning model B It is expressed as: ; Output value according to the third disease and the enhanced probability value of the first optimized output value , construct the loss function between deep learning model C and deep learning model A , get the corresponding weight coefficient; Loss function between deep learning model C and deep learning model A It is expressed as: ; According to the above operations, the relationship loss function between the output layers of deep learning model C and deep learning model B is established. The relationship loss function between the output layer of deep learning model C and deep learning model A , define the corresponding overall loss function between deep learning model A, deep learning model B and deep learning model C ; The relationship loss function between the output layers of deep learning model C and deep learning model B It is expressed as: ; in, is the loss function between deep learning model C and deep learning model B The weight coefficient of The probability value of the second preliminary output value is considered The loss function of the deep learning model B The weight coefficient of The relationship loss function between the output layer of deep learning model C and deep learning model A It is expressed as: ; in, is the loss function between deep learning model C and deep learning model A The weight coefficient of Specifically, the image characteristics of the broken wire of the inclined cable usually include small and irregular broken wire shapes, complex background noise, and illumination changes, which requires the model to have high-resolution feature extraction capabilities, strong background noise suppression capabilities, and robustness to illumination changes. The three models designed in the present invention can effectively achieve the above requirements through various levels of design. The specific functions of each layer structure are as follows: Input layer: receives the image data of the cable collected by the drone; Convolution layer: uses filters to extract the features of the image, including the local features of the broken wire shape. Multiple convolution layers can extract complex features from the bottom layer to the top layer; Activation layer: introduces nonlinearity to the network to help the network learn complex feature representations; Pooling layer: reduces the spatial size of the feature map, reduces the number of parameters and the amount of calculation, and at the same time improves the robustness of feature extraction, making the model less sensitive to small displacements and deformations (such as the subtle differences of broken wires); Merging layer: fuses feature information at different levels to help restore the details of the broken wire; Upsampling layer: improves the spatial resolution of the feature map, making the network It can locate the specific position of the broken wire more accurately; PANet layer: improves the information flow between low-level and high-level features, enhances the representation ability of features, and helps capture the fine features of broken wires; Pooling and splicing layer: refers to splicing the features of different pooling areas to further enhance the model's ability to capture features of different scales; SPP layer: through spatial pyramid pooling, the network can learn broken wire features of different sizes and enhance robustness to size changes; Output layer: maps the learned high-level features to the final recognition results, and outputs the probability of each category through the activation function, so that the model can predict whether there are broken wires in the image.
[0025] Furthermore, the S5 specifically includes the following steps: S51. Establish an inter-layer loss function from the enhanced model to the basic model, and combine the fused loss function to obtain the total loss function; S52. Based on the chain derivation rule, the three models are trained by error back propagation to obtain the trained deep learning model A, i.e., the optimal model; In the S51, the selected layers of the deep learning model A are connected to the selected layers of the deep learning model C, the selected layers in the deep learning model A include the second layer: convolution layer 1, the third layer: activation layer 1, the fourth layer: convolution layer 2 and the twenty-eighth layer: output layer, and the selected layers of the deep learning model C include the fourth layer: convolution layer 2, the sixth layer: convolution layer 3, the tenth layer: convolution layer 5 and the thirty-sixth layer: output layer; The selected layers of the deep learning model A are , the corresponding size is ,in, =2, 3, 4 and 28, is the selected number of layers of the deep learning model A, is the selected layer width of the deep learning model A, is the selected layer length of the deep learning model A, is the number of channels of the selected layer of deep learning model A, and the selected layer of deep learning model C is , the corresponding size is ,in, =4, 6, 10 and 36, is the selected number of layers of the deep learning model C, is the selected layer width of the deep learning model C, is the selected layer length of the deep learning model C, is the number of selected layer channels of the deep learning model C; when When , a convolution operation is added to the selected layer output of the deep learning model A, with a convolution kernel size of 1×1 and a number of convolution kernels. , get the selected layer output of deep learning model A , and the corresponding output value of each channel is , =1, 2, 3, …, , Output channel number for the selected layer of deep learning model A and the selected layer of deep learning model C The corresponding output values of each channel are , =1, 2, 3, …, , Output channel number for the selected layer of deep learning model C, and establish the loss function between the selected layer of deep learning model A and the selected layer of deep learning model C ; when When , the loss function between the selected layer of deep learning model A and the selected layer of deep learning model C is It is expressed as: ; when When , a convolution operation is added to the selected layer output of the deep learning model C, with a convolution kernel size of 1×1 and a number of convolution kernels. , the selected layer output of the deep learning model C is , and the corresponding output value of each channel is , the selected layers of the deep learning model C The corresponding output values of each channel are , establish the loss function between the selected layer of deep learning model A and the selected layer of deep learning model C ; when When , the loss function between the selected layer of deep learning model A and the selected layer of deep learning model C is It is expressed as: ; According to the above calculation, the loss function between the second layer of deep learning model A and the fourth layer of deep learning model C is obtained , the loss function between the third layer of deep learning model A and the sixth layer of deep learning model C And the loss function between the fourth layer of deep learning model A and the tenth layer of deep learning model C , combining the relationship loss function between the output layers of deep learning model B and deep learning model A , the relationship loss function between the output layers of deep learning model C and deep learning model B The relationship loss function between the output layer of deep learning model C and deep learning model A , define the total loss function ; Total loss function It is expressed as: ; in, is the loss function between the second layer of deep learning model A and the fourth layer of deep learning model C The corresponding weight coefficient is, is the loss function between the third layer of deep learning model A and the sixth layer of deep learning model C The corresponding weight coefficient is, is the loss function between the fourth layer of deep learning model A and the tenth layer of deep learning model C The corresponding weight coefficient; Specifically, deep learning model B is an auxiliary model. In order to enrich the feature representation capability of the basic deep learning model A, an inter-layer association relationship is established between the basic model (deep learning model A) and the enhanced model (deep learning model C), so that the features of different levels in the enhanced model can be transferred to the basic model, thereby further improving the performance improvement effect of the basic model. The trained deep learning model A is the optimal model. The deep learning model A has the highest level of lightweight and can be easily deployed in end devices such as edge devices to achieve fast and accurate identification of broken wires in inclined cables.
[0026] Furthermore, in S43, the layers in the network structure of the deep learning model A are connected in the following order: Input layer: image size 227×227×3; Convolutional layer 1: kernel size 3×3, number of kernels 32, stride: 1, batch normalization; Activation layer 1: LeakyReLU; Convolutional layer 2: kernel size 3×3, number of kernels 64, stride: 1, batch normalization; Activation layer 2: LeakyReLU; Convolution layer 3: convolution kernel size 3×3, number of convolution kernels 128, stride: 1, batch normalization; Activation layer 3: LeakyReLU; Convolutional layer 4: kernel size 3×3, number of kernels 256, stride: 1, batch normalization; Activation layer 4: LeakyReLU; SPP layer: extracts features of different scales and fuses them to increase the receptive field of the model; Pooling layer 1: Type: global average pooling, pooling size: 5×5, stride: 1; Pooling layer 2: Type: global average pooling, pooling size: 9×9, Stride: 1; Pooling layer 3: Type: global average pooling, pooling size: 13×13, Stride: 1; Pooling concatenation layer: connect the outputs of the three pooling layers; PANet layer: aggregates features at different levels and implements feature pyramids through upsampling and downsampling operations; Convolutional layer 5: kernel size 3×3, number of kernels 768, stride: 2, batch normalization; Activation layer 5: LeakyReLU; Convolutional layer 6: kernel size 3×3, number of kernels 768, stride: 2, batch normalization; Activation layer 6: LeakyReLU; Upsampling layer: uses the nearest neighbor interpolation method to upsample to the same size as the pooling splicing layer; Merging layer: The output feature map of the downsampling layer is merged with the output feature map of the upsampling layer; Convolutional layer 7: kernel size 3×3, number of kernels 256, stride: 1, batch normalization; Activation layer 7: LeakyReLU; Convolutional layer 8: kernel size 3×3, number of kernels 128, stride: 1, batch normalization; Activation layer 8: LeakyReLU; Convolutional layer 9: kernel size 3×3, number of kernels 64, stride: 1, batch normalization; Activation layer 9: LeakyReLU; Output layer: Type: Softmax, Number of neurons: 4.
[0027] Furthermore, in S43, the layers in the network structure of the deep learning model B are connected in the following order: Input layer: image size 227×227×3; Convolutional layer 1: kernel size 3×3, number of kernels 32, stride: 1, batch normalization; Activation layer 1: LeakyReLU; Convolutional layer 2: kernel size 3×3, number of kernels 64, stride: 1, batch normalization; Activation layer 2: LeakyReLU; Convolution layer 3: convolution kernel size 3×3, number of convolution kernels 128, stride: 1, batch normalization; Activation layer 3: LeakyReLU; Convolutional layer 4: kernel size 3×3, number of kernels 256, stride: 1, batch normalization; Activation layer 4: LeakyReLU; SPP layer: extracts features of different scales and fuses them to increase the receptive field of the model; Pooling layer 1: Type: global average pooling, pooling size: 5×5, stride: 1; Pooling layer 2: Type: global average pooling, pooling size: 7×7, Stride: 1; Pooling layer 3: Type: global average pooling, pooling size: 9×9, Stride: 1; Pooling layer 4: Type: global average pooling, pooling size: 11×11, Stride: 1; Pooling layer 5: Type: global average pooling, pooling size: 13×13, stride: 1; Pooling concatenation layer: connect the outputs of the five pooling layers; PANet layer: aggregates features at different levels and implements feature pyramids through upsampling and downsampling operations; Convolution layer 5: kernel size 3×3, number of kernels 1280, stride: 2, batch normalization; Activation layer 5: LeakyReLU; Convolutional layer 6: kernel size 3×3, number of kernels 1280, stride: 2, batch normalization; Activation layer 6: LeakyReLU; Upsampling layer: uses the nearest neighbor interpolation method to upsample to the same size as the pooling splicing layer; Merging layer: The output feature map of the downsampling layer is merged with the output feature map of the upsampling layer; Convolutional layer 7: kernel size 3×3, number of kernels 512, stride: 1, batch normalization; Activation layer 7: LeakyReLU; Convolutional layer 7: kernel size 3×3, number of kernels 256, stride: 1, batch normalization; Activation layer 7: LeakyReLU; Convolutional layer 8: kernel size 3×3, number of kernels 128, stride: 1, batch normalization; Activation layer 8: LeakyReLU; Convolutional layer 9: kernel size 3×3, number of kernels 64, stride: 1, batch normalization; Activation layer 9: LeakyReLU; Output layer: Type: Softmax, Number of neurons: 4.
[0028] Furthermore, in S43, the layers in the network structure of the deep learning model C are connected in the following order: Input layer: image size 227×227×3; Convolutional layer 1: kernel size 3×3, number of kernels 32, stride: 1, batch normalization; Activation layer 1: LeakyReLU; Convolutional layer 2: kernel size 3×3, number of kernels 64, stride: 1, batch normalization; Activation layer 2: LeakyReLU; Convolution layer 3: convolution kernel size 3×3, number of convolution kernels 128, stride: 1, batch normalization; Activation layer 3: LeakyReLU; Convolutional layer 4: convolution kernel size 3×3, number of convolution kernels 128, stride: 1, batch normalization; Activation layer 4: LeakyReLU; Convolutional layer 5: convolution kernel size 3×3, number of convolution kernels 256, stride: 1, batch normalization; Activation layer 5: LeakyReLU; Convolutional layer 6: kernel size 3×3, number of kernels 256, stride: 1, batch normalization; Activation layer 6: LeakyReLU; SPP layer: extracts features of different scales and fuses them to increase the receptive field of the model; Pooling layer 1: Type: global average pooling, pooling size: 5×5, stride: 1; Pooling layer 2: Type: global average pooling, pooling size: 7×7, Stride: 1; Pooling layer 3: Type: global average pooling, pooling size: 9×9, Stride: 1; Pooling layer 4: Type: global average pooling, pooling size: 11×11, Stride: 1; Pooling layer 5: Type: global average pooling, pooling size: 13×13, stride: 1; Pooling concatenation layer: connect the outputs of the three pooling layers; PANet layer: aggregates features at different levels and implements feature pyramids through upsampling and downsampling operations; Convolutional layer 7: kernel size 3×3, number of kernels 1280, stride: 2, batch normalization; Activation layer 7: LeakyReLU; Convolutional layer 8: kernel size 3×3, number of kernels 1280, stride: 2, batch normalization; Activation layer 8: LeakyReLU; Upsampling layer: uses the nearest neighbor interpolation method to upsample to the same size as the pooling splicing layer; Merging layer: The output feature map of the downsampling layer is merged with the output feature map of the upsampling layer; Convolutional layer 9: kernel size 3×3, number of kernels 512, stride: 1, batch normalization; Activation layer 9: LeakyReLU; Convolution layer 10: kernel size 3×3, number of kernels 256, stride: 1, batch normalization; Activation layer 10: LeakyReLU; Convolutional layer 11: kernel size 3×3, number of kernels 128, stride: 1, batch normalization; Activation layer 11: LeakyReLU; Convolutional layer 12: kernel size 3×3, number of kernels 64, stride: 1, batch normalization; Activation layer 12: LeakyReLU; Output layer: Type: Softmax, Number of neurons: 4.
[0029] Although the present invention has been described according to a limited number of embodiments, it will be apparent to those skilled in the art, with the benefit of the above description, that other embodiments may be envisioned within the scope of the invention thus described. In addition, it should be noted that the language used in this specification is selected primarily for readability and teaching purposes, rather than for explaining or defining the subject matter of the present invention. Therefore, many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the appended claims. The disclosure of the present invention is illustrative, not restrictive, with respect to the scope of the present invention, which is defined by the appended claims.
Claims
1. A method for safety detection, early warning and broken wire identification of a stay cable based on a drone, characterized in that: The following steps are involved: S1. Plan the shortest flight path of the drone based on the drone parameter information and operation information; S2. Collect the location information, temperature information, wind information and humidity information of the drone during flight along the shortest flight path of the drone, set corresponding thresholds, conduct comprehensive warning of drone detection and grade the warning degree; S3. Collect the cable image along the shortest flight path of the UAV, divide it into equal parts and update the image pixels to obtain an image with updated pixels; S4. construct a data set based on the pixel updated image combined with the disease information, input the data set into the established basic model, auxiliary model and enhanced model for training, fuse the loss functions of the three models, and obtain the fused loss function; S5. Establish the inter-layer loss function from the enhanced model to the basic model, combine the fused loss function to obtain the total loss function, train the three models, use the trained basic model as the final model, and output the final broken wire recognition result.
2. The method for safety detection, early warning and broken wire identification of a stay cable based on a drone according to claim 1 is characterized in that: The S1 specifically includes the following steps: S11. Determine the take-off and landing points of the drone and select the observation position of the operator; S12. Determine the flight altitude AVh according to the focal length and sensor size of the camera carried by the drone, obtain the image coverage and resolution, and set the drone flight speed to AVv; S13. Develop the shortest flight path based on the take-off and landing points of the UAV, and determine the coordinate information of the cable and the location information of the UAV shooting points; In S13, the take-off point coordinates are AVo (AVoX, AVoY, AVoZ). On this basis, all drone shooting points are determined. The number of any drone shooting point is p, and its corresponding drone shooting point coordinates are AVp (AVpX, AVpY, AVpZ). The number set of drone shooting points is expressed as {1, 2, 3, ..., XN}, and the corresponding drone shooting point coordinate set is expressed as {AV1 (AV1X, AV1Y, AV1Z), AV2 (AV2X, AV2Y, AV2Z), ..., AVXN (AVXNX, AVXNY, AVXNZ)}, where XN is the number of drone shooting points. The adjacent distance between any drone shooting point AVp and its adjacent drone shooting point AV (p+1) is calculated. ; Taking the take-off point AVo as the starting point, according to the distance between any drone shooting point and the take-off point , calculate the distance between all drone shooting points and the take-off point, select the three drone shooting points corresponding to the minimum distance, and randomly select them to obtain the first drone shooting point that the drone arrives at after departing from the take-off point; After determining the first shooting point that the drone arrives at, calculate the distance between all shooting points and the first shooting point that is arrived at, screen out the three drone shooting points corresponding to the smallest distance, and randomly select them to obtain the second shooting point that the drone arrives at after departing from the first shooting point it arrives at. Repeat the above steps until all the shooting points that the drone arrives at are obtained. According to the different random selections each time, calculate the total distance TDis1 of the drone's flight under the first flight mode. Repeat the above steps, the number of repetitions is CN, and integrate to obtain the total distance set of the drone's flight under different flight modes {TDis1, TDis2, TDis3, TDis4, …, TDisCN}, arrange them in order from small to large, and select the total distances and corresponding flight routes ranked in the top TN, TN is the preset value, count the intersection information of TN flight routes, and select the flight route with the largest number of intersections to obtain the shortest flight path of the drone.
3. The method for safety detection, early warning and broken wire identification of a stay cable based on a drone according to claim 2 is characterized in that: The S2 specifically includes the following steps: S21. Select the key warning points of the cable-stayed bridge, collect the position information of the drone during flight, and design the drone collision warning; S22. Set a safe temperature threshold, collect temperature information during the flight of the drone, and design a temperature warning for the drone; S23. Set a safe wind threshold, collect wind information during the flight of the drone, and design a wind warning for the drone; S24. Set a safe humidity threshold, collect humidity information during the flight of the drone, and design a humidity warning for the drone; S25. Consider drone collision warning, drone temperature warning, drone wind warning, and drone humidity warning, conduct comprehensive drone detection warning and classify the warning levels; In S21, the key warning points of the cable-stayed bridge are GJD1, GJD2, …, GJDgn respectively, where gn is the number of key warning points. Through the drone-borne positioning system, combined with the shortest flight path of the drone, the position information WRD during the flight of the drone is located in real time. The safety distance threshold SJLtr is set, and the distance SJLim between the position information during the flight of the drone and all the key warning points of the cable-stayed bridge is calculated. SJLim = |GJDim - WRD|, where GJDim is the key warning point of the cable-stayed bridge corresponding to the position information during the current drone flight, and im is the warning key point number, im = 1, 2, …, gn. According to the distance SJLim, a distance matrix SJL is constructed, SJL = [SJL1, SJL2, SJL3, …, SJLgn], and the minimum element value SJLmin in the distance matrix is calculated. SJLmin = min[SJL1, SJL2, SJL3, …, SJLgn], where min is the minimum value. Compare the relationship between the minimum element value SJLmin in the distance matrix and the safety distance threshold SJLtr. When SJLmin ≥ SJLtr, it is considered that the drone position is safe and no warning is required. When SJLmin < SJLtr, it is considered that the drone has a risk of colliding with the bridge and a warning is required; In S22, the safety temperature threshold is set to WDtr. Through the drone-borne temperature sensor, the temperature information WWD during the flight of the drone is collected in real time. Compare the relationship between it and the safety temperature threshold WDtr. When WWD ≥ WDtr, it is considered that the current temperature is suitable for drone detection and no warning is required. When WWD < WDtr, it is considered that the current temperature is not suitable for drone detection and a warning is required; In S23, the safety wind threshold is set to FDtr. Through the drone-borne wind speed sensor, the wind force information WFL during the flight of the drone is collected in real time. Compare the relationship between it and the safety wind threshold FDtr. When WFL ≥ FDtr, it is considered that the current wind force is suitable for drone detection and no warning is required. When WFL < FDtr, it is considered that the wind force is not suitable for drone detection and a warning is required; In S24, the safety humidity threshold is set to SDtr. Through the drone-borne humidity sensor, the humidity information WSD during the flight of the drone is collected in real time. Compare the relationship between it and the safety humidity threshold SDtr. When WSD ≥ SDtr, it is considered that the current humidity is suitable for drone detection and no warning is required. When WSD < SDtr, it is considered that the current humidity is not suitable for drone detection and a warning is required; In S25, the grade value YJL of the warning level classification is expressed as: YJL = YJpz + YJwd + YJfl + YJsd Among them, YJpz is the warning value corresponding to the UAV collision warning. When SJLmin ≥ SJLtr, YJpz = 0; when SJLmin < SJLtr, YJpz = 1. YJwd is the warning value corresponding to the temperature warning. When WWD ≥ WDtr, YJwd = 0; when WWD < WDtr, YJwd = 1. YJfl is the warning value corresponding to the wind warning. When WFL ≥ FDtr, YJfl = 0; when WFL < FDtr, YJfl = 1. YJsd is the warning value corresponding to the humidity warning. When WSD ≥ SDtr, YJsd = 0; when WSD < SDtr, YJsd = 1; When YJL > 0, it is necessary to give a warning to the UAV.
4. The method for safety detection, early warning and broken wire identification of a stay cable based on a drone according to claim 3 is characterized in that: In step S3, it specifically includes the following steps: S31. Use an on-board high-definition camera of the UAV to collect the images of the stay cables of the cable-stayed bridge along the determined shortest flight path. S32. Divide the pixel gray values of the stay cable images into 16 gray levels equally, and divide the stay cable images into n equal parts to obtain the divided images. S33. Count the pixel frequencies corresponding to each gray level in the equally divided image , calculate the grayscale frequency cumulative value corresponding to each grayscale level in the equally divided image ; S34.According to the gray frequency cumulative value , construct a pixel value update method and calculate the pixel update value corresponding to the gray level , get the image after pixel update; In S32, the pixel gray value is 0-255, and the gray level is , =1, 2, 3, …, 16, is the grayscale number, and the image after equal division is Pnum, num=1, 2, ..., n; In S33, each gray level in the equally divided image Pnum The corresponding gray frequency cumulative value It is expressed as: ; in, is less than or equal to The gray level number, Indicates the gray level number is Gray level; In S34, the pixel update value corresponding to the gray level It is expressed as: ; in, is the minimum non-zero value among all pixel frequency accumulation values, is the total number of pixels corresponding to the equally divided image Pnum, is the maximum value of the gray level.
5. The method for safety detection, early warning and broken wire identification of a stay cable based on a drone according to claim 4 is characterized in that: In step S4, it specifically includes the following steps: S41. Based on the image after pixel update, select all the images containing stay cable wire breaks, wear, outer sheath damage, and corrosion as the input images. S42. Through manual annotation, frame the stay cable wire break area and the other three diseases in the input images to construct a data set. According to the ratio of 8:2 in quantity, divide the data set into a training set and a test set. S43. According to the data set, construct a basic model, an auxiliary model, and a reinforcement model, calculate the loss functions of the three models and fuse them to obtain the fused loss function. In step S42, the wire break area is marked as Wire, and the three diseases, namely wear, outer sheath damage, and corrosion, are marked as other1, other2, and other3 in sequence. In step S43, construct the network structure of the basic model, that is, deep learning model A. Deep learning model A consists of 1 input layer, 9 convolutional layers, 9 activation layers, 3 pooling layers, 1 merging layer, 1 upsampling layer, 1 PANet layer, 1 pooling splicing layer, 1 SPP layer, and 1 output layer. Input the data set into the deep learning model A, calculate the weights and biases of the neural network units of the deep learning model A through the error back propagation algorithm, complete the training of the deep learning model A, and obtain the trained deep learning model A and its loss function ; The output value of the output unit node corresponding to the deep learning model A is , respectively, are the output values of the first output unit node , the output value of the second output unit node , the output value of the third output unit node and the output value of the fourth output unit node , according to the exponential function First, the output values of the four output unit nodes corresponding to the deep learning model A are initially converted into the probability values of the first preliminary output value and the first preliminary output value. , and then further optimize and transform it to obtain the first optimized output value and the enhanced probability value of the first optimized output value ; The enhanced probability value using the first optimized output value And the data set, combined with the error chain derivative rule, continue to train the deep learning model A. After the training is completed, the output value of the first disease is obtained The actual disease annotations corresponding to the nodes in the output layer of the deep learning model A , the output value of the first disease They are the output values of the first inclined cable broken wire respectively , the output value of the first wear , the first outer layer damage output value and the first rust output value ; Construct the auxiliary model, that is, deep learning model B. Deep learning model B consists of 1 input layer, 9 convolutional layers, 9 activation layers, 5 pooling layers, 1 merging layer, 1 upsampling layer, 1 PANet layer, 1 pooling splicing layer, 1 SPP layer, and 1 output layer. Input the data set into the deep learning model B, calculate the weights and biases of the neural network units of the deep learning model B through the error back propagation algorithm, complete the training of the deep learning model B, and obtain the trained deep learning model B and its loss function ; The output value of the output unit node corresponding to the deep learning model B is , respectively, are the output values of the fifth output unit node , the output value of the sixth output unit node , the output value of the seventh output unit node and the output value of the eighth output unit node , according to the exponential function First, the output values of the four output unit nodes corresponding to the deep learning model B are initially converted into the second preliminary output value and the probability value of the second preliminary output value , and then further optimize and transform it to obtain the second optimized output value and the enhanced probability value of the second optimized output value ; The enhanced probability value using the second optimized output value And the data set, combined with the error chain derivative rule, continue to train the deep learning model B. After the training is completed, the output value of the second disease is obtained The actual disease annotations corresponding to the nodes in the output layer of the deep learning model B , the output value of the second disease They are the output values of the second inclined cable broken wire respectively , the output value of the second wear , the second outer layer damage output value and the second rust output value ; Construct the reinforcement model, that is, deep learning model C. Deep learning model C consists of 1 input layer, 12 convolutional layers, 12 activation layers, 5 pooling layers, 1 merging layer, 1 upsampling layer, 1 PANet layer, 1 pooling splicing layer, 1 SPP layer, and 1 output layer. Input the data set into the deep learning model C, calculate the weights and biases of the neural network units of the deep learning model C through the error back propagation algorithm, complete the training of the deep learning model C, and obtain the trained deep learning model C and its loss function ; The output value of the output unit node corresponding to the deep learning model C is , respectively, are the output values of the ninth output unit node , the output value of the tenth output unit node , the output value of the eleventh output unit node and the output value of the twelfth output unit node , according to the exponential function First, the output values of the four output unit nodes corresponding to the deep learning model C are initially converted into the third preliminary output value and the probability value of the third preliminary output value , and then further optimize and transform it to obtain the third optimized output value and the enhanced probability value of the third optimized output value ; The enhanced probability value using the third optimized output value And the data set, combined with the error chain derivative rule, continue to train the deep learning model C. After the training is completed, the output value of the third disease is obtained The actual disease annotations corresponding to the nodes in the output layer of the deep learning model B , the third disease output value They are the output values of the third inclined cable broken wire respectively , Output value of the third wear , the third outer layer damage output value and the third rust output value ; According to the second disease output value is and the enhanced probability value of the first optimized output value , construct the loss function between deep learning model B and deep learning model A , get the corresponding weight coefficient; Probability value based on the first preliminary output value The actual disease annotations corresponding to the nodes in the output layer of the deep learning model A , construct the probability value considering the first preliminary output value The loss function of the deep learning model A is , get the corresponding weight coefficient; Based on the loss function between deep learning model B and deep learning model A and the probability value considering the first preliminary output value The loss function of the deep learning model A is , establish the relationship loss function between the output layer of deep learning model B and deep learning model A ; According to the above operations, the relationship loss function between the output layers of deep learning model C and deep learning model B is established. The relationship loss function between the output layer of deep learning model C and deep learning model A .
6. The method for safety detection, early warning and broken wire identification of a stay cable based on a drone according to claim 5 is characterized in that: In step S5, it specifically includes the following steps: S51. Establish the inter-layer loss function from the reinforcement model to the basic model, and combine it with the fused loss function to obtain the total loss function. S52. Based on the chain rule of differentiation, train the three models by the error backpropagation method to obtain the trained deep learning model A, that is, the optimal model. In the S51, the selected layers of the deep learning model A are connected to the selected layers of the deep learning model C, the selected layers in the deep learning model A include the second layer: convolution layer 1, the third layer: activation layer 1, the fourth layer: convolution layer 2 and the twenty-eighth layer: output layer, and the selected layers of the deep learning model C include the fourth layer: convolution layer 2, the sixth layer: convolution layer 3, the tenth layer: convolution layer 5 and the thirty-sixth layer: output layer; The selected layers of the deep learning model A are , the corresponding size is ,in, =2, 3, 4 and 28, is the selected number of layers of the deep learning model A, is the selected layer width of the deep learning model A, is the selected layer length of the deep learning model A, is the number of channels of the selected layer of deep learning model A, and the selected layer of deep learning model C is , the corresponding size is ,in, =4, 6, 10 and 36, is the selected number of layers of the deep learning model C, is the selected layer width of the deep learning model C, is the selected layer length of the deep learning model C, is the number of selected layer channels of the deep learning model C; when When , a convolution operation is added to the selected layer output of the deep learning model A, with a convolution kernel size of 1×1 and a number of convolution kernels. , get the selected layer output of deep learning model A , and the corresponding output value of each channel is , =1, 2, 3, …, , Output channel number for the selected layer of deep learning model A and the selected layer of deep learning model C The corresponding output values of each channel are , =1, 2, 3, …, , Output channel number for the selected layer of deep learning model C, and establish the loss function between the selected layer of deep learning model A and the selected layer of deep learning model C ; when When , a convolution operation is added to the selected layer output of the deep learning model C, with a convolution kernel size of 1×1 and a number of convolution kernels. , the selected layer output of the deep learning model C is , and the corresponding output value of each channel is , the selected layers of the deep learning model C The corresponding output values of each channel are , establish the loss function between the selected layer of deep learning model A and the selected layer of deep learning model C ; According to the above calculation, the loss function between the second layer of deep learning model A and the fourth layer of deep learning model C is obtained , the loss function between the third layer of deep learning model A and the sixth layer of deep learning model C And the loss function between the fourth layer of deep learning model A and the tenth layer of deep learning model C , combining the relationship loss function between the output layers of deep learning model B and deep learning model A , the relationship loss function between the output layers of deep learning model C and deep learning model B The relationship loss function between the output layer of deep learning model C and deep learning model A , define the total loss function ; Total loss function It is expressed as: ; in, is the loss function between the second layer of deep learning model A and the fourth layer of deep learning model C The corresponding weight coefficient is, is the loss function between the third layer of deep learning model A and the sixth layer of deep learning model C The corresponding weight coefficient is, is the loss function between the fourth layer of deep learning model A and the tenth layer of deep learning model C The corresponding weight coefficient.
7. The method for safety detection, early warning and broken wire identification of a stay cable based on a drone according to claim 6 is characterized in that: In S43, the layers in the network structure of the deep learning model A are connected in the following order: Input layer: image size 227×227×3; Convolutional layer 1: kernel size 3×3, number of kernels 32, stride: 1, batch normalization; Activation layer 1: LeakyReLU; Convolutional layer 2: kernel size 3×3, number of kernels 64, stride: 1, batch normalization; Activation layer 2: LeakyReLU; Convolution layer 3: convolution kernel size 3×3, number of convolution kernels 128, stride: 1, batch normalization; Activation layer 3: LeakyReLU; Convolutional layer 4: kernel size 3×3, number of kernels 256, stride: 1, batch normalization; Activation layer 4: LeakyReLU; SPP layer: extracts features of different scales and fuses them to increase the receptive field of the model; Pooling layer 1: Type: global average pooling, pooling size: 5×5, stride: 1; Pooling layer 2: Type: global average pooling, pooling size: 9×9, Stride: 1; Pooling layer 3: Type: global average pooling, pooling size: 13×13, Stride: 1; Pooling concatenation layer: connect the outputs of the three pooling layers; PANet layer: aggregates features at different levels and implements feature pyramids through upsampling and downsampling operations; Convolutional layer 5: kernel size 3×3, number of kernels 768, stride: 2, batch normalization; Activation layer 5: LeakyReLU; Convolutional layer 6: kernel size 3×3, number of kernels 768, stride: 2, batch normalization; Activation layer 6: LeakyReLU; Upsampling layer: uses the nearest neighbor interpolation method to upsample to the same size as the pooling splicing layer; Merging layer: The output feature map of the downsampling layer is merged with the output feature map of the upsampling layer; Convolutional layer 7: kernel size 3×3, number of kernels 256, stride: 1, batch normalization; Activation layer 7: LeakyReLU; Convolutional layer 8: kernel size 3×3, number of kernels 128, stride: 1, batch normalization; Activation layer 8: LeakyReLU; Convolutional layer 9: kernel size 3×3, number of kernels 64, stride: 1, batch normalization; Activation layer 9: LeakyReLU; Output layer: Type: Softmax, Number of neurons:
4.
8. The method for safety detection, early warning and broken wire identification of a stay cable based on a drone according to claim 7 is characterized in that: In S43, the layers in the network structure of the deep learning model B are connected in the following order: Input layer: image size 227×227×3; Convolutional layer 1: kernel size 3×3, number of kernels 32, stride: 1, batch normalization; Activation layer 1: LeakyReLU; Convolutional layer 2: kernel size 3×3, number of kernels 64, stride: 1, batch normalization; Activation layer 2: LeakyReLU; Convolution layer 3: convolution kernel size 3×3, number of convolution kernels 128, stride: 1, batch normalization; Activation layer 3: LeakyReLU; Convolutional layer 4: kernel size 3×3, number of kernels 256, stride: 1, batch normalization; Activation layer 4: LeakyReLU; SPP layer: extracts features of different scales and fuses them to increase the receptive field of the model; Pooling layer 1: Type: global average pooling, pooling size: 5×5, stride: 1; Pooling layer 2: Type: global average pooling, pooling size: 7×7, Stride: 1; Pooling layer 3: Type: global average pooling, pooling size: 9×9, Stride: 1; Pooling layer 4: Type: global average pooling, pooling size: 11×11, Stride: 1; Pooling layer 5: Type: global average pooling, pooling size: 13×13, stride: 1; Pooling concatenation layer: connect the outputs of the five pooling layers; PANet layer: aggregates features at different levels and implements feature pyramids through upsampling and downsampling operations; Convolution layer 5: kernel size 3×3, number of kernels 1280, stride: 2, batch normalization; Activation layer 5: LeakyReLU; Convolutional layer 6: kernel size 3×3, number of kernels 1280, stride: 2, batch normalization; Activation layer 6: LeakyReLU; Upsampling layer: uses the nearest neighbor interpolation method to upsample to the same size as the pooling splicing layer; Merging layer: The output feature map of the downsampling layer is merged with the output feature map of the upsampling layer; Convolutional layer 7: kernel size 3×3, number of kernels 512, stride: 1, batch normalization; Activation layer 7: LeakyReLU; Convolutional layer 7: kernel size 3×3, number of kernels 256, stride: 1, batch normalization; Activation layer 7: LeakyReLU; Convolutional layer 8: kernel size 3×3, number of kernels 128, stride: 1, batch normalization; Activation layer 8: LeakyReLU; Convolution layer 9: kernel size 3×3, number of kernels 64, stride: 1, batch normalization; Activation layer 9: LeakyReLU; Output layer: Type: Softmax, Number of neurons:
4.
9. The method for safety detection, early warning and broken wire identification of a stay cable based on a drone according to claim 8 is characterized in that: In S43, the layers in the network structure of the deep learning model C are connected in the following order: Input layer: image size 227×227×3; Convolutional layer 1: kernel size 3×3, number of kernels 32, stride: 1, batch normalization; Activation layer 1: LeakyReLU; Convolutional layer 2: kernel size 3×3, number of kernels 64, stride: 1, batch normalization; Activation layer 2: LeakyReLU; Convolution layer 3: convolution kernel size 3×3, number of convolution kernels 128, stride: 1, batch normalization; Activation layer 3: LeakyReLU; Convolutional layer 4: kernel size 3×3, number of kernels 128, stride: 1, batch normalization; Activation layer 4: LeakyReLU; Convolutional layer 5: kernel size 3×3, number of kernels 256, stride: 1, batch normalization; Activation layer 5: LeakyReLU; Convolutional layer 6: kernel size 3×3, number of kernels 256, stride: 1, batch normalization; Activation layer 6: LeakyReLU; SPP layer: extracts features of different scales and fuses them to increase the receptive field of the model; Pooling layer 1: Type: global average pooling, pooling size: 5×5, stride: 1; Pooling layer 2: Type: global average pooling, pooling size: 7×7, Stride: 1; Pooling layer 3: Type: global average pooling, pooling size: 9×9, Stride: 1; Pooling layer 4: Type: global average pooling, pooling size: 11×11, Stride: 1; Pooling layer 5: Type: global average pooling, pooling size: 13×13, stride: 1; Pooling concatenation layer: connect the outputs of the three pooling layers; PANet layer: aggregates features at different levels and implements feature pyramids through upsampling and downsampling operations; Convolutional layer 7: kernel size 3×3, number of kernels 1280, stride: 2, batch normalization; Activation layer 7: LeakyReLU; Convolutional layer 8: kernel size 3×3, number of kernels 1280, stride: 2, batch normalization; Activation layer 8: LeakyReLU; Upsampling layer: uses the nearest neighbor interpolation method to upsample to the same size as the pooling splicing layer; Merging layer: The output feature map of the downsampling layer is merged with the output feature map of the upsampling layer; Convolutional layer 9: kernel size 3×3, number of kernels 512, stride: 1, batch normalization; Activation layer 9: LeakyReLU; Convolution layer 10: kernel size 3×3, number of kernels 256, stride: 1, batch normalization; Activation layer 10: LeakyReLU; Convolutional layer 11: kernel size 3×3, number of kernels 128, stride: 1, batch normalization; Activation layer 11: LeakyReLU; Convolutional layer 12: kernel size 3×3, number of kernels 64, stride: 1, batch normalization; Activation layer 12: LeakyReLU; Output layer: Type: Softmax, Number of neurons: 4.
Citation Information
Patent Citations
Recognition method for inhaul cable wire breakage based on vibration signals
CN109682460A
Bridge cable wire breakage signal identification method and system based on long short-term memory network
CN114595733A
Cable structure appearance detection method based on unmanned aerial vehicle
CN109342439A
YOLOv6-based bridge disease inspection method and system
CN117151680A
Power transmission equipment target identification system based on unmanned aerial vehicle inspection
CN117369486A
Cited By
Low-altitude unmanned aerial vehicle intelligent early warning system and method based on multi-source data fusion
CN120431774A
