Bridge appearance disease detection method based on unmanned aerial vehicle formation

Abstract

The invention relates to the technical field of unmanned aerial vehicle formation bridge appearance disease detection, in particular to an unmanned aerial vehicle formation-based bridge appearance disease detection method, which comprises the following steps of performing three-dimensional model construction on a bridge through oblique photography and three-dimensional point cloud, extracting surface information according to a bridge model, and determining boundary information; performing region division through a structure according to the bridge model; the appearance position information of the bridge is partitioned and packaged into tasks; tasks are distributed to the multiple unmanned aerial vehicles, the unmanned aerial vehicles sequentially process each block task, and the optimal path of each unmanned aerial vehicle is calculated through a genetic algorithm; the unmanned aerial vehicle scans the bridge model, a scanned image is returned, and disease detection is carried out on the bridge appearance; according to the invention, damage detection is carried out by using the neural network detection technology of cracks, spalling and bending, and the problem of long working time of single unmanned aerial vehicles in a long-working-time and large-scale scene is solved through unmanned aerial vehicle formation cooperation.

Description

Technical Field

[0001] The present invention relates to the technical field of bridge appearance defect detection by unmanned aerial vehicle formations, and in particular to a method for bridge appearance defect detection based on unmanned aerial vehicle formations. Background Art

[0002] With the development of drone technology, drones have begun to emerge in all walks of life. With the continuous development of science and technology, drones have also been constantly updated and improved in the surveying and mapping engineering industry. The use of drone remote sensing technology can effectively collect and monitor data and realize data sharing. Drone remote sensing technology has the advantages of improving data monitoring efficiency, accelerating information processing, improving aviation and camera resolution, expanding monitoring range and improving sensor accuracy. At this stage, drone remote sensing technology is being used more and more widely in construction engineering surveying and mapping. For example, in surveying and mapping projects, staff can process data through drone remote sensing technology and select the required data information; data can be realized through base stations; for example, in urban planning, engineering and technical personnel can cleverly use drone remote sensing technology to quickly and accurately collect terrain information of designated areas, and create high-resolution terrain maps through comprehensive data analysis from different perspectives, thereby making the city's three-dimensional terrain model closer to real-time status.

[0003] On the one hand, traditional bridge inspection methods rely on manual inspection with the help of ladders, and inspectors face safety risks such as falling and collision when working in dangerous areas; bridge inspection vehicles may occupy the driving lane, affecting the normal operation of traffic and easily causing traffic accidents, and their purchase, maintenance and use costs are very high, and they also need to be equipped with professional operators and maintenance personnel; on the other hand, traditional manual inspection is affected by external factors, the observation results are inaccurate, and it is difficult to detect tiny cracks, rust, peeling and other diseases, and the detection accuracy is limited; in comparison, single drone operations also have their shortcomings; on the one hand, for larger inspection tasks, single drones need to start and stop multiple times in a cycle When the operation lasts for a long time and the battery backup is insufficient, waiting for charging will further delay the progress of the inspection, which is extremely disadvantageous in scenarios with long operation time and large scale. On the other hand, the endurance and load capacity of a single UAV are limited, and it is difficult to cover the entire large bridge or complex structure bridge in one flight. The operation can only be carried out in different areas and time periods, which increases the complexity and uncertainty of the inspection. In addition, a single UAV may be disturbed by factors such as airflow and magnetic field during flight, resulting in the images taken or the data collected being unstable and inaccurate, and the inspection can only be carried out from a single angle, which may miss the disease information of some key parts.

[0004] Therefore, in order to address the problem that a single UAV may be disturbed by factors such as airflow and magnetic field during flight, resulting in the captured images or collected data being unstable and inaccurate, and the disease information of some key parts may be missed, a method for bridge appearance disease detection based on UAV formation can be designed. Summary of the invention

[0005] In order to overcome the problem that a single UAV may be disturbed by factors such as airflow and magnetic field during flight, resulting in the captured images or collected data being unstable and inaccurate, and the disease information of some key parts may be missed.

[0006] The technical solution of the present invention is: a method for detecting bridge appearance defects based on a drone formation, comprising the following steps:

[0007] S1: Modeling the bridge: S1.1: Construct a 3D model of the bridge through oblique photography and 3D point cloud, extract surface information and determine boundary information based on the bridge model;

[0008] S1.2: Divide the area by structure according to the bridge model, subdivide the divided area and set the weight;

[0009] S2: Divide the appearance location information of the bridge into blocks and package them into tasks: Path the location information

[0010] Block division of locations;

[0011] S3: The tasks are assigned to multiple drones, which process each block task in turn, and the optimal path for each drone is calculated through a genetic algorithm;

[0012] S3.1: Build a graph model: Create a complete graph G(V,E); build a coding scheme and initialize the population;

[0013] S3.2: Define the usage function and define the genetic operation;

[0014] S3.3: perform an iterative optimization process;

[0015] S4: The drone scans the bridge model, transmits the scanned image back, and performs disease detection on the bridge appearance.

[0016] Preferably, V is the set of all nodes (including ground control stations and mission blocks), and E is the set of all edges.

[0017] Preferably, the boundary processing is to load and display it through three-dimensional map software and calculate the latitude and longitude position information of the Cartesian coordinate system to calculate the minimum approachable boundary of the bridge; through point cloud and high-precision images, the surface information of the bridge is extracted to provide a basis for regional division.

[0018] As a preference, based on the loading and display of the three-dimensional map software, select AR glasses or tablet devices to conduct real-time detection of the flight trajectory of the drone formation, the captured images of pests and diseases, and the three-dimensional model of the bridge; and integrate multi-source information such as high-precision images taken by the drone, point cloud data, historical inspection data, bridge design data, and ground monitoring equipment data.

[0019] As a preference, the divided structural units are subdivided and weights are set, and the inspection area is reasonably subdivided according to factors such as the geometric information of the bridge, the type of disease, and the detection requirements. Based on the bridge design and historical data, known disease-prone areas or high-risk areas are marked, and these areas are given higher detection weights.

[0020] Preferably, the surface feature extraction algorithm in computer vision can be used to extract the geometric features of the bridge surface to help identify the boundaries of different structural parts of the bridge.

[0021] Preferably, the blocks of the path position are divided into block information by dividing the scanned surface range and the position information.

[0022] As a preferred method, high-resolution thermal imaging sensors are used to detect the temperature distribution and appearance defects of the bridge structure, and multispectral sensors are used to obtain image information in different bands; integrated laser radars and infrared sensors are installed on the drone to ensure autonomous obstacle avoidance and navigation of the drone, so that the drone can automatically adjust its flight path to avoid bridge obstacles; symmetric encryption algorithm ASE or asymmetric encryption algorithm RSA and identity authentication mechanism are used to strengthen the communication security between the drone and the ground control station to prevent data from being stolen or tampered with. The specific formula is as follows:

[0023] The encryption formula of the symmetric encryption algorithm AS is:

[0024] C=E(K,P)

[0025] C is the ciphertext, i.e. the encrypted data, E is the encryption function, K is the key, and P is the plaintext, i.e. the data to be encrypted;

[0026] Decryption formula of symmetric encryption algorithm AS:

[0027] P=D(K,C)

[0028] D is the decryption function;

[0029] The encryption formula of the asymmetric encryption algorithm RSA is:

[0030] C=M^emodn

[0031] C is the ciphertext, M is the plaintext, and M is an integer less than n, e is the encryption exponent in the public key, n is the modulus in the public key, which is the product of two large prime numbers p and q;

[0032] Decryption formula of asymmetric encryption algorithm RSA:

[0033] M=C^dmodn

[0034] d is the decryption exponent in the private key.

[0035] As a preferred method, after the bridge pest and disease condition image is acquired, a convolutional neural network learning algorithm is used to extract and identify the image, and the convolution layer kernel extracts local features of the input image. The specific formula is as follows:

[0036] s(i,j)=(f*g)(i,j)=Σ m ∑ n f(m,n)g(im,jn)

[0037] s(i, j) is the pixel value of the output image at position (i, j) after convolution, and f(m, n) is the pixel value of the input image at position m, n ) The pixel value, g ( im,jn ) is the convolution kernel at position ( im,jn ) The weight value of

[0038] After the convolution operation, an activation function is used to increase the nonlinear expression ability of the network. The specific formula is as follows:

[0039] f ( x ) =max ( 0,x)

[0040] X is the output value of the convolution operation, and f(x) is the output value after being processed by the ReLU activation function;

[0041] After the activation function, the pooling operation is used to reduce the dimension of the feature map, reduce the amount of calculation and prevent overfitting. The specific formula is as follows:

[0042]

[0043] s ( i, j ) is the output value after pooling, f ( m,n ) is the pixel value of the input feature map in the pooling window R, m, n are the positions in the pooling window;

[0044] After the pooling operation, the neurons in the previous layer are connected through the fully connected layer. The specific formula is as follows:

[0045]

[0046] z represents the output value of the fully connected layer, w i Represents the weight value, x i represents the output value or input feature of the previous layer, and b represents the bias term.

[0047] The beneficial effects of the present invention are as follows: the position information of oblique photography bound with images and point clouds is used to fuse and eliminate the generated model to construct a fine three-dimensional model. At the same time, the neural network detection technology of cracks, peeling and bending is used for damage detection. Through the coordination of drone formations, the problem of long operation time of a single drone in scenes with long operation time and large scale is solved. DETAILED DESCRIPTION

[0048] The present invention is further described below in conjunction with embodiments.

[0049] The present invention provides an embodiment: a method for detecting bridge appearance defects based on a drone formation.

[0050] A three-dimensional model of the bridge is constructed through oblique photography and three-dimensional point clouds. Surface information is extracted and boundary information is determined based on the bridge model. Areas are divided according to the structure of the bridge model, and the divided areas are subdivided and weights are set. The appearance location information of the bridge is divided into blocks and packaged into tasks: the location information is divided into blocks of path locations; tasks are assigned to multiple drones, which process each block task in turn, and the optimal path for each drone is calculated through a genetic algorithm. The drone scans the bridge model, transmits the scanned image back, and performs disease detection on the bridge appearance.

[0051] Preferably, V is the set of all nodes (including ground control stations and mission blocks), and E is the set of all edges.

[0052] Preferably, the boundary processing is to load and display it through three-dimensional map software and calculate the latitude and longitude position information of the Cartesian coordinate system to calculate the minimum approachable boundary of the bridge; through point cloud and high-precision images, the surface information of the bridge is extracted to provide a basis for regional division.

[0053] As a preference, based on the loading and display of the three-dimensional map software, select AR glasses or tablet devices to conduct real-time detection of the flight trajectory of the drone formation, the captured images of pests and diseases, and the three-dimensional model of the bridge; and integrate multi-source information such as high-precision images taken by the drone, point cloud data, historical inspection data, bridge design data, and ground monitoring equipment data.

[0054] As a preference, the divided structural units are subdivided and weights are set, and the inspection area is reasonably subdivided according to factors such as the geometric information of the bridge, the type of disease, and the detection requirements. Based on the bridge design and historical data, known disease-prone areas or high-risk areas are marked, and these areas are given higher detection weights.

[0055] Preferably, the surface feature extraction algorithm in computer vision can be used to extract the geometric features of the bridge surface to help identify the boundaries of different structural parts of the bridge.

[0056] Preferably, the blocks of the path position are divided into block information by dividing the scanned surface range and the position information.

[0057] As a preferred method, high-resolution thermal imaging sensors are used to detect the temperature distribution and appearance defects of the bridge structure, and multispectral sensors are used to obtain image information in different bands; integrated laser radars and infrared sensors are installed on the drone to ensure autonomous obstacle avoidance and navigation of the drone, so that the drone can automatically adjust its flight path to avoid bridge obstacles; symmetric encryption algorithm ASE or asymmetric encryption algorithm RSA and identity authentication mechanism are used to strengthen the communication security between the drone and the ground control station to prevent data from being stolen or tampered with. The specific formula is as follows:

[0058] The encryption formula of the symmetric encryption algorithm AS is:

[0059] C=E(K,P)

[0060] C is the ciphertext, i.e. the encrypted data, E is the encryption function, K is the key, and P is the plaintext, i.e. the data to be encrypted;

[0061] Decryption formula of symmetric encryption algorithm AS:

[0062] P=D(K,C)

[0063] D is the decryption function;

[0064] The encryption formula of the asymmetric encryption algorithm RSA is:

[0065] C=M^emodn

[0066] C is the ciphertext, M is the plaintext, and M is an integer less than n, e is the encryption exponent in the public key, n is the modulus in the public key, which is the product of two large prime numbers p and q;

[0067] Decryption formula of asymmetric encryption algorithm RSA:

[0068] M=C^dmodn

[0069] d is the decryption exponent in the private key.

[0070] Example 1

[0071] When working, first a large drone equipped with high-precision sensors (such as high-resolution cameras, lidar or multispectral cameras) conducts a detailed scan of the bridge structure; the drone flies at a fixed altitude and a predetermined route to fully obtain high-density point cloud data and structural feature information on the bridge surface; then, multi-angle oblique photography technology is used to take high-resolution images of the bridge from different directions; these image data are combined with the point cloud data obtained by laser scanning to generate a three-dimensional model of the bridge, and based on the constructed bridge model, the geometric information and key parts of the bridge, such as the main beam, piers, guardrails, supporting structures, etc. are extracted; the surface feature extraction algorithm in computer vision is used to extract the geometric features of the bridge surface to help identify the boundaries of different structural parts of the bridge; on this basis, the three-dimensional map is used The software loads and displays and calculates the coordinates of the Cartesian coordinate system by itself, obtains the minimum boundary information that can be close to the bridge, converts the boundary information into the longitude and latitude information of the real world, configures the size range that the drone can scan, that is, the cone range of the drone, and divides the task blocks according to the cone range. On the basis of the loading and display of the 3D map software, select AR glasses or tablet devices to conduct real-time detection of the flight trajectory of the drone formation, the images of pests and diseases captured, and the 3D model of the bridge; integrates multi-source information such as high-precision images taken by drones, point cloud data, historical detection data, bridge design data, and ground monitoring equipment data, and uses high-resolution thermal imaging sensors to detect the temperature distribution and appearance defects of the bridge structure, and adopts multi-spectral sensors to obtain image information in different bands.

[0072] Example 2

[0073] Installing integrated lidar and infrared sensors on the drone ensures autonomous obstacle avoidance and navigation, so that the drone can automatically adjust its flight path to avoid bridge obstacles; using genetic algorithms to calculate the optimal path for each drone: the first step is to build a graph model: create a complete graph G(V,E); the second step is to build a coding scheme and initialize the population: the flight path of the drone is represented by the vertex sequence in the complete graph, and these sequences are used as the coding form to form the initial population; define the fitness function; when dealing with the problem of multiple drones performing flight missions, the total flight time is used as the core optimization target of the fitness function; the third step is to define the fitness function: when dealing with the problem of multiple drones performing flight missions, we use the total flight time as the core optimization target of the fitness function; the fourth step is to define the genetic operation: use RouletteWheel Selection) method is used as the selection operator to select individuals with higher fitness from the current population as the father of the next generation; single-point crossover technology is used as the crossover operator to create a new generation of individuals, ensuring that each mission point is visited by only one drone to maintain the effectiveness and rationality of path planning; the exchange mutation method is used as the mutation operator to locally optimize the new generation of individuals generated by the crossover operation to increase the diversity within the population; the fifth step is to perform an iterative optimization process: repeatedly perform selection, crossover, and mutation operations, and update the population after each iteration to retain the individual with the best fitness until a good enough solution is found, thereby ending the iterative optimization process; the symmetric encryption algorithm ASE or the asymmetric encryption algorithm RSA and the identity authentication mechanism are used to strengthen the communication security between the drone and the ground control station to prevent data from being stolen or tampered with. The specific formula is as follows:

[0074] The encryption formula of the symmetric encryption algorithm AS is:

[0075] C=E(K,P)

[0076] C is the ciphertext, i.e. the encrypted data, E is the encryption function, K is the key, and P is the plaintext, i.e. the data to be encrypted;

[0077] Decryption formula of symmetric encryption algorithm AS:

[0078] P=D(K,C)

[0079] D is the decryption function;

[0080] The encryption formula of the asymmetric encryption algorithm RSA is:

[0081] C=M^emodn

[0082] C is the ciphertext, M is the plaintext, and M is an integer less than n, e is the encryption exponent in the public key, n is the modulus in the public key, which is the product of two large prime numbers p and q;

[0083] Decryption formula of asymmetric encryption algorithm RSA:

[0084] M=C^dmodn

[0085] d is the decryption index in the private key. After the path calculation is completed, the returned data is combined and modeled. After data preprocessing, computer vision and image processing technology are used to detect and identify diseases on the image. After the bridge disease and insect pest image is obtained, the convolutional neural network learning algorithm is used to extract and identify the image. The convolution layer kernel extracts local features of the input image. The specific formula is as follows:

[0086] s(i,j)=(f*g)(i,j)=Σ m ∑ n f(m,n)g(im,jn)

[0087] s(i, j) is the pixel value of the output image at position (i, j) after convolution, f(m, n) is the pixel value of the input image at position (m, n), and g(im, jn) is the weight value of the convolution kernel at position (im, jn);

[0088] After the convolution operation, an activation function is used to increase the nonlinear expression ability of the network. The specific formula is as follows:

[0089] f(x)=max(0,x)

[0090] X is the output value of the convolution operation, and f(x) is the output value after being processed by the ReLU activation function;

[0091] After the activation function, the pooling operation is used to reduce the dimension of the feature map, reduce the amount of calculation and prevent overfitting. The specific formula is as follows:

[0092]

[0093] s(i, j) is the output value after pooling, f(m, n) is the pixel value of the input feature map in the pooling window R, and m, n are the positions in the pooling window;

[0094] After the pooling operation, the neurons in the previous layer are connected through the fully connected layer. The specific formula is as follows:

[0095]

[0096] z represents the output value of the fully connected layer, w i Represents the weight value, x i represents the output value or input feature of the previous layer, and b represents the bias term.

[0097] The embodiments of the present invention are described in detail above, but the present invention is not limited to the above embodiments, and various changes can be made within the knowledge scope of those skilled in the art without departing from the purpose of the present invention.

Claims

1. A method for detecting bridge appearance defects based on a drone formation, comprising the following steps: S1: Model the bridge; S1.1: Construct a 3D model of the bridge through oblique photography and 3D point cloud, extract surface information and determine boundary information based on the bridge model; S1.2: Divide the area by structure according to the bridge model, subdivide the divided area and set the weight; S2: Divide the appearance location information of the bridge into blocks and package them into tasks: Divide the location information into blocks of path locations; S3: The tasks are assigned to multiple drones, which process each block task in turn, and the optimal path for each drone is calculated through a genetic algorithm; S3.1: Build a graph model: Create a complete graph G(V,E); build a coding scheme and initialize the population; S3.2: Define the usage function and define the genetic operation; S3.3: perform an iterative optimization process; S4: The drone scans the bridge model, transmits the scanned image back, and performs disease detection on the bridge appearance.

2. The method for bridge appearance defect detection based on UAV formation according to claim 1 is characterized by: V is the set of all nodes (including ground control stations and mission blocks), and E is the set of all edges.

3. The method for bridge appearance defect detection based on UAV formation according to claim 2 is characterized by: Boundary processing is to load and display it through three-dimensional map software and calculate the latitude and longitude position information of the Cartesian coordinate system to calculate the minimum approach boundary of the bridge; through point cloud and high-precision images, the surface information of the bridge is extracted to provide a basis for regional division.

4. The method for detecting bridge appearance defects based on a UAV formation according to claim 3 is characterized by: Based on the loading and display of the 3D map software, select AR glasses or tablet devices to conduct real-time detection of the flight trajectory of the drone formation, the captured images of pests and diseases, and the 3D model of the bridge; integrate multi-source information such as high-precision images taken by the drone, point cloud data, historical inspection data, bridge design data, and ground monitoring equipment data.

5. The method for bridge appearance defect detection based on UAV formation according to claim 4 is characterized in that: The divided structural units are subdivided and weights are set. The inspection area is reasonably subdivided according to factors such as the geometric information of the bridge, the type of disease and the detection requirements. According to the bridge design and historical data, known disease-prone areas or high-risk areas are marked and assigned higher detection weights.

6. The method for detecting bridge appearance defects based on a UAV formation according to claim 5 is characterized in that: The surface feature extraction algorithm in computer vision is used to extract the geometric features of the bridge surface to help identify the boundaries of different structural parts of the bridge.

7. The method for bridge appearance defect detection based on UAV formation according to claim 6 is characterized by: The block division of the path position is divided into block information by scanning the surface range and position information.

8. The method for bridge appearance defect detection based on UAV formation according to any one of claims 1 to 7, characterized in that: Use high-resolution thermal imaging sensors to detect the temperature distribution and appearance defects of the bridge structure, and use multispectral sensors to obtain image information in different bands; install integrated lidar and infrared sensors on the drone to ensure autonomous obstacle avoidance and navigation, so that the drone can automatically adjust its flight path to avoid bridge obstacles; use symmetric encryption algorithm ASE or asymmetric encryption algorithm RSA and identity authentication mechanism to strengthen the communication security between the drone and the ground control station to prevent data from being stolen or tampered with. The specific formula is as follows: The encryption formula of the symmetric encryption algorithm AS is: C=E(K,P) C is the ciphertext, i.e. the encrypted data, E is the encryption function, K is the key, and P is the plaintext, i.e. the data to be encrypted; Decryption formula of symmetric encryption algorithm AS: P=D(K,C) D is the decryption function; The encryption formula of the asymmetric encryption algorithm RSA is: C=M^emodn C is the ciphertext, M is the plaintext, and M is an integer less than n. e is the encryption exponent in the public key, and n is the modulus in the public key, which is the product of two large prime numbers p and q. Decryption formula of asymmetric encryption algorithm RSA: M=C^dmodn d is the decryption exponent in the private key.

9. The method for bridge appearance defect detection based on UAV formation according to claim 8 is characterized by: After the bridge pest and disease image is acquired, the convolutional neural network learning algorithm is used to extract and identify the image. The convolution layer kernel extracts local features of the input image. The specific formula is as follows: s(i,j)=(f*g)(i,j)=∑ m ∑ n f(m,n)g(i-m,j-n) s(i, j) is the pixel value of the output image at position (i, j) after convolution, f(m, n) is the pixel value of the input image at position (m, n), and g(im, jn) is the weight value of the convolution kernel at position (im, jn); After the convolution operation, an activation function is used to increase the nonlinear expression ability of the network. The specific formula is as follows: f(x)=max(0,x) X is the output value of the convolution operation, and f(x) is the output value after being processed by the ReLU activation function; After the activation function, the pooling operation is used to reduce the dimension of the feature map, reduce the amount of calculation and prevent overfitting. The specific formula is as follows: s ( i, j ) is the output value after pooling, f ( m,n ) is the pixel value of the input feature map in the pooling window R, m, n are the positions in the pooling window; After the pooling operation, the neurons in the previous layer are connected through the fully connected layer. The specific formula is as follows: z represents the output value of the fully connected layer, w i Represents the weight value, x i represents the output value or input feature of the previous layer, and b represents the bias term.

Images