Automatic road and bridge inspection system based on unmanned aerial vehicle

Abstract

The invention relates to the technical field of bridge inspection, in particular to a road bridge automatic inspection system based on an unmanned aerial vehicle, which comprises the following steps: determining bridge structure information of an inspected bridge, and determining a bridge type according to the bridge structure information; determining a structure inspection grade table of the bridge according to the bridge type; according to the structure inspection grade table and the unmanned aerial vehicle information, the acquisition distance and the acquisition angle of the unmanned aerial vehicle at each position during inspection are determined, so that accurate bridge inspection information can be obtained during inspection, the probability of collision between the unmanned aerial vehicle and the bridge can be reduced, and the flight safety of the unmanned aerial vehicle is improved. By means of the mode, the problems that in the bridge inspection aspect, the control ability of an operator on the unmanned aerial vehicle is relied on, the problem cannot be found in time, and an air crash event is likely to happen are solved.

Description

Technical Field

[0001] The present invention relates to the technical field of bridge inspection, and in particular to an automated road and bridge inspection system based on an unmanned aerial vehicle (UAV) system. Background Art

[0002] With the continuous development of transportation facilities, bridges are an important part of transportation roads. The safety and stability of bridges are of great significance to ensuring people's lives and safety. The traditional bridge inspection method uses regular manual inspections to conduct bridge inspections, but this method has the problems of low detection efficiency and low detection accuracy, and it is difficult to achieve comprehensive inspections of all parts of the bridge. Therefore, drone automated inspections are currently widely used to assist in bridge inspections.

[0003] When drones are used to inspect roads and bridges, the stability of bridge slopes is directly related to traffic safety and the lifespan of infrastructure. Slope instability can trigger landslides and collapses, threatening the safety of bridge structures and disrupting traffic. Furthermore, if slope defects (such as cracks and soil erosion) are not detected promptly, they can develop into serious geological disasters, resulting in costly repairs.

[0004] Currently, drones for bridge inspections rely heavily on the operator's control capabilities, making them unable to detect problems promptly. This can easily lead to crashes, increasing operating costs, and lacking a drone control system for automated bridge inspections. Furthermore, the massive amount of high-precision bridge image data collected by drones requires distortion correction, 3D modeling, and defect identification. This data processing is complex, relying on manual analysis, and suffers from inefficient algorithms (such as shadows and vegetation interference). This leads to slow analysis speeds and inefficiencies. Furthermore, due to the lack of targeted analysis of different bridge structures, key locations are often overlooked when collecting bridge data, or the imagery is not accurately captured.

[0005] Therefore, the present invention provides an automated road and bridge inspection system based on drones to solve the above problems. Summary of the Invention

[0006] In view of the above situation, in order to overcome the defects of the existing technology, the present invention provides an automated road and bridge inspection system based on drones to solve the problem that the above-mentioned drones are highly dependent on the operator's control ability in bridge inspection, cannot detect problems in time, and are prone to crashes.

[0007] In order to achieve the above object, the technical solution adopted by the present invention is:

[0008] An automated road and bridge inspection system based on drones, comprising:

[0009] a bridge information acquisition module, which acquires bridge location information, and uses a first drone to acquire first image data of the bridge, and uses a second drone to acquire second image data of the bridge;

[0010] a bridge model construction module, forming a first bridge model based on the first image data; performing a matching analysis between the first bridge model and a preset bridge template model to determine a classification model for the first bridge model; adjusting the classification model based on the first image data to obtain a second bridge model; and determining a structural inspection grade table formed by key inspection information based on the second bridge model;

[0011] The flight path planning module determines the collection distance and angle of the second UAV inspection based on the structure inspection level table and UAV information; determines the inspection flight area based on the collection distance and angle; and uses a hybrid particle swarm algorithm and ant colony algorithm to optimize and analyze the inspection flight area, inspection structure, collection distance, and angle to determine the inspection planning path for the second UAV.

[0012] The bridge status analysis module performs a damage analysis on the second image data based on the second bridge model to determine the damage information; adjusts the structure inspection level table according to the damage information to obtain a dynamic structure inspection level table; and sends repair information to the supervisor when the damage information meets the repair conditions.

[0013] Preferably, the forming of the first bridge model based on the first image data includes: preprocessing the first image data, and performing image segmentation on the preprocessed first image data using the Otsu algorithm to obtain a bridge structure image; determining coordinate information based on the bridge structure image; structurally splicing the coordinate information and the bridge structure image based on a two-dimensional coordinate system to form multiple initial first bridge models; performing pixel difference analysis on each initial first bridge model, and taking the initial first bridge model with the largest number of pixel change values ​​greater than a preset value as the first bridge model.

[0014] Preferably, the matching analysis of the first bridge model and the preset bridge template model to determine the classification model of the first bridge model includes: scaling or enlarging the first bridge model to align the first bridge model and the preset bridge template model; performing similarity analysis on the pixel information of the first bridge model and the pixel information of the preset bridge template model to determine the matching value; and determining the classification model of the first bridge model based on the matching value.

[0015] Preferably, the adjusting the classification model according to the first image data to obtain the second bridge model includes: determining the corresponding surface of each initial first bridge model in the classification model according to the bridge structure relationship between each initial first bridge model and each surface of the classification model; and mapping the bridge structure characteristics of each initial first bridge model into the classification model based on the classification model to obtain the second bridge model.

[0016] Preferably, the method of determining the acquisition distance and acquisition angle of the second drone inspection according to the structure inspection level table and drone information includes: determining the acquisition angle and the maximum acquisition distance according to the detection accuracy corresponding to the structure inspection level, the camera parameters and the inclination angle of the structure surface; the maximum acquisition distance D max The calculation formula is:

[0017]

[0018] θ=-α,

[0019] Among them, L is the detection accuracy, P width is the number of horizontal pixels, α is the tilt angle of the structure surface detected in real time, FOV h is the horizontal field of view angle, and θ is the acquisition angle.

[0020] Preferably, the maximum acquisition distance is determined according to the detection accuracy, camera parameters and structural surface inclination angle corresponding to the structure inspection level, including: determining the field of view width according to the acquisition distance and camera parameters; determining the ground sampling distance according to the field of view width and camera parameters; and determining the maximum acquisition distance according to the constraint relationship between the ground sampling distance and the detection accuracy.

[0021] Preferably, the hybrid particle swarm algorithm and ant colony algorithm are used to optimize and analyze the inspection flight area, inspection structure, collection distance and collection angle to determine the inspection planning path of the second UAV, including: determining the inspection point information and constraint information; constructing a fitness function based on the inspection point information and constraint information; initializing the encoding of particles and initializing the pheromone matrix based on the inspection information; adjusting the speed update formula in the particle swarm optimization algorithm according to the pheromone in the ant colony algorithm; updating the particle speed and position, and calculating the fitness according to the fitness function; constructing a new path based on the pheromone and heuristic function, and updating the pheromone matrix; using the global optimal solution of the particle swarm optimization algorithm to correct the pheromone distribution of the ant colony algorithm and using the high-intensity pheromone of the ant colony algorithm to guide particle movement; when the maximum number of iterations is reached, the inspection planning path is determined based on the particle information corresponding to the fitness.

[0022] Preferably, the damage analysis of the second image data based on the second bridge model to determine the damage information includes: aligning the coordinates of the second bridge model and each second image data; for single image data, using the Canny edge detection method and the morphological skeleton extraction method to obtain crack features; using the RGB threshold segmentation method and the GLCM texture analysis method to obtain corrosion features; using the SIFT feature matching method and the optical flow displacement calculation method to obtain deformation features; determining the damage level based on the area and position of the crack features, corrosion features and deformation features; and integrating the crack features, corrosion features, deformation features and the damage level to obtain the damage information.

[0023] Preferably, when the damage information meets the repair conditions, the repair information is sent to the supervisor, including: when the damage area of ​​the damage structure in the damage information is greater than the preset area, or when the damage depth of the damage structure in the damage information is greater than the preset depth, generating corresponding first repair information; when the damage position of the damage structure in the damage information belongs to the preset key position, generating corresponding second repair information; and sending the repair information obtained by integrating the first repair information and the second repair information to the supervisor.

[0024] Preferably, the flight path planning module further includes: adjusting the inspection planning path according to the dynamic structure inspection level table to obtain a revised inspection planning path.

[0025] The beneficial effects of the present invention are:

[0026] 1. The present invention can determine the structural information of the inspected bridge, determine the bridge type based on the structural information, determine the structural inspection grade table of the bridge based on the bridge type, and determine the collection distance and angle at each location during the drone inspection based on the structural inspection grade table and drone information. This ensures accurate bridge inspection information is obtained during the inspection, reduces the probability of collision between the drone and the bridge, and improves the safety of drone flight. Through the above-mentioned approach, the present invention relies heavily on the operator's ability to control the drone in terms of bridge inspection, and cannot detect problems in a timely manner, which is prone to crashes.

[0027] 2. Secondly, after analyzing the acquired images obtained by the drone, determine the damage location, damage level and other damage information of the bridge; adjust the structural inspection level table according to the damage information to obtain a dynamic structural inspection level table for inspection that conforms to the current bridge status; and control the collection distance and collection angle of the second drone according to the dynamic structural inspection level table. In this way, the present invention can set different collection distances and collection angles for different bridge structures, so that when processing the obtained second image data, it can quickly perform image analysis and determine the damage status information of the bridge. When the damage level meets the repair level, upload repair information including the bridge name, bridge location, damage structure, damage location, damage range, and repair materials so that maintenance personnel can perform emergency maintenance to ensure safe passage of the bridge.

[0028] 3. When planning the flight path of the second drone, the present invention uses a method to determine the second drone's inspection collection distance and angle based on the structural inspection level table and drone information; determine the inspection flight area based on the collection distance and angle; and use a hybrid particle swarm algorithm and ant colony algorithm to optimize and analyze the inspection flight area, inspection structure, collection distance, and angle to determine the second drone's inspection planning path. Through this method, the present invention can obtain the optimal flight path information for collecting images of the bridge's key structures under the aforementioned constraints, allowing for accurate acquisition of bridge image data while avoiding obstacles. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 This is a schematic diagram of an automated road and bridge inspection system based on drones according to the present invention. DETAILED DESCRIPTION

[0030] The following will refer to the attached Figure 1 The embodiments of the present invention are described in detail. It should be understood by those skilled in the art that these embodiments are only used to explain the technical principles of the present invention and are not intended to limit the scope of protection of the present invention.

[0031] The automatic inspection system for roads and bridges based on drones is shown in the attached Figure 1 Shown, including:

[0032] The bridge information acquisition module obtains the bridge location information, and uses a first drone to obtain first image data of the bridge, and uses a second drone to obtain second image data of the bridge.

[0033] Among them, the first drone is used to collect the first image data formed by the general structural data of the bridge for the first time, so as to determine the structural model of the bridge, that is, the classification model of the bridge after analyzing the first image data; various types of classification models are stored in a model collection library of three-dimensional structural models corresponding to the preset bridge template model; the classification models include simply supported beam bridges, cantilever beam bridges, stone arch bridges, reinforced concrete arch bridges, suspension bridges, and cable-stayed bridges.

[0034] The bridge model construction module forms a first bridge model based on the first image data; performs a matching analysis between the first bridge model and a preset bridge template model to determine a classification model for the first bridge model; adjusts the classification model based on the first image data to obtain a second bridge model; and determines a structural inspection level table formed by key inspection information based on the second bridge model.

[0035] The preset bridge template models are two-dimensional model images that highlight the most bridge features of various bridges, so as to perform comparative analysis with the first bridge model and determine the classification model of the first bridge model. The classification model is a three-dimensional model image corresponding to each preset bridge template model.

[0036] Preferably, the key inspection information includes the pavement layer of the driving lane, expansion joints, drainage system, guardrails, main beams, slopes, bridge decks, supports, piers, abutments, etc.

[0037] Specifically, by identifying and processing the first image data, the classification model of the bridge in the first image data is determined; then, based on the classification model of the bridge, the structural inspection level table corresponding to the bridge in the first image data and sorted by structural level is determined so as to carry out subsequent processing steps.

[0038] The flight path planning module determines the maximum collection distance and collection angle of the second UAV inspection based on the structural inspection level table and UAV information; determines the inspection flight area based on the maximum collection distance and safe obstacle avoidance distance; and uses a hybrid particle swarm algorithm and ant colony algorithm to optimize and analyze the inspection flight area, inspection structure, maximum collection distance and collection angle to determine the inspection planning path of the second UAV.

[0039] Specifically, the present invention determines the maximum acquisition distance and acquisition angle of the camera carried by the second drone during the inspection through the inspection level and drone information in the structural inspection level table of the bridge; and then uses a hybrid particle swarm algorithm and an ant colony algorithm to optimize and analyze the inspection flight area, inspection structure, maximum acquisition distance and acquisition angle, so as to determine the optimal inspection planning path, which helps to reasonably collect images of the key structures of the bridge, and then helps to quickly analyze the status of the bridge later.

[0040] The bridge status analysis module performs a damage analysis on the second image data based on the second bridge model to determine the damage information; adjusts the structure inspection level table according to the damage information to obtain a dynamic structure inspection level table; and sends repair information to the supervisor when the damage information meets the repair conditions.

[0041] The repair information includes the bridge name, bridge location, damaged structure, damage location, repair materials, etc. The second image data is processed through image recognition using the second bridge model to determine the specific location and level of the bridge damage, thereby obtaining damage information for each key structure of the current bridge.

[0042] Specifically, the present invention can determine the bridge structure information of the inspected bridge, determine the bridge type based on the bridge structure information; determine the structural inspection level table of the bridge based on the bridge type; determine the collection distance and collection angle at each position during the drone inspection based on the structural inspection level table and drone information, and can ensure the acquisition of accurate bridge inspection information during the inspection. At the same time, it can reduce the probability of collision between the drone and the bridge, and improve the safety of the drone flight.

[0043] After analyzing the acquired images obtained by the drone, the damage information such as the location and level of the bridge is determined; the structural inspection level table is adjusted according to the damage information to obtain a dynamic structural inspection level table for inspection that conforms to the current state of the bridge; the collection distance and collection angle of the second drone are controlled according to the dynamic structural inspection level table. In this way, the present invention can set different collection distances and collection angles for different bridge structures, so that when processing the obtained second image data, image analysis can be quickly performed to determine the damage status information of the bridge. When the damage level meets the repair level, the repair information including the bridge name, bridge location, damage structure, damage location, damage range, and repair materials is uploaded so that maintenance personnel can perform emergency maintenance and ensure the safe passage of the bridge.

[0044] In one embodiment of the present invention, a first bridge model is formed based on first image data, including: preprocessing the first image data, and performing image segmentation on the preprocessed first image data using the Otsu algorithm to obtain a bridge structure image; determining coordinate information based on the bridge structure image; structurally splicing the coordinate information and the bridge structure image based on a two-dimensional coordinate system to form multiple initial first bridge models; performing pixel difference analysis on each initial first bridge model, and taking the initial first bridge model with the largest number of pixel change values ​​greater than a preset value as the first bridge model.

[0045] The preset value can be 50-80, and the specific value can be adjusted according to actual conditions.

[0046] The preprocessing process includes scaling the image according to the acquisition distance of the first image data to form image data of uniform scale, so that after image segmentation, a two-dimensional first bridge model can be formed according to the coordinate information and the shooting angle of the image.

[0047] During image segmentation, the first image data is separated into foreground and background using the adaptive threshold in the Otsu algorithm to obtain the bridge structure image in the first image data. This is primarily based on the separation of the bridge slope from background elements such as vegetation, soil, and river water, resulting in a bridge structure image bounded by the slope and the bridge edge structure. Then, based on a two-dimensional coordinate system, the coordinate information and the edges of the bridge structure image are structurally spliced ​​to form initial first bridge models for the top, front, back, left, and right directions of the bridge. Of these five initial first bridge models, the one with the most features and two-dimensional form is selected as the first bridge model. This first bridge model is then compared and analyzed with a preset bridge template model to determine the classification model of the bridge in the first image data, which in turn facilitates subsequent analysis.

[0048] In one embodiment of the present invention, a matching analysis is performed on the first bridge model and the preset bridge template model to determine the classification model of the first bridge model, including: scaling or enlarging the first bridge model to align the first bridge model and the preset bridge template model; performing similarity analysis on the pixel information of the first bridge model and the pixel information of the preset bridge template model to determine the matching value; and determining the classification model of the first bridge model based on the matching value.

[0049] Specifically, the first bridge model is scaled or enlarged, and the first bridge model and the preset bridge template model are aligned to facilitate subsequent pixel value difference analysis; when performing similarity analysis on the pixel information of the first bridge model and the pixel information of the preset bridge template model, the first bridge model and the preset bridge template model are divided into multiple blocks; the overall matching value of the first bridge model and the preset bridge template model is determined based on the average value of the pixel difference values ​​of each block, so as to improve the data processing speed by adopting distributed parallel processing; and the classification model of the first bridge model in each preset bridge template model is determined based on the maximum value of each matching value.

[0050] Through the configuration of this embodiment, the present invention can determine the classification model of the first bridge model according to the matching result between the first bridge model and the preset bridge template model, so as to provide a data basis for subsequent processing and analysis.

[0051] In one embodiment of the present invention, the classification model is adjusted according to the first image data to obtain the second bridge model, including: determining the corresponding surface of each initial first bridge model in the classification model according to the bridge structure relationship between each initial first bridge model and each surface of the classification model; and mapping the bridge structure characteristics of each initial first bridge model into the classification model based on the classification model to obtain the second bridge model.

[0052] Specifically, according to the correspondence between each initial first bridge model and the bridge structure in the upper, front, rear, left and right surfaces of the classification model, the corresponding surface to which each initial first bridge model belongs in the classification model is determined; based on the classification model, the bridge structure characteristics of each initial first bridge model are added to the three-dimensional classification model to obtain a second bridge model of part of the current bridge state.

[0053] Through the setting method of this embodiment, the present invention, based on the acquired classification model, makes detailed adjustments to the classification model according to the bridge structure image in the first image data, obtains a second bridge model, and preliminarily constructs a three-dimensional bridge structure model under the current circumstances; so as to facilitate subsequent analysis of the bridge status information.

[0054] Furthermore, a structural inspection level table formed by key inspection information is determined based on the second bridge model, including: determining the corresponding structural inspection level table in the preset inspection structure database based on the classification model of the second bridge model; the structural inspection level table contains key inspection information that requires key inspection for this type of bridge; the key inspection information includes expansion joints, drainage systems, guardrails, main beams, slopes, bridge decks, supports, piers, abutments, etc.

[0055] In one embodiment of the present invention, the acquisition distance and acquisition angle of the second drone inspection are determined according to the structure inspection level table and the drone information, including: determining the acquisition angle and the maximum acquisition distance according to the detection accuracy corresponding to the structure inspection level, the camera parameters and the inclination angle of the structure surface; the maximum acquisition distance D max The calculation formula is:

[0056]

[0057] θ=-α,

[0058] Among them, L is the detection accuracy, P width is the number of horizontal pixels, α is the tilt angle of the structure surface detected in real time, FOV h is the horizontal field of view angle, and θ is the acquisition angle.

[0059] In this embodiment, the structural inspection level table contains the inspection levels corresponding to each bridge structure; the drone information includes the camera parameters of the camera, namely the horizontal field of view FOV h and the horizontal pixel number Pwidth ; L is the minimum detectable accuracy corresponding to the inspection level, α is the inclination angle of the structure surface detected in real time, and θ is the acquisition angle; for different inclined areas of the bridge (such as cables and arch ribs), the maximum acquisition distance and acquisition angle need to be calculated in sections.

[0060] Furthermore, in one embodiment of the present invention, the maximum acquisition distance is determined based on the detection accuracy, camera parameters and inclination angle of the structure surface corresponding to the structure inspection level, including: determining the field of view width based on the acquisition distance and camera parameters; determining the ground sampling distance based on the field of view width and camera parameters; and determining the maximum acquisition distance based on the constraint relationship between the ground sampling distance and the detection accuracy.

[0061] Among them, the field of view width W surface The calculation formula is:

[0062]

[0063] Where D is the acquisition distance, FOV h is the horizontal field of view angle, and α is the dynamically changing inclination angle of the structure surface.

[0064] The calculation formula of the ground sampling distance G is:

[0065]

[0066] Among them, W surface is the field of view width, P width is the number of horizontal pixels;

[0067] According to the constraint relationship between ground sampling distance and detection accuracy L: G≤L, the maximum acquisition distance D is obtained. max Then, the inspection flight area is determined based on the maximum acquisition distance and the safe obstacle avoidance distance S; the height range of the inspection flight area is [S,D max ], acquisition angle θ = -α, horizontal direction: the UAV projection needs to be within the range of the bridge surface (to avoid offset that causes invalid ground sampling distance).

[0068] Through the setting method of this embodiment, the present invention can specifically calculate the maximum acquisition distance based on the detection accuracy, camera parameters and structural surface inclination angle corresponding to the structural inspection level; and then can determine the inspection flight area of ​​the UAV, so as to subsequently optimize and analyze the inspection flight area, inspection structure, acquisition distance and acquisition angle through a hybrid particle swarm algorithm and an ant colony algorithm, determine the inspection planning path of the second UAV, and complete accurate data collection.

[0069] In one embodiment of the present invention, a hybrid particle swarm algorithm and an ant colony algorithm are used to optimize and analyze the inspection flight area, inspection structure, collection distance and collection angle to determine the inspection planning path of the second UAV, including: determining the inspection point information and constraint information; constructing a fitness function based on the inspection point information and constraint information; initializing the encoding of particles and initializing the pheromone matrix based on the inspection information; adjusting the speed update formula in the particle swarm optimization algorithm based on the pheromone in the ant colony algorithm; updating the particle speed and position, and calculating the fitness based on the fitness function; constructing a new path based on the pheromone and the heuristic function, and updating the pheromone matrix; using the global optimal solution of the particle swarm optimization algorithm to correct the pheromone distribution of the ant colony algorithm and using the high-intensity pheromone of the ant colony algorithm to guide particle movement; when the maximum number of iterations is reached, determining the inspection planning path with the particle information corresponding to the fitness.

[0070] Specifically, the inspection point information and constraint information are determined; the inspection point information includes the information of each collection point of the second UAV around the bridge, including the collection distance, collection angle and three-dimensional coordinate position; the constraint information includes the constraint range information of the collection distance, and the angle constraint information of the collection angle perpendicular to the surface of the bridge structure; the distance between adjacent inspection points needs to meet the UAV kinematic constraint information, such as the minimum turning path.

[0071] The fitness function is constructed based on the inspection point information and constraint information; the fitness function f(X) is:

[0072]

[0073] Among them, L total is the minimum total path length, D i is the actual collection distance of the i-th inspection point, D max is the maximum allowed height, S is the safe obstacle avoidance distance, θ i is the collection angle of the i-th inspection point, α i is the structural surface inclination angle of the i-th inspection point, λ D ,λ θ It is a penalty coefficient that needs to be adjusted according to the strictness of the constraints. Too large a value may lead to unstable solutions. It is used to suppress solutions that violate the constraints.

[0074] The particles are initialized and encoded according to the inspection information, and the pheromone matrix is ​​initialized; each particle position represents a candidate path, that is, a set of n inspection points arranged in sequence.

[0075] According to the pheromone in the ant colony algorithm, the speed update formula in the particle swarm optimization algorithm is adjusted; the speed update formula of the i-th particle is:

[0076]

[0077] in, is the updated speed, ω is the inertia weight, a larger value enhances the global search ability, and a smaller value promotes local convergence. is the speed before updating, c1 and c2 are learning factors, r1 and r2 are random numbers, and p besti is the historical optimal position of the particle, g best is the global optimal position, is the particle position before updating, γ is the weight coefficient, is the pheromone concentration at the particle position, d preferred is the direction vector of the neighborhood with the highest pheromone concentration.

[0078] The position update formula is:

[0079] in, is the updated particle position, is the particle position before updating, The updated speed.

[0080] Pheromone concentration τ at path (i, j) ij The update formula is:

[0081] Where ρ is the volatility rate; a high volatility rate prevents premature convergence, while a low volatility rate enhances path utilization. Q is a constant, and f(X) is the path fitness. Furthermore, pheromone updates in the ant colony algorithm prioritize the determination of the particle's historical optimal position and the global optimal position in the particle swarm algorithm to enhance high-quality paths.

[0082] Update the particle speed and position, and calculate the fitness according to the fitness function; construct a new path based on pheromones and heuristic functions, and update the pheromone matrix; use the global optimal solution of the particle swarm optimization algorithm to correct the pheromone distribution of the ant colony algorithm and use the high-intensity pheromone of the ant colony algorithm to guide particle movement; when the maximum number of iterations is reached, determine the inspection planning path based on the particle information corresponding to the fitness.

[0083] Through the above-mentioned approach, the present invention utilizes a hybrid particle swarm algorithm and an ant colony algorithm to optimize and analyze the inspection flight area, inspection structure, acquisition distance, and acquisition angle, determining the planned inspection path for the second drone to accurately capture image data corresponding to each bridge structure, thereby enabling rapid identification of bridge damage information in subsequent steps. Furthermore, through the above-mentioned method, the present invention is able to obtain optimal flight path information for acquiring images of key bridge structures within the aforementioned constraints, enabling accurate acquisition of bridge image data while avoiding obstacles.

[0084] In one embodiment of the present invention, a damage analysis is performed on the second image data based on the second bridge model to determine damage information, including: aligning the coordinates of the second bridge model and each second image data; for single image data, obtaining crack features using a Canny edge detection method and a morphological skeleton extraction method; obtaining corrosion features using an RGB threshold segmentation method and a GLCM texture analysis method; obtaining deformation features using a SIFT feature matching method and an optical flow displacement calculation method; determining a damage level based on the area and position of crack features, corrosion features, and deformation features; and obtaining damage information by integrating crack features, corrosion features, deformation features, and the damage level.

[0085] Specifically, the pixel coordinates of the second image data were converted to the model coordinate system using the parameters used during the drone inspection (rotation matrix, translation vector, etc.). The ORB / SIFT algorithm was used to extract key feature points between the image and the model, and the RANSAC algorithm was used to eliminate mismatches and optimize registration accuracy. Histogram equalization or the Retinex algorithm was used to reduce the effects of uneven illumination in the image data. When using the Canny edge detection method and the morphological skeleton extraction method to obtain crack features, the Canny edge detection method used Gaussian filtering (σ=1.0σ=1.0) for noise reduction; the gradient amplitude and direction were calculated, and dual threshold segmentation was performed (high threshold 70, low threshold 30); edge pixels were connected with a hysteresis to generate a binary edge map; and when using the morphological skeleton extraction method, edge gaps were filled using a closing operation (kernel size 3×3), and the edges were refined into single-pixel skeletons using the Zhang-Suen algorithm. The two methods were combined to obtain crack features.

[0086] When using RGB threshold segmentation and GLCM texture analysis to obtain corrosion features, the image data is converted to HSV space using the RGB threshold segmentation method. The hue range (e.g., H∈[0,30] for the corrosion area) and the saturation threshold (S>100) are set. A binary corrosion mask is then generated. The gray-level co-occurrence matrix (distance d=1, directions 0° / 45° / 90° / 135°) is calculated using the GLCM texture analysis method. The contrast (CC) and inverse distance (HH) are extracted and combined with thresholds (e.g., C>50 and H<0.3) to determine the corrosion area and obtain the corrosion features.

[0087] When using SIFT feature matching and optical flow displacement calculation methods to obtain deformation features, SIFT feature matching is used to extract SIFT feature points from multiple frames of images. BBF matching (distance threshold 0.6) is used to obtain the same-name points. RANSAC is then used to eliminate mismatches and calculate the fundamental matrix FF. When using optical flow displacement calculation methods, the Lucas-Kanade optical flow method is used to calculate the pixel displacement vector (u, v)(u, v). Displacement field threshold segmentation (e.g., |u|+|v|>5|u|+|v|>5 pixels) is used to identify the deformed area and obtain deformation features.

[0088] Then, the damage grade is determined based on the area and location of the crack characteristics, corrosion characteristics and deformation characteristics; and the damage information is obtained by integrating the crack characteristics, corrosion characteristics, deformation characteristics and the damage grade.

[0089] Through the configuration of this embodiment, the present invention can perform damage analysis on the second image data based on the second bridge model and accurately determine the damage information of the bridge.

[0090] In one embodiment of the present invention, when the damage information meets the repair conditions, repair information is sent to the supervisor, including: when the damage area of ​​the damage structure in the damage information is greater than the preset area, or when the damage depth of the damage structure in the damage information is greater than the preset depth, generating corresponding first repair information; when the damage position of the damage structure in the damage information belongs to the preset key position, generating corresponding second repair information; and sending the repair information obtained by integrating the first repair information and the second repair information to the supervisor.

[0091] Through the configuration of this embodiment, the present invention can send information about bridges that are determined to be in need of repair to supervisors, so that supervisors can promptly arrange for maintenance personnel to carry out bridge maintenance and repairs to ensure the safety of pedestrians on the bridges.

[0092] In one embodiment of the present invention, the flight path planning module further includes: adjusting the inspection planning path according to the dynamic structure inspection level table to obtain a revised inspection planning path.

[0093] Specifically, when a new dynamic structure inspection level table is obtained, the hybrid particle swarm algorithm and ant colony algorithm are used again to optimize and analyze the inspection flight area, detection accuracy of the inspection structure, collection distance and collection angle, and determine the revised inspection planning path for the second UAV to collect bridge image data next time.

[0094] Through the configuration of this embodiment, the present invention can adjust the inspection planning path according to the dynamic structural inspection level table, and obtain a revised inspection planning path for the second drone when it next inspects the bridge. This allows for the collection of image data with corresponding detection accuracy for different levels of bridge structures, so that subsequent damage information analysis can quickly obtain the results of the damage analysis. When the damage level meets the repair level, repair information including the bridge name, bridge location, damaged structure, damage location, damage range, and repair materials is uploaded to facilitate maintenance personnel to carry out emergency maintenance and ensure safe passage of the bridge.

[0095] Various embodiments of the systems and techniques described herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on a chip (SOCs), programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments can include being implemented in one or more computer programs that are executable and / or interpreted on a programmable system comprising at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0096] It should be noted that, in the description of the present invention, the terms "first", "second" and "third" are used for descriptive purposes only and should not be understood as indicating or implying relative importance.

[0097] The program code for implementing the method of the present disclosure can be written in any combination of one or more programming languages. These program codes can be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device so that when the program code is executed by the processor or controller, the functions / operations specified in the flow chart and / or block diagram are implemented. The program code can be executed entirely on the machine, partially on the machine, as a stand-alone software package, partially on the machine and partially on a remote machine, or entirely on a remote machine or server.

[0098] In the context of the present disclosure, a machine-readable medium can be a tangible medium that can contain or store a program for use by or in conjunction with an instruction execution system, device or equipment. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium can include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0099] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0100] The systems and techniques described herein can be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer having a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network (LAN), a wide area network (WAN), and the Internet.

[0101] A computer system may include a client and a server. The client and server are generally remote from each other and typically interact through a communication network. The client-server relationship arises through computer programs running on the respective computers and having a client-server relationship with each other. The server may be a cloud server, a server in a distributed system, or a server integrated with a blockchain.

[0102] Thus far, the technical solutions of the present invention have been described in conjunction with the preferred embodiments shown in the accompanying drawings. However, it will be readily understood by those skilled in the art that the scope of protection of the present invention is obviously not limited to these specific embodiments. Without departing from the principles of the present invention, those skilled in the art may make equivalent changes or substitutions to the relevant technical features, and the technical solutions after such changes or substitutions will fall within the scope of protection of the present invention.

Claims

1. An automated road and bridge inspection system based on drones, characterized in that: include: a bridge information acquisition module, which acquires bridge location information, and uses a first drone to acquire first image data of the bridge, and uses a second drone to acquire second image data of the bridge; a bridge model construction module, forming a first bridge model based on the first image data; performing a matching analysis between the first bridge model and a preset bridge template model to determine a classification model for the first bridge model; adjusting the classification model based on the first image data to obtain a second bridge model; and determining a structural inspection grade table formed by key inspection information based on the second bridge model; The flight path planning module determines the collection distance and angle of the second UAV inspection based on the structure inspection level table and UAV information; determines the inspection flight area based on the collection distance and angle; and uses a hybrid particle swarm algorithm and ant colony algorithm to optimize and analyze the inspection flight area, inspection structure, collection distance, and angle to determine the inspection planning path for the second UAV. a bridge state analysis module, performing a damage analysis on the second image data based on the second bridge model to determine damage information; Adjust the structural inspection grade table according to the disease information to obtain a dynamic structural inspection grade table; When the disease information meets the repair conditions, the repair information is sent to the supervisor.

2. The automated inspection system according to claim 1, characterized in that: The forming of the first bridge model based on the first image data includes: preprocessing the first image data and performing image segmentation on the preprocessed first image data using the Otsu algorithm to obtain a bridge structure image; determining coordinate information based on the bridge structure image; structurally splicing the coordinate information and the bridge structure image based on a two-dimensional coordinate system to form multiple initial first bridge models; performing pixel difference analysis on each initial first bridge model, and taking the initial first bridge model with the largest number of pixel change values ​​greater than a preset value as the first bridge model.

3. The automated inspection system according to claim 1, characterized in that: The matching analysis between the first bridge model and the preset bridge template model to determine the classification model of the first bridge model includes: scaling or enlarging the first bridge model to align the first bridge model and the preset bridge template model; performing similarity analysis on the pixel information of the first bridge model and the pixel information of the preset bridge template model to determine a matching value; and determining the classification model of the first bridge model based on the matching value.

4. The automated inspection system according to claim 1, characterized in that: The adjusting of the classification model according to the first image data to obtain the second bridge model includes: determining the corresponding surface of each initial first bridge model in the classification model according to the bridge structure relationship between each initial first bridge model and each surface of the classification model; and mapping the bridge structure characteristics of each initial first bridge model into the classification model based on the classification model to obtain the second bridge model.

5. The automated inspection system according to claim 1, characterized in that: The method of determining the acquisition distance and acquisition angle of the second drone inspection according to the structure inspection level table and drone information includes: determining the acquisition angle and maximum acquisition distance according to the detection accuracy corresponding to the structure inspection level, camera parameters and the inclination angle of the structure surface; the maximum acquisition distance D max The calculation formula is: θ=-α, Among them, L is the detection accuracy, P width is the number of horizontal pixels, α is the tilt angle of the structure surface detected in real time, FOV h is the horizontal field of view angle, and θ is the acquisition angle.

6. The automated inspection system according to claim 5, characterized in that: The method of determining the maximum acquisition distance according to the detection accuracy, camera parameters and structural surface inclination angle corresponding to the structural inspection level includes: determining the field of view width according to the acquisition distance and camera parameters; determining the ground sampling distance according to the field of view width and camera parameters; and determining the maximum acquisition distance according to the constraint relationship between the ground sampling distance and the detection accuracy.

7. The automated inspection system according to claim 1, characterized in that: The hybrid particle swarm algorithm and ant colony algorithm are used to optimize and analyze the inspection flight area, inspection structure, collection distance and collection angle to determine the inspection planning path of the second UAV, including: determining the inspection point information and constraint information; constructing a fitness function based on the inspection point information and constraint information; initializing the encoding of particles and initializing the pheromone matrix based on the inspection information; adjusting the speed update formula in the particle swarm optimization algorithm based on the pheromone in the ant colony algorithm; updating the particle speed and position, and calculating the fitness based on the fitness function; constructing a new path based on the pheromone and heuristic function, and updating the pheromone matrix; using the global optimal solution of the particle swarm optimization algorithm to correct the pheromone distribution of the ant colony algorithm and using the high-intensity pheromone of the ant colony algorithm to guide particle movement; when the maximum number of iterations is reached, determining the inspection planning path with the particle information corresponding to the fitness.

8. The automated inspection system according to claim 1, characterized in that: The damage analysis of the second image data based on the second bridge model to determine the damage information includes: aligning the coordinates of the second bridge model and each second image data; for single image data, using the Canny edge detection method and the morphological skeleton extraction method to obtain crack features; using the RGB threshold segmentation method and the GLCM texture analysis method to obtain corrosion features; using the SIFT feature matching method and the optical flow displacement calculation method to obtain deformation features; determining the damage level based on the area and position of the crack features, corrosion features, and deformation features; and integrating the crack features, corrosion features, deformation features, and the damage level to obtain the damage information.

9. The automated inspection system according to claim 1, characterized in that: When the damage information meets the repair conditions, the repair information is sent to the supervisor, including: when the damage area of ​​the damage structure in the damage information is greater than the preset area, or when the damage depth of the damage structure in the damage information is greater than the preset depth, generating corresponding first repair information; when the damage position of the damage structure in the damage information belongs to the preset key position, generating corresponding second repair information; and sending the repair information obtained by integrating the first repair information and the second repair information to the supervisor.

10. The automated inspection system according to claim 1, characterized in that: The flight path planning module also includes: adjusting the inspection planning path according to the dynamic structure inspection level table to obtain a revised inspection planning path.

Images