Unmanned aerial vehicle monitoring method for quality inspection of large-span bridge pier column

Abstract

The invention relates to the technical field of bridge detection, in particular to an unmanned aerial vehicle monitoring method for quality inspection of large-span bridge pier studs, comprising the following steps: S1, configuring a multi-modal sensor on an unmanned aerial vehicle to obtain pier stud layout information, and planning an unmanned aerial vehicle route based on the pier stud layout information; s2, automatically executing a flight task according to the route of the unmanned aerial vehicle, and collecting a multi-modal image and spatial position information; when the device is used, the visual, depth and thermal information is fused, so that the device is beneficial to more comprehensively and accurately identifying the structural abnormality and thermal damage of the pier stud, improving the image acquisition precision, constructing a three-dimensional model of structural defects, accurately measuring the length, width, depth and other size information of cracks, and binding the identification result with an actual structural model; and generating a three-dimensional visual defect marking graph and a statistical report, and finally outputting a visual defect result, thereby being beneficial to identifying the damage type of the pier stud and improving the precision of an identification mode.

Description

Technical Field

[0001] The present invention relates to the technical field of bridge detection, and in particular to a drone monitoring method for quality inspection of long-span bridge piers. Background Art

[0002] Using drones to inspect the quality of long-span bridge piers is an efficient and safe method, especially in inaccessible or dangerous environments. Drones can carry high-resolution cameras, infrared cameras, and other sensors, providing detailed visual and thermal imaging data to help inspectors identify potential problems.

[0003] The patent publication number is CN114486908A, and its description states that "the present invention provides a bridge pier disease identification and detection drone, including a drone and a calibration device, a camera device and an image processing unit carried on the drone; the calibration device is arranged in front of the drone, and the calibration device includes a plurality of calibration round heads, the rear end of each calibration round head is connected to the drone through a telescopic spring, and the front ends of all calibration round heads are flush; the camera device is used to capture an image of the bridge pier in which all calibration round heads are in contact with the surface of the bridge pier; the image processing unit stores a set reference pattern in which all calibration round heads are in contact with the vertical surface of the bridge pier, and the image processing unit is used to screen out the pattern in which each calibration round head is in contact with the bridge pier in the bridge pier image as a reference pattern. The measured reference pattern is used to perform angle correction on the entire pier image. The corrected pier image is then compared with the set reference pattern to calculate the actual size of the defect features. The beneficial effects of this invention include resolving the issue of drones being unable to maintain uniform imaging distance for defects, effectively improving bridge inspection accuracy. While the aforementioned technology uses grayscale conversion, segmentation, morphological processing, and stretching and compression to correct the measured reference pattern to align with the set reference pattern, achieving simultaneous correction of defect images, traditional image acquisition and recognition methods lack accuracy. Conventional drones primarily capture two-dimensional images, lacking the ability to analyze three-dimensional features such as crack depth and structural deformation. Image recognition often relies on manual post-processing, resulting in low efficiency and high subjectivity.

[0004] In summary, developing a drone monitoring method for quality inspection of large-span bridge piers is still a key issue that needs to be urgently addressed in the field of bridge inspection technology. Summary of the Invention

[0005] The purpose of the present invention is to solve the problem in the prior art that although the above-mentioned technology corrects the measured reference pattern to be consistent with the set reference pattern through graying, segmentation, morphological processing and stretching and compression, thereby achieving the purpose of synchronously correcting the disease image, the traditional image acquisition and recognition methods are insufficient in accuracy. Conventional drones mostly acquire two-dimensional images and have poor ability to analyze three-dimensional morphologies such as crack depth and structural deformation. Image recognition mostly relies on post-manual interpretation, which is inefficient and highly subjective.

[0006] To achieve the above object, the present invention provides the following technical solutions:

[0007] The present invention provides a drone monitoring method for long-span bridge pier quality inspection, comprising the following steps: S1, acquiring pier layout information by configuring a multimodal sensor on a drone, and planning a drone route based on the pier layout information;

[0008] S2. Automatically execute the flight mission according to the UAV route and collect multimodal images and spatial position information;

[0009] S3. performing data fusion and preprocessing on the multimodal images and spatial position information to generate a multimodal dataset in a unified format;

[0010] S4. Perform AI recognition and defect modeling based on the multimodal data set, and output defect results;

[0011] S5. Based on the comparison of the defect results with historical data, the degradation trend is evaluated, and a test report is automatically generated and uploaded to the cloud for remote expert collaborative diagnosis and maintenance recommendation push.

[0012] Furthermore, in step S1, a multimodal sensor is configured on the drone to obtain pier column layout information, and a method for planning the drone route based on the pier column layout information is as follows:

[0013] The multimodal sensor includes a high-resolution RGB camera, a laser radar, and an infrared thermal imager. The pier layout information includes comprehensive collection of visual, depth, and thermal anomaly information on the pier surface. The UAV route is planned based on the pier layout information by solving the shortest path problem. The UAV route is set by the path point Q i ={Q1,Q2,…,Q n}, and the expression formula of the drone route is obtained: Where Min is minimization, which means to find the minimum objective function value. The summation symbol indicates that the summation is performed from i=1 to n-1, dte(Q i ,Q i+1 ) is the distance function representing point Q i and Q i+1 The distance between i is the position of the i-th point in the path, Q i+1 is the position of the i+1th point in the path, Q i ∈W is the constraint condition representing point Q iIt must belong to the set W, where W is the pier column layout information. After planning is completed, the generated route is executed by the drone's control system. During execution, the drone uses the GPS sensor to track the path and adjusts the heading in real time to avoid obstacles. The path tracking is optimized by model predictive control, expressed as: Where E(t) is the control input vector at time t, is the variable value when minimizing the objective function, R(t) is the current state vector at time t, R des (t) is the expected state vector at time t, ||R(t)-R des (t)|| 2 is the square of the Euclidean distance between the current state and the desired state, is a regularization weight factor, ||R(t)|| 2 is the energy (squared norm) of the control input.

[0014] Furthermore, in step S2, the method for automatically executing the flight mission according to the drone route and collecting multimodal images and spatial position information is:

[0015] According to the drone route Q i ={Q1,Q2,…,Q n}, automatically execute flight missions and perform navigation task allocation. During the task allocation, each path point on the route will synchronously collect multimodal images and position information. The multimodal images include RGB images, depth images, and thermal images. The expression formula is: Where Y(x,y,t) is the RGB image, U(x,y,t) is the depth image, and I(x,y,t) is the thermal image. is the spatial position of the drone, Indicates the location The RGB image captured by the camera sensor at RGB (x, y, t) is the noise term of RGB image acquisition, U(x, y, t) represents the noise term of the image acquired from position t at time t. The depth image collected, β(r) is the reflection coefficient of the lidar, α U (x,y,t) is the noise of the depth measurement, I(x,y,t) is the noise of the thermal imaging sensor at position The collected thermal image, T rea (x, y, t) is the actual thermal distribution of the pier, δ(·) is the imaging transformation function of the sensor, α I (x,y,t) is the noise in the thermal image.

[0016] Furthermore, in step S2, the method for automatically executing the flight mission according to the drone route and collecting multimodal images and spatial position information is:

[0017] The position information H(t) is obtained through the GPS sensor, and the sensor data and position data are synchronized using the timestamp t. During the execution of the mission, the UAV continuously uses the feedback control mechanism based on reinforcement learning to control the flight and execute the mission according to the real-time feedback. The expression formula is: in is the target position coordinate set by the task, TC t is the score of task completion, TP t is the time penalty term, o t is the reward value at time t used to evaluate the quality of the current task behavior, ε1 is the weight coefficient of the control position error penalty term, ε2 is the weight coefficient of the control task completion reward term, and ε3 is the weight coefficient of the control time penalty term. is the current position vector of the UAV at time t, is the current drone position With target location The squared Euclidean distance between them.

[0018] Furthermore, in step S3, the multimodal image and the spatial position information are subjected to data fusion and preprocessing to generate a multimodal data set in a unified format.

[0019] The data fusion and preprocessing include time synchronization and spatial registration, image enhancement, and denoising. The time synchronization interpolates the timestamps of the multimodal images including RGB images, lidar images, infrared thermal imaging images, and GPS position data through an interpolation method, and sets a unified time grid {t1, t2, ..., t m}, align data with different timestamps to a unified time point t syn The spatial registration is performed by setting the images acquired by the multimodal sensor to have a transformation matrix {O RGB ,O U ,O I}, and then transformed into a unified spatial coordinate system

[0020] Furthermore, in step S3, the multimodal image and the spatial position information are subjected to data fusion and preprocessing to generate a multimodal data set in a unified format.

[0021] The image enhancement and denoising are expressed as follows:

[0022] Among them, P dee,RGB(x, y, t) is the RGB image pixel value after Gaussian filtering and denoising at pixel position (x, y) and time t, P RGB (x, y, t) is the pixel value of the original RGB color image at pixel position (x, y) and time t, A G (·) is the Gaussian filter function, x, y are the horizontal and vertical coordinates of the pixel position in the image, t is the timestamp of the image frame, x0, y0 are the coordinates of the center of the current filter kernel (convolution kernel), σ is the standard deviation of the Gaussian distribution, is the normalization constant of the Gaussian function, exp(·) is the exponential function, (x-x0) 2 +(y-y0) 2 is the square of the Euclidean distance between the current pixel position and the filter center point, S dee,U (x, y, t) is the luminance image denoised by bilateral filtering at pixel position (x, y) and time t, Bil(·) is the bilateral filtering algorithm, S U (x, y, t) is the original brightness image (grayscale image) at pixel position (x, y) and time t, D dee,I (x, y, t) is the denoised version of the infrared thermal image at pixel position (x, y) and time t, NLM(·) is the non-local mean filtering algorithm, and D I (x, y, t) is the temperature value of the original infrared thermal image at the pixel point (x, y) and time t. After completing time synchronization, spatial registration, image enhancement and denoising, all multimodal data are fused, expressed as follows:

[0023] F={P dee,RGB (x,y,t),S dee,U (x,y,t),D dee,I (x,y,t),H sync (t)}, Where F is a multimodal dataset, H sync (t) represents the positioning of the UAV in three-dimensional space at time t, t∈{t1,t2,…,t m} represents the time point of data collection, and a multimodal dataset F in a unified format is generated based on spatial and temporal information.

[0024] Furthermore, in step S4, AI recognition and defect modeling are performed based on the multimodal data set, and the method for outputting defect results is:

[0025] The AI recognition and defect modeling includes preliminary positioning of defect areas, fine recognition and classification, and spatial modeling and size measurement; the preliminary positioning of defect areas uses a lightweight target detection model to locate suspected defect areas and generate candidate area frames. The lightweight target detection model f detInput the preprocessed multimodal dataset F and output the bounding box J of each candidate region i =[x min ,y min ,x max ,y max ] and assign a confidence K to each candidate region i , expression formula: Where φ is the sigmoid function, f conf It is a subnetwork used to calculate the confidence of each candidate box, J i is the bounding box of the i-th candidate defect area, f det is a lightweight target detection model, F is a multimodal dataset, K i is the confidence score of the i-th candidate region, and non-maximum suppression is used to remove redundant candidate region boxes, and the threshold is set to The candidate box with the highest confidence is retained.

[0026] Furthermore, in step S4, AI recognition and defect modeling are performed based on the multimodal data set, and the method for outputting defect results is:

[0027] The refined identification and classification uses a deep learning model to perform pixel-level segmentation on the candidate area, identify the types of cracks and erosion, and extract boundary and morphological features. The expression formula is: where γ·J i is the gradient amplitude of the defect boundary used to extract the edge strength of the i-th defect, is the gradient modulus of the image brightness function L(x′,y′) at the pixel position (x′,y′), L(x′,y′) is the brightness value of the image at the pixel position (x′,y′), is the partial derivative of image brightness with respect to x′, is the partial derivative of image brightness with respect to y′, L(x′, y′)∈{0,1,2,...,N} represents the category of defect type, and the spatial modeling and dimensional measurement map the recognition results to the point cloud, perform 3D modeling and geometric measurement, and output the spatial dimensional information of the length, width, and depth of the crack. The output defect result is bound to the structure using visual output to generate a 3D defect annotation map and statistical table, and bound to the structural coordinate model.

[0028] Furthermore, in step S5, the method for comparing the defect results with historical data, evaluating the degradation trend, automatically generating a test report, and uploading it to the cloud for remote expert collaborative diagnosis and maintenance suggestion push is as follows:

[0029] The degradation trend ΔC is evaluated by calculating the weighted sum of the characteristic differences, and the defect result is defined as The characteristic vector of the historical data is in are the length, width and depth of the crack respectively, and the expression formula is: Where ΔC is the degradation trend, C cur,i is the i-th defect result in the current inspection, C his,i is the i-th defect result in the history, η i is the weight coefficient of each defect feature, is the dimension index, is a summation symbol. According to the degradation trend ΔC, the degradation trend index ι is further generated. The expression formula is: Where ι is the deterioration trend index, C his,i is the i-th defect result in the history, ΔC is the degradation trend, max(C his,i ) is the historical maximum value of the i-th dimension. ι>1 indicates that the structure has undergone significant degradation, and ι≤1 indicates that the degradation is relatively mild.

[0030] Furthermore, in step S5, the method for comparing the defect results with historical data, evaluating the degradation trend, automatically generating a test report, and uploading it to the cloud for remote expert collaborative diagnosis and maintenance suggestion push is as follows:

[0031] The test report includes a heat map and a risk level report. ther It is obtained by normalizing the temperature value, and the expression formula is: Where V ther (x″, y″) means that the heatmap value at position (x″, y″) is normalized to the range of [0, 1], B ther It is an infrared image. The risk level report is automatically generated based on the degradation trend index ι, combined with the defect type and size, and is expressed as: in It's low risk. It's medium risk. If it is a high risk, the defect 3D annotation heat map and risk level report will be uploaded to the external cloud platform μ cloud , and perform encryption and security processing on the external cloud platform μ cloud In the process, remote experts can further analyze the defects through collaborative diagnostic tools.

[0032] Beneficial effects

[0033] Compared with the known public technology, the technical solution provided by the present invention has the following advantages:

[0034] Beneficial effects:

[0035] When used, the present invention, by fusing visual, depth and thermal information, is conducive to more comprehensive and accurate identification of structural anomalies and thermal damage of piers, improving the accuracy of image acquisition, constructing a three-dimensional model of structural defects, accurately measuring the length, width, depth and other dimensional information of cracks, binding the identification results with the actual structural model, generating a three-dimensional visual defect annotation map and statistical report, and finally outputting a visual defect result, which is conducive to identifying the type of pier damage and improving the accuracy of the identification method.

[0036] When used, the present invention quantifies the degree of difference through a weighted summation method, thereby judging whether the structural degradation has worsened, which is conducive to reducing the reliance on later manual interpretation, thereby improving the efficiency of quality inspection of large-span bridge piers. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1 The present invention is a flow chart of a drone monitoring method for quality inspection of long-span bridge piers. DETAILED DESCRIPTION

[0038] In order to enable those skilled in the art to better understand the solutions of the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the embodiments described are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts should fall within the scope of protection of the present invention.

[0039] It should be noted that the terms "first," "second," and the like in the description and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or precedence. It should be understood that the numbers used in this manner are interchangeable where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having," as well as any variations thereof, are intended to cover non-exclusive inclusions. For example, a process, method, system, product, or apparatus comprising a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units that are not explicitly listed or that are inherent to these processes, methods, products, or apparatuses.

[0040] The present invention is described in further detail below with reference to the accompanying drawings:

[0041] Example:

[0042] like Figure 1As shown, the present invention provides a drone monitoring method for quality inspection of long-span bridge piers, comprising the following steps: S1, acquiring pier layout information by configuring a multimodal sensor on a drone, and planning a drone route based on the pier layout information;

[0043] Furthermore, in step S1, a multimodal sensor is configured on the drone to obtain pier column layout information, and a method for planning the drone route based on the pier column layout information is as follows:

[0044] The multimodal sensor includes a high-resolution RGB camera, a laser radar, and an infrared thermal imager. The pier layout information includes comprehensive collection of visual, depth, and thermal anomaly information on the pier surface. The UAV route is planned based on the pier layout information by solving the shortest path problem. The UAV route is set by the path point Q i ={Q1,Q2,…,Q n}, and the expression formula of the drone route is obtained: Where Min is minimization, which means to find the minimum objective function value. The summation symbol indicates that the summation is performed from i=1 to n-1, dte(Q i ,Q i+1 ) is the distance function representing point Q i and Q i+1 The distance between i is the position of the i-th point in the path, Q i+1 is the position of the i+1th point in the path, Q i ∈W is the constraint condition representing point Q i It must belong to the set W, where W is the pier column layout information. After planning is completed, the generated route is executed by the drone's control system. During execution, the drone uses the GPS sensor to track the path and adjusts the heading in real time to avoid obstacles. The path tracking is optimized by model predictive control, expressed as: Where E(t) is the control input vector at time t, is the variable value when minimizing the objective function, R(t) is the current state vector at time t, R des (t) is the expected state vector at time t, ||R(t)-R des (t)|| 2 is the square of the Euclidean distance between the current state and the desired state, is a regularization weight factor, ||R(t)|| 2 is the energy (squared norm) of the control input.

[0045] In this embodiment, the method integrates three types of sensors on the drone: a high-resolution RGB camera is used to obtain visual images of the pier, a lidar is used to obtain the three-dimensional depth structure of the pier, and an infrared thermal imager is used to detect possible thermal anomalies, such as temperature differences caused by cracks. To achieve path tracking, the drone will continuously adjust its flight trajectory through GPS positioning. By fusing visual, depth, and thermal information, it is conducive to more comprehensive and accurate identification of structural anomalies and thermal damage to the pier.

[0046] S2. Automatically execute the flight mission according to the UAV route and collect multimodal images and spatial position information;

[0047] Furthermore, in step S2, the method for automatically executing the flight mission according to the drone route and collecting multimodal images and spatial position information is:

[0048] According to the drone route Q i ={Q1,Q2,…,Q n}, automatically execute flight missions and perform navigation task allocation. During the task allocation, each path point on the route will synchronously collect multimodal images and position information. The multimodal images include RGB images, depth images, and thermal images. The expression formula is: Where Y(x,y,t) is the RGB image, U(x,y,t) is the depth image, and I(x,y,t) is the thermal image. is the spatial position of the drone, Indicates the location The RGB image captured by the camera sensor at RGB (x, y, t) is the noise term of RGB image acquisition, U(x, y, t) represents the noise term of the image acquired from position t at time t. The depth image collected, β(r) is the reflection coefficient of the lidar, α U (x,y,t) is the noise of the depth measurement, I(x,y,t) is the noise of the thermal imaging sensor at position The collected thermal image, T rea (x, y, t) is the actual thermal distribution of the pier, δ(·) is the imaging transformation function of the sensor, α I (x,y,t) is the noise in the thermal image.

[0049] Furthermore, in step S2, the method for automatically executing the flight mission according to the drone route and collecting multimodal images and spatial position information is:

[0050] The position information H(t) is obtained through the GPS sensor, and the sensor data and position data are synchronized using the timestamp t. During the execution of the mission, the UAV continuously uses the feedback control mechanism based on reinforcement learning to control the flight and execute the mission according to the real-time feedback. The expression formula is: in is the target position coordinate set by the task, TC t is the score of task completion, TP t is the time penalty term, o t is the reward value at time t used to evaluate the quality of the current task behavior, ε1 is the weight coefficient of the control position error penalty term, ε2 is the weight coefficient of the control task completion reward term, and ε3 is the weight coefficient of the control time penalty term. is the current position vector of the UAV at time t, is the current drone position With target location The squared Euclidean distance between them.

[0051] In this embodiment, the method automatically executes flight missions based on a planned drone route and simultaneously collects multimodal images and location information. Specifically, at each pre-set path point, the method uses multiple sensors to capture RGB, depth, and thermal images, while also recording the GPS coordinates of the current location. This data is synchronized using timestamps to ensure a precise match between the image and spatial location. The drone utilizes a reinforcement learning control mechanism, which autonomously assesses the efficiency of the current flight strategy by setting a target position and real-time feedback scores, and promptly adjusts the path to ensure mission continuity and safety. All images and location information are clearly timestamped and geotagged, enabling direct use for trend analysis, structural health monitoring, and digital archiving.

[0052] S3. performing data fusion and preprocessing on the multimodal images and spatial position information to generate a multimodal dataset in a unified format;

[0053] Furthermore, in step S3, the multimodal image and the spatial position information are subjected to data fusion and preprocessing to generate a multimodal data set in a unified format.

[0054] The data fusion and preprocessing include time synchronization and spatial registration, image enhancement, and denoising. The time synchronization interpolates the timestamps of the multimodal images including RGB images, lidar images, infrared thermal imaging images, and GPS position data through an interpolation method, and sets a unified time grid {t1, t2, ..., t m}, align data with different timestamps to a unified time point t synThe spatial registration is performed by setting the images acquired by the multimodal sensor to have a transformation matrix {O RGB ,O U ,O I}, and then transformed into a unified spatial coordinate system

[0055] Furthermore, in step S3, the multimodal image and the spatial position information are subjected to data fusion and preprocessing to generate a multimodal data set in a unified format.

[0056] The image enhancement and denoising are expressed as follows:

[0057] Among them, P dee,RGB (x, y, t) is the RGB image pixel value after Gaussian filtering and denoising at pixel position (x, y) and time t, P RGB (x, y, t) is the pixel value of the original RGB color image at pixel position (x, y) and time t, A G (·) is the Gaussian filter function, x, y are the horizontal and vertical coordinates of the pixel position in the image, t is the timestamp of the image frame, x0, y0 are the coordinates of the center of the current filter kernel (convolution kernel), σ is the standard deviation of the Gaussian distribution, is the normalization constant of the Gaussian function, exp(·) is the exponential function, (x-x0) 2 +(y-y0) 2 is the square of the Euclidean distance between the current pixel position and the filter center point, S dee,U (x, y, t) is the luminance image denoised by bilateral filtering at pixel position (x, y) and time t, Bil(·) is the bilateral filtering algorithm, S U (x, y, t) is the original brightness image (grayscale image) at pixel position (x, y) and time t, D dee,I (x, y, t) is the denoised version of the infrared thermal image at pixel position (x, y) and time t, NLM(·) is the non-local mean filtering algorithm, and D I (x, y, t) is the temperature value of the original infrared thermal image at the pixel point (x, y) and time t. After completing time synchronization, spatial registration, image enhancement and denoising, all multimodal data are fused, expressed as follows:

[0058] F={P dee,RGB (x,y,t),S dee,U (x,y,t),D dee,I (x,y,t),H sync (t)}, Where F is a multimodal dataset, H sync(t) represents the positioning of the UAV in three-dimensional space at time t, t∈{t1,t2,…,t m} represents the time point of data collection, and a multimodal dataset F in a unified format is generated based on spatial and temporal information.

[0059] In this embodiment, this method fuses and preprocesses the multimodal images (including RGB images, lidar depth maps, and infrared thermal images) collected by the drone with GPS spatial location information to generate a multimodal data set in a unified format, so that all images are aligned to a unified time and space point, the images are clearer, and the temperature distribution is more accurate for subsequent AI analysis and defect modeling. Through time interpolation and spatial alignment, the time and space errors between multi-source data are eliminated, which is conducive to improving the analysis accuracy. The unified coordinate data can be seamlessly used for 3D modeling and thermal map overlay, supporting intuitive structural status display.

[0060] S4. Perform AI recognition and defect modeling based on the multimodal data set, and output defect results;

[0061] Furthermore, in step S4, AI recognition and defect modeling are performed based on the multimodal data set, and the method for outputting defect results is:

[0062] The AI recognition and defect modeling includes preliminary positioning of defect areas, fine recognition and classification, and spatial modeling and size measurement; the preliminary positioning of defect areas uses a lightweight target detection model to locate suspected defect areas and generate candidate area frames. The lightweight target detection model f det Input the preprocessed multimodal dataset F and output the bounding box J of each candidate region i =[x min ,y min ,x max ,y max ] and assign a confidence K to each candidate region i , expression formula: Where φ is the sigmoid function, f conf It is a subnetwork used to calculate the confidence of each candidate box, J i is the bounding box of the i-th candidate defect area, f det is a lightweight target detection model, F is a multimodal dataset, K i is the confidence score of the i-th candidate region, and non-maximum suppression is used to remove redundant candidate region boxes, and the threshold is set to The candidate box with the highest confidence is retained.

[0063] Furthermore, in step S4, AI recognition and defect modeling are performed based on the multimodal data set, and the method for outputting defect results is:

[0064] The refined identification and classification uses a deep learning model to perform pixel-level segmentation on the candidate area, identify the types of cracks and erosion, and extract boundary and morphological features. The expression formula is: where γ·J i is the gradient amplitude of the defect boundary used to extract the edge strength of the i-th defect, is the gradient modulus of the image brightness function L(x′,y′) at the pixel position (x′,y′), L(x′,y′) is the brightness value of the image at the pixel position (x′,y′), is the partial derivative of image brightness with respect to x′, is the partial derivative of image brightness with respect to y′, L(x′, y′)∈{0,1,2,...,N} represents the category of defect type, and the spatial modeling and dimensional measurement map the recognition results to the point cloud, perform 3D modeling and geometric measurement, and output the spatial dimensional information of the length, width, and depth of the crack. The output defect result is bound to the structure using visual output to generate a 3D defect annotation map and statistical table, and bound to the structural coordinate model.

[0065] In this embodiment, after the multimodal data processing is completed, this method uses AI technology through a lightweight target detection model to quickly detect suspected defect areas, such as cracks or erosion locations, from the multimodal image and generate candidate area frames. Each frame has a confidence score, which indicates the probability that it may be a defect. Non-maximum suppression (NMS) is used to eliminate overlapping areas, and only the most credible candidate frames are retained. A deep learning segmentation model is used to extract the boundaries and morphological classification of defects, and the geometric features of the edges are extracted. The recognition results are mapped to a three-dimensional point cloud. Combined with the drone positioning information, a three-dimensional model of the structural defect is constructed, and the length, width, depth and other dimensional information of the cracks are accurately measured. The recognition results are bound to the actual structural model to generate a three-dimensional visual defect annotation map and statistical report. Finally, a visual defect result is output, which is conducive to identifying the type of pier damage. The defects are restored to a three-dimensional geometric form based on point cloud technology, providing an intuitive and realistic display of pier structure damage.

[0066] S5. Compare the defect results with historical data, evaluate degradation trends, automatically generate a test report, and upload it to the cloud for remote expert collaborative diagnosis and maintenance recommendations.

[0067] Furthermore, in step S5, the method for comparing the defect results with historical data, evaluating the degradation trend, automatically generating a test report, and uploading it to the cloud for remote expert collaborative diagnosis and maintenance suggestion push is as follows:

[0068] The degradation trend ΔC is evaluated by calculating the weighted sum of the characteristic differences, and the defect result is defined as The characteristic vector of the historical data is in are the length, width and depth of the crack respectively, and the expression formula is: Where ΔC is the degradation trend, C cur,i is the i-th defect result in the current inspection, C his,i is the i-th defect result in the history, η i is the weight coefficient of each defect feature, is the dimension index, is a summation symbol. According to the degradation trend ΔC, the degradation trend index ι is further generated. The expression formula is: Where ι is the deterioration trend index, C his,i is the i-th defect result in the history, ΔC is the degradation trend, max(C his,i ) is the historical maximum value of the i-th dimension. ι>1 indicates that the structure has undergone significant degradation, and ι≤1 indicates that the degradation is relatively mild.

[0069] Furthermore, in step S5, the method for comparing the defect results with historical data, evaluating the degradation trend, automatically generating a test report, and uploading it to the cloud for remote expert collaborative diagnosis and maintenance suggestion push is as follows:

[0070] The test report includes a heat map and a risk level report. ther It is obtained by normalizing the temperature value, and the expression formula is: Where V ther (x″, y″) means that the heatmap value at position (x″, y″) is normalized to the range of [0, 1], B ther It is an infrared image. The risk level report is automatically generated based on the degradation trend index ι, combined with the defect type and size, and is expressed as: in It's low risk. It's medium risk. If it is a high risk, the defect 3D annotation heat map and risk level report will be uploaded to the external cloud platform μ cloud , and perform encryption and security processing on the external cloud platform μ cloud In the process, remote experts can further analyze the defects through collaborative diagnostic tools.

[0071] In this embodiment, the method compares the currently identified defects with historical inspection data, calculates the differences based on the key features of the defects (such as crack length, width, and depth), quantifies the degree of difference through a weighted summation method, and thus determines whether the structural degradation has worsened. In order to more intuitively represent the degree of degradation, a degradation trend index is calculated based on the maximum value of the historical data and the current trend value. The higher the value, the more dangerous the structural condition. With the help of trend assessment and index quantification, long-term monitoring and management are facilitated, and experts in multiple locations are supported to collaborate in determining the severity of defects, which is conducive to improving the professionalism and efficiency of diagnosis.

[0072] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements will not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the various embodiments of the present invention.

Claims

1. A drone monitoring method for quality inspection of long-span bridge piers, characterized in that: The following steps are involved: S1. Configuring a multimodal sensor on a drone to obtain pier column layout information, and planning a drone route based on the pier column layout information; S2. Automatically execute the flight mission according to the UAV route and collect multimodal images and spatial position information; S3. performing data fusion and preprocessing on the multimodal images and spatial position information to generate a multimodal dataset in a unified format; S4. Perform AI recognition and defect modeling based on the multimodal data set, and output defect results; S5. Based on the comparison of the defect results with historical data, the degradation trend is evaluated, and a test report is automatically generated and uploaded to the cloud for remote expert collaborative diagnosis and maintenance recommendation push.

2. The drone monitoring method for quality inspection of long-span bridge piers according to claim 1 is characterized in that: In step S1, a multimodal sensor is configured on a drone to obtain pier column layout information, and a method for planning a drone route based on the pier column layout information is as follows: The multimodal sensor includes a high-resolution RGB camera, a laser radar, and an infrared thermal imager. The pier layout information includes comprehensive collection of visual, depth, and thermal anomaly information on the pier surface. The UAV route is planned based on the pier layout information by solving the shortest path problem. The UAV route is set by the path point Q i ={Q1,Q2,…,Q n }, and the expression formula of the drone route is obtained: Where Min is minimization, which means to find the minimum objective function value. The summation symbol indicates that the summation is performed from i=1 to n-1, dte(Q i ,Q i+1 ) is the distance function representing point Q i and Q i+1 The distance between i is the position of the i-th point in the path, Q i+1 is the position of the i+1th point in the path, Q i ∈W is the constraint condition representing point Q i It must belong to the set W, where W is the pier column layout information. After planning is completed, the generated route is executed by the drone's control system. During execution, the drone uses the GPS sensor to track the path and adjusts the heading in real time to avoid obstacles. The path tracking is optimized by model predictive control, expressed as: Where E(t) is the control input vector at time t, is the variable value when minimizing the objective function, R(t) is the current state vector at time t, R des (t) is the expected state vector at time t, ||R(t)-R des (t)|| 2 is the square of the Euclidean distance between the current state and the desired state, θ is a regularization weight factor, ||R(t)|| 2 is the energy (squared norm) of the control input.

3. The drone monitoring method for quality inspection of long-span bridge piers according to claim 2 is characterized in that: In step S2, the method for automatically executing the flight mission according to the drone route and collecting multimodal images and spatial position information is as follows: According to the drone route Q i ={Q1,Q2,…,Q n }, automatically execute flight missions and perform navigation task allocation. During the task allocation, each path point on the route will synchronously collect multimodal images and position information. The multimodal images include RGB images, depth images, and thermal images. The expression formula is: Where Y(x,y,t) is the RGB image, U(x,y,t) is the depth image, and I(x,y,t) is the thermal image. is the spatial position of the drone, Indicates the location The RGB image captured by the camera sensor at RGB (x, y, t) is the noise term of RGB image acquisition, U(x, y, t) represents the noise term of the image acquired from position t at time t. The depth image collected, β(r) is the reflection coefficient of the lidar, α U (x,y,t) is the noise of the depth measurement, I(x,y,t) is the noise of the thermal imaging sensor at position The collected thermal image, T rea (x, y, t) is the actual thermal distribution of the pier, δ(·) is the imaging transformation function of the sensor, α I (x,y,t) is the noise in the thermal image.

4. The drone monitoring method for quality inspection of long-span bridge piers according to claim 3 is characterized in that: In step S2, the method for automatically executing the flight mission according to the drone route and collecting multimodal images and spatial position information is as follows: The position information H(t) is obtained through the GPS sensor, and the sensor data and position data are synchronized using the timestamp t. During the execution of the mission, the UAV continuously uses the feedback control mechanism based on reinforcement learning to control the flight and execute the mission according to the real-time feedback. The expression formula is: in is the target position coordinate set by the task, TC t is the score of task completion, TP t is the time penalty term, o t is the reward value at time t used to evaluate the quality of the current task behavior, ε1 is the weight coefficient of the control position error penalty term, ε2 is the weight coefficient of the control task completion reward term, and ε3 is the weight coefficient of the control time penalty term. is the current position vector of the UAV at time t, is the current drone position With target location The squared Euclidean distance between them.

5. The drone monitoring method for quality inspection of long-span bridge piers according to claim 4 is characterized in that: In step S3, the multimodal image and spatial position information are subjected to data fusion and preprocessing to generate a multimodal dataset in a unified format as follows: The data fusion and preprocessing include time synchronization and spatial registration, image enhancement, and denoising. The time synchronization interpolates the timestamps of the multimodal images including RGB images, lidar images, infrared thermal imaging images, and GPS position data through an interpolation method, and sets a unified time grid {t1, t2, ..., t m }, align data with different timestamps to a unified time point t syn The spatial registration is performed by setting the images acquired by the multimodal sensor to have a transformation matrix {O RGB ,O U ,O I }, and then transformed into a unified spatial coordinate system 6. The drone monitoring method for quality inspection of long-span bridge piers according to claim 5 is characterized in that: In step S3, the multimodal image and spatial position information are subjected to data fusion and preprocessing to generate a multimodal dataset in a unified format as follows: The image enhancement and denoising are expressed as follows: in P dee,RGB (x, y, t) is the RGB image pixel value after Gaussian filtering and denoising at pixel position (x, y) and time t, P RGB (x, y, t) is the pixel value of the original RGB color image at pixel position (x, y) and time t, A G (·) is the Gaussian filter function, x, y are the horizontal and vertical coordinates of the pixel position in the image, t is the timestamp of the image frame, x0, y0 are the coordinates of the center of the current filter kernel (convolution kernel), σ is the standard deviation of the Gaussian distribution, is the normalization constant of the Gaussian function, exp(·) is the exponential function, (x-x0) 2 +(y-y0) 2 is the square of the Euclidean distance between the current pixel position and the filter center point, S dee,U (x, y, t) is the luminance image denoised by bilateral filtering at pixel position (x, y) and time t, Bil(·) is the bilateral filtering algorithm, S U (x, y, t) is the original brightness image (grayscale image) at pixel position (x, y) and time t, D dee,I (x, y, t) is the denoised version of the infrared thermal image at pixel position (x, y) and time t, NLM(·) is the non-local mean filtering algorithm, and D I (x, y, t) is the temperature value of the original infrared thermal image at the pixel point (x, y) and time t. After completing time synchronization, spatial registration, image enhancement and denoising, all multimodal data are fused, expressed as follows: Where F is a multimodal dataset, H sync (t) represents the positioning of the UAV in three-dimensional space at time t, t∈{t1,t2,…,t m } represents the time point of data collection, and a multimodal dataset F in a unified format is generated based on spatial and temporal information.

7. The drone monitoring method for quality inspection of long-span bridge piers according to claim 6 is characterized in that: In step S4, AI recognition and defect modeling are performed based on the multimodal data set, and the method for outputting defect results is: The AI recognition and defect modeling includes preliminary positioning of defect areas, fine recognition and classification, and spatial modeling and size measurement; the preliminary positioning of defect areas uses a lightweight target detection model to locate suspected defect areas and generate candidate area frames. The lightweight target detection model f det Input the preprocessed multimodal dataset F and output the bounding box J of each candidate region i =[x min ,y min ,x max ,y max ] and assign a confidence K to each candidate region i , expression formula: Where φ is the sigmoid function, f conf It is a subnetwork used to calculate the confidence of each candidate box, J i is the bounding box of the i-th candidate defect area, f det is a lightweight target detection model, F is a multimodal dataset, K i is the confidence score of the i-th candidate region, and non-maximum suppression is used to remove redundant candidate region boxes, and the threshold is set to The candidate box with the highest confidence is retained.

8. The drone monitoring method for quality inspection of long-span bridge piers according to claim 7 is characterized in that: In step S4, AI recognition and defect modeling are performed based on the multimodal data set, and the method for outputting defect results is: The refined identification and classification uses a deep learning model to perform pixel-level segmentation on the candidate area, identify the types of cracks and erosion, and extract boundary and morphological features. The expression formula is: where γ·J i is the gradient magnitude of the defect boundary L(x′,y′)∈{0,1,2,...,N} used to extract the edge strength of the i-th defect, is the gradient modulus of the image brightness function L(x′,y′) at the pixel position (x′,y′), L(x′,y′) is the brightness value of the image at the pixel position (x′,y′), is the partial derivative of image brightness with respect to x′, is the partial derivative of image brightness with respect to y′, L(x′, y′)∈{0,1,2,...,N} represents the category of defect type, and the spatial modeling and dimensional measurement map the recognition results to the point cloud, perform 3D modeling and geometric measurement, and output the spatial dimensional information of the length, width, and depth of the crack. The output defect result is bound to the structure using visual output to generate a 3D defect annotation map and statistical table, and bound to the structural coordinate model.

9. The drone monitoring method for quality inspection of long-span bridge piers according to claim 8, characterized in that: In step S5, the method for comparing the defect results with historical data, evaluating the degradation trend, automatically generating a test report, and uploading it to the cloud for remote expert collaborative diagnosis and maintenance suggestion push is as follows: The degradation trend ΔC is evaluated by calculating the weighted sum of the characteristic differences, and the defect result is defined as The characteristic vector of the historical data is in are the length, width and depth of the crack respectively, and the expression formula is: Where ΔC is the degradation trend, C cur,i is the i-th defect result in the current inspection, C his,i is the i-th defect result in the history, η i is the weight coefficient of each defect feature, is the dimension index, is a summation symbol. According to the degradation trend ΔC, the degradation trend index ι is further generated. The expression formula is: Where ι is the deterioration trend index, C his,i is the i-th defect result in the history, ΔC is the degradation trend, max(C his,i ) is the historical maximum value of the i-th dimension. ι>1 indicates that the structure has undergone significant degradation, and ι≤1 indicates that the degradation is relatively mild.

10. The drone monitoring method for quality inspection of long-span bridge piers according to claim 8, characterized in that: In step S5, the method for comparing the defect results with historical data, evaluating the degradation trend, automatically generating a test report, and uploading it to the cloud for remote expert collaborative diagnosis and maintenance suggestion push is as follows: The test report includes a heat map and a risk level report. ther It is obtained by normalizing the temperature value, and the expression formula is: Where V ther (x″, y″) means that the heatmap value at position (x″, y″) is normalized to the range of [0, 1], B ther It is an infrared image. The risk level report is automatically generated based on the degradation trend index ι, combined with the defect type and size, and is expressed as: in It's low risk. It's medium risk. If it is a high risk, the defect 3D annotation heat map and risk level report will be uploaded to the external cloud platform μ cloud , and perform encryption and security processing on the external cloud platform μ cloud In the process, remote experts can further analyze the defects through collaborative diagnostic tools.

Images