A UAV inspection method for highway slope engineering

Abstract

The present invention relates to the field of image data processing technology, and in particular to a drone inspection method for highway slope engineering, comprising: using a drone to collect a video frame sequence of a highway slope; obtaining two fitting surfaces of pixel points in each frame of the video frame sequence; determining the degree of noise interference of the pixel points; calculating the preferred degree of the window radius of the pixel points and determining the final window radius; according to the final window radius, using the LK algorithm to obtain an aligned video frame sequence, and then obtain a panoramic view of the highway slope engineering, so as to realize the inspection of the highway slope engineering. The present invention uses surface fitting and structural similarity analysis, combined with iterative calculations, to adaptively determine the window radius for each pixel point, so that the algorithm can accurately capture pixel motion information and significantly improve the accuracy of video frame alignment; the present invention effectively avoids the splicing problem caused by frame dislocation and blur, and the highway slope panoramic view produced is clearer and more accurate, providing high-quality images for subsequent inspections.

Description

Technical Field

[0001] The present invention relates to the technical field of image data processing, and more specifically, to a drone inspection method for highway slope engineering. Background Art

[0002] With the rapid development of drone technology, the application of drones in various fields is increasing. Drones have the characteristics of fast flight speed, wide coverage, and flexible operation. They can well solve the disadvantages of traditional manual inspection methods of field observation and recording, such as high labor costs and high safety risks. The application of drones in the inspection of highway slopes can give full play to their advantages and improve inspection efficiency. During the drone inspection process, the images collected by the drone need to be spliced ​​to merge the continuous images collected by the drone into a complete highway slope panorama or three-dimensional model. In order to ensure that the overlapping areas can be perfectly matched during splicing and avoid dislocation or blur, it is necessary to align the adjacent video frames taken by the drone. The Lucas-Kanade (LK) algorithm can achieve efficient alignment of adjacent frames in the video taken by the drone by accurately estimating the motion vector between adjacent frames.

[0003] In the prior art, the study on real-time display of flow field based on optical flow algorithm published in CNKI with document number 1672-9897(2024)00-0001-10 disclosed: combining two types of algorithms, the LK optical flow algorithm with higher efficiency and the LS optical flow algorithm with stronger physical properties, to achieve high-precision real-time display of flow field. Through artificial synthetic particle image testing and towing water tank device, the efficiency, accuracy and robustness of the algorithm were systematically tested and compared with the cross-correlation algorithm. In the test of synthetic image data set, the calculation time of LK-I algorithm was the lowest, only 15ms, and the calculation accuracy and robustness of LK-S&LS algorithm were the highest, with root mean square and angular errors of 0.073 and 1.56 respectively, and the strongest anti-noise ability. The time consumption of cross-correlation algorithm was 1500ms, and the root mean square error and angular error were 0.128 and 3.13, which was significantly inferior to other optical flow algorithms. In the experimental test based on the towing tank, both the LK and LK&LS coupling algorithms can realize the rapid display of the velocity field and vorticity field in the near-wake region. Among them, the LK&LS algorithm has the least flow field noise, and the cross-correlation algorithm has the most noise.

[0004] However, in the process of using drones to shoot highway slope videos and align them, the existing LK algorithm has noise in the video frames due to the influence of the surrounding environment when shooting highway slope videos by drones. The existing LK algorithm uses a fixed window radius, which results in that the feature points in the window may not accurately reflect the actual motion situation under noise interference, because the noise pixels will have a certain influence on the optical flow calculation, thereby affecting the accuracy of the video frame alignment, resulting in a decrease in the stitching quality of the spliced ​​highway slope panorama, which in turn affects the accuracy and efficiency of the inspection of highway slope projects. Summary of the invention

[0005] In order to solve the problem that the existing LK algorithm adopts a fixed window radius, thereby affecting the accuracy of video frame alignment, resulting in reduced quality of the subsequent highway slope panorama, and further affecting the accuracy and efficiency of the inspection of the highway slope project, the present invention proposes a drone inspection method for highway slope projects, the method comprising the following steps:

[0006] A drone is used to collect a video frame sequence of a highway slope; for each pixel point in the video frame sequence, a surface fitting is performed on the actual local range and the reference local range of the pixel point to obtain two fitting surfaces of the pixel point; the noise interference degree of the pixel point is determined based on the two fitting surfaces and the structural similarity between the pixel point and the actual local range of the corresponding point of the pixel point in the previous frame; the preferred degree of the window radius of the pixel point is obtained by iteration, and the final window radius of the pixel point when performing the LK algorithm is determined. In each round of iteration: the preferred degree of the window radius of the pixel point is calculated, , For the Pixels during round iteration The preferred degree of window radius, and Respectively Pixels during round iteration The mean and the first The noise interference degree of each pixel, For the Pixels during round iteration The number of pixels in the window. is the natural exponential function, is an absolute value symbol; according to the final window radius, the LK algorithm is used to obtain the aligned video frame sequence, and then a panoramic view of the highway slope project is obtained to realize the inspection of the highway slope project.

[0007] By performing surface fitting on the local range of pixel points and dynamically adjusting the window radius in combination with the structural similarity between the pixel points and the corresponding points of the previous frame, the accuracy of each pixel point in the alignment process is optimized. Compared with the traditional method of fixing the window radius, the present invention can better adapt to the pixel characteristics in different scenes and improve the accuracy of the alignment results; by introducing the measurement and optimization iterative process of the noise interference degree, the noise interference in the image can be effectively identified and suppressed, because in actual scenes, factors such as slope terrain, illumination changes and image quality differences may cause the noise of the video frame to be more obvious; the optimized LK algorithm can align video frames more accurately, thereby generating a higher quality highway slope panorama, which helps to improve the accuracy and efficiency of slope inspection and facilitate subsequent engineering evaluation and problem detection; by dynamically adjusting the window radius, it can adapt to different slope terrains, illumination and shooting conditions, thereby improving the application effect in various complex environments, and through the automated pixel alignment and optimization process, the need for manual intervention is reduced, making the drone inspection process more efficient, and can quickly process and generate accurate slope panoramas, thereby improving the overall efficiency of the inspection operation.

[0008] Furthermore, the actual local range is based on pixel points. A window is constructed with the preset length as the center and the preset length as the side length.

[0009] Furthermore, the reference local range is obtained by using cubic spline interpolation to interpolate pixel points. The window after replacement.

[0010] Furthermore, the surface fitting is performed based on the least squares method.

[0011] Furthermore, the noise interference degree satisfies the following relationship:

[0012] ; In the formula, Pixel The noise interference level, Pixel The double integral of the two fitting surfaces of in the definition domain of the actual local range and the reference local range, and Pixels The actual local range and the fitting surface of the reference local range, Pixel and pixels At the corresponding point in the previous frame The actual local structural similarity of is the normalization function, is the absolute value symbol.

[0013] Combining local surface differences and structural similarities, a more accurate noise interference assessment is provided, which improves the accuracy of image processing. Extreme noise interference is suppressed through the normalization function, which enhances the stability and adaptability of the algorithm in complex environments. SSIM is used to evaluate structural similarity to further improve the accuracy of noise detection and adapt to dynamically changing images. Through comprehensive integral operations and structural similarity, the noise interference weight is automatically adjusted to adapt to different slope terrains and shooting conditions.

[0014] Furthermore, in the process of each round of iteration, it also includes: in response to the preference degree of the window radius of the pixel points obtained in three consecutive rounds of iterations including the current round of iteration showing a downward trend, the iteration is terminated, and the window radius corresponding to the preference degree of the window radius of the pixel points obtained in the first round of iteration among the three rounds of iteration is used as the final window radius.

[0015] By judging the downward trend of the window radius optimization degree in three consecutive rounds of iterations, the iteration is terminated in time, which avoids unnecessary repeated calculations and saves computing resources and time; the window radius of the first round of three iterations is selected as the final window radius to ensure the stability and reliability of the final result and avoid the fluctuations caused by subsequent iterations; by terminating the iteration in time and selecting the appropriate window radius, the algorithm can maintain good stability and adaptability in a dynamically changing environment, thereby enhancing the robustness of the algorithm.

[0016] Furthermore, the method of obtaining the aligned video frame sequence using the LK algorithm includes: obtaining the motion vectors of all pixels in each frame using the LK algorithm to obtain the optical flow field of each frame, and transforming each frame and its previous frame to obtain the aligned video frame sequence.

[0017] By using the LK algorithm to calculate the motion vector of pixel points, the motion information between video frames can be accurately estimated, improving the accuracy of motion tracking; by transforming each frame and its previous frame, the difference between frames can be effectively eliminated, ensuring a smooth transition of video frames and reducing screen jitter or dislocation; retaining inter-frame details to avoid image information loss and ensure that fine motion changes in dynamic video processing are better displayed.

[0018] Furthermore, the panoramic image is obtained by stitching based on image stitching technology.

[0019] By generating panoramas through image stitching technology, multiple local images can be seamlessly integrated to expand the field of view, enhance the image's sense of space and detail, and is suitable for the display of wide scenes or the reconstruction of complex environments.

[0020] The present invention has the following beneficial effects:

[0021] In view of the defect of the fixed window radius of the traditional LK algorithm, the present invention uses surface fitting and structural similarity analysis combined with iterative calculation to adaptively determine the window radius for each pixel point, so that the algorithm can accurately capture pixel motion information and significantly improve the accuracy of video frame alignment; the present invention effectively avoids the splicing problems caused by frame dislocation and blur, and the produced highway slope panorama is clearer and more accurate, providing high-quality images for subsequent inspections; high-quality panorama can help inspection personnel accurately judge the hidden dangers of slope engineering, reduce misjudgment and repeated collection, reduce time and resource costs, and improve the accuracy and efficiency of inspections. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] By reading the following detailed description with reference to the accompanying drawings, the above and other objects, features and advantages of the exemplary embodiments of the present invention will become readily understood. In the accompanying drawings, several embodiments of the present invention are shown in an exemplary and non-limiting manner, and the same or corresponding reference numerals represent the same or corresponding parts, wherein:

[0023] Figure 1 The present invention is a flowchart of a method for using an unmanned aerial vehicle to inspect a highway slope project according to an embodiment of the present invention. DETAILED DESCRIPTION

[0024] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.

[0025] In the LK algorithm, the window radius defines the size of the local neighborhood used to calculate the motion vector, so the selection of the window radius will directly affect the accuracy of the optical flow estimation, and then affect the accuracy of the video frame alignment. The traditional LK algorithm often uses a fixed window radius, which will make it difficult for the algorithm to adapt to various noises in the highway slope video frame. If the window radius is set too small, the LK algorithm will be greatly disturbed by the noise, and if the window radius is set too large, the calculation efficiency will decrease. Therefore, the present invention will calculate the degree of noise interference by analyzing the grayscale changes in the highway slope video frame image, and then use the noise interference degree in the local range of the pixel point to adapt the pixel point to the window radius of the LK algorithm, so that the LK algorithm maintains the calculation accuracy while maintaining the calculation efficiency, thereby improving the accuracy of the highway slope video frame alignment.

[0026] The specific embodiments of the present invention are described in detail below with reference to the accompanying drawings.

[0027] See also Figure 1, which shows a flowchart of a method for unmanned aerial vehicle inspection of a highway slope project provided by an embodiment of the present invention, the method comprising the following steps:

[0028] S1: Video frame sequence of highway slope collected by UAV.

[0029] Use a drone to take pictures of the highway slope. The drone should be close to the slope surface during shooting. Shooting should be done during the day to ensure normal sunlight intensity and avoid overexposure or underexposure. Scan and shoot detailed pictures of the highway slope in a "J" shape from top to bottom according to the planned route to obtain a video frame sequence of the highway slope.

[0030] Implementers can set the distance from the slope according to specific implementation conditions, for example, 10 meters.

[0031] S2: Obtain two fitting surfaces of the pixel points in each frame in the video frame sequence.

[0032] It should be noted that, because of the influence of the surrounding environment during the process of shooting highway slope videos by drones, there may be noise in the video frames of the highway slope videos. Such noise often manifests itself as mutation points within the local range of pixels. Therefore, the mutation of pixels within the local range can be used to calculate the degree of noise interference for each pixel in the video frame. The mutation of a pixel point within its local range can be represented by the gap between the actual local range of the current pixel point and the reference local range, wherein the actual local range is the actual local range of the current pixel point in the video frame, and the reference local range is the local range after the current data point is replaced by an interpolation algorithm based on the actual local range of the current pixel point; and the gap between the actual local range and the reference local range can be represented by the volume enclosed by the fitting surfaces of the two local ranges, because the mutation point will not only affect its own fitting process but also the fitting process of the surrounding pixels. Therefore, if the mutation of the pixel point is higher, the fitting error of the fitting surface of its actual local range will be greater. After the reference local range is interpolated, its mutation becomes weaker, and the fitting error of the fitting surface of the reference local range will become smaller. Therefore, the larger the volume enclosed by the fitting surfaces of the actual local range and the reference local range, the greater the gap and the higher the mutation.

[0033] For a pixel point in each frame in the video frame sequence, surface fitting is performed on the actual local range and the reference local range of the pixel point to obtain two fitting surfaces of the pixel point.

[0034] Specifically, the actual local range is based on pixel points A window is constructed with the preset length as the center and the preset length as the side length.

[0035] Implementers can set the actual local range size according to specific implementation conditions, for example, in pixels. centered window.

[0036] Specifically, the reference local range is to use the cubic spline interpolation method to interpolate the pixel points The window after replacement.

[0037] Specifically, the surface fitting is performed based on the least square method.

[0038] S3: Determine the noise interference degree of the pixel point.

[0039] It should be noted that in the video frame of the highway slope, the edge positions of stones, sand, grass, etc. in the image may also cause large grayscale changes in the image, that is, lead to higher mutation. Therefore, the present invention uses the structural similarity between two adjacent frames of images (the adjacent frame in this case is the previous frame of the current video frame, and the first frame does not have a previous frame of the image, so the first frame of the image is not processed) as the credibility of the mutation, corrects the mutation, and thus obtains the degree of noise interference; because the changes in adjacent video frames in the video are small, the higher the structural similarity between adjacent images, the more it indicates that the current mutation point is caused by noise, and the higher the credibility of the mutation characterizing the degree of noise interference.

[0040] The noise interference degree of the pixel point is determined according to the two fitting surfaces and the structural similarity between the pixel point and the actual local range of the corresponding point of the pixel point in the previous frame.

[0041] Specifically, the noise interference degree satisfies the following relationship:

[0042] ;

[0043] In the formula, Pixel The noise interference level, Pixel The double integral of the two fitting surfaces of in the definition domain of the actual local range and the reference local range, and Pixels The actual local range and the fitting surface of the reference local range, Pixel and pixels At the corresponding point in the previous frame The actual local structural similarity of is the normalization function, is the absolute value symbol.

[0044] in, The structural similarity index is an indicator that measures the similarity between two images. It comprehensively evaluates the similarity between images from three aspects: brightness, contrast, and structure. Represents pixel and pixels At the corresponding point in the previous frame The actual local structural similarity of arrive , the larger the value, the more similar they are. Represents the linear normalization operation of the structural similarity value. The normalized structural similarity can represent the pixel point Mutability characterizes the credibility of the degree of noise interference.

[0045] S4: Calculate the preferred degree of the window radius of the pixel point and determine the final window radius.

[0046] It should be noted that when performing LK algorithm calculations, the role of the pixels in the window to which the current pixel belongs is to calculate the optical flow estimate of the current pixel. Therefore, it is necessary to ensure that the data in the window is less affected by noise interference, that is, enough reliable pixels are required to participate in the calculation; therefore, if the noise performance of the pixels in the window corresponding to a window radius is lower, the higher its preference; wherein, the noise performance is first represented by the mean of the noise interference of the pixels in the window, and then what is required is that there are enough real pixels in the window, and when the mean represents the noise performance, there may be a problem that the noise interference of individual pixels is larger, resulting in a larger overall noise performance; therefore, the average outlier of the noise interference of the pixels in the window is used to characterize the credibility of the window noise performance, and the higher the average outlier, the higher the credibility of the noise performance.

[0047] The optimal degree of the window radius of the pixel point is obtained by iteration, and the final window radius of the pixel point when performing the LK algorithm is determined. In each round of iteration:

[0048] Calculate the preferred degree of the window radius of the pixel:

[0049] ;

[0050] In the formula, For the Pixels during round iteration The preferred degree of window radius, and Respectively Pixels during round iteration The mean and the first The noise interference degree of each pixel, For the Pixels during round iteration The number of pixels in the window. is the natural exponential function, is the absolute value symbol.

[0051] Implementers can set the initial window radius (window radius in the first iteration) and iteration step size according to specific implementation conditions. For example, the initial window radius is , the iteration step length is ; Among them, the radius of the first round window When the pixel When the window is centered, the window size is , corresponding to the number of pixels in the window , the radius of the second round window is , then the corresponding window size is , the number of pixels in the window at this time .

[0052] in, Pixel The difference between the noise interference degree of and the mean value of the noise interference degree in the current window indicates its outlier nature, then The average outlier of the current window represents the confidence level of the window noise performance.

[0053] Specifically, each round of iteration also includes:

[0054] In response to the fact that the preference degree of the window radius of the pixel points obtained in three consecutive iterations including the current iteration shows a downward trend, the iteration is terminated, and the window radius corresponding to the preference degree of the window radius of the pixel points obtained in the first iteration of the three iterations is used as the final window radius.

[0055] S5: According to the final window radius, the LK algorithm is used to obtain an aligned video frame sequence, and then a panoramic view of the highway slope project is obtained to realize the inspection of the highway slope project.

[0056] Specifically, the LK algorithm is used to obtain the motion vectors of all pixels in each frame, and the optical flow field of each frame is obtained (the motion vector constitutes the optical flow field of this frame relative to the previous frame, and the optical flow field is essentially a vector field that describes the direction and speed of pixel movement). Each frame and its previous frame are transformed to obtain an aligned video frame sequence.

[0057] Specifically, the panoramic image is obtained by stitching based on image stitching technology.

[0058] Through the above steps, a panoramic picture of the highway slope is obtained. Then, AI technology or manual work can be used to identify defects in the panoramic image of the highway slope project to determine whether there are cracks, landslides, collapses and other defects on the slope. Based on the results of disease identification and analysis, a detailed inspection report is generated and fed back to relevant departments and personnel so that targeted governance measures can be taken to complete the drone inspection of highway slope projects.

[0059] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A drone inspection method for highway slope engineering, characterized in that: include: Use drones to collect video frame sequences of highway slopes; For each pixel point in a video frame sequence, surface fitting is performed on the actual local range and the reference local range of the pixel point to obtain two fitting surfaces of the pixel point; Determining the noise interference degree of the pixel point based on the two fitted surfaces and the structural similarity between the pixel point and the actual local range of the corresponding point of the pixel point in the previous frame; The optimal degree of the window radius of the pixel point is obtained by iteration, and the final window radius of the pixel point when performing the LK algorithm is determined. In each round of iteration: Calculate the preferred degree of the window radius of the pixel point, , For the Pixels during round iteration The preferred degree of window radius, and Respectively Pixels during round iteration The mean and the first The noise interference degree of each pixel, For the Pixels during round iteration The number of pixels in the window. is the natural exponential function, is the absolute value symbol; According to the final window radius, the LK algorithm is used to obtain an aligned video frame sequence, and then a panoramic view of the highway slope project is obtained to realize the inspection of the highway slope project.

2. The method for unmanned aerial vehicle inspection of highway slope engineering according to claim 1, characterized in that: The actual local range is in pixels. A window is constructed with the preset length as the center and the preset length as the side length.

3. The method for unmanned aerial vehicle inspection for highway slope engineering according to claim 2, characterized in that: The reference local range is obtained by using cubic spline interpolation to interpolate pixel points The window after replacement.

4. The method for unmanned aerial vehicle inspection for highway slope engineering according to claim 1, characterized in that: The surface fitting is performed based on the least square method.

5. The method for unmanned aerial vehicle inspection for highway slope engineering according to claim 1, characterized in that: The noise interference degree satisfies the following relationship: ; In the formula, Pixel The noise interference level, Pixel The double integral of the two fitting surfaces of in the definition domain of the actual local range and the reference local range, and Pixels The actual local range and the fitting surface of the reference local range, Pixel and pixels At the corresponding point in the previous frame The actual local structural similarity of is the normalization function, is the absolute value symbol.

6. The method for unmanned aerial vehicle inspection for highway slope engineering according to claim 1, characterized in that: In each round of iteration, the following steps are also included: In response to the fact that the preference degree of the window radius of the pixel points obtained in three consecutive iterations including the current iteration shows a downward trend, the iteration is terminated, and the window radius corresponding to the preference degree of the window radius of the pixel points obtained in the first iteration of the three iterations is used as the final window radius.

7. The method for unmanned aerial vehicle inspection for highway slope engineering according to claim 1, characterized in that: The method of obtaining the aligned video frame sequence by using the LK algorithm includes: The LK algorithm is used to obtain the motion vectors of all pixels in each frame, obtain the optical flow field of each frame, and transform each frame and its previous frame to obtain an aligned video frame sequence.

8. The method for unmanned aerial vehicle inspection for highway slope engineering according to claim 1, characterized in that: The panoramic image is obtained by stitching based on image stitching technology.

Images