High-speed construction safety monitoring method based on unmanned aerial vehicle inspection

Abstract

The present application relates to the technical field of image processing, in particular to a high-speed construction safety monitoring method based on unmanned aerial vehicle inspection. After each frame of construction image is divided into multiple local areas, the first anchor point area in each frame of construction image is screened out through image similarity analysis. The spatial distance between each local area and the first anchor point area and the splicing contribution degree similarity are analyzed, and the splicing influence degree of the local area relative to its adjacent local area is combined to obtain the anchor point potential degree of each local area. The second anchor point area is screened out based on the anchor point potential degree, and the image is faded and spliced, and then the final monitoring image is transmitted. The present application can effectively fade the local area without splicing reference, and retain the features of the area with strong splicing reference, so as to reduce the image calculation amount while retaining the important features for splicing, improve the efficiency of image splicing, and ensure the efficiency of monitoring information transmission during unmanned aerial vehicle inspection.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and specifically to a high-speed construction safety monitoring method based on drone inspection. Background Technology

[0002] Traditional methods of monitoring highway construction often rely on fixed ground cameras or manual patrols. These methods have limited field of view and struggle to cover large areas and complex terrains (such as high slopes, deep cuts, and overpasses), resulting in blind spots. Furthermore, the real-time performance and security of these inspections are often delayed or limited. Consequently, monitoring data acquired using traditional methods is mostly static images or partial video, making it difficult to achieve efficient and high-quality construction safety monitoring. Utilizing drones for inspection is a novel approach to monitoring highway construction processes. By leveraging the high flexibility of drones combined with real-time data transmission technology, it is possible to achieve large-scale, all-weather monitoring without blind spots.

[0003] In the process of using drones for inspection and safety monitoring of high-speed construction, it is often necessary to monitor key areas such as high slopes and deep road cuts, as well as the overall construction distribution comparison of large areas. Therefore, during drone inspection, it is necessary to perform image fusion and stitching on consecutive frames to obtain a panoramic view of the monitored area. Existing image stitching algorithms usually select a sufficient number of feature points on two images for comparison and matching before stitching, which is computationally redundant and inefficient. Summary of the Invention

[0004] To address the technical problem that existing technologies, when stitching images from UAV inspections, directly rely on feature point matching, resulting in redundant computation and hindering the rapid and efficient stitching and transmission of monitoring information, this invention aims to provide a high-speed construction safety monitoring method based on UAV inspections. The specific technical solution adopted is as follows: This invention proposes a method for high-speed construction safety monitoring based on unmanned aerial vehicle (UAV) inspection, the method comprising: Acquire continuous multi-frame construction images collected during drone inspections; divide the construction images into multiple local regions; Based on the image similarity between each local region and all local regions in the adjacent frame construction image, the stitching contribution of each local region is obtained and a first anchor point region is selected from each frame construction image. For each frame of construction image, the spatial distance between each local region (excluding the first anchor point region) and the first anchor point region is obtained; the splicing influence degree of the local region is obtained based on the uniformity of the splicing contribution between the local region and its adjacent local regions; and the anchor point potential degree of the local region is obtained based on the spatial distance, the splicing influence degree, and the similarity of the splicing contribution between the local region and the first anchor point region. The second anchor point region is selected based on the anchor point potential, and all anchor point regions are then faded. The faded continuous frame construction images are then stitched together, and the stitched panoramic construction image is transmitted as monitoring information.

[0005] Furthermore, the filtering method for the first anchor point region includes: For each local region, the image similarity between the local region and all local regions in the next frame of the construction image is calculated to obtain the stitching contribution of each local region. Select the local area with the largest splicing contribution as the first anchor point area.

[0006] Furthermore, the method for obtaining the splicing contribution includes: For each local region, the maximum image similarity of all local regions in the next frame of the construction image is taken as the stitching contribution.

[0007] Furthermore, the method for obtaining image similarity includes: The mean square error of the gray values ​​of pixels at the same position between two local regions is obtained, and the cosine similarity of the overall gray-level gradient direction between the two local regions is obtained; the image similarity is obtained based on the mean square error and the cosine similarity.

[0008] Furthermore, the method for obtaining the degree of splicing influence includes: The difference in splicing contribution between the local region and each adjacent local region is obtained. The average value of the splicing contribution difference is negatively correlated and normalized to obtain the degree of splicing influence.

[0009] Furthermore, the method for obtaining the anchor point potential includes: The product of the spatial distance, the degree of splicing influence, and the similarity of splicing contribution is used as the anchor point potential.

[0010] Furthermore, the method for obtaining the splicing contribution similarity includes: The absolute value of the difference between the splicing contribution of the local region and the first anchor point region is negatively correlated and normalized to obtain the splicing contribution similarity.

[0011] Furthermore, the dilution process includes: The local area where the anchor point potential is greater than the preset potential threshold is designated as the second anchor point area; Using the first and second anchor regions as centers, a fade-out algorithm based on contribution decay radius is used to fade out other local regions. In the fade-out algorithm, the local regions of non-anchor regions obtain fade-out weights by referencing the nearest anchor region.

[0012] Furthermore, the step of stitching together the faded, consecutive construction images includes: Feature points in adjacent construction images after fading are obtained using the Gaussian pyramid algorithm. The feature value of each feature point is obtained using the scale-invariant feature transformation algorithm. Feature points between adjacent construction images are matched based on the feature values. The homography matrix between adjacent construction images is calculated based on the matching results. The homography matrix is ​​then used for image calibration.

[0013] Furthermore, the spatial distance is the distance between the centroids of the two regions.

[0014] The present invention has the following beneficial effects: To reduce the computational burden caused by feature point redundancy during image stitching, this invention employs a blurring strategy to blur and fade areas with weak stitching features in the image, thereby reducing the number of feature points and improving image stitching efficiency. In the blurring strategy, considering that the reference area for feature points that can be used for stitching should be a shared area between adjacent construction images, this invention divides each frame of construction images into multiple local regions and then uses image similarity analysis to select the first anchor point region in each frame. Furthermore, this invention considers that in addition to the first anchor point region, there should also be feature anchor points that can be used for image stitching. Therefore, it analyzes the spatial distance and stitching contribution similarity between each local region and the first anchor point region, and combines this with the stitching influence of each local region relative to its adjacent local regions to obtain the anchor point potential of each local region. The greater the anchor point potential, the farther the local area is from the first anchor point area, while having a similar stitching reference degree and a strong influence degree reflected in the neighborhood information. Therefore, by selecting the second anchor point area based on the anchor point potential degree and fading the image, local areas without stitching reference degree can be effectively faded, while features are preserved for areas with strong stitching reference degree. This reduces the amount of image computation while retaining important features for stitching, improving the efficiency of image stitching and ensuring the efficiency of monitoring information transmission during UAV inspection. Attached Figure Description

[0015] To more clearly illustrate the technical solutions and advantages in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0016] Figure 1 This is a flowchart of a high-speed construction safety monitoring method based on drone inspection, provided as an embodiment of the present invention. Detailed Implementation

[0017] To further illustrate the technical means and effects adopted by the present invention to achieve its intended purpose, the following, in conjunction with the accompanying drawings and preferred embodiments, details the specific implementation, structure, features, and effects of a high-speed construction safety monitoring method based on unmanned aerial vehicle (UAV) inspection proposed according to the present invention. In the following description, different "one embodiment" or "another embodiment" do not necessarily refer to the same embodiment. Furthermore, specific features, structures, or characteristics in one or more embodiments can be combined in any suitable form.

[0018] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains.

[0019] The following description, in conjunction with the accompanying drawings, details a specific scheme for a high-speed construction safety monitoring method based on unmanned aerial vehicle (UAV) inspection provided by this invention.

[0020] Please see Figure 1 The diagram illustrates a flowchart of a high-speed construction safety monitoring method based on unmanned aerial vehicle (UAV) inspection, according to an embodiment of the present invention. The method includes: Step S1: Acquire multiple consecutive construction images collected during the drone inspection process; divide the construction images into multiple local areas.

[0021] This invention addresses high-speed construction safety monitoring, employing drones to capture images at the construction site. It's important to note that because the monitoring objective is to transmit a panoramic image of the construction site, the drone's flight speed and image capture frequency need adjustment to ensure overlap between adjacent frames and that each frame is the same size. Specific drone camera parameter settings are well-known to those skilled in the art and will not be limited or elaborated upon here. Spatially overlapping content in adjacent frames allows for more accurate feature matching and image calibration, resulting in better stitching performance.

[0022] It should be noted that, in this embodiment of the invention, after acquiring the construction images, image preprocessing is required to facilitate subsequent image processing. Image preprocessing includes denoising and grayscale conversion. The specific preprocessing methods are well-known to those skilled in the art and will not be elaborated here.

[0023] In existing technologies, image stitching based on UAV inspections requires global feature point matching, resulting in significant computational redundancy. In reality, not every image feature point needs matching. For example, if two consecutive time frames both contain blue sky and white clouds as a background, the reference value of selected feature points is not particularly high. Therefore, this invention aims to minimize the impact of these less relevant regions on image stitching, retaining more relevant regions for matching analysis to improve efficiency. Thus, this invention divides each construction image frame into multiple local regions using the same block-segmentation method, and subsequently analyzes the characteristics of these local regions.

[0024] It should be noted that the method for segmenting construction images can be set according to the specific image resolution, as long as the segmentation strategy is the same for each frame of construction images. This embodiment of the invention does not impose any limitations on this method. In one specific implementation of this embodiment, the construction image is segmented into 25×25 pixel blocks. If the edge portion cannot form a 25×25 pixel image, zero-padding is performed along the edge to ensure that each image block is 25×25 pixels in size. The specific zero-padding algorithm is a technique well-known to those skilled in the art and will not be described in detail here.

[0025] Step S2: Based on the image similarity between each local region and all local regions in the adjacent frame construction image, obtain the stitching contribution of each local region and select a first anchor point region in each frame construction image.

[0026] For adjacent construction images, since image stitching is required, it is ensured that there is common image information between adjacent frames during shooting. This means that there are a small number of overlapping areas and a large number of non-overlapping areas between the two images. The overlapping areas are the areas in this embodiment of the invention that are intended to retain more features for image stitching reference; while the non-overlapping areas are areas that need to be faded to reduce computational load. For overlapping areas, the same visual information will inevitably be generated between the two frames. Therefore, this embodiment of the invention can select the first anchor point region in each frame of construction images based on the image similarity between each local region and all local regions in the adjacent frame of construction images. That is, the first anchor point region is the local region in one frame of construction images that has the highest similarity to the next frame of construction images. This indicates that it is an overlapping region with a strong stitching reference value, that is, the greater the stitching contribution.

[0027] Preferably, in this embodiment of the invention, the method for filtering the first anchor point region includes: For each local region, the image similarity between that local region and all local regions in the next frame of the construction image is calculated to obtain the stitching contribution of each local region. That is, the larger the stitching contribution, the stronger the image similarity between that local region and the local regions in the next frame of the construction image. Therefore, the local region with the largest stitching contribution is selected as the first anchor point region.

[0028] Furthermore, in this embodiment of the invention, the method for obtaining the splicing contribution includes: For each local region, the maximum image similarity among all local regions in the next frame of the construction image is used as the stitching contribution. That is, the maximum value is calculated and used as a reference value to select the first anchor point region based on the stitching contribution.

[0029] Furthermore, in this embodiment of the invention, the method for obtaining image similarity includes: The mean square error of the grayscale values ​​of pixels at the same position between two local regions is obtained, and the cosine similarity of the overall grayscale gradient direction between the two local regions is obtained. The image similarity is obtained based on the mean square error and the cosine similarity. That is, for two local regions, the smaller the mean square error, the smaller the difference between pixels at the same position, and the greater the similarity of the image information. Because the angle of the drone may have changed when the two images were taken, even if the two regions describe the same object, the grayscale values ​​will have a certain difference due to the angle deviation. In this case, the mean square error may be large. Therefore, it is necessary to further consider the similarity of the overall grayscale gradient direction. The overall grayscale gradient direction is the sum of the gradient directions of all pixels in a local region. The greater the cosine similarity between the overall grayscale gradient directions, the more similar the grayscale change directions are between the two local regions. Although the shooting angle of the same region may change, the grayscale change trend will not change significantly. For example, if the previous frame image shows a complete structure, and the next frame image shows a missing part due to the movement of the drone, although a part is missing, the gradient direction of the grayscale value is likely to be the same as the gradient direction of the overall object grayscale value. Therefore, the final cosine similarity should be positively correlated with the image similarity, while the mean square error should be negatively correlated with the image similarity.

[0030] As a specific example, in one implementation of this invention, the mean squared error plus a preset parameter is used as the denominator, and the cosine similarity is used as the numerator. The resulting ratio is used as the image similarity. That is, the correlation is constructed using a ratio. The preset parameter in the denominator prevents the denominator from being zero; in this embodiment, it can be 0.1. In other implementations of this invention, the correlation can be constructed in other ways, which are not limited or elaborated here.

[0031] Step S3: For each frame of construction image, obtain the spatial distance between each local region (excluding the first anchor point region) and the first anchor point region; obtain the splicing influence degree of the local region based on the image similarity between the local region and its adjacent local regions; obtain the anchor point potential degree of the local region based on the spatial distance, the splicing influence degree, and the similarity of the splicing contribution degree between the local region and the first anchor point region.

[0032] Since each frame of the construction image is segmented in step S1, although a first anchor point region can be determined through image similarity in step S2, this region cannot completely encompass all overlapping areas. If only the first anchor point region is retained and all other local regions are faded, the effective information will be diluted, thus affecting the stitching quality. Therefore, this embodiment of the invention needs to further analyze each other local region, quantifying the stitching reference value of other local regions by analyzing their spatial relationship and similarity relationship with the first anchor point region.

[0033] Firstly, spatial analysis is performed. This embodiment of the invention considers that the overlapping area may be a relatively large region compared to a local area. Therefore, in areas far from the first anchor point region and its neighborhood, other anchor point regions may still exist. Because the image fading process gradually increases the fading degree based on the anchor point region as the center, if fading is only performed based on the first anchor point region, other unidentified anchor point regions will be given a greater fading degree, thus fading useful information. Although this reduces computational load, it increases stitching error. Therefore, this embodiment of the invention further analyzes each local region in each frame of the construction image, excluding the first anchor point region. Considering that regions farther from the first anchor point region are more likely to be new anchor point regions, the spatial distance between the local region and the first anchor point region is first obtained. Furthermore, considering that a local region is an anchor point region, meaning it belongs to an overlapping region, it indicates that the local region is similar to the first anchor point region. Its adjacent local regions should also have strong splicing reference value, that is, their splicing contribution should be similar and exhibit a relatively uniform distribution. Therefore, the splicing influence degree of the local region is obtained based on the uniformity of the splicing contribution between the local region and its adjacent local regions. Combining the similarity of the splicing contribution between the local region and the first anchor point region—that is, the greater the similarity of the splicing contribution, the more likely the local region is to be an anchor point region—all features are statistically analyzed to obtain the anchor point potential of each local region. In other words, the greater the anchor point potential, the more likely the local region is to be a new anchor point region.

[0034] Preferably, in this embodiment of the invention, the method for obtaining the degree of splicing influence includes: The difference in splicing contribution between the local region and each adjacent local region is obtained. The average value of the splicing contribution difference is negatively correlated and normalized to obtain the degree of splicing influence. That is, the greater the degree of splicing influence, the more uniform the distribution of splicing contribution between the local region and adjacent local regions, which indicates that the local region is more likely to be located within the overlapping region and can act as an anchor region to influence the surrounding local regions.

[0035] In this embodiment of the invention, the difference in splicing contribution is the absolute value of the difference between the splicing contributions. The method for negative correlation mapping and normalization can be implemented using the exp(-x) function, an exponential function with the natural constant as its base. Substituting the difference in splicing contribution into this function achieves negative correlation mapping and normalization. Other implementations of this embodiment can also use other basic mathematical methods to achieve negative correlation mapping and normalization, which will not be elaborated upon here. In this embodiment of the invention, adjacent local regions are local regions that share an edge and are directly adjacent to each other.

[0036] Furthermore, methods for obtaining the similarity of splicing contributions include: The absolute value of the difference between the splicing contribution of the local region and the first anchor point region is negatively correlated and normalized to obtain the splicing contribution similarity. Similar to the splicing influence described above, the same exponential function can be used to achieve negative correlation mapping and normalization in this embodiment of the invention, which will not be elaborated further.

[0037] Preferably, since the spatial distance, the degree of splicing influence, and the similarity of splicing contribution should all be positively correlated with the anchor point potential, the product of the spatial distance, the degree of splicing influence, and the similarity of splicing contribution is taken as the anchor point potential.

[0038] It should be noted that, because the similarity of the splicing influence and the splicing contribution are normalized in one specific implementation of this invention, the spatial distance also needs to be normalized before being multiplied by the other two features in order to unify the dimensions. The normalization method can be range standardization, that is, statistically analyzing the maximum and minimum spatial distances and normalizing each spatial distance data point. This is a technique well-known to those skilled in the art and will not be elaborated upon here. The spatial distance refers to the distance between the centroids of two local regions, which can be represented by the Euclidean distance of the centroid coordinates; further details will not be provided.

[0039] Step S4: Select the second anchor point area based on the anchor point potential, and perform fade processing on all anchor point areas; stitch the faded continuous frame construction images together, and transmit the stitched panoramic construction image as monitoring information.

[0040] After obtaining the anchor point potential of each local region, the second anchor point region can be selected based on the anchor point potential. It should be noted that there are multiple second anchor point regions, which, together with the first anchor point region, are collectively referred to as the anchor point region. Fading processing can be performed using the anchor point region as the center. Feature extraction and stitching of the faded consecutive frames of construction images reduces computational load while retaining more useful information for feature matching, thus improving stitching efficiency. The stitched panoramic construction image can then be transmitted.

[0041] Preferably, in this embodiment of the invention, the fading process includes: The local area with anchor point potential exceeding a preset potential threshold is designated as the second anchor point area. In this embodiment of the invention, in the initial stage, the top 20% of local areas in the construction image, ranked from largest to smallest anchor point potential, can be designated as the second anchor point area, i.e., the potential threshold is set to the 20th percentile. After the algorithm is executed, more sample data can be collected to adjust the threshold value for optimization. Specific techniques are well-known to those skilled in the art and will not be elaborated upon here.

[0042] Using the first and second anchor point regions as centers, a fade-out algorithm based on the contribution decay radius is applied to fade other local regions. Because a local region may receive fade-out results from multiple anchor point regions simultaneously, the fade-out algorithm uses the nearest anchor point region as a reference to obtain the fade-out weight for local regions outside the anchor point region.

[0043] In this embodiment of the invention, the fade weight can be directly set as the result of a negative correlation mapping and normalization of the distance between the local region and the nearest anchor point region; alternatively, it can be a negative correlation mapping and normalization of the ratio between the square of this distance and the square of the preset attenuation radius. The specific attenuation radius needs to be set according to the image size. The implementation of negative correlation mapping and normalization can also be achieved using the exponential function described above, which will not be elaborated here. After obtaining the weight, smoothing can be achieved using alpha adjustment or grayscale smoothing algorithms.

[0044] Preferably, in this embodiment of the invention, stitching together the faded consecutive construction images includes: The Gaussian pyramid algorithm is used to obtain feature points in adjacent frames of construction images after fading. The scale-invariant feature descriptor describes each feature point with good accuracy. Therefore, the scale-invariant feature transformation algorithm is used to obtain the feature value of each feature point.

[0045] Feature points between adjacent construction images are matched based on the feature values, and the homography matrix between adjacent construction images is calculated based on the matching results. The homography matrix is ​​then used for image calibration.

[0046] It should be noted that since there is no next frame of construction image after the last frame, the last frame of construction image can be compared and analyzed with the previous frame of construction image to obtain the corresponding faded result and then stitched with the previous frame image, thus achieving the stitching of all consecutive frame construction images.

[0047] In summary, this invention divides each frame of construction images into multiple local regions and then uses image similarity analysis to select the first anchor point region in each frame. It analyzes the spatial distance and stitching contribution similarity between each local region and the first anchor point region, and combines this with the stitching influence of each local region relative to its adjacent local regions to obtain the anchor point potential of each local region. Based on the anchor point potential, a second anchor point region is selected, and the image is then faded. The faded result is then stitched together, and the final monitoring image is transmitted. This invention can effectively fade local regions without stitching reference, while preserving features for regions with strong stitching reference. This reduces image computation while retaining important features for stitching, improving image stitching efficiency and ensuring efficient transmission of monitoring information during UAV inspections.

[0048] It should be noted that the order of the above embodiments of the present invention is merely for descriptive purposes and does not represent the superiority or inferiority of the embodiments. The processes depicted in the accompanying drawings do not necessarily require a specific or sequential order to achieve the desired result. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0049] The various embodiments in this specification are described in a progressive manner. The same or similar parts between the various embodiments can be referred to each other. Each embodiment focuses on describing the differences from other embodiments.

Claims

1. A method for high-speed construction safety monitoring based on unmanned aerial vehicle (UAV) inspection, characterized in that, The method includes: Acquire continuous multi-frame construction images collected during drone inspections; divide the construction images into multiple local regions; Based on the image similarity between each local region and all local regions in the adjacent frame construction image, the stitching contribution of each local region is obtained and a first anchor point region is selected from each frame construction image. For each frame of construction image, the spatial distance between each local region (excluding the first anchor point region) and the first anchor point region is obtained; the splicing influence degree of the local region is obtained based on the uniformity of the splicing contribution between the local region and its adjacent local regions; and the anchor point potential degree of the local region is obtained based on the spatial distance, the splicing influence degree, and the similarity of the splicing contribution between the local region and the first anchor point region. The second anchor point area is selected based on the anchor point potential, and all anchor point areas are faded; the continuous frame construction images after the faded processing are stitched together, and the stitched panoramic construction image is transmitted as monitoring information. The filtering method for the first anchor point region includes: For each local region, the image similarity between the local region and all local regions in the next frame of the construction image is calculated to obtain the stitching contribution of each local region. Select the local region with the largest splicing contribution as the first anchor point region; The method for obtaining the degree of splicing influence includes: The difference in splicing contribution between the local region and each adjacent local region is obtained. The average value of the splicing contribution difference is negatively correlated and normalized to obtain the degree of splicing influence. The method for obtaining the anchor point potential includes: The product of the spatial distance, the degree of splicing influence, and the similarity of splicing contribution is used as the anchor point potential degree. Methods for obtaining the similarity of splicing contributions include: The absolute value of the difference between the splicing contribution of the local region and the first anchor point region is negatively correlated and normalized to obtain the splicing contribution similarity.

2. The method for high-speed construction safety monitoring based on UAV inspection according to claim 1, characterized in that, The method for obtaining the splicing contribution includes: For each local region, the maximum image similarity of all local regions in the next frame of the construction image is taken as the stitching contribution.

3. The method for high-speed construction safety monitoring based on UAV inspection according to claim 1, characterized in that, The method for obtaining image similarity includes: The mean square error of the gray values ​​of pixels at the same position between two local regions is obtained, and the cosine similarity of the overall gray-level gradient direction between the two local regions is obtained; the image similarity is obtained based on the mean square error and the cosine similarity.

4. The method for high-speed construction safety monitoring based on UAV inspection according to claim 1, characterized in that, The desalination process includes: The local area where the anchor point potential is greater than the preset potential threshold is designated as the second anchor point area; Using the first and second anchor regions as centers, a fade-out algorithm based on contribution decay radius is used to fade out other local regions. In the fade-out algorithm, the local regions of non-anchor regions obtain fade-out weights by referencing the nearest anchor region.

5. A method for high-speed construction safety monitoring based on unmanned aerial vehicle (UAV) inspection according to claim 1, characterized in that, The step of stitching together the faded, consecutive construction images includes: Feature points in adjacent construction images after fading are obtained using the Gaussian pyramid algorithm. The feature value of each feature point is obtained using the scale-invariant feature transformation algorithm. Feature points between adjacent construction images are matched based on the feature values. The homography matrix between adjacent construction images is calculated based on the matching results. The homography matrix is ​​then used for image calibration.

6. A method for high-speed construction safety monitoring based on unmanned aerial vehicle (UAV) inspection according to claim 1, characterized in that, The spatial distance is the distance between the centroids of the two regions.

Images