Bridge dynamic displacement high-precision identification method based on digital image processing

Abstract

The invention relates to the technical field of image enhancement, in particular to a bridge dynamic displacement high-precision identification method based on digital image processing, which comprises the following steps of: taking any frame of video image in a video image sequence of a bridge to be identified as a target video image, and according to an LBP value and a gradient magnitude of each pixel point in the target video image, calculating the dynamic displacement of the bridge according to the LBP value and the gradient magnitude of each pixel point; determining each fuzzy region of the target video image; determining a motion law index of each fuzzy region of the target video image in adjacent frames and a motion track stability index of each fuzzy region in continuous frames, and further determining an influence degree of each fuzzy region on bridge displacement recognition precision; and performing enhancement processing on the target video image by using the influence degree to obtain the target video image after enhancement processing, and further obtaining each frame of video image after enhancement processing so as to perform bridge dynamic displacement identification. According to the method, the noise influence of uneven illumination and bridge vibration in the video image is overcome to a certain extent, and the recognition of the dynamic displacement of the bridge is facilitated.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of image enhancement, in particular to a bridge dynamic displacement high-precision identification method based on digital image processing. BACKGROUND

[0002] During long-term service, the bearing capacity and durability of the bridge structure are continuously reduced due to environmental erosion, cyclic load and other effects, thereby affecting the safety during the operation of the road bridge. In the bridge structure, there are many factors affecting the structural performance, and detailed and comprehensive safety detection of the bridge structure is the basis for corresponding maintenance, reinforcement and reconstruction of the structure. In bridge safety monitoring, the displacement change of the bridge is one of the most important indicators in bridge monitoring indicators, which can reflect the stiffness of the beam body in the vertical direction, so as to judge the stability and safety of the bridge structure.

[0003] In the bridge dynamic displacement monitoring, the image sequence of the bridge structure is obtained by the image acquisition device, and the displacement change of the structure is calculated by using the image processing algorithm. However, due to the change of natural light conditions, such as cloud shielding, and the vibration of the bridge caused by the driving of vehicles on the bridge, the recognition and matching of the key points of the bridge in the continuous frame images are prone to errors, so that the bridge dynamic displacement high-precision identification is not accurate. SUMMARY

[0004] In order to solve the above-mentioned technical problem of inaccurate bridge dynamic displacement high-precision identification, the purpose of the present application is to provide a bridge dynamic displacement high-precision identification method based on digital image processing, and the technical solution adopted is as follows: An embodiment of the present application provides a bridge dynamic displacement high-precision identification method based on digital image processing, which comprises the following steps: obtaining a video image sequence of a bridge to be identified; taking any frame video image in the video image sequence as a target video image, determining each fuzzy area of the target video image according to the LBP value and gradient amplitude of each pixel point in the target video image; the fuzzy area is an area affected by external environmental factors; determining the motion law index of each fuzzy area of the target video image in the adjacent frame and the motion trajectory stability index in the continuous frame; fusing the motion law index and the motion trajectory stability index to obtain the influence degree of each fuzzy area on the bridge displacement identification precision; using the influence degree to perform enhancement processing on the fuzzy area in the target video image, to obtain an enhanced target video image; and then obtaining each frame of the enhanced video image to identify the bridge dynamic displacement.

[0005] Further, the method further comprises: dividing the target video image into a plurality of local regions; determining each local region corresponding to each bridge structure according to the LBP value of each pixel point in each local region; determining each target point in each local region and a matching value of each target point; the matching value is the minimum value of all LBP difference values corresponding to two local regions of the same bridge structure, and the target point is the two pixel points corresponding to the minimum value; determining the blur degree of each local region according to each target point and the matching value of each target point in each local region; and determining the average gradient amplitude value of each local region according to the gradient amplitude value of each pixel point in each local region; determining an interference factor of each local region for bridge displacement recognition according to the blur degree and the average gradient amplitude value of each local region; regarding the local region with the interference factor not less than a preset interference threshold as a blur region.

[0006] Further, the method further comprises: calculating a difference value between the LBP value of a target pixel point in a first local region and the LBP value of each pixel point in a second local region, denoted as an LBP difference value, and selecting the minimum LBP difference value; and regarding the average value of the minimum LBP difference values corresponding to all pixel points in the first local region as the texture difference degree between the first local region and the second local region; the first local region and the second local region are two different local regions in the target video image, and the target pixel point is any pixel point in the first local region; performing clustering processing on all local regions in the target video image to obtain a plurality of clustering clusters according to the texture difference degree between each two local regions; and regarding each local region in the same clustering cluster as each local region corresponding to the same bridge structure.

[0007] Further, the method further comprises: respectively for each local region, determining a feature point as a target point repeatedly appearing in a target point set corresponding to the local region, and counting the number of target points corresponding to each feature point, and the number of target points and the number of feature points in the local region; determine the blur degree of the local region according to the target point quantity corresponding to each feature point, the minimum matching value, and a first ratio of the feature point quantity and the target point quantity in the local region; The blur degree is negatively correlated with the target point quantity and the first ratio, and positively correlated with the minimum matching value.

[0008] Further, the determination of the interference factor of each local region on bridge displacement identification according to the blur degree and the average gradient amplitude of each local region comprises: For each local region, the texture difference degree between each two local regions corresponding to the bridge structure to which the local region belongs is obtained, and the minimum texture difference degree is selected; The blur degree, the average gradient amplitude, and the minimum texture difference degree of the local region are analyzed to determine the interference factor of the local region on bridge displacement identification; The interference factor is positively correlated with the blur degree, and negatively correlated with the average gradient amplitude and the minimum texture difference degree.

[0009] Further, the determination of the motion rule index of each blur region in the adjacent frame and the motion trajectory stability index of each blur region in the continuous frame of the target video image comprises: Each feature point in each blur region of the target video image is obtained, and then a matching point corresponding to each feature point in the next frame of the target video image is determined; The motion distance and the motion direction between each feature point and its matching point are determined to form a motion vector, and the motion rule index of each blur region in the adjacent frame is determined according to the motion vector and the position coordinates of each feature point in each blur region; Each feature point is matched and analyzed in each frame of the continuously distributed video image to obtain the number of continuous matching success frames corresponding to each feature point in each blur region; The motion trajectory stability index of each blur region in the continuous frame is determined according to the number of continuous matching success frames and the blur degree of each blur region in each frame of the continuously matching success frames.

[0010] Further, the determination of the motion rule index of each blur region in the adjacent frame according to the motion vector and the position coordinates of each feature point in each blur region comprises: For each blur region, the modulus difference of the motion vector, the included angle value of the motion vector, and the Euclidean distance between each two feature points are determined according to the motion vector and the position coordinates of each feature point in the blur region; Determine the clustering distance between each two feature points based on the module length difference, the angle value, and the Euclidean distance corresponding to each two feature points; cluster all feature points in the fuzzy area using the clustering distance to obtain clusters; According to the angle between the motion vectors of every two feature points in each cluster and the modulus of the motion vector of each feature point, the motion regularity index of the blurred area in adjacent frames is determined.

[0011] Furthermore, determining the motion regularity index of the blurred area in adjacent frames based on the angle value between the motion vectors of every two feature points in each cluster and the motion vector modulus of each feature point includes: Calculating the average value of all angle values ​​and the variance of the motion vector modulus in each cluster to obtain a motion consistency index for each cluster; taking the cluster whose motion consistency index is greater than a preset consistency threshold as a reference cluster; the motion consistency index is negatively correlated with both the average value and the variance; According to the number of clusters corresponding to the blurred area, the motion consistency index and the number of feature points of each reference cluster, the motion regularity index of the blurred area in adjacent frames is determined; the motion regularity index is negatively correlated with the number of clusters, and positively correlated with both the motion consistency index and the number of feature points.

[0012] Furthermore, determining the motion trajectory stability index of each blurred area in the continuous frames according to the number of consecutively matched successful frames and the blur degree of each blurred area in each consecutively matched successful video image frame includes: Obtaining the number of pixels in each blurred area in each frame of the video image that is successfully matched continuously, and using a second ratio of the number of pixels in the blurred area to the total number of pixels in the video image as a weight; Performing weighted summation processing on the blur degree according to the weights of each blur region in each frame of video image to obtain the influence degree of each frame of video image on the accuracy of bridge displacement recognition; For each feature point, a continuous matching index of each feature point is determined based on the degree of influence of each frame of video image on the accuracy of bridge displacement recognition and a third ratio of the number of consecutive matching successful frames to the total number of frames; the continuous matching index is negatively correlated with the degree of influence and positively correlated with the third ratio; The continuous matching indices of all feature points in the same blurred area in the target video image are comprehensively analyzed to obtain the motion trajectory stability index of each blurred area in consecutive frames.

[0013] Furthermore, the fusing of the motion law index and the motion trajectory stability index to obtain the degree of influence of each fuzzy area on the bridge displacement recognition accuracy includes: The product of the motion law index and the motion trajectory stability index of each fuzzy area is calculated respectively, and the product is normalized in a negative correlation manner to obtain the influence degree of the fuzzy area on the bridge displacement recognition accuracy.

[0014] The present application has the following advantages: The present application provides a bridge dynamic displacement high-precision recognition method based on digital image processing, which can overcome the influence of uneven illumination and bridge vibration noise in video images to some extent, enhance the accuracy of key point recognition and matching in the bridge dynamic displacement recognition process, and is beneficial to the recognition of bridge dynamic displacement and the improvement of the accuracy of bridge dynamic displacement recognition. Firstly, each fuzzy area is determined by combining the LBP value and gradient amplitude of each pixel point in the video image. The fuzzy area is an area affected by external environmental factors. By selecting the fuzzy area, the influence of uneven illumination on bridge dynamic displacement recognition can be analyzed, the data analysis amount can be reduced, and adaptive image enhancement processing of the fuzzy area can be performed to reduce the influence of uneven illumination in the video image. Secondly, the motion law index of each fuzzy area in the adjacent frame and the motion trajectory stability index in the continuous frame are determined, which indicates that the present application considers the influence of bridge vibration caused by vehicle driving in addition to the influence of illumination. Then, the motion law index and the motion trajectory stability index are used to quantify the influence degree of the fuzzy area on the bridge displacement recognition accuracy. Finally, the video image after enhancement processing is obtained by using the influence degree to enhance the corresponding fuzzy area in the video image. The accuracy of bridge dynamic displacement recognition realized by the adaptive enhanced video image is higher, which effectively guarantees the structural safety of the bridge. BRIEF DESCRIPTION OF DRAWINGS

[0015] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, and the advantages thereof, the drawings needed to be used in the embodiments or the prior art description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and for those skilled in the art, other drawings can be obtained without creative labor based on these drawings.

[0016] Figure 1 An embodiment of the present application provides a flowchart of a bridge dynamic displacement high-precision recognition method based on digital image processing. Figure 2 An embodiment of the present application provides a flowchart of a step of determining each fuzzy area of a target video image. Figure 3 An embodiment of the present application provides a flowchart of a step of determining the motion law index of each fuzzy area of a target video image in an adjacent frame and the motion trajectory stability index in a continuous frame. DETAILED DESCRIPTION

[0017] To further illustrate the technical means and effects employed by the present invention to achieve its intended objectives, the following, in conjunction with the accompanying drawings and preferred embodiments, describes in detail the specific implementations, structures, features, and effects of the technical solutions proposed by the present invention. In the following description, references to "one embodiment" or "another embodiment" do not necessarily refer to the same embodiment. Furthermore, specific features, structures, or characteristics of one or more embodiments may be combined in any suitable manner.

[0018] Unless defined otherwise, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs.

[0019] The application scenarios targeted by the present invention may be: During their service life, bridges are subjected to constant loads from vehicles, pedestrians, and other factors, while also adapting to changes in the natural environment. This can lead to fatigue damage and deformation of the bridge structure. To ensure the bridge's load-bearing capacity and safety, bridge displacement identification is necessary. However, during the dynamic displacement detection process, the video images captured by cameras may contain noise, resulting in unclear areas in the video image, making it difficult to identify the bridge's dynamic displacement.

[0020] In order to enhance the quality of video images, that is, to eliminate the influence of noise on video images, and to achieve high-precision recognition of bridge dynamic displacement, an embodiment of the present invention provides a high-precision recognition method of bridge dynamic displacement based on digital image processing, such as Figure 1 As shown, the following steps are included: S1, obtaining a video image sequence of a bridge to be identified.

[0021] In one embodiment, within a preset time period, a high-frame-rate, high-resolution industrial camera or drone or other equipment is used to continuously capture images of the bridge structure to be identified, and the captured video images are preprocessed to obtain a video image sequence.

[0022] Among them, the preset time period can be the current 7 days, and the video image acquisition frame rate can be more than 25 frames per second, which can be set by the implementer according to the specific actual situation; image preprocessing includes but is not limited to: image segmentation, image denoising and other processing methods.

[0023] In one embodiment, each frame of the video image in the video image sequence may be a grayscale image processed by a segmentation neural network, which can effectively remove the influence of background factors such as the sky area other than the bridge structure.

[0024] The segmentation neural network can be Mask R-CNN (Mask Regional Convolutional Neural Network), and a cross-entropy loss function is used.

[0025] Of course, the collected video images can also be subjected to noise removal, contrast enhancement, removal of illumination changes, etc., to make the video images clearer and provide accurate image data for subsequent analysis. Common preprocessing methods include Gaussian blur.

[0026] It should be noted that current noise suppression and deblurring techniques, such as Gaussian blur removal and image sharpening, are mainly aimed at static and relatively simple blurring. For complex blurring caused by external environmental factors (such as weather changes), traditional image processing methods currently cannot fully restore image details, and may even introduce new artifacts. Therefore, based on the noise characteristics caused by external environmental factors in the video image sequence, adaptive noise reduction and enhancement processing is performed on each frame of video image in the video image sequence to ensure the accuracy of the bridge dynamic displacement recognition realized based on the video image sequence.

[0027] At this point, the video image sequence of the bridge structure of the bridge to be recognized is obtained.

[0028] S2, any frame of video image in the video image sequence is taken as a target video image, and each blur region of the target video image is determined according to the LBP value and gradient amplitude of each pixel point in the target video image.

[0029] Here, the blur region is a region in the target video image affected by external environmental factors.

[0030] In the actual bridge dynamic monitoring process, the camera is often in an external environment, and the lighting conditions are affected by time, weather, season, and the environment around the bridge. For example, during the day and at night, on sunny and cloudy days, when the bridge is in direct sunlight or in the shadow area, the brightness of the bridge surface can vary greatly. Especially between consecutive frames, changes in lighting can cause the brightness or contrast of some regions to change significantly, resulting in complex noise in the video images captured by the camera, causing details in the image to be lost or blurred, and blur regions, i.e. unclear regions, to appear.

[0031] Each frame of video image in the video image sequence needs to be analyzed, but for ease of understanding the scheme, an example of image enhancement analysis is given, and any frame of video image in the video image sequence can be taken as a target video image.

[0032] As an exemplary embodiment, the determination of the blur regions of the target video image is as follows: Figure 2As shown, comprising: S21, dividing the target video image into a plurality of local regions.

[0033] In the video image of the bridge structure, some regions may become unclear due to noise, uneven illumination, etc. At this time, if the regions affected by noise and uneven illumination are mixed with regions with clear texture for analysis, it may cause errors in the clarity evaluation of the video image. Therefore, according to the structural characteristics of the bridge, the regions with similar texture in the video image are clustered, so as to avoid the interference of noise on the clear evaluation result and improve the accuracy of bridge displacement identification.

[0034] Among them, the structural characteristics of the bridge refer to the existence of a large number of regions with similar texture, for example, the bridge usually contains many repeated components such as beams, columns, trusses, etc. These components appear as similar texture in the image.

[0035] In one embodiment, the target video image is divided into a plurality of regions with similar texture, color or structural characteristics using a superpixel segmentation algorithm, denoted as a local region.

[0036] Among them, one local region corresponds to one component region. The implementation process of the superpixel segmentation algorithm is prior art, which will not be repeated here. Of course, the implementer can also use other image division methods such as region growing algorithm or gray level co-occurrence matrix, which is not specifically limited here.

[0037] S22, according to the LBP value of each pixel point in each local region, analyzing the texture difference degree between each two local regions, and determining each local region corresponding to each bridge structure.

[0038] Here, determining each local region corresponding to each bridge structure means determining each local region belonging to the same type of bridge structure, which is beneficial to the analysis of the interference situation of each local region of the same type of bridge structure for bridge displacement identification.

[0039] In one embodiment, the LBP value of each pixel point in each local region is obtained using the local binary pattern (Local Binary Pattern, LBP).

[0040] Among them, the local binary pattern generates a binary pattern through the neighborhood pixel values around the pixel, which is used to describe the texture features. Since the LBP algorithm focuses on the relative size relationship between the pixel point and its neighborhood pixels, rather than the absolute gray value. Therefore, even if the overall brightness changes, as long as the relative size relationship between adjacent pixels remains unchanged, the LBP value will not be affected, so the LBP value of the pixel point is used for data analysis here.

[0041] As an exemplary embodiment, each local region corresponding to each bridge structure is determined, including: In a first step, a difference value between the LBP value of the target pixel point in the first local region and the LBP value of each pixel point in the second local region is calculated, denoted as an LBP difference value, and the minimum LBP difference value is selected; the average value of the minimum LBP difference values corresponding to all pixel points in the first local region is taken as the texture difference degree between the first local region and the second local region.

[0042] Here, the texture difference degree refers to the texture dissimilarity of two local regions, and the first local region and the second local region are two different local regions in the target video image, and the target pixel point is any pixel point in the first local region.

[0043] In an embodiment, the first local region is denoted as the qth region, the second local region is denoted as the pth region, and the target pixel point is denoted as the kth pixel point, q, p and k are all positive integers; the LBP difference value between the LBP value of the kth pixel point in the qth region and the LBP value of all pixel points in the pth region is obtained, the minimum LBP difference value is taken as the texture difference factor of the kth pixel point in the qth region, and then the average value of the texture difference factors of all pixel points in the qth region is taken as the texture difference degree between the qth region and the pth region. Wherein, the LBP difference value is the absolute value of the difference between the LBP of two pixel points.

[0044] In a second step, all local regions in the target video image are clustered to obtain each cluster according to the texture difference degree between each two local regions; each local region in the same cluster is determined as each local region corresponding to the same bridge structure.

[0045] In an embodiment, the texture difference degree is taken as a clustering distance, and all local regions in the target video image are clustered by using the clustering distance between each two local regions, so that each cluster can be obtained, and each cluster represents the same bridge structure. Wherein, the implementation process of the clustering algorithm is a prior art, which will not be described here.

[0046] It is worth noting that, since the local regions in each cluster belong to the same type of component in physics, their texture features should have high consistency, which can effectively avoid the texture difference interference of different components and improve the accuracy of identifying the fuzzy region.

[0047] S23, determining each target point in each local region and its matching value.

[0048] Here, the matching value is the minimum value among all LBP difference values corresponding to two local regions of the same bridge structure, and the target points are the two pixel points corresponding to the minimum value. The greater the matching value, the more obvious the texture difference between the two target points.

[0049] In one embodiment, for the jth bridge structure, the minimum value among all LBP values of all pixel points in the qth region and all LBP difference values of all pixel points in the pth region is obtained; the two pixel points corresponding to the minimum value are recorded as a target point in the qth region and a target point in the pth region; and the minimum value is taken as the matching value of the target points in the qth region and the pth region. Wherein, j is a positive integer.

[0050] Referring to the determination method of the target point in the qth region and the target point in the pth region, and the matching value thereof, each target point in each local region and the matching value thereof can be obtained.

[0051] S24, according to each target point in each local region and the matching value thereof, determine the blur degree of each local region; according to the gradient amplitude of each pixel point in each local region, determine the mean value of the gradient amplitude of each local region.

[0052] Here, the blur degree refers to the unsharpness of the local region. The greater the blur degree, the greater the interference degree of the local region to the bridge displacement recognition.

[0053] As an exemplary embodiment, determining the blur degree of each local region comprises: First, for each local region, determine the target points in the target point set corresponding to the local region that appear repeatedly as feature points, count the number of target points corresponding to each feature point, and count the number of target points and feature points in the local region.

[0054] Here, the feature point refers to a target point in a local region and a target point in multiple local regions of the same bridge structure forming the minimum LBP difference value; the more feature points in the local region, the higher the clarity of the local region, and vice versa, the lower the blur degree.

[0055] Second, according to the number of target points corresponding to each feature point and the minimum matching value, and the first ratio of the number of feature points and the number of target points in the local region, determine the blur degree of the local region.

[0056] Here, the minimum matching value refers to the minimum value among all matching values ​​of the target points corresponding to the feature point. If no minimum value exists, an arbitrary matching value is used as the minimum matching value. The degree of blur is negatively correlated with the number of target points and the first ratio, and positively correlated with the minimum matching value. Negative correlation means that the larger the independent variable, the smaller the dependent variable, while positive correlation means that the larger the independent variable, the larger the dependent variable.

[0057] As an example, the calculation formula for the blur degree of the qth region can be: Where, Indicates the degree of blur in the qth region, exp represents an exponential function with a natural constant as the base, exp(-) is used to implement negative correlation processing, L represents the number of feature points in the qth region, and N represents the number of target points in the qth region. represents the first ratio of the number of feature points to the number of target points in the qth region, l represents the sequence number of the feature point, Indicates the number of target points corresponding to the lth feature point in the qth region, Indicates the minimum matching value corresponding to the lth feature point in the qth region.

[0058] In the calculation formula of the degree of blur, the ratio of the number of feature points to the number of target points is called the first ratio in order to distinguish it from other ratios in this embodiment, namely the second ratio and the third ratio; the first ratio The smaller it is, the fewer the number of feature points matched in the qth region, the less clear the qth region is, and the greater the degree of blur; the number of target points corresponding to the feature points The larger the value, the higher the clarity of the qth region, the smaller the blur, and the minimum matching value corresponding to the feature point. The smaller it is, the more closely the target point in the qth region matches the target points in other regions of the same bridge structure, and the higher the clarity of the qth region.

[0059] As an exemplary embodiment, determining the mean gradient amplitude of each local area includes: In the first step, edge detection is performed on each local area in the target video image to determine the gradient amplitude of each pixel in each local area.

[0060] In one embodiment, Canny edge detection is used to perform edge detection on the local area.

[0061] In the second step, based on the gradient amplitude of each pixel in each local area, the mean gradient amplitude of all pixels in the same local area is calculated.

[0062] The smaller the gradient amplitude mean value is, the more likely the corresponding local region is disturbed by noise and uneven illumination, lacks obvious edge information, and is more blurred.

[0063] S25, according to the blur degree and the gradient amplitude mean value of each local region, determine the interference factor of each local region to the bridge displacement identification.

[0064] Here, the interference factor refers to the influence of the image features of the local region on the bridge displacement identification process.

[0065] As an exemplary embodiment, determining the interference factor of each local region to the bridge displacement identification includes: First, for each local region, obtain the texture difference degree between each two local regions corresponding to the bridge structure to which the local region belongs, and select the minimum texture difference degree.

[0066] In this embodiment, the greater the minimum texture difference degree is, the smaller the structural texture similarity between all local regions corresponding to the bridge structure to which the local region belongs, further indicating that the clustering result in determining the bridge structure is less reliable, and the confidence of the blur degree and the gradient amplitude mean value of the local region is smaller, i.e. less reliable.

[0067] Second, fuse and analyze the blur degree, the gradient amplitude mean value and the minimum texture difference degree of the local region to determine the interference factor of the local region to the bridge displacement identification.

[0068] Here, the interference factor is positively correlated with the blur degree, and negatively correlated with the gradient amplitude mean value and the minimum texture difference degree.

[0069] As an example, the calculation formula of the interference factor of the qth region of the jth bridge structure to the bridge displacement identification can be: ; in the formula, indicates the interference factor of the qth region of the jth bridge structure of the zth video image to the bridge displacement identification, norm indicates a normalization function, indicates the blur degree of the qth region, indicates the minimum texture difference degree corresponding to the jth bridge structure of the zth video image, indicates the gradient amplitude mean value of the qth region of the jth bridge structure of the zth video image.

[0070] In the calculation formula of the interference factor, and Generally, there is no possibility of zero, but in order to avoid extreme cases, a non-zero constant is added to the denominator of the fraction, such as taking an empirical value of 0.001. Similarly, it also applies to fractions in other calculation formulas.

[0071] S26 , taking a local area where the interference factor is not less than a preset interference threshold as a fuzzy area.

[0072] Here, the larger the interference factor is, the more likely it is that the local area is a fuzzy area.

[0073] In one embodiment, the interference factor ranges from 0 to 1, and the preset interference threshold can be set to 0.7. Local areas with an interference factor of not less than 0.7, that is, local areas with an interference factor greater than or equal to 0.7, are used as fuzzy areas to obtain various fuzzy areas in the target video image. The preset interference threshold can be set by the implementer based on actual circumstances and is not specifically limited here.

[0074] So far, this embodiment has obtained various blurred areas in the target video image.

[0075] S3, determining the motion regularity index of each blurred area of ​​the target video image in adjacent frames and the motion trajectory stability index in continuous frames; fusing the motion regularity index and the motion trajectory stability index to obtain the degree of influence of each blurred area on the bridge displacement recognition accuracy.

[0076] During the video image acquisition process of the bridge structure, vehicles may be traveling on the bridge deck. When the vehicles are driving on the bridge, their own movement and the relative motion of the wheels and the bridge deck cause high-frequency vibrations in the local structure of the bridge (such as the beams and bridge deck where the vehicles pass). At this time, directly enhancing the acquired video images may amplify the noise in the blurred areas, interfering with the subsequent displacement calculation.

[0077] Noise typically manifests as random, discontinuous changes. In temporally continuous video images, noise typically exhibits a random variation pattern within each frame, making it impossible to predict with simple temporal consistency. Therefore, the more irregular or discontinuous the changes in the blurred area, the greater the impact of that local area on bridge displacement identification accuracy.

[0078] As an exemplary embodiment, the motion regularity index of each blurred area of ​​the target video image in adjacent frames and the motion trajectory stability index in continuous frames are determined, such as Figure 3 As shown, including: S31 , obtaining each feature point in each blurred area of ​​the target video image, and then determining a matching point corresponding to each feature point in the next frame of the target video image.

[0079] In one embodiment, a feature matching algorithm is used to determine a matching point corresponding to each feature point in the next frame of the target video image with a similar descriptor. The implementation process of the feature matching algorithm is known in the art, and will not be described here.

[0080] It is worth noting that the feature points without matching points do not participate in subsequent calculation and analysis.

[0081] S32, determine the motion distance and motion direction between each feature point and its matching point to form a motion vector; determine the motion regularity index of each blur region in the adjacent frame according to the motion vector and position coordinates of each feature point in each blur region.

[0082] In one embodiment, each feature point in the blur region in the zth frame of video image is mapped to the z+1th frame of video image, and the Euclidean distance and direction between each feature point and the matching matching point are determined as the motion distance and motion direction, respectively.

[0083] The motion vector of a feature point shows random or inconsistent changes in adjacent frames, which means that the feature point may be affected by noise, so the motion regularity index of the blur region in the adjacent frame is determined by analyzing the motion consistency of multiple feature points in the blur region.

[0084] As an exemplary embodiment, the motion regularity index of each blur region in the adjacent frame includes: First, for each blur region, the length difference of each motion vector, the angle value of each motion vector, and the Euclidean distance between each two feature points are determined according to the motion vector and position coordinates of each feature point in the blur region.

[0085] In one embodiment, for each two feature points in the blur region, the length of each motion vector is first determined, and then the length difference of each two motion vectors is determined; the angle value between each two motion vector directions is obtained; the Euclidean distance between each two feature points is calculated according to the position coordinates of each two feature points.

[0086] Second, the clustering distance of each two feature points is determined according to the length difference, angle value and Euclidean distance corresponding to each two feature points; all feature points in the blur region are clustered using the clustering distance to obtain each cluster.

[0087] In one embodiment, the length difference, angle value and Euclidean distance are first standardized to unify the dimension, and then the clustering distance of each two feature points is calculated based on the standardized length difference, angle value and Euclidean distance.

[0088] As an example, the calculation formula of the clustering distance of each two feature points can be: ; in the formula, D represents the clustering distance of the two feature points, represents the normalized value of the length difference corresponding to the two feature points, represents the normalized value of the angle value corresponding to the two feature points, represents the normalized value of the Euclidean distance corresponding to the two feature points.

[0089] In an embodiment, based on the clustering distance of each two feature points in the same fuzzy region, the K-medoids clustering algorithm is used to perform clustering analysis on all feature points in the fuzzy region, and a plurality of class clusters are obtained. The implementation process of the K-medoids clustering algorithm is a prior art, which will not be described here.

[0090] Thirdly, according to the angle value between the motion vectors of each two feature points in each class cluster and the motion vector length of each feature point, the motion regularity index of the fuzzy region in the adjacent frame is determined.

[0091] In a bridge structure, the real displacement (such as the up and down vibration of the main beam) usually has consistent motion characteristics, that is, the motion direction and amplitude of the feature points in the same class cluster have little difference. Based on this, if the feature points in a class cluster fluctuate greatly, it may be local noise or non-structural motion, and the motion regularity is poor.

[0092] As an example embodiment, determining the motion regularity index of the fuzzy region in the adjacent frame includes: Firstly, the average value of all angle values in each class cluster and the variance of the motion vector length are calculated, and the motion consistency index of each class cluster is obtained; the class cluster with the motion consistency index greater than a preset consistency threshold is taken as a reference class cluster.

[0093] Here, the greater the average value of all angle values in the class cluster, the more chaotic the motion direction of the feature points in the class cluster, the lower the motion direction consistency, and the smaller the motion consistency index. The greater the variance of the motion vector length of all feature points in the class cluster, the greater the fluctuation of the motion distance of all feature points in the class cluster, and the smaller the motion consistency index. Therefore, the motion consistency index is negatively correlated with the average value and the variance.

[0094] As an example, the calculation formula of the motion consistency index of each class cluster can be: ; in the formula, Y represents the motion consistency index of the class cluster, exp represents the exponential function with the natural exponential e as the base, exp( ) is used to realize the normalization processing of the negative correlation of the data, represents the average value of all angle values in the class cluster, represents the variance of all motion vector lengths in the class cluster.

[0095] In one embodiment, in order to screen out the cluster with strong consistency of motion direction and motion amplitude in the blur area, so as to facilitate the subsequent calculation of the motion rule index, the cluster with the motion consistency index greater than the preset consistency threshold value is taken as the reference cluster, and the preset consistency threshold value can be 0.7.

[0096] The second sub-step is to determine the motion rule index of the blur area in the adjacent frame according to the number of clusters corresponding to the blur area, the motion consistency index of each reference cluster, and the number of feature points.

[0097] Here, the motion rule index is negatively correlated with the number of clusters, and positively correlated with the motion consistency index and the number of feature points.

[0098] As an example, the calculation formula of the motion rule index of the blur area in the adjacent frame can be: In the formula, indicates the motion rule index of the mth blur area in the zth video image in the adjacent frame, m indicates the serial number of the blur area, norm indicates the normalization function, indicates the number of clusters in the local area, C indicates the number of reference clusters, and c indicates the serial number of the cluster, indicates the motion consistency index of the cth reference cluster, indicates the number of feature points of the cth reference cluster, and n indicates the number of pixel points of the cth reference cluster.

[0099] In the calculation formula of the motion rule index, the greater the value is, the more the number of feature points with inconsistent motion in the blur area is, the worse the motion rule between the feature corner points is, and the greater the influence on the bridge displacement recognition accuracy is; the greater the value is, the greater the motion consistency index is, and the stronger the motion rule of the blur area in the adjacent frame is.

[0100] It is worth noting that for each blur area in the last video image in the video image, it can not participate in the subsequent dynamic displacement analysis process.

[0101] It should be noted that due to the similar descriptors or motion patterns of the blur area between two adjacent video images, the blur area can be misjudged as motion consistent. However, in the time-continuous matching process, the randomness of noise points will gradually be exposed, resulting in matching failure or trajectory interruption, so further tracking the motion trajectory of the blur area in the continuous frame, analyzing the influence of the blur area on the bridge displacement recognition accuracy, i.e. corresponding to the subsequent steps S33 to S34.

[0102] S33, each feature point in each frame of video image in continuous distribution is matched and analyzed, and the number of continuous matching success frames corresponding to each feature point in each fuzzy area is obtained.

[0103] Here, the greater the number of continuous matching success frames corresponding to the feature point, the higher the continuous matching rate of the feature point, and the smaller the influence on the bridge displacement recognition accuracy.

[0104] In one embodiment, for each feature point in each fuzzy area in the target video image, the continuous matching coordinates of each feature point are recorded in each frame of video image in continuous distribution, the trajectory sequence of each feature point is constructed, if the feature point fails to match in a frame of video image, i.e. the feature point does not exist in the video image Similar descriptor, interpolation or null processing is performed; the number of continuous matching success frames is counted from the trajectory sequence of each feature point, and the number of continuous matching success frames corresponding to each feature point in each fuzzy area is obtained.

[0105] It should be noted that the video images in time sequence analyzed in the embodiment refer to the target video image and each frame of video image located after the target video image acquisition time.

[0106] S34, according to the number of continuous matching success frames and the blur degree of each fuzzy area in each frame of video image with continuous matching success, the motion trajectory stability index of each fuzzy area in continuous frames is determined.

[0107] As an exemplary embodiment, according to the number of continuous matching success frames and the blur degree of each fuzzy area in each frame of video image with continuous matching success, the motion trajectory stability index of each fuzzy area in continuous frames is determined, including: First, the number of pixel points of each fuzzy area in each frame of video image with continuous matching success is obtained, and the second ratio of the number of pixel points of the fuzzy area to the total number of pixel points of the video image is taken as the weight.

[0108] The greater the weight, the greater the proportion of the fuzzy area in the video image it is located in, and the higher the credibility of the blur degree of the fuzzy area, which is convenient for subsequent weighted sum analysis of the blur degree of the fuzzy area based on the weight.

[0109] Second, the blur degree of each fuzzy area in each frame of video image is weighted and summed according to the weight, and the influence degree of each frame of video image on the bridge displacement recognition accuracy is obtained.

[0110] As an example, the calculation formula of the influence degree of the z-th frame of video image on the bridge displacement recognition accuracy can be: ; In the formula, , wherein z represents the z-th frame of video images, norm represents a linear normalization function, M represents the number of blur areas in the z-th frame of video images, m represents the serial number of the blur area, and represents the blur degree of the m-th blur area in the z-th frame of video images, represents the number of pixel points of the m-th blur area in the z-th frame of video images, represents the total number of pixel points of the z-th frame of video images, represents a second ratio, i.e., the weight.

[0111] Referring to the influence degree of the z-th frame of video images on the bridge displacement recognition accuracy, the influence degree of each frame of video images on the bridge displacement recognition accuracy can be obtained.

[0112] In the third step, the continuous matching index of each feature point is determined according to the influence degree of each frame of video images on the bridge displacement recognition accuracy, the third ratio of the number of continuous matching successful frames to the total number of frames, respectively for each feature point.

[0113] Here, the continuous matching index is negatively correlated with the influence degree, and positively correlated with the third ratio.

[0114] As an example, the calculation formula of the continuous matching index of each feature point can be: , wherein R represents the continuous matching index of the feature point, norm represents a normalization function, represents the number of continuous matching successful frames, Z represents the total number of frames of all video images in the video image sequence, represents the third ratio of the number of continuous matching successful frames to the total number of frames, represents the cumulative value of the influence degree of each frame of video images on the bridge displacement recognition accuracy which is successfully matched continuously.

[0115] In the calculation formula of the continuous matching index, the number of continuous matching successful frames corresponding to different feature points can be different, the video images which are successfully matched continuously can be different, and therefore the cumulative value of the influence degree may be different; the greater the cumulative value of the influence degree , the more serious the blur degree of the video images which are successfully matched, the worse the image quality, and the more likely to reduce the matching stability and reliability of the feature point, so that even if the feature point is successfully matched, there can be high error or interference, and the credibility of the third ratio is low.

[0116] In the fourth step, the continuous matching indexes of all feature points in the same blur area in the target video image are comprehensively analyzed to obtain the motion trajectory stability index of each blur area in the continuous frames.

[0117] In one embodiment, the continuous matching indices of all feature points in the same blurred area are averaged, and the average value of the continuous matching indices of the blurred area is used as the motion trajectory stability index of the blurred area in consecutive frames.

[0118] As an exemplary embodiment, the influence of each fuzzy area on the bridge displacement recognition accuracy is obtained by integrating the motion law index and the motion trajectory stability index, including: Here, the motion law index and motion trajectory stability index are both dimensionless indicators. The larger the motion law index and motion trajectory stability index, the more stable the feature points in the blurred area are in continuous frames and can be continuously tracked. The more stable the motion trajectory of the feature points, the more likely it is the true displacement of the bridge. The smaller the impact of the blurred area on the accuracy of bridge displacement recognition, the smaller the degree of denoising enhancement in the blurred area.

[0119] In one embodiment, for each fuzzy area, the product of the motion law index of the fuzzy area and the motion trajectory stability index is calculated; the product can be negatively correlated and normalized using the exp(-) function to obtain the degree of influence of the fuzzy area on the bridge displacement identification accuracy.

[0120] Thus, this embodiment has obtained the degree of influence of each fuzzy area in the target video image on the bridge displacement recognition accuracy.

[0121] S4, using the influence degree to enhance the corresponding blurred area in the target video image to obtain the enhanced target video image; and then obtaining each frame of the enhanced video image to identify the dynamic displacement of the bridge.

[0122] As an exemplary embodiment, enhancing the corresponding blurred area in the target video image using the influence degree to obtain the enhanced target video image includes: In the first step, the sharpening intensity of each pixel in the target video image is determined according to the influence of each blurred area in the target video image on the accuracy of bridge displacement recognition.

[0123] In one embodiment, the sharpening intensity of each pixel in each blurred area of ​​the target video image is set to the influence degree of the corresponding blurred area, and the sharpening intensity of each pixel in the target video image except the blurred area is set to zero, thereby obtaining the sharpening intensity of each pixel in the target video image. The sharpening intensity of each pixel in the same blurred area is the same.

[0124] In the second step, the target video image is sharpened according to the sharpening intensity of each pixel to obtain the enhanced target video image.

[0125] The process of sharpening a video image based on the sharpening intensity is a prior art and will not be described in detail here.

[0126] As an exemplary embodiment, obtaining each frame of the enhanced video image to identify the dynamic displacement of the bridge includes: In the first step, each frame of the enhanced video image is obtained by referring to the acquisition process of the enhanced target video image.

[0127] The second step is to identify the dynamic displacement of the bridge based on each frame of the enhanced video image.

[0128] As an exemplary embodiment, the identification of the dynamic displacement of the bridge is performed based on each frame of the enhanced video image, including: In the first sub-step, feature point detection algorithms such as AKAZE, SIFT, and BRIEF are used to extract key points from each frame of the enhanced video image, and algorithms such as FLANN and PROSAC are used to match key points to determine the position changes of key points in the video image that are continuously distributed in time.

[0129] Get the coordinates of the key points in the z-th frame video image after enhancement processing, recorded as After matching by FLANN and PROSAC, the corresponding points are obtained in the enhanced z+1 frame video image, which is recorded as , then the expression of key point displacement can be: ; Where, represents the lateral displacement of the key point, Indicates the longitudinal displacement of the key point.

[0130] In the second sub-step, the displacement between key points is calculated by the sub-pixel matching algorithm, and the pixel displacement is converted into physical displacement by combining the scaling factor. The expression can be: ; Where, Indicates the lateral physical displacement of the key point, represents the scale factor, Indicates the longitudinal physical displacement of the key point.

[0131] In the third sub-step, the dynamically changing physical displacement is reconstructed by fusing accelerometer data or using a multi-scale filtering method to determine the dynamic displacement results of the bridge.

[0132] It should be noted that reconstructing the dynamic physical displacement can smooth and optimize the displacement data, eliminate the instability caused by local errors in image processing or feature matching, effectively extract the real dynamic change trend of the object instead of local short-time errors, and thus improve the accuracy of bridge unique recognition.

[0133] So far, the embodiment completes accurate identification of the bridge dynamic displacement.

[0134] The application divides a plurality of blur areas in a single frame video image, analyzes the influence degree of different blur areas on the bridge displacement identification accuracy, and is used for self-adaptive sharpening enhancement of the video image, so as to improve the bridge dynamic displacement identification accuracy and guarantee the structural safety of the bridge.

[0135] The above-described embodiments are only used to illustrate the technical solutions of the present application, rather than limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that the technical solutions recorded in the foregoing embodiments can be modified, or some technical features can be replaced by equivalents; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the scope of the technical solutions of the embodiments of the present application, and should be included in the protection scope of the present application.

Claims

1. A high-precision bridge dynamic displacement recognition method based on digital image processing, characterized in that, The method comprises the following steps: obtaining a video image sequence of a bridge to be identified; taking any frame video image in the video image sequence as a target video image, and determining each fuzzy region of the target video image according to the LBP value and gradient amplitude of each pixel point in the target video image; the fuzzy region is a region affected by external environmental factors; determining the motion law index of each fuzzy region in the adjacent frame and the motion trajectory stability index of each fuzzy region in the continuous frame of the target video image; and fusing the motion law index and the motion trajectory stability index to obtain the influence degree of each fuzzy region on the bridge displacement identification accuracy; performing enhancement processing on the fuzzy region in the target video image by using the influence degree to obtain an enhanced target video image; and then obtaining each frame video image after enhancement processing to identify the dynamic displacement of the bridge.

2. The high-precision bridge dynamic displacement recognition method based on digital image processing according to claim 1, characterized in that, The method of determining each fuzzy region of the target video image according to the LBP value and gradient amplitude of each pixel point in the target video image comprises the following steps: dividing the target video image into a plurality of local regions; analyzing the texture difference degree between each two local regions according to the LBP value of each pixel point in each local region, and determining each local region corresponding to each bridge structure; determining each target point and its matching value in each local region; the matching value is the minimum value of all LBP difference values corresponding to two local regions of the same bridge structure, and the target point is the two pixel points corresponding to the minimum value; determining the fuzzy degree of each local region according to each target point and its matching value of each local region, and determining the gradient amplitude mean value of each local region according to the gradient amplitude of each pixel point in each local region; determining the interference factor of each local region on bridge displacement identification according to the fuzzy degree and the gradient amplitude mean value of each local region; regarding the local region with an interference factor not less than a preset interference threshold as a fuzzy region.

3. The high-precision bridge dynamic displacement recognition method based on digital image processing according to claim 2, characterized in that, The method of analyzing the texture difference degree between each two local regions according to the LBP value of each pixel point in each local region, and determining each local region corresponding to each bridge structure comprises the following steps: calculating the difference value between the LBP value of a target pixel point in a first local region and the LBP value of each pixel point in a second local region, denoted as LBP difference value, and selecting the minimum LBP difference value; and taking the average value of the minimum LBP difference value corresponding to all pixel points in the first local region as the texture difference degree between the first local region and the second local region; the first local region and the second local region are two different local regions in the target video image, and the target pixel point is any pixel point in the first local region; performing clustering processing on all local regions in the target video image according to the texture difference degree between each two local regions to obtain each cluster; and determining each local region in the same cluster as each local region corresponding to the same bridge structure.

4. The high-precision bridge dynamic displacement recognition method based on digital image processing according to claim 3, characterized in that, The method of determining the fuzzy degree of each local region according to each target point and its matching value of each local region comprises the following steps: For each local region, determine the target points repeatedly appearing in the target point set corresponding to the local region as feature points, and count the number of target points corresponding to each feature point, as well as the number of target points and the number of feature points in the local region; Determine the blur degree of the local region according to the number of target points corresponding to each feature point and the minimum matching value, and the first ratio of the number of feature points and the number of target points in the local region. The blur degree is negatively correlated with the target point number and the first ratio, and positively correlated with the minimum matching value.

5. The high-precision bridge dynamic displacement recognition method based on digital image processing according to claim 4, characterized in that, The determination of the interference factor of each local region to the bridge displacement recognition according to the blur degree and the average gradient amplitude of each local region comprises: For each local region, obtain the texture difference degree between each two local regions corresponding to the bridge structure to which the local region belongs, and select the minimum texture difference degree; Fuse and analyze the blur degree, the average gradient amplitude and the minimum texture difference degree of the local region to determine the interference factor of the local region to the bridge displacement recognition; The interference factor is positively correlated with the blur degree, and negatively correlated with the average gradient amplitude and the minimum texture difference degree.

6. The high-precision bridge dynamic displacement recognition method based on digital image processing according to claim 4, characterized in that, The determination of the motion rule index of each blur region in the adjacent frame and the motion trajectory stability index of each blur region in the continuous frame of the target video image comprises: Obtain each feature point in each blur region of the target video image, and then determine the matching point corresponding to each feature point in the next frame of the target video image; Determine the motion distance and motion direction between each feature point and its matching point to form a motion vector, and determine the motion rule index of each blur region in the adjacent frame according to the motion vector and the position coordinates of each feature point in each blur region; Perform matching analysis on each feature point in each frame of the continuously distributed video image to obtain the number of continuous matching success frames corresponding to each feature point in each blur region; Determine the motion trajectory stability index of each blur region in the continuous frame according to the number of continuous matching success frames and the blur degree of each blur region in each frame of the continuously matching success video image.

7. The high-precision bridge dynamic displacement recognition method based on digital image processing according to claim 6, characterized in that, The determination of the motion rule index of each blur region in the adjacent frame according to the motion vector and the position coordinates of each feature point in each blur region comprises: For each blur region, determine the length difference of the motion vector, the angle value of the motion vector and the Euclidean distance between each two feature points according to the motion vector and the position coordinates of each feature point in the blur region; Determine the clustering distance of each two feature points according to the length difference, the angle value and the Euclidean distance corresponding to each two feature points, and cluster all feature points in the blur region by using the clustering distance to obtain each cluster; Determine the motion rule index of the blur region in the adjacent frame according to the angle value between the motion vectors of each two feature points in each cluster and the length of the motion vector of each feature point.

8. The high-precision bridge dynamic displacement recognition method based on digital image processing according to claim 7, characterized in that, The motion rule index of the blur area in the adjacent frame is determined according to an included angle value between motion vectors of each two feature points in each cluster and a motion vector module length of each feature point, and the method comprises the following steps: An average value of all included angle values and a variance of motion vector module lengths in each cluster are calculated to obtain a motion consistency index of each cluster; a cluster with a motion consistency index greater than a preset consistency threshold is regarded as a reference cluster; the motion consistency index is negatively correlated with the average value and the variance; A motion rule index of the blur area in the adjacent frame is determined according to the number of clusters corresponding to the blur area, the motion consistency index of each reference cluster and the number of feature points; the motion rule index is negatively correlated with the number of clusters and positively correlated with the motion consistency index and the number of feature points.

9. The high-precision bridge dynamic displacement recognition method based on digital image processing according to claim 6, characterized in that, The motion trajectory stability index of each blur area in the continuous frames is determined according to the number of continuous matching successful frames and the blur degree of each blur area in each video image with continuous matching success, and the method comprises the following steps: The number of pixel points of each blur area in each video image with continuous matching success is obtained, and a second ratio of the number of pixel points of the blur area to the total number of pixel points of the video image is taken as a weight; The blur degree is weighted and summed according to the weight of each blur area in each video image to obtain the influence degree of each video image on the bridge displacement recognition accuracy; The continuous matching index of each feature point is determined according to the influence degree of each video image on the bridge displacement recognition accuracy, a third ratio of the number of continuous matching successful frames to the total number of frames, respectively for each feature point; the continuous matching index is negatively correlated with the influence degree and positively correlated with the third ratio; The continuous matching indexes of all feature points in the same blur area in the target video image are comprehensively analyzed to obtain the motion trajectory stability index of each blur area in the continuous frames.

10. The high-precision bridge dynamic displacement recognition method based on digital image processing according to claim 1, characterized in that, The influence degree of each blur area on the bridge displacement recognition accuracy is obtained by fusing the motion rule index and the motion trajectory stability index, and the method comprises the following steps: The product of the motion rule index and the motion trajectory stability index of each blur area is calculated, respectively for each blur area; the product is negatively correlated and normalized to obtain the influence degree of the blur area on the bridge displacement recognition accuracy.

Images