Real-time monitoring method, system and device for bridge vibration based on visual enhancement

Abstract

The present invention discloses a real-time monitoring method and system for bridge vibration based on visual enhancement, the method comprising: real-time acquisition of bridge structure vibration images; region selection of bridge structure vibration images within the acquisition period, and cropping of the selected regions to obtain cropped images; lossless data compression and lossy data compression of the cropped images respectively; extraction of vibration signals from lossless data compressed images by combining template matching and LK optical flow method; frequency domain analysis of vibration signals to obtain the fundamental frequency of the bridge structure; video synthesis of lossy data compressed images; and visualization of bridge structure vibration and micro-motion amplification using multi-layer filtering technology based on the fundamental frequency and video. The present invention can accurately extract the vibration signals of the bridge structure by combining template matching and LK sub-pixel optical flow, and can realize visualization and amplification of micro-vibrations of the bridge by multi-layer filtering technology.

Description

Technical Field

[0001] The present invention relates to the technical field of bridge health monitoring, and in particular to a method, system, device, terminal equipment and computer-readable storage medium for real-time monitoring of bridge vibration based on visual enhancement. Background Art

[0002] The various structures of bridges will vibrate under the influence of vehicle loads, environment and other factors. The vibration characteristics will reflect the health status of the bridge structure. When the vibration of each part is abnormal, it often indicates that the bridge components are damaged and need to be dealt with in time. Therefore, it is necessary to monitor the vibration of the bridge in real time. Real-time acquisition of the bridge status is the basis for ensuring the safety and stability of the bridge.

[0003] At present, there are two main types of bridge vibration monitoring: contact and non-contact. Contact vibration monitoring mainly obtains acceleration and displacement data by burying sensors. The data obtained by this method is one-dimensional data. Although the accuracy is high and the data is relatively reliable, it needs to be pre-buried or later laid, which greatly increases the workload and difficulty of work. The laying process will cause damage to the bridge structure. Non-contact monitoring methods use GPS, laser, and radar technology to obtain vibration data. GPS has low accuracy and limited sampling frequency, which has limitations for data acquisition of small bridges or small vibrations. Laser technology equipment is relatively precise. Although the data acquisition accuracy and sampling frequency are high, it is easily affected by the outdoor environment and has high requirements for the power of the laser beam. Radar technology has a strong adaptability to outdoor environments, but the equipment cost is high and it is not easy to promote in engineering.

[0004] Compared with the above methods, the vision-based non-contact vibration monitoring method has the advantage of multiple measurement points. By using each pixel in the image as a sensor to obtain vibration data, the vibration monitoring efficiency is greatly improved. However, the existing vision-based non-contact structural vibration measurement is the extraction of one-dimensional signals. Since digital images need to be obtained, the storage space required is relatively large, and short-term temporary detection is the main method. There is a lack of long-term, real-time detection methods. At present, the vision-based vibration measurement method lacks accuracy and stability in the measurement of small displacements, lacks engineering application conditions and structural small vibration visualization methods, and all current vibration monitoring methods do not have real-time processing capabilities on mobile terminals. Summary of the invention

[0005] In order to address the deficiencies of the above-mentioned prior art, the present invention provides a real-time monitoring method, system, device, terminal device and computer-readable storage medium for bridge vibration based on visual enhancement, which performs large-scale rapid area matching in each frame of image through template matching, and extracts vibration displacement signals with high precision in combination with LK sub-pixel optical flow; realizes rapid and accurate signal extraction of structural vibration displacement through coarse alignment and fine extraction; removes noise caused by jitter of view acquisition equipment and amplification algorithm through multi-layer filtering technology, and proposes vibration visualization technology suitable for all motion forms (large motion, small motion) through image phase information, so as to realize visual amplification and presentation of tiny vibrations.

[0006] The first object of the present invention is to provide a real-time monitoring method for bridge vibration based on visual enhancement.

[0007] The second object of the present invention is to provide a real-time monitoring system for bridge vibration based on visual enhancement.

[0008] The third object of the present invention is to provide a real-time monitoring device for bridge vibration based on visual enhancement.

[0009] A fourth object of the present invention is to provide a terminal device.

[0010] A fifth object of the present invention is to provide a computer-readable storage medium.

[0011] The first object of the present invention can be achieved by adopting the following technical solutions:

[0012] A real-time monitoring method for bridge vibration based on visual enhancement, the method comprising:

[0013] Real-time acquisition of bridge structure vibration images;

[0014] Performing region selection on the bridge structure vibration image within the acquisition period, and cropping the selected region to obtain a cropped image;

[0015] Perform lossless data compression and lossy data compression on the cropped image respectively;

[0016] The vibration signal is extracted from the lossless data compressed image by combining template matching and LK optical flow method.

[0017] Perform frequency domain analysis on the vibration signal to obtain the fundamental frequency of the bridge structure;

[0018] Perform video synthesis on lossy data compressed images;

[0019] Based on fundamental frequency and video, multi-layer filtering technology is used to realize the visualization of bridge structure vibration and amplification of tiny motion; wherein, the multi-layer filtering technology includes equipment natural frequency filtering, second-order acceleration filtering and motion filtering. The equipment natural frequency filtering is used to remove the influence of image acquisition equipment on video, the second-order acceleration filtering is used to realize nonlinear tiny motion visualization, and the motion filtering is used to select the stationary area in the video, and filter the video based on the vibration signal extracted from the stationary area to filter out the noise amplified by the phase motion of the frame image.

[0020] Furthermore, the method of extracting vibration signals from lossless data compressed images by combining template matching and LK sub-pixel optical flow includes:

[0021] Using the first image as the reference image, manually select the target area from it and use it as a template to calculate the similarity coefficient of the same-sized area in other cropped images;

[0022] The area corresponding to the maximum similarity coefficient is taken as the matching area, that is, a rough matching is achieved;

[0023] Vibration signal extraction is performed based on LK sub-pixel optical flow in the target area and all matching areas, thus achieving fine matching.

[0024] Furthermore, a diamond search strategy is used when calculating the similarity coefficient to reduce the number of calculations and improve the matching speed.

[0025] Furthermore, the frequency domain analysis of the vibration signal to obtain the fundamental frequency of the bridge structure includes:

[0026] The power spectrum density obtained according to the vibration signal is:

[0027]

[0028] Where PSD is the power spectral density, X(f) is the Fourier transform of the vibration signal, and T is the duration of the vibration signal;

[0029] Find the peak value of the power spectrum density. The minimum frequency corresponding to the peak value is the fundamental frequency of the bridge structure.

[0030] Furthermore, the bridge structure vibration visualization and micro-motion amplification are realized by using multi-layer filtering technology based on fundamental frequency and video, including:

[0031] The frame image signal in the video is decomposed into multiple frequency bands through a complex-valued controllable pyramid, and the time domain is pass-band filtered, and the filtering range is determined according to the base frequency. At the same time, a Laplace filter is used in the time domain to perform second-order acceleration filtering to filter the frequency of directional movement deviation. The image phase difference after pass-band filtering and the directional movement deviation after second-order acceleration filtering are amplified by combining the amplification factor, and a new vibration amplification visualization video is synthesized.

[0032] The vibration signal of the video after the stable area in the video frame is magnified is extracted, and the noise amplified by the image phase motion is filtered out through motion filtering to improve the visualization and magnification effect of the tiny motion of the bridge structure.

[0033] Furthermore, a circular storage method is used when storing lossless data compressed images: an image with the longest storage time is deleted and then a newly acquired image is stored to ensure that the most images are stored.

[0034] Furthermore, the image acquisition device includes a zoom lens, an industrial camera and a single-chip microcomputer, and the single-chip microcomputer is used to execute external trigger acquisition instructions of the industrial camera.

[0035] Furthermore, ambient light sensors and adjustable fill lights are arranged on the left and right sides of the industrial camera. The ambient light sensors sense the light intensity of the target area. When the brightness is lower than a set threshold, the adjustable fill light is turned on, and the fill light intensity is dynamically adjusted according to the ambient brightness, so that the brightness of the monitored area remains constant.

[0036] The second object of the present invention can be achieved by adopting the following technical solutions:

[0037] A real-time monitoring system for bridge vibration based on visual enhancement, the system comprising:

[0038] An image acquisition module is used to acquire vibration images of bridge structures in real time;

[0039] An image cropping module is used to select a region of the bridge structure vibration image within a collection period and crop the selected region to obtain a cropped image;

[0040] A data compression module, used for performing lossless data compression and lossy data compression on the cropped image;

[0041] A signal extraction module is used to extract vibration signals from lossless data compressed images by combining template matching and LK sub-pixel optical flow;

[0042] The frequency domain analysis module is used to perform frequency domain analysis on the vibration signal to obtain the fundamental frequency of the bridge structure;

[0043] The visualization and motion amplification module is used to synthesize videos from images compressed with lossy data; based on the baseband and video, multi-layer filtering technology is used to realize the visualization of bridge structure vibration and small motion amplification; wherein, the multi-layer filtering technology includes equipment natural frequency filtering, second-order acceleration filtering and motion filtering, the equipment natural frequency filtering is used to remove the influence of the image acquisition device on the video, the second-order acceleration filtering is used to realize nonlinear small motion visualization, and the motion filtering is used to select the stationary area in the video, and filter the video based on the vibration signal extracted from the stationary area to filter out the noise amplified by the phase motion of the frame image.

[0044] The third object of the present invention can be achieved by adopting the following technical solutions:

[0045] A real-time monitoring device for bridge vibration based on visual enhancement, the device comprising an image acquisition module, a mobile terminal and a cloud server, wherein:

[0046] The image acquisition module is used to acquire the vibration image of the bridge structure in real time;

[0047] The mobile terminal is used to implement the above-mentioned bridge vibration real-time monitoring method;

[0048] The cloud server is used to store lossless data compressed images.

[0049] The fourth object of the present invention can be achieved by adopting the following technical solutions:

[0050] A terminal device includes a processor and a memory for storing a program executable by the processor. When the processor executes the program stored in the memory, the above-mentioned real-time monitoring method of bridge vibration based on visual enhancement is implemented.

[0051] The fifth object of the present invention can be achieved by adopting the following technical solutions:

[0052] A computer-readable storage medium stores a program, and when the program is executed by a processor, the above-mentioned real-time monitoring method of bridge vibration based on visual enhancement is implemented.

[0053] The present invention has the following beneficial effects compared with the prior art:

[0054] 1. The present invention uses a combination of template matching and LK optical flow method, which can first quickly perform rough matching in each frame of the image, and then use LK optical flow method to extract high-precision signals in the matching area of ​​each frame, which not only improves the calculation efficiency but also ensures the accuracy of the extracted signal. At the same time, it makes up for the limitation of traditional template matching in extracting small vibration signals, and improves the problem of insufficient applicability of LK optical flow method in large vibration scenes;

[0055] 2. The present invention uses multi-layer filtering technology to amplify the phase information and acceleration information between images, thereby realizing the visualization of bridge structure vibration and the amplification of small movements: the device self-oscillation frequency filtering technology is used to remove the influence of the external environment on the monitoring image of the image acquisition device; a second-order acceleration filter is used to realize the visualization of nonlinear small movements under large movements, and solve the problem that phase entanglement occurs in motion amplification under large movements, thereby causing the target object to be easily seriously distorted; motion filtering is used to remove the amplified noise, thereby improving the vibration amplification and visualization effects;

[0056] 3. The present invention realizes real-time acquisition of bridge structure vibration images, greatly reduces the amount of calculation through image cropping and data compression, and adopts a parallel computing strategy to realize vibration signal extraction and visualization amplification, thereby being capable of real-time processing on a mobile terminal. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the structures shown in these drawings without paying creative work.

[0058] Figure 1 A simplified flow chart of a method for real-time monitoring of bridge vibration based on visual enhancement according to Embodiment 1 of the present invention;

[0059] Figure 2 is a detailed flow chart of a real-time monitoring method for bridge vibration based on visual enhancement according to Embodiment 1 of the present invention;

[0060] Figure 3 is a functional relationship between the power spectral density (PSD) and the signal frequency in Embodiment 1 of the present invention;

[0061] Figure 4 It is a structural block diagram of a real-time monitoring system for bridge vibration based on visual enhancement according to Embodiment 2 of the present invention;

[0062] Figure 5 This is a structural block diagram of the terminal device of embodiment 3 of the present invention. DETAILED DESCRIPTION

[0063] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments in the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention. It should be understood that the specific embodiments described are only used to explain the present application and are not used to limit the present application.

[0064] Embodiment 1:

[0065] like Figure 1 , 2 As shown, this embodiment provides a real-time monitoring method for bridge vibration based on visual enhancement, which specifically includes the following steps:

[0066] S101. Collect vibration images of bridge structures in real time.

[0067] In this embodiment, the bridge structure vibration image is collected in real time by an image collection device.

[0068] Image acquisition devices include zoom lenses, industrial cameras and single-chip microcomputers.

[0069] The zoom lens is manual zoom with a focal length range of 12.5-100mm, which can meet the image data acquisition needs of close and long distances.

[0070] The acquisition mode of the industrial camera is grayscale mode, which reduces the storage space occupied by a single image. The acquisition frame rate of the industrial camera is above 100FPS to meet the monitoring needs of different vibration scenarios.

[0071] The single-chip microcomputer executes the external trigger acquisition command of the camera. Compared with the soft trigger acquisition, the hard trigger can ensure the stability of the time interval of continuous image acquisition, ensure the video effect, and avoid the inconsistency with the movement of the bridge target of concern.

[0072] The industrial camera is placed in the center of the front of the bridge to ensure the acquisition field of view. Use a zoom lens to monitor targets at different distances. Manually adjust the focal length according to the monitoring site environment and the monitoring target shape. After determining the installation position of the equipment, use the checkerboard calibration plate and the "Zhang calibration method" to calibrate the lens, obtain the camera's internal and external parameters, and determine the calibration coefficient. According to the Nyquist theorem, the sampling frequency must be greater than twice the maximum frequency of the target signal. Based on the experience of bridge vibration frequency detection, it is recommended that the sampling frequency should not be less than 30HZ, that is, the shooting frame rate of the industrial camera should not be less than 30FPS. At the same time, try to use low frame mode for acquisition while ensuring the frame rate that meets the signal acquisition requirements to reduce the amount of data required. Adjust the exposure time according to the monitoring environment. The higher the exposure time, the brighter the picture, and the exposure time is inversely proportional to the frame rate. The core computing module controls the microcontroller to trigger the industrial camera acquisition command for continuous data image acquisition.

[0073] This embodiment uses a single-chip microcomputer to perform external hard triggering operations on the industrial camera, and allocates an independent thread to trigger image acquisition, which can ensure that the acquisition is performed at a strict frame rate without interacting with other calculations, avoiding frame loss. Ensure that the video synthesized according to the acquisition frame rate is consistent with the actual object movement, avoiding acceleration or frame skipping. The use of a telephoto lens can meet the task requirements of different monitoring scenarios. Avoid the impact of the layout position and target size on the vibration monitoring of the bridge structure.

[0074] In this embodiment, while the industrial camera is used to collect vibration images of the bridge structure, the vibration data of the industrial camera itself is also collected.

[0075] The three-axis acceleration sensor is installed parallel to the imaging plane of the industrial camera to sense the vibration of the industrial camera itself and obtain vibration data for subsequent equipment self-frequency filtering processing. It also determines whether the position of the industrial camera needs to be adjusted by sensing its own posture.

[0076] In order to ensure high-quality image acquisition, this embodiment also provides a light sensing unit, which is mainly composed of an ambient light brightness sensor and an adjustable fill light, which are arranged on the left and right sides of the industrial camera. The ambient light brightness sensor senses the light intensity of the target area, turns on the fill light when the external environment is dim and the brightness is lower than the set threshold, and dynamically adjusts the fill light intensity according to the ambient brightness to keep the brightness of the monitored area constant.

[0077] S102 , preprocessing the bridge structure vibration image within the acquisition period to obtain a lossless data compressed image and a lossy data compressed image.

[0078] (1) Data preprocessing.

[0079] In order to achieve long-term vibration monitoring, a large amount of data needs to be stored in the limited local storage space. The original image is manually selected and cropped by the valuable area in the frame, and only the cropped continuous image is stored. The cropped image is compressed, and lossless data compression and lossy data compression are performed respectively.

[0080] This embodiment performs synchronous operations of cropping and data compression while collecting images, and sets the collection period to one minute. After completing one minute of image collection and preprocessing, video synthesis is performed on the lossy data compressed images; vibration signals are extracted from the lossless data compressed images, and frequency domain analysis is performed based on the one-minute vibration signal to obtain the frequency domain range that needs to be amplified and perform motion amplification operations.

[0081] The compression rate of lossy data compression is determined by the time required to process a one-minute video zoom operation, ensuring that the sum of the zoom time and the image preprocessing time is less than or equal to one-minute image acquisition time, thereby completing local real-time visualization processing.

[0082] (2) Data storage.

[0083] The lossless data compressed images are transmitted and backed up to the cloud server through wireless transmission means such as WIFI and Bluetooth. The lossless compressed images are stored locally in a circular manner: the image with the longest storage time is deleted and then a newly collected image is stored to ensure that the most lossless compressed images are stored locally.

[0084] Lossy compression reduces the resolution of the image through bilinear interpolation, and the compression ratio is determined by data preprocessing and sliced ​​video enlargement calculation time.

[0085] Cropping and lossless compression can minimize the computational cost while retaining effective information. For bridge structure vibration visualization and amplification, real-time processing and high-precision post-processing can be performed according to different monitoring requirements.

[0086] S103: Extracting the vibration signal of the bridge according to the lossless data compressed image.

[0087] The vibration signal of the bridge structure is extracted from the lossless data compressed image, and a combination of a coarse registration strategy of similarity matching and a fine extraction strategy of LK sub-pixel optical flow is adopted for fast real-time extraction.

[0088] (1) Use similarity matching to complete rough matching.

[0089] Manually select target area A mn, use this area as a template to calculate the similarity coefficient of the same-sized area in the image, use the diamond search strategy to quickly match the area in each frame of the image, and use the zero-mean normalized cross-correlation function to calculate the similarity coefficient r of two images of the same size:

[0090]

[0091] in, Target area A mn The average value of the image matrix, For image and A mn Image pixel matrix B of the same size mn The average value, m, n are the target area A mn The dimensions of the matrix.

[0092] The diamond search adopted in this embodiment can reduce the number of matching calculations and improve the matching speed.

[0093] The image with the largest similarity coefficient is the most similar area, which means that the rough matching is completed.

[0094] (2) Based on the coarse matching area, the LK optical flow method is used to extract the bridge vibration signal.

[0095] Sub-pixel vibration signal extraction is performed based on the LK optical flow method in the area where each frame image is roughly aligned with the target area.

[0096] The LK optical flow method regards the brightness value of a pixel in a video image frame at a certain time as a three-dimensional equation I(x, y, t), where x is the horizontal coordinate of the pixel, y is the vertical coordinate of the pixel, and t represents time. For high-speed video, the next frame after time point t can be described as I(x+δ x ,y+δ y ,t+δ t ), where δ x , δ y Respectively represent small horizontal and vertical displacements, δ t Represents the time increment. Under the basic assumption of constant brightness and small motion of the LK optical flow method, we can get:

[0097] I(x,y,t)=I(x+δ x ,y+δ y ,t+δ t )

[0098] Expanding the above formula based on Taylor yields:

[0099]

[0100] According to the above formula, we can get:

[0101]

[0102] In the formula, They represent the optical flow gradients of the image in the horizontal, vertical and time directions respectively, and can be calculated using the gradient operator Barron. δt is the time difference between the previous and next frames, and its value is 1; δx and δy are the displacements in the horizontal and vertical directions respectively.

[0103] By solving δx and δy based on the least squares method, the sub-pixel displacement of the target area in the horizontal and vertical directions can be obtained.

[0104] By combining the coarse registration strategy of similarity matching with the fine extraction strategy of LK sub-pixel optical flow, the signals of large motion and small vibration can be extracted without being restricted by the motion amplitude, which can ensure both the extraction speed and the calculation accuracy, thereby realizing the real-time extraction of vibration signal I(x, t).

[0105] The extracted vibration signal is calculated through the camera calibration coefficient to obtain the real vibration amplitude of the structure. The vibration amplitude is the size of the vibration signal, which reflects the amplitude of the structural vibration. Based on this vibration amplitude, it can be judged whether the vibration is too large.

[0106] The vibration signal amplitude value is in pixels, and the actual vibration amplitude is obtained by combining the camera calibration coefficient. The camera calibration coefficient refers to the actual physical size represented by each pixel in the image. There are two methods for calculation, one is marker conversion and the other is parameter calculation. Marker conversion is to shoot a marker of known physical size at a fixed position, count the number of pixels of the marker in the image, and thus calculate the actual physical size represented by each pixel. The parameter calculation method uses the camera parameters: pixel size, detection distance and lens focal length. The calculation formula is as follows:

[0107]

[0108] Where k is the camera calibration coefficient, α is the pixel size, H is the detection distance, and f is the lens focal length.

[0109] S104. Perform frequency domain analysis on the vibration signal to obtain the fundamental frequency of the bridge structure.

[0110] The extracted vibration signal is analyzed in the frequency domain. In view of the discreteness and randomness of the bridge vibration signal, the power spectral density (PSD) of the vibration signal is calculated to determine the fundamental frequency of the bridge structure:

[0111]

[0112] Where PSD is the power spectral density, X(f) is the Fourier transform of the vibration signal I(x, t), and T is the duration of the vibration signal.

[0113] Find the minimum frequency corresponding to the peak value of the signal power spectrum density, which is the fundamental frequency of the structure.

[0114] Specifically, the peak value of the power spectrum density is determined by smoothing and filtering the power spectrum density, finding local extreme points, and automatically determining the peak value by interpolation near the extreme points.

[0115] The change of the fundamental vibration frequency of the bridge structure is an important basis for judging whether the structure is damaged. Figure 3 is the function relationship between power spectral density (PSD) and signal frequency.

[0116] In this embodiment, the structural fundamental frequency is determined by automatically finding the peak of the power spectrum density, the frequency range of vibration visualization and amplification is determined according to the structural fundamental frequency, and support and basis for structural performance evaluation are provided through the change of the structural fundamental frequency.

[0117] S105. Perform video synthesis on the lossy data compressed image; and realize visual amplification of bridge structure vibration based on the baseband and the video.

[0118] The overall structure of the bridge is relatively macroscopic, and it is difficult to visually observe the vibration characteristics and vibration forms when vibration occurs. The amplitude of structural vibration is relatively small and it is difficult to observe directly with the naked eye. The vibration information of the bridge structure is visualized and amplified by visual enhancement, which can intuitively show the small movement forms and states of the bridge structure, assist in discovering and judging the problems of the structure, and timely warn and maintain.

[0119] This embodiment analyzes the phase information between images and adopts a multi-layer filtering technology to solve the problem that phase entanglement occurs in motion amplification under large movements, thereby causing the target object to be easily seriously distorted.

[0120] The multi-layer filtering includes equipment self-frequency filtering, second-order acceleration filtering and motion filtering. First, the equipment self-frequency filtering is used to obtain the self-frequency of the industrial camera through the three-axis acceleration sensor to filter out the influence of external factors on the camera's self-frequency. Then, the second-order acceleration filtering is used to visualize the nonlinear small movements under large movements through the second-order acceleration filter. Finally, the motion filtering is used to select a theoretically stable and immobile area in the video, and the entire video is filtered based on the vibration signal extracted from the area to remove the pseudo-motion caused by the amplification of noise by the image phase motion amplification.

[0121] For a one-dimensional vibration signal I(x, t), the motion signal after the signal is translated by a magnitude of δ(t) is expressed as I(x, t) = f(x + d(t)), and the signal after amplification a is expressed as The second-order signal of the second-order Taylor expansion of the signal about x is:

[0122]

[0123] The nonlinear part of the amplified signal, i.e. the acceleration, is:

[0124]

[0125] After the images are synthesized into a video according to the acquisition frame rate, the device vibration frequency collected by the three-axis acceleration sensor is filtered using the device's self-frequency to remove the impact of the industrial camera's own movement on the video results.

[0126] Specifically, a high-frequency three-axis acceleration sensor obtains the acceleration of the camera along the X and Y directions of the image, obtains the displacement of the camera in two directions between adjacent frames through quadratic integration, calculates the smooth trajectory of camera shake through motion smoothing, performs motion compensation on each frame of the image, and interpolates the image edges.

[0127] Image phase motion amplification uses a complex-valued controllable pyramid to perform multi-band decomposition of the image signal of each frame, and passband filtering is performed in the time domain. The filtering range is determined by the structural fundamental frequency detected according to the power spectral density. At the same time, a Laplace filter is used in the time domain to perform second-order acceleration filtering to screen the frequency of directional movement deviation. The phase difference after passband filtering and the directional movement deviation after second-order acceleration filtering are amplified by combining the amplification factor, and a new vibration amplification visualization video is synthesized. Optionally, the vibration signal of the amplified video of the stable moving area in the video frame, such as the wall, is extracted, and the noise of the image phase motion amplification is filtered out through precise motion filtering to improve the visualization amplification effect of the small movement of the bridge structure.

[0128] This embodiment also provides a real-time monitoring device for bridge vibration based on visual enhancement, including an image acquisition module, a mobile terminal and a cloud server, wherein the image acquisition module is used to acquire real-time vibration images of the bridge structure; the mobile terminal is used to implement the above-mentioned real-time monitoring method for bridge vibration based on visual enhancement; the cloud server is used to store images with lossless data compression to facilitate long-term monitoring of bridge vibration.

[0129] Those skilled in the art will appreciate that all or part of the steps in the method for implementing the above embodiments may be completed by instructing related hardware through a program, and the corresponding program may be stored in a computer-readable storage medium.

[0130] It should be noted that although the method operations of the above embodiments are described in a specific order in the accompanying drawings, this does not require or imply that these operations must be performed in this specific order, or that all the operations shown must be performed to achieve the desired results. On the contrary, the steps depicted can be performed in a different order. Additionally or alternatively, some steps can be omitted, multiple steps can be combined into one step, and / or one step can be decomposed into multiple steps.

[0131] Embodiment 2:

[0132] like Figure 4 As shown, this embodiment provides a real-time monitoring system for bridge vibration based on visual enhancement, which includes an image acquisition module 401, an image cropping module 402, a data compression module 403, a sub-signal extraction module 404, a frequency domain analysis module 405 and a visualization and motion amplification module 406, wherein:

[0133] An image acquisition module 401 is used to acquire a vibration image of the bridge structure in real time;

[0134] An image cropping module 402 is used to select a region of the bridge structure vibration image within a collection period, and crop the selected region to obtain a cropped image;

[0135] A data compression module 403, used for performing lossless data compression and lossy data compression on the cropped image;

[0136] A signal extraction module 404 is used to extract vibration signals from lossless data compressed images by combining template matching and LK sub-pixel optical flow;

[0137] The frequency domain analysis module 405 is used to perform frequency domain analysis on the vibration signal to obtain the fundamental frequency of the bridge structure;

[0138] The visualization and motion amplification module 406 is used to perform video synthesis on the image compressed by lossy data; based on the baseband and the video, a multi-layer filtering technology is used to realize the visualization of bridge structure vibration and the amplification of small motion; wherein, the multi-layer filtering technology includes equipment natural frequency filtering, second-order acceleration filtering and motion filtering, the equipment natural frequency filtering is used to remove the influence of the image acquisition device on the video, the second-order acceleration filtering is used to realize nonlinear small motion visualization, and the motion filtering is used to select the stationary area in the video, and filter the video based on the vibration signal extracted from the stationary area to filter out the noise of the frame image phase motion amplification.

[0139] The specific implementation of each module in this embodiment can refer to the above-mentioned embodiment 1, which will not be described one by one here; it should be noted that the system provided in this embodiment is only illustrated by the division of the above-mentioned functional modules. In actual applications, the above-mentioned functions can be assigned to different functional modules as needed, that is, the internal structure can be divided into different functional modules to complete all or part of the functions described above.

[0140] Embodiment 3:

[0141] This embodiment provides a terminal device, which can be a computer. Figure 5As shown, it comprises a processor 502, a memory, an input device 503, a display 504 and a network interface 505 connected via a system bus 501. The processor is used to provide computing and control capabilities. The memory comprises a non-volatile storage medium 506 and an internal memory 507. The non-volatile storage medium 506 stores an operating system, a computer program and a database. The internal memory 507 provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. When the processor 502 executes the computer program stored in the memory, the real-time monitoring method for bridge vibration based on visual enhancement of the above-mentioned embodiment 1 is implemented.

[0142] Embodiment 4:

[0143] This embodiment provides a computer-readable storage medium storing a computer program. When the computer program is executed by a processor, the real-time monitoring method for bridge vibration based on visual enhancement of the above-mentioned embodiment 1 is implemented.

[0144] It should be noted that the computer-readable storage medium of the present embodiment may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0145] The above is only a preferred embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can make equivalent replacements or changes according to the technical solution and inventive concept of the present invention within the scope disclosed by the present invention, which shall fall within the protection scope of the present invention.

Claims

1. A real-time monitoring method for bridge vibration based on visual enhancement, characterized in that: The method comprises: Real-time acquisition of bridge structure vibration images; Performing region selection on the bridge structure vibration image within the acquisition period, and cropping the selected region to obtain a cropped image; Perform lossless data compression and lossy data compression on the cropped image respectively; For the lossless data compressed image, the target area is selected from the first cropped image and used as a template. Template matching is performed on each of the other cropped images to obtain the matching area of ​​each of the other cropped images. The vibration signal is extracted from the target area and all matching areas based on the LK sub-pixel optical flow. Perform frequency domain analysis on the vibration signal to obtain the fundamental frequency of the bridge structure; Perform video synthesis on lossy data compressed images; Based on the fundamental frequency and video, a multi-layer filtering technology is used to realize the visualization of bridge structure vibration and micro-motion amplification, including: after filtering the video with the device self-oscillation frequency, the frame image signal in the video is decomposed into multiple bands through a complex-valued controllable pyramid, and the time domain is pass-band filtered, and the filtering range is determined according to the fundamental frequency; at the same time, a Laplace filter is used in the time domain to perform second-order acceleration filtering to screen the frequency of directional movement deviation; the image phase difference after pass-band filtering and the directional movement deviation after second-order acceleration filtering are amplified by combining the amplification factor, and a new vibration amplification visualization video is synthesized; the stationary area in the video is selected, and the video is filtered based on the vibration signal extracted from the stationary area to enhance the visualization and amplification effect of the micro-motion of the bridge structure; wherein, the device self-oscillation frequency filtering is used to remove the influence of the image acquisition device on the video, and the second-order acceleration filtering is used to realize nonlinear micro-motion visualization.

2. The bridge vibration real-time monitoring method according to claim 1 is characterized in that: The target area is selected from the first cropped image and used as a template, and template matching is performed on each of the other cropped images to obtain a matching area of ​​each of the other cropped images, including: Using the first image as the reference image, manually select the target area from it and use it as a template to calculate the similarity coefficient of the same-sized area in other cropped images; The area corresponding to the maximum similarity coefficient is used as the matching area of ​​the corresponding cropped image, that is, a rough match is achieved.

3. The bridge vibration real-time monitoring method according to claim 2 is characterized in that: A diamond search strategy is used when calculating similarity coefficients to reduce the number of calculations and improve matching speed.

4. The real-time monitoring method for bridge vibration according to any one of claims 1 to 3, characterized in that: The method of performing frequency domain analysis on the vibration signal to obtain the fundamental frequency of the bridge structure includes: The power spectrum density obtained according to the vibration signal is: In the formula, PSD is the power spectral density, X (f) is the Fourier transform of the vibration signal, T is the duration of the vibration signal; Find the peak value of the power spectrum density. The minimum frequency corresponding to the peak value is the fundamental frequency of the bridge structure.

5. The real-time monitoring method for bridge vibration according to any one of claims 1 to 3, characterized in that: When storing lossless data compressed images, a circular storage method is used: after deleting the image with the longest storage time, a newly acquired image is stored to ensure that the most images are stored.

6. The real-time monitoring method for bridge vibration according to any one of claims 1 to 3, characterized in that: The image acquisition device comprises a zoom lens, an industrial camera and a single-chip microcomputer, wherein the single-chip microcomputer is used for executing an external trigger acquisition instruction of the industrial camera.

7. The bridge vibration real-time monitoring method according to claim 6 is characterized in that: Ambient light sensors and adjustable fill lights are set on the left and right sides of the industrial camera. The ambient light sensors sense the light intensity of the target area. When the brightness is lower than the set threshold, the adjustable fill light is turned on and the fill light intensity is dynamically adjusted according to the ambient brightness to keep the brightness of the monitored area constant.

8. A real-time monitoring system for bridge vibration based on visual enhancement, characterized in that: The system comprises: An image acquisition module is used to acquire vibration images of bridge structures in real time; An image cropping module is used to select a region of the bridge structure vibration image within a collection period and crop the selected region to obtain a cropped image; A data compression module, used for performing lossless data compression and lossy data compression on the cropped image; The signal extraction module is used to select the target area from the first cropped image of the lossless data compressed image and use it as a template, perform template matching on each of the other cropped images, and obtain the matching area of ​​each of the other cropped images; extract the vibration signal based on the LK sub-pixel optical flow in the target area and all matching areas; The frequency domain analysis module is used to perform frequency domain analysis on the vibration signal to obtain the fundamental frequency of the bridge structure; The visualization and motion amplification module is used to synthesize lossy data compressed images into videos; based on baseband and video, multi-layer filtering technology is used to achieve bridge structure vibration visualization and micro-motion amplification; The method of realizing bridge structure vibration visualization and micro-motion amplification based on baseband and video by using multi-layer filtering technology includes: After the video is filtered by the device's natural frequency, the frame image signal in the video is decomposed into multiple bands through a complex-valued controllable pyramid, and the time domain is pass-band filtered, and the filtering range is determined according to the base frequency; at the same time, a Laplace filter is used in the time domain to perform second-order acceleration filtering to screen the directional movement deviation frequency; the image phase difference after pass-band filtering and the directional movement deviation after second-order acceleration filtering are amplified by combining the amplification factor, and a new vibration amplification visualization video is synthesized; the immobile area in the video is selected, and the video is filtered based on the vibration signal extracted from the immobile area to enhance the visualization and amplification effect of the small movement of the bridge structure; wherein the device natural frequency filtering is used to remove the influence of the image acquisition device on the video, and the second-order acceleration filtering is used to realize nonlinear small movement visualization.

9. A real-time monitoring device for bridge vibration based on visual enhancement, characterized in that: The device includes an image acquisition module, a mobile terminal and a cloud server, wherein: The image acquisition module is used to acquire the vibration image of the bridge structure in real time; The mobile terminal is used to implement the bridge vibration real-time monitoring method according to any one of claims 1 to 7; The cloud server is used to store lossless data compressed images.

Images