Structural vibration identification method and device based on computer vision

Abstract

The invention discloses a structural vibration identification method and device based on computer vision, and the method comprises the steps: spatial domain decomposition, resampling and reconstruction, reweighted residual sparseness processing, and temporal correlation enhancement, achieves the processing and identification from a video to vibration information, effectively resists the noise, reduces the phase step problem, and improves the recognition efficiency. And a high-precision structural vibration analysis result is provided.

Description

Technical Field

[0001] The present application relates to but is not limited to the field of video image processing technology, and in particular to a computer vision method and device for structural vibration recognition. Background Art

[0002] The non-contact motion recognition method based on optical measurement of digital cameras is a technology that uses digital cameras and image processing technology to detect and analyze the movement of objects. The optical measurement method based on cameras is non-contact and does not require the traditional sensor to be attached to the object being measured, so it does not introduce any additional mass and does not change the natural motion state of the object. When using a digital camera for measurement, the image or video of the entire scene is captured simultaneously by a single or multiple synchronized cameras. Since all motion data comes from the same video stream or a sequence of images taken at the same time, the image frames are naturally synchronized, avoiding the problem of time differences between multiple sensors. In other words, the non-contact motion recognition method based on optical measurement of digital cameras can effectively solve the problems of adding additional mass to traditional sensors and asynchronous wireless sensors.

[0003] However, although the non-contact motion recognition method based on digital camera optical measurement has significant advantages in many aspects, the traditional computer vision measurement method requires a large number of high-contrast marks or stripe patterns to be arranged on the surface of the structure to be measured, which is a relatively time-consuming process. The limitations of this method are obvious when the area to be measured is large or difficult to reach. In addition, for traditional optical measurement systems, using only a small amount of energy to stimulate the structure to be measured may result in the inability to capture subtle vibration responses.

[0004] Structural vibration identification methods based on phase motion amplification technology can address the above challenges. However, structural vibration identification methods based on phase motion amplification technology are based on a basic assumption that small movements can be converted into phase shifts of pixel colors. In fact, the spatial displacement encoded in the phase information is limited, which may lead to phase step phenomena. In addition, phase estimation is an ill-posed problem in which observation noise is amplified and propagated into the estimation of phase and amplitude, which may lead to image blur. Therefore, there is an urgent need for a structural vibration identification method that can effectively resist noise and solve the phase step problem. Summary of the invention

[0005] The present application provides a computer vision structural vibration recognition method and device, which can effectively resist noise and solve the phase step problem.

[0006] The embodiment of the present invention provides a computer vision structural vibration recognition method, comprising:

[0007] converting the original structural vibration video into a first image sequence;

[0008] Performing spatial domain decomposition on the first image sequence frame by frame to obtain a first spatial domain signal, and calculating initial phase difference information of the second image sequence after the spatial domain decomposition;

[0009] Resampling and reconstructing the first spatial domain signal to obtain a third image sequence of the second spatial domain signal;

[0010] Processing the third image sequence using the reweighted residual sparsity to enhance the temporal correlation of the third image sequence;

[0011] Based on the phase difference information in the third image sequence, structural vibration information is obtained using conversion calculation.

[0012] In an exemplary embodiment, converting the original structural vibration video into a first image sequence includes:

[0013] Converting the original structural vibration video into the first image sequence;

[0014] The first image sequence is converted from the RGB color space to the YIQ color space.

[0015] In an exemplary embodiment, the first image sequence is converted from the RGB color space to the YIQ color space according to the following formula:

[0016]

[0017] Among them, Y represents the brightness signal, I represents the color difference signal to which the human eye is sensitive, Q represents the color difference signal to which the human eye is not sensitive, R represents the red channel image signal, G represents the green channel image signal, and B represents the blue channel image signal.

[0018] In an exemplary embodiment, performing spatial domain decomposition on the first image sequence frame by frame to obtain a first spatial domain signal, and calculating initial phase difference information of the second image sequence after the spatial domain decomposition, includes:

[0019] Using a complex-valued steerable pyramid model to perform frame-by-frame spatial domain decomposition on the first image sequence, and calculating a second image sequence corresponding to each frequency, thereby obtaining the first spatial domain signal;

[0020] For each frequency feature of the second image sequence, the phase information of the current image is subtracted from the phase information of the first frame image, and the initial phase difference information Δψ of each frame image corresponding to the first frame image is calculated;

[0021] The initial phase difference information of the second image sequence is obtained according to the initial phase difference information of all frequency components.

[0022] In an exemplary embodiment, the complex-valued steerable pyramid model is shown as follows:

[0023]

[0024] Where A represents the amplitude, ψ represents the initial phase, and θ represents the frequency. represents the convolution operation, represents the real part of the convolution kernel with frequency θ, represents the imaginary part of the convolution kernel with frequency θ, and M is the image;

[0025] The initial phase difference information Δψ is calculated as: Δψ θ (t) = ψ θ (t)-ψ θ (1), where t represents time.

[0026] In an exemplary embodiment, the step of resampling and reconstructing the first spatial domain signal to obtain a third image sequence of the second spatial domain signal includes:

[0027] According to the initial phase difference information of the second image sequence, the first spatial domain information of the video image sequence is resampled and reconstructed using the compressed sensing sparsity theorem to obtain a third image sequence of the second spatial domain signal.

[0028] In an exemplary embodiment, the reconstruction is as follows:

[0029]

[0030] Where i represents the image block number; P represents the image block size; Φ P represents the transformation matrix from image to block; M represents a video frame, which is used to represent a frame of image in a video; M P (i,t) is the resampling mode of M(i,t); is a hypothesis of M(i,t); H(i,t) is a single hypothesis, is the weight vector to be solved, which is used to represent the linear combination of the hypothesis H(i,t); the constraint is

[0031] In an exemplary embodiment, it also includes:

[0032] The uncertainty of the third image sequence is estimated using sparse coding, and the noise-free block domain matrix M is calculated. s (t).

[0033] In an exemplary embodiment, the block domain uncertainty estimate is as follows:

[0034]

[0035] in, represents the optimized sparse coefficient matrix; C represents the coefficient matrix of sparse coding; represents the noisy image block matrix at time t; ‖·‖ F represents the Frobenius norm; ‖·‖1 represents the L1 norm; D represents the data dictionary, and γ represents the regularization parameter;

[0036] The noise-free block domain matrix M s (t) is:

[0037] In an exemplary embodiment, the third image sequence is processed by the following formula to enhance the time correlation of the third image sequence:

[0038]

[0039] stT=M(t),

[0040] Wherein, T represents the intermediate variable of frame M(t); M(t) represents the original vibration video frame at time t; represents the reconstructed video frame at time t; represents the noise-free block domain matrix of the key video frame; key represents the selected video key frame; Φ represents the image-to-block transformation matrix; β is the regularization parameter, nB represents the total number of blocks in the frame, and F is the weighted residual; the constraint condition T=M(t).

[0041] In an exemplary embodiment, the step of obtaining structural vibration information by using conversion calculation based on phase difference information in the third image sequence includes:

[0042] Acquiring a structural motion response in the video based on the phase difference information in the third image sequence;

[0043] According to the actual length of the structure, the obtained structural motion response in the video is converted into the structural vibration information.

[0044] In an exemplary embodiment, based on the phase difference information in the third image sequence, the structural motion response in the video is calculated by the following formula:

[0045]

[0046] Where u represents the structural motion response in the x direction, v represents the structural motion response in the y direction, ψ0 represents the phase of the structure in the x direction, and ψ π / 2 represents the phase of the structure in the y direction.

[0047] In an exemplary embodiment, converting the obtained structural motion response in the video into structural vibration information includes:

[0048] Calculate the ratio between the length of a known structure and the number of pixels the structure spans in the image;

[0049] The product of the structural motion response in the video and the obtained ratio is calculated to obtain the structural vibration information.

[0050] The embodiment of the present application also provides a computer-readable storage medium storing computer-executable instructions, wherein the computer-executable instructions are used to execute any of the computer vision structural vibration identification methods described above.

[0051] An embodiment of the present application further provides a computer device, including a memory and a processor, wherein the memory stores the following instructions executable by the processor: used to execute the steps of any of the above-mentioned computer vision structural vibration identification methods.

[0052] The present application embodiment further provides a computer vision structural vibration recognition device, comprising: a conversion module, a decomposition processing module, a reconstruction module, an enhancement processing module, and an acquisition module; wherein,

[0053] A conversion module, used for converting the original structural vibration video into a first image sequence;

[0054] A decomposition processing module, used for performing spatial domain decomposition on the first image sequence frame by frame to obtain a first spatial domain signal, and calculating initial phase difference information of the second image sequence after the spatial domain decomposition;

[0055] A reconstruction module, used for resampling and reconstructing the first spatial domain signal to obtain a third image sequence of the second spatial domain signal;

[0056] An enhancement processing module, used for processing the third image sequence of the reconstructed second spatial domain signal using the reweighted residual sparsity to enhance the time correlation of the third image sequence of the reconstructed second spatial domain signal;

[0057] The acquisition module is used to acquire structural vibration information by using conversion calculation based on the phase difference information in the processed third image sequence.

[0058] The computer vision structural vibration identification method provided by the embodiment of the present application, including spatial decomposition, resampling and reconstruction, reweighted residual sparsity processing, and time correlation enhancement, realizes the processing and identification of vibration information from video, effectively resists noise, reduces phase step problems, and provides high-precision structural vibration analysis results.

[0059] Other features and advantages of the present invention will be described in the following description, and partly become apparent from the description, or understood by practicing the present invention. The purpose and other advantages of the present invention can be realized and obtained by the structures particularly pointed out in the description, claims and drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] The accompanying drawings are used to provide further understanding of the technical solution of the present application and constitute a part of the specification. Together with the embodiments of the present application, they are used to explain the technical solution of the present application and do not constitute a limitation on the technical solution of the present application.

[0061] Figure 1 It is a flowchart of the structural vibration recognition method of computer vision in the embodiment of the present application;

[0062] Figure 2 It is a schematic diagram of the structure of the computer vision structural vibration recognition device in the embodiment of the present application;

[0063] FIG3(a) is an enlarged view of the motion of the three-layer frame selected in the embodiment of the present application and a schematic diagram of the motion shown in the slices thereof. Figure 1 ;

[0064] FIG3( b ) is an enlarged view of the motion of a three-layer frame selected in an embodiment of the present application and a schematic diagram of the motion shown in a slice thereof; and;

[0065] FIG3( c ) is a third schematic diagram of the movement of a three-layer frame selected in an embodiment of the present application and a slice thereof;

[0066] FIG3( d ) is an enlarged view of the motion of a three-layer frame selected in an embodiment of the present application and a fourth schematic diagram of the motion shown in a slice thereof;

[0067] FIG4( a ) is a quantitative index curve diagram of frame similarity with noise variance varying from 0.01 to 0.145 using peak signal-to-noise ratio in an embodiment of the present application;

[0068] FIG4( b ) is a quantitative index curve diagram of frame similarity with noise variance varying from 0.01 to 0.145 using structural similarity in an embodiment of the present application;

[0069] FIG5( a ) is a plan view of a stone curtain wall structure vibration table in an embodiment of the present application;

[0070] FIG5( b ) is a top view of a vibration table test device for a stone curtain wall structure in an embodiment of the present application;

[0071] FIG6( a ) is a schematic diagram of the vibration curve of the stone curtain wall structure at measuring point 1 in the embodiment of the present application. Figure 1 ;

[0072] FIG6( b ) is a schematic diagram of the vibration curve of the stone curtain wall structure at measuring point 1 in the embodiment of the present application. Figure 2 ;

[0073] FIG6( c ) is a third schematic diagram of the vibration curve of the stone curtain wall structure at measuring point 1 in an embodiment of the present application;

[0074] FIG6(d) is a schematic diagram of the vibration curve of the stone curtain wall structure at measuring point 2 in the embodiment of the present application. Figure 1 ;

[0075] FIG6(e) is a schematic diagram of the vibration curve of the stone curtain wall structure at measuring point 2 in the embodiment of the present application. Figure 2 ;

[0076] FIG6( f ) is a third schematic diagram of the vibration curve of the stone curtain wall structure at measuring point 2 in an embodiment of the present application;

[0077] FIG. 7( a ) is a diagram showing the vibration error of the stone curtain wall structure at measuring point 1 in an embodiment of the present application;

[0078] FIG. 7( b ) is a diagram showing the vibration error of the stone curtain wall structure at measuring point 2 in an embodiment of the present application. DETAILED DESCRIPTION

[0079] In order to make the purpose, technical solution and advantages of the present application more clear, the embodiments of the present application will be described in detail below with reference to the accompanying drawings. It should be noted that the embodiments and features in the embodiments of the present application can be combined with each other arbitrarily without conflict.

[0080] In order to facilitate understanding of the present application, the present application will be described more fully below with reference to the relevant drawings. Embodiments of the present application are provided in the drawings. However, the present application can be implemented in many different forms and is not limited to the embodiments described herein. On the contrary, the purpose of providing these embodiments is to make the disclosure of the present application more thorough and comprehensive.

[0081] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein in the specification of this application are only for the purpose of describing specific embodiments and are not intended to limit this application.

[0082] It is understood that the terms "first" and "second" used in this application are only used for descriptive purposes and cannot be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features. In the description of this application, the meaning of "plurality" is at least two, such as two, three, etc., unless otherwise clearly and specifically defined.

[0083] It can be understood that the “connection” in the following embodiments should be understood as “electrical connection”, “communication connection”, etc. if the connected circuits, modules, units, etc. have electrical signals or data transmission between each other.

[0084] When used herein, the singular forms "a", "an", and "said / the" may also include plural forms, unless the context clearly indicates otherwise. It should also be understood that the terms "include / comprise" or "have" and the like specify the presence of stated features, wholes, steps, operations, components, parts, or combinations thereof, but do not exclude the possibility of the presence or addition of one or more other features, wholes, steps, operations, components, parts, or combinations thereof. At the same time, the term "and / or" used in this specification includes any and all combinations of the relevant listed items.

[0085] Figure 1 Schematic diagram of the flow of the structural vibration recognition method of computer vision in the embodiment of the present application, such as Figure 1 As shown, it may include:

[0086] Step 100: Convert the original structural vibration video into a first image sequence.

[0087] In one exemplary embodiment, step 100 may include:

[0088] converting the original structural vibration video into a first image sequence;

[0089] The converted first image sequence is converted from the RGB color space to the YIQ color space.

[0090] In an exemplary embodiment, the continuous vibration video is decomposed into a first image sequence composed of frame-by-frame images so that each frame can be analyzed independently. In one embodiment, a video processing library (such as OpenCV) can be used to decompose the video frame by frame and save it as a separate image file.

[0091] In an exemplary embodiment, in the obtained first image sequence, each frame of the image is usually represented in the RGB color space. For further processing, especially when processing color information or doing certain specific analysis (such as phase information extraction), the image needs to be converted from the RGB color space to the YIQ or other color space. The YIQ color space helps to separate color information and brightness information. In one embodiment, the first image sequence can be converted from the RGB color space to the YIQ color space using formula (1):

[0092]

[0093] In formula (1), Y represents a brightness signal, I represents a color difference signal to which the human eye is sensitive, Q represents a color difference signal to which the human eye is not sensitive, R represents a red channel image signal, G represents a green channel image signal, and B represents a blue channel image signal.

[0094] Step 101: performing spatial domain decomposition on a first image sequence frame by frame to obtain a first spatial domain signal, and calculating initial phase difference information of a second image sequence after the spatial domain decomposition.

[0095] In an exemplary embodiment, a complex-valued steerable pyramid model, wavelet transform, Laplace pyramid combined with other phase extraction methods can be used to perform spatial domain decomposition on the image sequence frame by frame, and calculate and obtain initial phase difference information.

[0096] In an exemplary embodiment, when the complex-valued steerable pyramid model is used to perform spatial domain decomposition on the first image sequence frame by frame, and initial phase difference information of the second image sequence after the spatial domain decomposition is calculated, step 101 may include:

[0097] A complex-valued steerable pyramid model is used to perform frame-by-frame spatial domain decomposition on a first image sequence in a YIQ color space, and a second image sequence corresponding to each frequency is calculated to obtain a first spatial domain signal;

[0098] For each frequency feature of the second image sequence, the phase information of the current image is subtracted from the phase information of the first frame image, and the initial phase difference information Δψ of each frame image corresponding to the first frame image is calculated.

[0099] In step 101, first, each frame image in the YIQ color space is spatially decomposed, and the complex-valued steerable pyramid model is used to decompose the image into multiple frequency components (i.e., features of different scales and directions); then, for each frame image, the phase information is calculated on each frequency component, and a corresponding frequency image sequence is generated. These image sequences represent the features of each frame image at different scales and directions. Afterwards, for each frame image, the phase information is calculated on each frequency component, and a corresponding frequency image sequence is generated, which represents the features of each frame image at different scales and directions. Then, for each frequency feature, the phase information of the current frame is subtracted from the phase information of the first frame image, and the initial phase difference information Δψ of each frame image relative to the first frame image is calculated. Finally, the phase difference information of all frequency components is integrated to obtain the initial phase difference data of the complete second image sequence. The initial phase difference information represents the subtle changes of the image between different frames, which is the key data for identifying vibration.

[0100] In one embodiment, the complex-valued steerable pyramid model can be expressed as shown in formula (2):

[0101]

[0102] In formula (2), A represents the amplitude, ψ represents the initial phase, and θ represents the frequency. represents the convolution operation, represents the real part of the convolution kernel with frequency θ, represents the imaginary part of the convolution kernel with frequency θ, and M is the image.

[0103] In one embodiment, the phase difference Δψ θ (t) is calculated as shown in formula (3):

[0104] Δψ θ (t) = ψ θ (t)-ψ θ (1) (3)

[0105] In formula (3), t represents time.

[0106] Step 102: resample and reconstruct the first spatial domain signal to obtain a third image sequence of the second spatial domain signal.

[0107] In an exemplary instance, the first spatial domain signal may be resampled and a third image sequence of the second spatial domain signal may be reconstructed using methods such as multi-hypothesis prediction methods, interpolation techniques, Laplace reconstruction, wavelet reconstruction, deep learning methods, and frequency domain reconstruction.

[0108] In one embodiment, step 102 may include:

[0109] According to the calculated initial phase difference information of the second image sequence, the compressed sensing sparsity theorem is used to resample and reconstruct the first spatial domain information of the video image sequence to obtain a third image sequence of the second spatial domain signal.

[0110] In one embodiment, the spatial domain reconstruction is as shown in formula (4):

[0111]

[0112] In formula (4), i represents the image block number. The entire image can be divided into multiple small blocks, each of which has a unique number; P represents the image block size; Φ P represents the image-to-block conversion matrix, i.e., the matrix operation used to convert the entire image into individual image blocks; M represents a video frame, which is used to represent a frame of image in a video; M P (i, t) is the resampling mode of M(i, t), that is, the resampling result of a specific image block i extracted from the video frame M at time t; is a hypothesis of M(i,t). H(i,t) is a single hypothesis, is the weight vector that needs to be solved, used to represent the linear combination of the hypothesis H(i,t).

[0113] By using formula (4), we can achieve the minimization of M P (i,t) and Φ P The bi-norm error between H(i,t)ω. The bi-norm ‖·‖2 represents the Euclidean distance. The goal of the optimization problem is to find the optimal weight vector ω so that the signal Φ obtained by linear combination of the hypothesis H(i,t) P H(i,t)ω is closest to the resampled signal M P (i,t). The constraints are Means the reconstructed image block Must be equal to the optimal weight vector This process ensures that the final image patch is consistent with the linear combination of the hypothesized model.

[0114] In one embodiment, the structural changes in each frame of the image are estimated based on a multi-hypothesis prediction method, and the resampled signal is more accurate or has better spatiotemporal characteristics than the original signal, and the accuracy of the signal is enhanced by resampling. After resampling, spatial signal reconstruction is also performed so that the spatial features of the image sequence are reconstructed, ensuring that vibration information can be accurately represented at different scales.

[0115] In an exemplary embodiment, step 102 may further include:

[0116] The uncertainty region in the third image sequence is identified and processed by sparse coding based on the weight matrix. In one embodiment, it may include: using sparse coding to estimate the uncertainty of the third image sequence, thereby calculating the noise-free block domain matrix M s (t).

[0117] In one embodiment, the block domain uncertainty estimate is as shown in formula (5):

[0118]

[0119] In formula (5), represents the optimized sparse coefficient matrix, which is the sparse representation obtained by minimizing the objective function; C represents the coefficient matrix of sparse coding; represents the noisy image block matrix at time t; ‖·‖ F represents the Frobenius norm, which is usually used to measure the difference between matrices; ‖·‖1 represents the L1 norm, which is usually used to achieve the regularization representation of sparsity; D represents the data dictionary, and γ represents the regularization parameter. It should be noted that

[0120] Solution The process can be implemented by the alternating direction multiplier method (ADMM). The specific implementation is not used to limit the protection scope of the present application and will not be repeated here.

[0121] In one embodiment, the noise-free block-domain matrix M s (t) is the calculation result of the reconstruction after sparse coding as shown in formula (6):

[0122]

[0123] In the process of phase motion estimation, due to the existence of phase limitation, the phase information exceeding the limitation will be compressed, resulting in inaccurate phase estimation results. In addition, phase estimation will also be affected by ill-conditioned estimation, and its accuracy will also be greatly affected. Therefore, through step 102, the structural phase motion information is resampled and reconstructed to avoid phase step and ill-conditioned estimation problems.

[0124] Step 103: Process the reconstructed third image sequence of the second spatial domain signal using the reweighted residual sparsity to enhance the time correlation of the reconstructed third image sequence of the second spatial domain signal.

[0125] In one embodiment, the third image sequence of the reconstructed second spatial domain signal may be processed by formula (7) to enhance the temporal correlation of the third image sequence:

[0126]

[0127] stT=M(t)

[0128] In formula (7), T represents the intermediate variable of frame M(t); M(t) represents the original video frame at time t; Represents the reconstructed video frame at time t, which is the target to be solved in the optimization problem; represents the noise-free block domain matrix of the key video frame. The key frame is usually a frame with a higher sampling rate or importance, which is used to guide the reconstruction of other frames; key represents the selected video key frame, that is, it has a higher sampling rate; Φ represents the image-to-block conversion matrix; β is the regularization parameter, nB represents the total number of blocks in the frame, and F is the weighted residual. The constraint T = M(t) means that the intermediate variable T is actually a representation of the original frame M(t), ensuring that the final solved frame is a reconstructed version based on the original frame.

[0129] The objective function of formula (7) consists of two parts: Is a keyframe The error between the intermediate variable T and the key frame is to ensure that the intermediate variable T is as close as possible to the key frame after conversion, so as to maintain consistency between frames; β∑ 1≤i≤nB ‖F(M s (i, t))‖1 is to weight and sum the sparse residuals of each image block in the video frame, encouraging the sparsity of the residuals, which means that we hope that the reconstruction error of the image block is concentrated on a few blocks as much as possible to maintain the temporal correlation between frames. Formula (7) solves the problem by minimizing this objective function. and The reconstructed frame can maintain a high consistency with the key frame (temporal correlation is enhanced) and maintain sparsity at the block level within the frame, thereby reducing noise and uncertainty.

[0130] Since spatial and block domain resampling and reconstruction will lose video information in the time domain, the embodiment of the present application uses reweighted residual sparsity to enhance the temporal correlation of the image sequence, and step 103 realizes the linear combination of the reweighted sparsity of the residuals between blocks of the enhanced key video frame and the multi-hypothesis prediction of the non-key video frame. The temporal correlation of the third image sequence of the second spatial domain signal obtained after the enhanced reconstruction in step 103 improves the quality of the reconstructed video, making the transition between frames smoother, thereby reducing artifacts or inconsistencies introduced by the reconstruction.

[0131] Step 104: Based on the phase difference information in the processed third image sequence, use conversion calculation to obtain structural vibration information.

[0132] In an exemplary embodiment, the optimized phase difference information includes structural motion information in the video, and the relevant motion information needs to be converted into actual structural vibration information of the structure. Step 104 may include:

[0133] Obtaining a structural motion response in the video based on phase difference information in the processed third image sequence;

[0134] According to the actual length of the structure, the structural motion response in the obtained video is converted into structural vibration information.

[0135] In an exemplary embodiment, based on the phase difference information in the optimized third image sequence, the structural motion response in the video can be calculated by formula (8) and formula (9):

[0136]

[0137] In formula (8) and formula (9), u represents the structural motion response in the x direction, v represents the structural motion response in the y direction, ψ0 represents the phase of the structure in the x direction, and ψ π / 2 represents the phase of the structure in the y direction.

[0138] In an exemplary embodiment, without considering image deformation, converting the obtained structural motion response in the video into structural vibration information may include:

[0139] Calculate the ratio between the length of a known structure and the number of pixels the structure spans in the image;

[0140] The product of the structural motion response in the video and the obtained ratio is calculated to obtain the structural vibration information.

[0141] The computer vision structural vibration identification method provided in the embodiment of the present application includes spatial decomposition, resampling and reconstruction, reweighted residual sparsity processing, and time correlation enhancement, which realizes the processing and identification of vibration information from video, effectively resists noise, reduces phase step problems, and provides high-precision structural vibration analysis results.

[0142] The computer vision structural vibration recognition method provided in the embodiments of the present application can also be applied to computer vision micro-vibration recognition based on sparse enhanced compressed sensing.

[0143] The present application also provides a computer-readable storage medium storing computer-executable instructions, wherein the computer-executable instructions are used to execute any of the computer vision structural vibration identification methods described above.

[0144] The present application further provides a computer device, including a memory and a processor, wherein the memory stores the following instructions executable by the processor: used to execute the steps of any of the above-mentioned computer vision structural vibration identification methods.

[0145] The present application also provides a computer vision structural vibration recognition device, which may include: a conversion module, a decomposition processing module, a reconstruction module, an enhancement processing module, and an acquisition module; wherein:

[0146] A conversion module, used for converting the original structural vibration video into a first image sequence;

[0147] A decomposition processing module, used for performing spatial domain decomposition on the first image sequence frame by frame to obtain a first spatial domain signal, and calculating initial phase difference information of the second image sequence after the spatial domain decomposition;

[0148] A reconstruction module, used for resampling and reconstructing the first spatial domain signal to obtain a third image sequence of the second spatial domain signal;

[0149] An enhancement processing module, used for processing the third image sequence of the reconstructed second spatial domain signal using the reweighted residual sparsity to enhance the time correlation of the third image sequence of the reconstructed second spatial domain signal;

[0150] The acquisition module is used to acquire structural vibration information by using conversion calculation based on the phase difference information in the processed third image sequence.

[0151] The computer vision structural vibration identification device provided in the embodiment of the present application realizes the processing and identification of vibration information from video through spatial decomposition, resampling and reconstruction, reweighted residual sparsity processing, and time correlation enhancement, effectively resists noise, reduces phase step problems, and provides high-precision structural vibration analysis results.

[0152] The structural vibration recognition method based on computer vision provided in the embodiment of the present application can be evaluated from two aspects: motion amplification and vibration recognition. In the motion amplification analysis process, a three-layer frame vibration video was selected in this embodiment. In order to ensure the test of the stability of the algorithm, the method of this embodiment was compared with the traditional inference-based and phase-based methods. In order to increase the difficulty of the analysis, Gaussian noise with a variance range of 0.01 to 0.145 was artificially added in advance to all analyzed video signals.

[0153] In the evaluation of the frame structure magnification results, the horizontal slice line selected to record the structural movement is parallel to the floor and is located on the 2nd floor, see Figure 3(a)-Figure 3(d) . In the unamplified original video as shown in Figure 3(a), it can be found that the original vibration information is basically imperceptible. When the tiny movement is amplified 40 times, as shown in Figure 3(b) and Figure 3(c), it can be found that the phase-based and inference-based methods can roughly reveal the movement of the structure. However, the original noise is amplified, and the movement revealed by the shape of the frame structure and the horizontal tangent is basically severely blurred, as shown in the global view of Figure 3(b) and the white area of ​​Figure 3(c). In contrast, in Figure 3(d), the video motion processed using the structural vibration recognition method of computer vision provided in an embodiment of the present application (i.e., the proposed method in the figure) is clearer and there is almost no noise. The quantitative comparison of video motion amplification is shown in Figure 4(a) and Figure 4(b). For each method, the peak signal-to-noise ratio (PSNR) and structural similarity (SSIM) are used to calculate the similarity between the video amplified by the three methods and the original frame by frame. In Figure 4(a) and Figure 4(b), the dotted line represents the phase-based method, the dashed line represents the inference-based method, and the solid line represents the structural vibration recognition method based on computer vision provided in the embodiments of the present application. It can be found that the structural vibration recognition method of computer vision provided in the embodiments of the present application has significantly better performance in amplifying small structural motions.

[0154] The vibration table test used to verify the effectiveness of the computer vision in the structural vibration identification process provided in the embodiment of the present application can be shown in Figures 5(a) and 5(b). Figure 5(a) is a plan layout diagram of the stone curtain wall structure vibration table in the embodiment of the present application. The plane size of the structure is 3000mm (x direction) × 3000mm (y direction), and the total height is 4260mm. The sampling frequency of the camera is 50Hz, and the resolution of the video image sequence is 1920×1080. Figure 5(b) is a top view of the stone curtain wall structure vibration table test device in the embodiment of the present application, in which measuring point 1 (shown by a circular black dot) is located in the layer, and measuring point 2 (shown by a square black dot) is located on the layer. The results of vibration identification are compared. Figure 6(a)-Figure 6(f) As shown, the error comparison is shown in Figure 7(a) and Figure 7(b), wherein the results of measuring point 1 are shown in Figure 6(a), Figure 6(b), Figure 6(c), and Figure 7(a), and the results of point 2 are shown in Figure 6(d), Figure 6(e), Figure 6(f), and Figure 7(b). Although the phase-based and reasoning-based methods can accurately estimate partial structural vibrations, when large vibrations occur, large deviations will occur. In addition, the results obtained by the phase-based and reasoning-based methods have far more abnormal values ​​than the results obtained by the computer vision structural vibration identification method provided in the embodiment of the present application (i.e., the proposed method in the figure), as shown in Figure 7(a) and Figure 7(b). In summary, the computer vision structural vibration identification method provided in the embodiment of the application has better accuracy.

[0155] Although the embodiments disclosed in this application are as above, the contents described are only embodiments adopted to facilitate understanding of this application and are not intended to limit this application. Any technician in the field to which this application belongs can make any modifications and changes in the form and details of implementation without departing from the spirit and scope disclosed in this application, but the scope of patent protection of this application shall still be based on the scope defined in the attached claims.

Claims

1. A computer vision structural vibration recognition method, characterized in that: include: converting the original structural vibration video into a first image sequence; Performing spatial domain decomposition on the first image sequence frame by frame to obtain a first spatial domain signal, and calculating initial phase difference information of the second image sequence after the spatial domain decomposition; Resampling and reconstructing the first spatial domain signal to obtain a third image sequence of the second spatial domain signal; Processing the third image sequence using the reweighted residual sparsity to enhance the temporal correlation of the third image sequence; Based on the phase difference information in the third image sequence, structural vibration information is obtained using conversion calculation.

2. The structural vibration identification method according to claim 1, wherein: The step of converting the original structural vibration video into a first image sequence comprises: Converting the original structural vibration video into the first image sequence; The first image sequence is converted from the RGB color space to the YIQ color space.

3. The structural vibration identification method according to claim 2, wherein: The first image sequence is converted from RGB color space to YIQ color space according to the following formula: Among them, Y represents the brightness signal, I represents the color difference signal to which the human eye is sensitive, Q represents the color difference signal to which the human eye is not sensitive, R represents the red channel image signal, G represents the green channel image signal, and B represents the blue channel image signal.

4. The structural vibration identification method according to claim 1, 2 or 3, wherein: The step of performing spatial domain decomposition on the first image sequence frame by frame to obtain a first spatial domain signal, and calculating initial phase difference information of the second image sequence after the spatial domain decomposition, comprises: Using a complex-valued steerable pyramid model to perform frame-by-frame spatial domain decomposition on the first image sequence, and calculating a second image sequence corresponding to each frequency, thereby obtaining the first spatial domain signal; For each frequency feature of the second image sequence, the phase information of the current image is subtracted from the phase information of the first frame image, and the initial phase difference information Δψ of each frame image corresponding to the first frame image is calculated; The initial phase difference information of the second image sequence is obtained according to the initial phase difference information of all frequency components.

5. The structural vibration identification method according to claim 4, wherein: The complex-valued steerable pyramid model is shown below: Where A represents the amplitude, ψ represents the initial phase, and θ represents the frequency. represents the convolution operation, represents the real part of the convolution kernel with frequency θ, represents the imaginary part of the convolution kernel with frequency θ, and M is the image; The initial phase difference information Δψ is calculated as: Δψ θ (t) = ψ θ (t)-ψ θ (1), where t represents time.

6. The structural vibration identification method according to claim 1, 2 or 3, wherein: The method of resampling and reconstructing the first spatial domain signal to obtain a third image sequence of the second spatial domain signal includes: According to the initial phase difference information of the second image sequence, the first spatial domain information of the video image sequence is resampled and reconstructed using the compressed sensing sparsity theorem to obtain a third image sequence of the second spatial domain signal.

7. The structural vibration identification method according to claim 6, wherein: The reconstruction is shown below: Where i represents the image block number; P represents the image block size; Φ P represents the transformation matrix from image to block; M represents a video frame, which is used to represent a frame of image in a video; M P (i,t) is the resampling mode of M(i,t); is a hypothesis of M(i,t); H(i,t) is a single hypothesis, is the weight vector to be solved, which is used to represent the linear combination of the hypothesis H(i,t); the constraint is 8. The structural vibration identification method according to claim 7, further comprising: The uncertainty of the third image sequence is estimated using sparse coding, and the noise-free block domain matrix M is calculated. s (t).

9. The structural vibration identification method according to claim 8, wherein: The block domain uncertainty estimation is as follows: in, represents the optimized sparse coefficient matrix; C represents the coefficient matrix of sparse coding; represents the noisy image block matrix at time t; ‖·‖ F represents the Frobenius norm; ‖·‖1 represents the L1 norm; D represents the data dictionary, and γ represents the regularization parameter; The noise-free block domain matrix M s (t) is:

10. The structural vibration identification method according to claim 1, 2 or 3, wherein: The third image sequence is processed by the following formula to enhance the time correlation of the third image sequence: stT=M(t), Wherein, T represents the intermediate variable of frame M(t); M(t) represents the original vibration video frame at time t; represents the reconstructed video frame at time t; represents the noise-free block domain matrix of the key video frame; key represents the selected video key frame; Φ represents the image-to-block transformation matrix; β is the regularization parameter, nB represents the total number of blocks in the frame, and F is the weighted residual; the constraint condition T=M(t).

11. The structural vibration identification method according to claim 1, 2 or 3, wherein: The method of obtaining structural vibration information by using conversion calculation based on phase difference information in the third image sequence includes: Acquiring a structural motion response in the video based on the phase difference information in the third image sequence; According to the actual length of the structure, the obtained structural motion response in the video is converted into the structural vibration information.

12. The structural vibration identification method according to claim 11, wherein: Based on the phase difference information in the third image sequence, the structural motion response in the video is calculated by the following formula: Where u represents the structural motion response in the x direction, v represents the structural motion response in the y direction, ψ0 represents the phase of the structure in the x direction, and ψ π / 2 represents the phase of the structure in the y direction.

13. The structural vibration identification method according to claim 11, wherein: The step of converting the obtained structural motion response in the video into structural vibration information includes: Calculate the ratio between the length of a known structure and the number of pixels the structure spans in the image; The product of the structural motion response in the video and the obtained ratio is calculated to obtain the structural vibration information.

14. A computer-readable storage medium storing computer-executable instructions, wherein the computer-executable instructions are used to execute the computer vision structural vibration recognition method according to any one of claims 1 to 13.

15. A computer device comprising a memory and a processor, wherein: The memory stores the following instructions executable by the processor: used to execute the steps of the computer vision structural vibration identification method described in any one of claims 1 to 13.

16. A computer vision structural vibration recognition device, characterized in that: include: A conversion module, a decomposition processing module, a reconstruction module, an enhancement processing module, and an acquisition module; wherein, A conversion module, used for converting the original structural vibration video into a first image sequence; A decomposition processing module, used for performing spatial domain decomposition on the first image sequence frame by frame to obtain a first spatial domain signal, and calculating initial phase difference information of the second image sequence after the spatial domain decomposition; A reconstruction module, used for resampling and reconstructing the first spatial domain signal to obtain a third image sequence of the second spatial domain signal; An enhancement processing module, used for processing the third image sequence of the reconstructed second spatial domain signal using the reweighted residual sparsity to enhance the time correlation of the third image sequence of the reconstructed second spatial domain signal; The acquisition module is used to acquire structural vibration information by using conversion calculation based on the phase difference information in the processed third image sequence.

Images