Method and system for structural vibration measurement based on video images

Abstract

The application provides a kind of structure vibration measurement method and system based on video image, belong to computer vision technical field.The method includes determining the measurable pixel in vibration video;Respectively construct target motion signal matrix and background motion signal matrix;Background motion signal matrix is decomposed to construct camera shake reference base;Determine vibration frequency and construct vibration reference base;Pixel by pixel compare the energy proportion of each motion signal explained by two kinds of reference bases to determine the first vibration pixel set;Reliable weak response pixels are selected to be merged into the second vibration pixel set with the first vibration pixel set;According to the reliability score, the residual motion signal of each pixel after camera shake suppression is weighted and fused to obtain the vibration signal of the target detection object.The application can effectively distinguish the real target vibration from the camera shake interference, improve the accuracy and reliability of target detection object vibration measurement.

Description

Technical Field

[0001] This application belongs to the field of computer vision technology, and in particular relates to a method and system for measuring structural vibration based on video images. Background Technology

[0002] Video-based structural vibration measurement technology obtains the dynamic response of a structure by analyzing minute image motions in a video. It offers significant advantages such as being non-contact, enabling simultaneous multi-point measurement across the entire field, and avoiding the influence of sensor-added mass. It has broad application prospects in non-contact vibration measurement, operational status monitoring, and dynamic characteristic analysis of industrial equipment, building structures, mechanical structures, materials and components, and electronic components. With the continuous development of computer vision and digital image processing technologies, phase-based subpixel motion estimation methods can effectively capture minute structural vibrations in videos, providing an important non-contact measurement method for engineering scenarios where it is difficult to deploy a large number of contact sensors.

[0003] In existing technologies, video-based vibration measurement typically relies on the premise that the camera is approximately stationary during acquisition. However, in actual industrial sites or outdoor environments, cameras are usually mounted on tripods, temporary support platforms, or fixed structures. Factors such as ground vibration, equipment operation disturbances, wind loads, and personnel activities can all induce undesired motion (i.e., camera shake) in the camera and its support system, resulting in additional apparent motion in the video that is superimposed on the actual vibration of the object being measured. To address this issue, researchers have proposed various processing methods. For example, Zhu et al. (“VisualVibration Detection under Camera Motion Interference: A Structured PhaseManifold Decoupling Method with Spatial Heterogeneity Constraints”, IEEE Sensors Journal, 2026) proposed a structured phase manifold decoupling method with spatial heterogeneity constraints. This method extracts regional phase-time signals from the target region and multiple spatially discrete background regions. It constructs a structured feature matrix using constant, linear, square, and interaction terms of multiple background signals. It then fits the camera motion interference in the target region through regression and algebraically subtracts the estimated camera interference from the phase signal of the target region to obtain the purified regional vibration signal and main frequency information.

[0004] However, the existing methods described above are essentially still regional-level interference fitting and elimination frameworks. Their core lies in establishing a camera interference-side model, while target vibration is passively obtained only as a residual signal after interference elimination. This method does not establish independent source attribution criteria for each pixel within the target region, thus failing to determine at the pixel-by-pixel level whether the motion response of a measurable pixel primarily originates from the actual vibration of the target structure or from camera shake fitting residue or noise interference. Especially when the vibration frequency of the measured object is close to the camera shake frequency, regional-level algebraic subtraction is insufficient to effectively separate the two. The true vibration peak in the residual signal may be weakened, and shake residue may be misjudged as a vibration response, leading to a decrease in the accuracy and reliability of the final measurement results. This makes it difficult to meet the engineering requirements for high-reliability vibration measurement in complex field environments. Summary of the Invention

[0005] This application provides a structural vibration measurement method and system based on video images. By constructing a target vibration reference base and a camera jitter reference base respectively, and comparing the proportion of energy of the motion signal of each measurable pixel explained by the two types of reference bases on a pixel-by-pixel basis to determine its source, the method solves the problem that existing regional interference cancellation frameworks cannot distinguish between real target vibration and camera jitter at the pixel-by-pixel level, especially when the frequencies of the two are close, resulting in insufficient measurement accuracy.

[0006] In a first aspect, this application provides a method for measuring structural vibration based on video images, including: Identify measurable pixels in the vibration video; After determining the target area where the target object is located and the reference area where the background is located from the vibration video, the target motion signal matrix and the background motion signal matrix are constructed respectively based on the motion signals of the measurable pixels in the target area and the reference area. The background motion signal matrix is ​​decomposed, and the common temporal components of multiple measurable pixels in the reference area are extracted to construct a camera shake reference base. The vibration frequency of the target object is determined by using measurable pixels within the target area and a camera shake reference base, so as to construct a vibration reference base for the target object; Using a vibration reference base and a camera jitter reference base, the proportion of energy of the motion signal of each measurable pixel in the target area is determined by each reference base to determine the first set of vibration pixels; After selecting reliable weak response pixels from the measurable pixels that were not selected into the first vibration pixel set, all reliable weak response pixels are merged with the first vibration pixel set to form the second vibration pixel set. Based on the reliability score of each pixel in the second set of vibration pixels, the remaining motion signals of each pixel after suppressing camera shake are weighted and fused to obtain the vibration signal of the target object.

[0007] Optionally, determine the measurable pixels in the vibration video, including: Extracting the motion signal of each pixel over time from a vibration video including the target object; An imaging noise model is constructed based on the imaging conditions of the vibration video, and measurable pixels in the vibration video are selected based on the imaging noise model.

[0008] Optionally, after determining the target region where the target object is located and the reference region where the background is located from the vibration video, a target motion signal matrix and a background motion signal matrix are constructed respectively based on the motion signals of measurable pixels in the target region and the reference region, including: Based on the motion signals of each measurable pixel in the target area at each time, the motion amplitude of each measurable pixel at each time is arranged in chronological order to obtain the motion signal column vector of each measurable pixel. Arrange the column vectors of motion signals of all measurable pixels within the target area row by row to obtain the target motion signal matrix; Based on the motion signals of each measurable pixel in the reference area at each time, the motion amplitudes of each measurable pixel at each time are arranged in chronological order to obtain the motion signal column vector of each measurable pixel. The motion signal column vectors of all measurable pixels within the reference area are arranged row by row to obtain the background motion signal matrix.

[0009] Optionally, the background motion signal matrix is ​​decomposed to extract the common temporal components of multiple measurable pixels within the reference region, in order to construct a camera shake reference base, including: Singular value decomposition is performed on the background motion signal matrix to obtain multiple common time curves and the singular values ​​corresponding to the multiple common time curves; The common time curves corresponding to the first preset number of singular values ​​are selected as the column vectors of the camera shake reference basis, in descending order of singular values.

[0010] Optionally, the vibration frequency of the target object is determined using measurable pixels within the target area and a camera shake reference base to construct a vibration reference base for the target object, including: Based on the structural position of the target object, a preset target sub-region is determined within the target area, and measurable pixels with motion signal amplitude greater than a preset threshold are selected from the preset target sub-region as seed pixels of the target object. The motion signals of various sub-pixels are projected onto the camera shake reference base to obtain the fitting coefficients of the motion signals of various sub-pixels on the camera shake reference base. The camera shake component in the motion signal of each sub-pixel is reconstructed based on the fitting coefficients. The camera shake component is then removed from the original motion signal to obtain the remaining motion signal of each sub-pixel. Perform Fourier transform on the residual motion signals of various sub-pixels to obtain the spectral amplitude of each sub-pixel at each frequency. The median of the spectral amplitude of all seed pixels at each frequency is used to obtain the aggregated spectral amplitude of various sub-pixels. Within the preset frequency band of interest, the frequency corresponding to the maximum amplitude of the aggregated spectrum is determined as the vibration frequency of the target object. Using the vibration frequency as the center, the residual motion signals of various sub-pixels are subjected to narrowband filtering, and the median of the narrowband filtering results of various sub-pixels is taken at each time step to obtain the vibration reference signal of the target object. Perform a Hilbert transform on the vibration reference signal to obtain the orthogonal phase components; The vibration reference signal and its corresponding orthogonal phase components are used to form the initial vibration reference matrix of the target object. After removing components in the initial vibration reference matrix that can be interpreted by the camera jitter reference basis, orthogonal normalization is performed to obtain the vibration reference basis of the target object.

[0011] Optionally, using a vibration reference base and a camera jitter reference base, the proportion of energy of the motion signal of each measurable pixel in the target area interpreted by each reference base is determined to determine the first set of vibration pixels, including: Construct the first projection matrix corresponding to the vibration reference base and the second projection matrix corresponding to the camera jitter reference base, respectively; The motion signals of each measurable pixel in the target area are projected onto the first projection matrix and the second projection matrix respectively to obtain the energy of the vibration projection component of each measurable pixel and the energy of the camera shake projection component. The ratio of the energy of the vibration projection component of each measurable pixel to the total energy of the motion signal, and the ratio of the energy of the camera shake projection component to the total energy of the motion signal are obtained respectively, and used as the first energy ratio and the second energy ratio respectively. After combining the vibration reference base and the camera jitter reference base, orthogonal normalization is performed to obtain the joint reference base; Construct a joint projection matrix corresponding to the joint reference base to determine the proportion of residual energy in the motion signal of each measurable pixel that cannot be explained by the joint projection matrix; The competition score for vibration and jitter of each measurable pixel is calculated based on the difference between the first energy ratio and the second energy ratio and the residual energy ratio. The selection threshold is determined based on the competition score. Measurable pixels with competition scores greater than or equal to the selection threshold are selected to obtain the first set of vibration pixels.

[0012] Optionally, after selecting reliable weak response pixels from the measurable pixels that were not included in the first vibration pixel set, all reliable weak response pixels are merged with the first vibration pixel set to form a second vibration pixel set, including: Pixels within the target area that belong to the measurable pixel set but were not selected into the first vibration pixel set are considered as candidate pixels; Remove the components that can be interpreted by the camera shake reference base from the motion signal of each candidate pixel to obtain the remaining signal of each candidate pixel; A reliability score is calculated based on the remaining signal of each candidate pixel; Candidate pixels whose reliability scores meet the preset conditions are identified as reliable weak response pixels; All reliable weak response pixels are merged with the first set of vibration pixels to form the second set of vibration pixels.

[0013] Optionally, a reliability score is calculated based on the remaining signal of each candidate pixel, including: The proportion of energy in the remaining signal of the candidate pixel that can be explained by the vibration reference base is calculated as the first indicator; The proportion of the remaining signal spectral energy of the candidate pixel near the vibration frequency to the sum of the spectral energy near the vibration frequency and the camera shake frequency is calculated as a second indicator. Identify local neighbor pixels that are spatially adjacent to the candidate pixel and belong to the same structural part of the target object. Use the relative magnitude of the frequency band energy of the candidate pixel and the frequency band energy of other measurable pixels in the local neighbor pixels as a third indicator. The product of the first, second, and third indicators is used as the reliability score.

[0014] Optionally, based on the reliability score of each pixel in the second set of vibration pixels, the remaining motion signals of each pixel after suppressing camera shake are weighted and fused to obtain the vibration signal of the target object, including: Using the reliability score of each pixel in the second set of vibration pixels as the weight, the value of the remaining motion signal of each pixel after suppressing camera shake at the target time is weighted and averaged to obtain the vibration signal value at the target time. The vibration signal values ​​at each moment are arranged in chronological order to obtain the vibration signal of the target object.

[0015] Secondly, this application provides a structural vibration measurement system based on video images, comprising: The first determining module is used to determine the measurable pixels in the vibration video; The first construction module is used to determine the target area where the target object is located and the reference area where the background is located from the vibration video, and then construct the target motion signal matrix and the background motion signal matrix respectively based on the motion signals of the measurable pixels in the target area and the reference area. The second construction module is used to decompose the background motion signal matrix and extract the time component common to multiple measurable pixels in the reference area to construct a camera shake reference base. The third construction module is used to determine the vibration frequency of the target object by using measurable pixels in the target area and camera shake reference base, so as to construct the vibration reference base of the target object; The second determining module is used to determine the proportion of energy of the motion signal of each measurable pixel in the target area interpreted by each reference base using the vibration reference base and the camera shake reference base, so as to determine the first set of vibration pixels. The pixel merging module is used to select reliable weak response pixels from the measurable pixels that have not been selected into the first vibration pixel set, and then merge all reliable weak response pixels with the first vibration pixel set to form a second vibration pixel set. The weighted fusion module is used to perform weighted fusion on the remaining motion signals of each pixel after suppressing camera shake, based on the reliability score of each pixel in the second set of vibration pixels, to obtain the vibration signal of the target object.

[0016] Beneficial effects: The structural vibration measurement method based on video images provided in this application first determines the measurable pixels to ensure that all pixels involved in subsequent calculations have stable motion measurability, excluding pixels with excessive imaging noise that cannot provide effective motion information. Then, a camera shake reference base is constructed based on a reference region, and a vibration reference base for the target object is constructed based on the measurable pixels within the target region and the camera shake reference base, establishing time reference bases for two different sources: background shake and target vibration. Furthermore, unlike existing technologies that directly average or subtract regional interference from the motion signal of the entire target region, this application calculates, for each measurable pixel within the target region, the proportion of its motion signal energy that can be explained by the vibration reference base and the proportion that can be explained by the camera shake reference base. Based on this, at the pixel-by-pixel level, it quantifies and determines whether the motion response of each pixel mainly originates from real structural vibration or from camera shake, thereby determining the first set of vibrating pixels. Furthermore, this application selects reliable weak-response pixels from the candidate pixels that were not selected for the first vibration pixel set, based on their reliability scores. These pixels, while exhibiting weak vibration responses, are still reliable in origin and are merged with the first vibration pixel set to form a second vibration pixel set. This ensures that, while maintaining the reliability of the source attribution, pixels with weak vibration responses that still reflect the vibration of the real structure can be included in the measurement support domain, avoiding insufficient spatial coverage caused by retaining only strong-response pixels. Finally, based on the reliability scores of each pixel in the second vibration pixel set, the remaining motion signals of each pixel after suppressing camera shake are weighted and fused, so that pixels with high reliability dominate the final vibration signal, while the contributions of pixels with low reliability are correspondingly weakened. Through the synergistic effect of the above solutions, this application can change the processing paradigm of simply relying on frequency filtering or interference cancellation in the prior art, and transform the video vibration measurement problem under camera shake interference into a pixel-by-pixel source attribution problem; especially when the vibration frequency of the target object and the camera shake frequency are close to each other, it can effectively distinguish between real vibration and shake interference without relying on the difference in spectral position, significantly suppressing the interference of camera shake components on the final measurement results, while retaining the amplitude and frequency information of the real structural vibration, thereby obtaining a target object vibration signal with high accuracy and high reliability. Attached Figure Description

[0017] To more clearly illustrate the technical solution of this application, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 A flowchart illustrating a structural vibration measurement method based on video images, provided for an embodiment of this application; Figure 2 A schematic diagram illustrating the division of the target area and reference area in a lamp holder vibration experiment provided in this application embodiment; Figure 3 Diagnostic results of vibration frequency and camera shake frequency provided in the embodiments of this application; Figure 4 The diagram shows the construction results of the vibration dominance score and the reliable vibration pixel set provided in the embodiments of this application; Figure 5 A comparison diagram of the spectra of motion signals obtained by different pixel sets and jitter suppression methods in the lamp holder vibration experiment provided in the embodiments of this application; Figure 6 A schematic diagram illustrating the division of the target area and reference area in a humidifier experiment provided in this application embodiment; Figure 7 Diagram showing the diagnostic results of the humidifier vibration frequency and camera shake frequency provided in the embodiments of this application; Figure 8 This application provides a diagram showing the target area core support, weak reliability support, and reliability weighting results in an embodiment. Figure 9 A comparison of the spectra of motion signals before and after jitter suppression in different pixel sets during a humidifier experiment provided in this application embodiment; Figure 10 This is a schematic diagram of a structural vibration measurement system based on video images, provided as an embodiment of this application. Detailed Implementation

[0019] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0020] Example 1 like Figure 1 As shown in the figure, this application provides a structural vibration measurement method based on video images, including: Step 101: Determine the measurable pixels in the vibration video.

[0021] For example, motion signals of each pixel over time are extracted from a vibration video including the target object.

[0022] Specifically, the vibration video is converted into multiple vibration images I(p,d) ordered chronologically, where d=1,2,…,D; D is the total number of frames in the vibration video; and p represents the number of pixels in the frame image.

[0023] Perform multi-scale, multi-directional complex filtering on each frame of the image to obtain the pixel p in the d-th frame at different scales. Local complex response under direction angle θ :

[0024] in, Indicates the pixel p in the d-th frame of the image at scale The local amplitude under the direction angle θ; exp(·) represents an exponential function with the natural constant e as the base; j is the imaginary unit; Indicates the pixel p in the d-th frame of the image at scale Local phase under direction angle θ.

[0025] Local phase change of pixel p in frame d and frame (d-1) of the (vibration) video Represented as:

[0026] Under minute vibrations, the local phase change approximately satisfies the following relationship with the pixel displacement:

[0027] Where T represents the matrix transpose; Let represent the local wave vector of the corresponding complex filter; Δd(p,d) represents the displacement of pixel p between the d-th and (d-1)-th frames of the video image; for each selected complex filter, its local wave vector... The local wave vector of the filter is determined by its spatial frequency parameters; for example, for a Gabor filter, it can be expressed as: ,in For this filter at scale The center frequency is given by θ, where θ is the direction angle of the filter. The local wave vector is used to establish an approximately linear relationship between the local phase change and the actual small displacement of the pixel on the image plane.

[0028] For pixel p, at different scales Local phase change under direction θ By performing a weighted average, we obtain the one-dimensional motion amplitude x(p,d) corresponding to pixel p in the d-th frame:

[0029] Where S is the set of all different scale-direction angles; ε represents a positive number used to prevent the denominator from being zero.

[0030] Arrange the one-dimensional motion amplitudes corresponding to each frame image according to the frame number to obtain the one-dimensional motion signal. N is the total number of time sampling points; Let be the set of real numbers.

[0031] An imaging noise model is constructed based on the imaging conditions of the vibration video, and the set of measurable pixels in the vibration video is obtained by screening based on the imaging noise model.

[0032] Specifically, a nearly static video segment with camera exposure time, gain, frame rate, and lighting conditions identical to the vibration video is selected as the calibration video. The grayscale value of pixel p' in the q-th frame of the calibration video is denoted as I. cal (p',q); q=1,2,…,N c N c This is used to determine the total number of frames in the calibrated video.

[0033] Calculate the average gray level of pixel p' in the calibration video. :

[0034] Calculate the sample variance of the gray level of pixel p' over time. This is used to characterize the imaging noise intensity at that grayscale location:

[0035] To avoid misinterpreting real edges, textures, or slight structural motion as imaging noise, pixels with significant variations are removed based on the spatial gradient of the average image, while flat areas are retained.

[0036] Among them, Ω flat Represents a set of pixels in a flat region; τ represents the 2-norm; g This represents the gradient threshold used to distinguish between flat and edge regions.

[0037] The average gray level is divided into several gray level intervals, and the average sample variance of flat pixels within each interval is calculated:

[0038] Where, f(I) k ) represents the gray values ​​I of k gray ranges. k The variance of imaging noise at time; mean{·} represents the average of the values ​​in the set; C k This represents the k-th grayscale interval.

[0039] Interpolation or fitting is performed on the discrete data points obtained in each grayscale range to obtain the imaging noise level function:

[0040] in, f(I) represents the pixel grayscale value; f(I) represents the grayscale value as... The variance of imaging noise at that time; σ I This represents the corresponding standard deviation of the imaging noise. If a suitable approximate still video segment does not exist, f(I) can be estimated using calibrated videos independently acquired by the same camera under the same exposure, gain, and illumination conditions, or a pre-calibrated noise level function can be used.

[0041] Select the grayscale value I of pixel p from the reference frame image in the vibration video. ref (p). Two sets of independent simulated noise are superimposed on the same reference frame image to obtain the m-th noisy image pair:

[0042] in, This represents the k'-th noisy image obtained from the m-th noise simulation; The standard normal noise sample is represented; N(0,1) represents a normal distribution with a mean of 0 and a variance of 1; iid represents that each noise sample is independent and follows the same distribution; M represents the total number of noise simulations; the two noisy images are generated from the same reference frame image, so there is no real structural displacement between them, and the calculated motion is only caused by noise disturbance.

[0043] The two noisy images generated by the m-th noise simulation are used as the first and second frames, respectively. The same multi-scale, multi-directional angle local phase motion calculation method as described above is employed to calculate the phase change at each pixel, thus obtaining the pixel... Phase change:

[0044] Where, n (m) (p) represents the phase change of pixel p in a frame of a noisy video (i.e., a noisy image); and These represent the phase changes of pixel p in the horizontal x and vertical y directions of a frame image of a noisy video, respectively. Since noisy images do not exhibit true displacement, all calculated motion is considered as pseudo-motion resulting from local phase disturbances caused by imaging noise.

[0045] Calculate the mean phase change of pixel p in the noisy video. :

[0046] Next, calculate the phase change covariance matrix of pixel p in the noisy video. :

[0047] Eigenvalue decomposition is performed on the phase change covariance matrix of pixel p in the noisy video:

[0048] Where, V(p)=[d min (p),d max [p];0≤λ min (p)≤λ max (p); λ min (p) and λ max (p) represent the smaller and larger variances of the two-dimensional pseudo-motion in two mutually orthogonal directions, respectively; d min (p) and d max (p) represents the unit direction vectors corresponding to the two; V(p) represents the orthogonal matrix formed by the columns of these two unit direction vectors. The direction corresponding to the smaller eigenvalue indicates that the motion fluctuation at pixel p caused by noise is smaller, therefore d min (p) is determined as the motion direction of pixel p that is less affected by noise.

[0049] Due to λ min (p) indicates the phase change along d min (p) The variance in the direction, therefore, λ min (p) The square root is defined as the pixel p in d min (p) Standard deviation of phase change in the direction:

[0050] Where, σ noise The smaller (p) is, the more likely pixel p is to move along d. min (p) Motion calculations in the direction of motion are less susceptible to imaging noise.

[0051] Set screening criteria based on the standard deviation of motion noise:

[0052] Among them, Ω meas τ represents the set of measurable pixels in a real video (vibration video) image; Ω represents the set of all pixels in a real video image. noise This represents the measurable pixel selection threshold, used to identify pixels that are retained after filtering. The measurable pixel selection threshold can be determined based on the statistical distribution, sorting, or percentile of the standard deviation of noise pseudo-motion for each pixel.

[0053] This step determines whether pixel motion can be stably measured under the current imaging noise conditions, but does not determine whether the motion comes from structural vibration of the target object or camera shake.

[0054] Step 102: After determining the target area where the target object is located and the reference area where the background is located from the vibration video, construct the target motion signal matrix and the background motion signal matrix respectively based on the motion signals of the measurable pixels in the target area and the reference area.

[0055] For example, based on the motion signals of each measurable pixel in the target area at each time, the motion amplitudes of each measurable pixel at each time are arranged in chronological order to obtain the motion signal column vector of each measurable pixel.

[0056] The target motion signal matrix is ​​obtained by arranging the column vectors of motion signals of all measurable pixels within the target area row by row.

[0057] Based on the motion signals of each measurable pixel in the reference area at each time, the motion amplitudes of each measurable pixel at each time are arranged in chronological order to obtain the motion signal column vector of each measurable pixel.

[0058] The motion signal column vectors of all measurable pixels within the reference area are arranged row by row to obtain the background motion signal matrix.

[0059] Specifically, the selected target region and reference region are Ω respectively. tar and Ω ref Then the corresponding set of measurable pixels is:

[0060] Among them, Ω eff Ω represents the set of measurable pixels within the target area; bg Represents the set of measurable pixels within the reference area; It represents the intersection of sets.

[0061] For any measurable pixel p'', arrange its motion signals at each time step into a column vector:

[0062] This represents the motion signal of the measurable pixel p'' at the Nth time.

[0063] Let Ω eff ={p1,p2,…, The motion signals of the measurable pixels in the target area are arranged row by row to obtain the target motion signal matrix X. eff :

[0064] Where, N eff This represents the number of measurable pixels used for calculation within the target area.

[0065] Let Ω bg ={p1,p2,…, Arrange the motion signals of the background pixels row by row to obtain the background motion signal matrix:

[0066] Where, N bg This represents the number of measurable pixels used for calculation within the reference area.

[0067] Step 103: Decompose the background motion signal matrix and extract the time component shared by multiple measurable pixels in the reference area to construct a camera shake reference base.

[0068] For example, singular value decomposition is performed on the background motion signal matrix to obtain multiple common time curves and the singular values ​​corresponding to the multiple common time curves.

[0069] Specifically, for the background motion signal matrix X bg Perform singular value decomposition:

[0070] Among them, U bg =[u1,u2,…,u r ];V bg =[v1,v2,…,v r ];Σ bg =diag(σ1,σ2,…,σ r ); σ1≥σ2≥…≥σ r >0.

[0071] Singular value decomposition (SVD) is used to represent the motion signals of multiple background pixels (i.e., multiple measurable pixels within a reference region) as a weighted combination of several common time curves. r represents the number of non-zero singular values ​​obtained from SVD; v r This represents the r-th common time curve extracted from the motion signals of multiple background pixels, containing the motion values ​​at all N sampling times; u r σ represents the relative coefficient of the r-th common time curve in the motion signal of each background pixel; r Σ represents the r-th singular value, used to characterize the overall contribution of the corresponding common temporal curve to the motion signal of all background pixels; bg This represents a diagonal matrix composed of singular values.

[0072] The common time curves corresponding to the first preset number of singular values ​​are selected as the column vectors of the camera shake reference basis, in descending order of singular values.

[0073] The larger the singular value, the greater the proportion of the corresponding common time curve in the total motion signal of all background pixels. Following the order of singular values ​​from largest to smallest, the common time curves v1, v2, ..., v corresponding to the first R' larger singular values ​​are selected.R' Since these common time curves are matrices V obtained from singular value decomposition. bg The column vectors are already orthogonal and have a length of 1. Therefore, arranging them directly by column yields the camera shake reference base D. jit :

[0074] Where R' represents the number of common time curves used to construct the camera shake reference base, which can be determined according to the preset number or the cumulative energy ratio; N represents the number of time sampling points contained in each common time curve; Represents an N-row, R'-column matrix of real numbers.

[0075] Step 104: Determine the vibration frequency of the target object using the measurable pixels in the target area and the camera shake reference base, so as to construct the vibration reference base of the target object.

[0076] For example, a preset target sub-region is determined within the target area based on the structural position of the target object, so as to select measurable pixels with motion signal amplitude greater than a preset threshold from the preset target sub-region as seed pixels of the target object.

[0077] Based on the structural location of the target object, a small area within the target region that is confirmed to belong to the target object is pre-determined and denoted as the pre-defined target sub-region Ω. prior For a seed pixel p''' that is simultaneously located in the preset target sub-region and whose motion can be reliably measured, calculate the root mean square amplitude a(p''') of its motion signal:

[0078] Where, x p''' (n) represents the amplitude of the motion signal of the seed pixel p''' at time n.

[0079] From the intersection of measurable pixels and the preset target sub-region, select pixels with larger motion signal amplitudes as seed pixels for target detection:

[0080] Among them, Ω seed Ω represents the set of seed pixels for the target object; seed It is only used to determine the vibration frequency of the target object and to construct an initial vibration reference base, and is not used as the final reliable set of vibration pixels; τ seed This indicates the preset threshold for selecting the amplitude of the seed pixel motion signal.

[0081] The motion signals of various sub-pixels are projected onto the camera shake reference base to obtain the fitting coefficients of the motion signals of various sub-pixels on the camera shake reference base.

[0082] Specifically, due to camera shake reference base D jit Each column has been orthogonally normalized. For any seed pixel p''', its motion signal x p''' Fitting coefficients on the camera shake reference basis for:

[0083] The camera shake component in the motion signal of each sub-pixel is reconstructed based on the fitting coefficients. The camera shake component is then removed from the original motion signal to obtain the remaining motion signal of each sub-pixel.

[0084] Specifically, based on the fitting coefficients, the components in the motion signal of the seed pixel p''' that can be explained by the camera shake reference base are obtained. :

[0085] Among them, P jit is the projection matrix of the camera shake reference base.

[0086] The camera shake-related components are removed from the original motion signal to obtain the residual motion signals of various sub-pixels. :

[0087] Where I' represents the identity matrix, and its dimensions are sums to P. jit Maintain consistency.

[0088] Perform Fourier transform on the residual motion signals of various sub-pixels to obtain the spectral amplitude of each sub-pixel at each frequency.

[0089] Specifically, perform a Fourier transform on the remaining motion signal of each seed pixel:

[0090] Where, x res,p''' (n) represents the remaining motion signal of seed pixel p''' at time n; F{·} represents the Fourier transform; X res,p''' (f) represents the spectral amplitude of the remaining motion signal of the seed pixel p''' at frequency f.

[0091] By taking the median of the spectral amplitudes of all seed pixels at each frequency, the aggregated spectral amplitudes of various sub-pixels are obtained:

[0092] Where median(·) represents taking the median of the spectral amplitude for all seed pixels; A seed,res(f) represents the aggregated spectral amplitude of all seed pixels. Median aggregation can reduce the impact of abnormal peaks and random noise in individual seed pixels.

[0093] Within a preset frequency band of interest, the frequency corresponding to the maximum amplitude of the aggregated spectrum is determined as the vibration frequency of the target object:

[0094] Among them, [f min ,f max ] indicates a pre-defined frequency band of interest; arg max A seed,res (f) represents the frequency at which the aggregated spectrum amplitude reaches its maximum value; f t This represents the final determined vibration frequency, which is used for subsequent narrowband filtering and vibration reference base construction.

[0095] Using the vibration frequency as the center, narrowband filtering is performed on the residual motion signals of various sub-pixels, and the median of the narrowband filtering results of various sub-pixels is taken at each time step to obtain the vibration reference signal of the target object:

[0096] Wherein, s0(n) represents the vibration reference signal of the target object at the nth time; Indicated by f t The narrowband filtering operation is centered on the seed pixels; taking the median at each time step means sorting the narrowband filtering results of all seed pixels at each time step and taking the median value to reduce the influence of individual abnormal seed pixels and random noise.

[0097] The vibration reference signal is subjected to Hilbert transform to obtain the orthogonal phase components.

[0098] Specifically, a Hilbert transform is performed on the vibration reference signal: h0(n)=H{s0(n)} Where H{·} denotes the Hilbert transform; h0(n) is the orthogonal phase component that is 90° out of phase with s0(n).

[0099] The vibration reference signal and its corresponding orthogonal phase components are used to form the initial vibration reference matrix of the target object; the two reference signals are arranged into column vectors s0 and h0 according to time to form the initial vibration reference matrix Z. vib =[s0,h0].

[0100] After removing components in the initial vibration reference matrix that can be interpreted by the camera jitter reference basis, orthogonal normalization is performed to obtain the vibration reference basis of the target object.

[0101] Specifically, by using the orthogonal complement projection of the camera jitter reference space, the signal components in the initial target vibration reference matrix that can still be explained by the camera jitter reference basis are removed, and then the remaining column vectors are orthogonally normalized: D vib =orth((I'-P jit )Z vib ) Among them, D vib This is the final vibration reference basis; orth(·) denotes the orthogonal normalization operation; (I'-P jit )Z vib Indicates removing Z vib The components that fall into the learned camera shake reference space are then transformed into mutually orthogonal basis vectors of length 1 by an orthogonal normalization operation, so that the proportion of energy of each pixel motion signal that can be explained by the vibration reference basis can be stably calculated later.

[0102] This step removes signal components that can be interpreted by the camera shake reference base, rather than making unverified claims of removing all real camera shake.

[0103] Step 105: Using the vibration reference base and the camera jitter reference base, determine the proportion of energy of the motion signal of each measurable pixel in the target area that is interpreted by each reference base, so as to determine the first set of vibration pixels.

[0104] For example, a first projection matrix corresponding to the vibration reference base and a second projection matrix corresponding to the camera jitter reference base are constructed respectively.

[0105] Specifically, the vibration reference base D vib and camera shake reference base D jit The corresponding projection matrices are:

[0106] Among them, P vib P is the first projection matrix corresponding to the vibration reference base; jit This is the second projection matrix corresponding to the camera shake reference base.

[0107] The motion signals of each measurable pixel within the target area are projected onto the first projection matrix and the second projection matrix, respectively, to obtain the energy of the vibration projection component of each measurable pixel and the energy of the camera shake projection component.

[0108] The ratio of the energy of the vibration projection component of each measurable pixel to the total energy of the motion signal, and the ratio of the energy of the camera shake projection component to the total energy of the motion signal are obtained respectively, and are used as the first energy ratio and the second energy ratio respectively.

[0109] Specifically, for the motion signal x of measurable pixel p'' within the target area p'' The ratio of the energy of the projected component of the vibration reference base to the total energy of the motion signal of the measurable pixel p'' for:

[0110] Here, ε represents a positive number used to prevent the denominator from being zero.

[0111] The ratio of the energy of the camera shake reference base projection component to the total energy of the motion signal of the measurable pixel p'' is:

[0112] After combining the vibration reference base and the camera jitter reference base and performing orthogonal normalization, a joint reference base is obtained:

[0113] Among them, D joint The joint reference basis matrix is ​​formed by the vibration reference basis D. vib With camera shake reference base D jit P is obtained by concatenating columns and then orthogonally normalizing them. joint It is a joint projection matrix used to project the motion signal of the measurable pixel onto a joint subspace composed of the vibration reference base and the camera jitter reference base.

[0114] Construct a joint projection matrix corresponding to the joint reference base to determine the proportion C of residual energy in the motion signal of each measurable pixel that cannot be explained by the joint projection matrix. res (p''):

[0115] The competition score for vibration and jitter of each measurable pixel is calculated based on the difference between the first energy ratio and the second energy ratio, as well as the residual energy ratio.

[0116] Specifically, based on the difference between the first energy ratio and the second energy ratio, and by penalizing the anomalous response using the residual energy ratio of the joint reference base, the vibration and jitter competition score of the measurable pixel p'' is obtained: Q(p'')=(max(0,C vib (p'')-C jit (p'')))(1-C res (p'')) γ Where max(·,·) represents taking the maximum of the two values ​​within the parentheses; γ is the residual penalty strength; when the interpretability of the camera shake reference base is not lower than that of the vibration reference base, the competition score is zero; when the joint reference base still cannot fully interpret the motion signal of the measurable pixel p'', a larger residual ratio will reduce the competition score.

[0117] The selection threshold is determined based on the percentile of the competition score. Measurable pixels with competition scores greater than or equal to the selection threshold are selected to obtain the first set of vibration pixels.

[0118] Specifically, let ρ core The selection threshold τ is determined based on the percentile of the competition scores of all measurable pixels, with a preset core vibration pixel retention ratio. core :

[0119] in, This represents quantile operations, which involve arranging the values ​​within the parentheses in ascending order and taking the quantile value at position 1-ρ. core The value at the quantile.

[0120] The core vibration pixel set, i.e., the first vibration pixel set Ω, is obtained based on the threshold. core :

[0121] The core vibration pixel simultaneously satisfies the following conditions: The vibration reference base can interpret the motion signal of the pixel well.

[0122] The explanatory power of the vibration reference base is higher than that of the camera shake reference base.

[0123] The pixel motion signal can be fully explained by a joint reference base consisting of a vibration reference base and a camera shake reference base, rather than noise or anomalous responses that cannot be explained by either type of reference base.

[0124] Step 106: After selecting reliable weak response pixels from the measurable pixels that were not selected into the first vibration pixel set, merge all reliable weak response pixels with the first vibration pixel set to form the second vibration pixel set.

[0125] For example, pixels within the target region that belong to the set of measurable pixels but were not selected into the first set of vibration pixels are selected as candidate pixels to obtain the candidate pixel set Ω. cand .

[0126] Remove the components that can be interpreted by the camera shake reference base from the motion signal of each candidate pixel to obtain the remaining signal of each candidate pixel.

[0127] Specifically, for any candidate pixel , From its original motion signal Remove components that can be explained by the camera shake reference base to obtain candidate pixels. The remaining signal :

[0128] A reliability score is calculated based on the remaining signal of each candidate pixel.

[0129] Furthermore, the proportion of energy in the remaining signal of the candidate pixel that can be explained by the vibrational reference basis is calculated as the first metric:

[0130] in, Indicates candidate pixels The proportion of energy in the remaining signal that can be explained by the vibrational reference base.

[0131] The proportion of the remaining signal spectral energy of the candidate pixel near the vibration frequency to the sum of the spectral energy near the vibration frequency and near the camera shake frequency is calculated as a second indicator.

[0132] Specifically, perform a Fourier transform on the remaining motion signal of the candidate pixels:

[0133] in, Indicates candidate pixels The spectral amplitude of the remaining motion signal at frequency f; Indicates candidate pixels The remaining motion signal at time n; n=1,2,…,N.

[0134] Let the vibration frequency of the target object be f. t The camera shake frequency is f jit The corresponding frequency bands for the two are f t -Δf t ≤B t ≤f t +Δf t and f jit -Δf jit ≤B jit ≤f jit +Δf jit ;Δf t >0, Δf t B represents the vibration frequency band of the target object being detected. t Half bandwidth; Δf jit >0, Δf jit Indicates camera shake frequency band B jit Half bandwidth.

[0135] Candidate pixels The spectral energy within the two frequency bands mentioned above is defined as follows:

[0136] in, Indicates candidate pixels The spectral energy of the remaining motion signal within the vibration frequency band of the target object; Indicates candidate pixels The spectral energy of the remaining motion signal within the camera shake frequency band.

[0137] The second indicator is expressed as:

[0138] in, express exist and The proportion in the sum; The larger the value, the more likely it is to be a candidate pixel. In the remaining motion signal, the vibration frequency component of the target object is more dominant than the camera shake frequency component.

[0139] Identify local neighboring pixels that are spatially adjacent to the candidate pixel and belong to the same structural part of the target object. Use the relative magnitude of the frequency band energy of the candidate pixel and the frequency band energy of other measurable pixels in the local neighboring pixels as a third indicator.

[0140] Specifically, for Perform logarithmic compression:

[0141] in, This represents the vibrational frequency band energy after logarithmic compression.

[0142] With candidate pixels Centered on the target region, select candidate pixels among the measurable pixels. Pixels belonging to the same structural region of the target object and located within a preset radius are selected as candidate pixels. local neighborhood :

[0143] in, Represents pixels The coordinates of the frame image in the image coordinate system; Indicates candidate pixels The coordinates of in the image coordinate system; r is the preset local neighborhood radius; Represents pixels The label of the local structural region to which it belongs; Indicates candidate pixels The label of the local structural region to which it belongs; the same structural region refers to the structural region with the same label obtained through image segmentation or edge detection; Represents pixels and candidate pixels They belong to the same local structural area (taking a lamp holder as an example, for instance, they all belong to the pixels on the lamp holder body). This represents the Euclidean distance between two pixels.

[0144] set up and Within the neighborhood The lower robust percentile and the higher robust percentile, then the candidate pixel The local energy relative value is:

[0145] in, Indicates candidate pixels The relative local energy values ​​are used to compare candidate pixels. The corresponding frequency band energy and the frequency band energy of other measurable pixels in the local neighborhood; clip [0,1] (·) indicates that the result of the calculation within the parentheses is limited to between 0 and 1.

[0146] The product of the first, second, and third indices is used as the candidate pixel. Reliability score :

[0147] The product form requires candidate pixels to simultaneously meet the following three conditions: the remaining signal can be interpreted by the vibration reference base, the frequency band energy accounts for a relatively high proportion, and the frequency band energy can be distinguished in the local neighborhood of the same structural region.

[0148] Candidate pixels whose reliability scores meet preset criteria are identified as reliable weak response pixels:

[0149] Among them, Ω weak For a reliable set of weakly responding pixels; τ weak The threshold for selecting reliable weak response pixels can be determined based on the sorting, percentile, or preset value of the reliability score.

[0150] Merge all reliable weak response pixels with the first set of vibration pixels into a second set of vibration pixels:

[0151] ∪ represents the union of sets; Indicates an inclusion relationship; Ω rel This represents the set of reliable vibration pixels, i.e., the second set of vibration pixels. Ω weak Instead of a simple spatial expansion of the core region, it consists of weak-response pixels that, after jitter-related component suppression, still simultaneously meet the three performance requirements, forming an Ω. rel Then, a reliability score is calculated for all pixels, which is used as a weighting coefficient in the subsequent weighted calculation of vibration signals.

[0152] Step 107: Based on the reliability score of each pixel in the second set of vibration pixels, the remaining motion signals of each pixel after suppressing camera shake are weighted and fused to obtain the vibration signal of the target object.

[0153] For example, using the reliability score of each pixel in the second set of vibration pixels as weights, the values ​​of the remaining motion signals of each pixel after suppressing camera shake at the target time are weighted and averaged to obtain the vibration signal value at the target time:

[0154] in, Let be the vibration signal value at the nth time. Represents the (measurable) pixels in the second set of vibration pixels. Reliability score; Represents pixels The remaining motion signal at time n.

[0155] The vibration signal values ​​at each moment are arranged in chronological order to obtain the vibration signal of the target object.

[0156] In this step, the weighted fusion does not directly average the pixel motion signals across the entire target area, nor does it use only the core pixel with the strongest vibration. Instead, it utilizes both the first vibrating pixel and verified reliable weak-response pixels, while reducing the impact of low-reliability pixels on the final result.

[0157] This embodiment further discloses specific examples: In one example, the vibration measurement of a lamp holder under near-frequency camera shake interference is taken as an example. The lamp holder 2 is used as the target object. A sinusoidal excitation with an amplitude of 0.1V and a frequency of 8Hz is applied to the lamp holder 2 using an exciter 3. The camera 4 is mounted on a tripod. During video acquisition, the tripod is disturbed to create experimental conditions where the vibration of the target object and camera shake coexist within a similar frequency range. The area containing the lamp holder structure is designated as the first target area 6, and the checkerboard background area 1, which does not contain the lamp holder structure vibration but is affected by camera shake along with the first target area, is designated as the first reference area 5. An example of the experimental setup is shown below. Figure 2 As shown in (a) above, the region is divided as follows: Figure 2 As shown in (b) of the diagram.

[0158] The frequency diagnosis results obtained by processing the acquired video using the method described in this application are as follows: Figure 3 As shown, the residual motion signal after suppressing camera shake components at the seed pixel exhibits a significant peak at 7.998Hz, consistent with the set excitation frequency of 8Hz; the camera shake reference base shows a significant peak at 8.6133Hz. Therefore, 7.998Hz and 8.613Hz are determined as the vibration frequencies f, respectively. t and (camera) shake frequency f jit .

[0159] The vibration dominance score and the construction results of the reliable vibration pixel set are as follows: Figure 4 As shown, Figure 4 (a) shows the proportion of energy in the remaining motion signal after pixel-suppressed camera shake that can be explained by the vibration reference base; Figure 4 (b) shows the proportion of the spectral energy of the pixel's residual signal near the vibration frequency to the sum of the spectral energy near the vibration frequency and near the camera shake frequency; Figure 4 (c) shows the relative values ​​of local energy; Figure 4 (d) shows the pixel reliability score; Figure 4 (a) through (d) are all normalized results; Figure 4 (e) shows the core set of vibrating pixels (i.e., the first set of vibrating pixels); Figure 4 (f) shows a set of reliable weak-response pixels; Figure 4 Figure (g) shows the final reliable vibration pixel set (i.e., the second vibration pixel set). A total of 5455 core vibration pixels and 3021 reliable weak response pixels were selected, resulting in 8476 reliable vibration pixels, accounting for approximately 13.92% of the effective pixels in the first target area. Among them, the core vibration pixels are mainly concentrated in the lamp head area where vibration is more obvious, while the reliable weak response pixels mainly supplement the pixels near the lamp head and on the upper part of the lamp holder with weak amplitude but still consistent with the vibration changes of the target object. The final reliable vibration pixel set is not a simple spatial expansion of the core vibration area, but is composed of pixels that have been verified by vibration-jitter source comparison and weak response reliability.

[0160] The spectrum obtained by different measurement methods and the quantitative comparison results are as follows: Figure 5 As shown in Table 1. To facilitate comparison at the same scale, Figure 5 The spectral amplitude of each measurement method is based on the average motion signal of all effective pixels within the first target area at the vibration frequency f. t The nearby spectral peaks are converted using a common reference. Let A... tThis indicates that, after conversion based on the aforementioned common reference, the vibration frequency f t Maximum spectral amplitude within a nearby local frequency window; A jit f represents the camera shake frequency. jit ( Figure 5 Chinese f jit Use f j (represented by) the maximum spectral amplitude within a local frequency window near (A); then A t / A jit This represents the peak value ratio of the target vibration component to the camera shake component. Because different measurement methods use the same peak value as a common reference, rather than normalizing to their respective maximum values, therefore... Figure 5 The amplitude of some spectra in the image can be greater than 1. When directly averaging all effective pixels within the first target region, subtracting from the background reference signal, and suppressing signal components within the narrow frequency band where camera shake occurs, A... t / A jit The ratios were 1.28, 2.05, and 3.20, respectively; when using the method of this application, the ratio increased to 8.63. Taking the result obtained by directly averaging all effective pixels as a reference, the suppression ratio of jitter-related components reached 6.25, while retaining the vibration peak of the target object at 7.998Hz.

[0161] Table 1. Quantitative Comparison Results of Different Measurement Methods under Near-Frequency Vibration-Shaking Conditions

[0162] This example illustrates that when the vibration frequency of the target object is close to the camera's shaking frequency, this application can filter out pixels that mainly reflect the vibration of the target object, reduce the influence of shaking components on the measurement results, and obtain a motion signal with a stable vibration frequency of the target object.

[0163] In another example, the application effect under conditions of uneven texture distribution on the surface of the target object is taken as an example. Using a humidifier as the target object, to achieve effective testing, patterns with different contrasts and texture intensities are set on its surface, so that the target object surface simultaneously includes areas with clear texture, areas with weak texture, and areas with low texture. The surface of the humidifier is designated as the second target area 7, and the stationary area next to the humidifier that does not contain target structural vibrations but is affected by camera shake along with the second target area is designated as the second reference area 8. Figure 6 As shown.

[0164] The method described in this application is used to process the acquired video. First, a camera shake reference base is constructed from the pixel motion signals of the reference region; then, components that can be explained by the camera shake reference base are suppressed from the motion signals of the seed pixels, and spectral analysis is performed on the remaining motion signals. The frequency diagnostic results are as follows: Figure 7As shown, the blue curve represents the spectrum of the remaining motion signal after the seed pixel suppresses camera shake, and the orange curve represents the spectrum of the first principal component of the camera shake reference base. The blue curve shows a significant peak at 14.560Hz, therefore 14.560Hz is determined as the target vibration frequency; the orange curve shows a significant peak at 13.125Hz, therefore 13.125Hz is determined as the camera shake frequency.

[0165] The construction result of the reliable vibration pixel set is as follows Figure 8 As shown. Figure 8 (a) in the image shows the target object to be detected; Figure 8 (b) shows the core vibration pixel set; Figure 8 (c) in the diagram shows a set of reliable weak-response pixels; Figure 8 In the diagram, (d) represents the normalized reliability weighted graph. After determining the vibration frequency and camera shake frequency, the motion signal of each pixel within the second target region is evaluated using both the vibration reference base and the camera shake reference base to obtain the core vibration pixels. Then, weak response reliability is assessed for pixels not included in the core vibration pixel set (i.e., the first vibration pixel set), and reliable weak response pixels that still contain vibration information of the target object are selected. In this embodiment, there are a total of 86,507 pixels within the second target region that have been screened and retained. Of these, 37,182 pixels are selected as core vibration pixels, and 7,802 pixels are selected as reliable weak response pixels, resulting in 44,984 reliable vibration pixels. Reliable weak response pixels account for approximately 17.3% of the final reliable vibration pixels.

[0166] The comparison results of the spectra of different pixel sets before and after jitter suppression are as follows: Figure 9 As shown, the original motion signal spectrum of all retained pixels contains several obvious non-target frequency components. Averaging all pixels alone makes it difficult to determine which pixels primarily reflect the vibration of the target object. The motion signals of the core vibration pixel, the final reliable vibration pixel, and the reliable weak response pixel all retain a significant peak around 14.560Hz, indicating that the selected reliable weak response pixel has the same vibration frequency as the core vibration pixel. After further suppressing signal components that can be explained by the camera shake reference base, the target frequency peak of the final reliable vibration pixel is still retained, while the relative amplitudes of the shake components and other non-target frequency components decrease.

[0167] The above results show that, under the condition of uneven distribution of surface texture strength, this application can not only identify the vibration frequency of the target object, but also filter out reliable pixels that still reflect the vibration of the target object from weak response pixels, thereby expanding the range of reliable vibration pixels and reducing the influence of camera shake on the final motion signal.

[0168] The results of the above-mentioned lamp holder experiment show that, under the condition that the vibration frequency of the target object (7.998Hz) and the camera shake frequency (8.613Hz) are close, the vibration-shake peak ratio obtained by the method of this application reaches 8.63, which is significantly better than existing processing methods such as directly averaging all effective pixels (1.28), subtracting the background reference signal (2.05), and notch filtering on the narrow frequency band where the shake frequency is located (3.20). Moreover, the reliable vibration pixel set can adaptively shrink as the camera shake increases, and the core vibration pixels and reliable weak response pixels together constitute the final support. The humidifier experiment further verified that the method can accurately identify the target vibration frequency of 14.560Hz without manual marking under the condition of uneven distribution of surface texture intensity. It can also recover reliable pixels that still reflect the target vibration from the weak texture area, indicating that the present application can effectively distinguish between real target vibration and camera shake interference. In particular, it can maintain high reliability measurement results when the frequencies of the two are close. At the same time, it achieves a reasonable balance between source credibility and spatial coverage through the weak reliability response recovery mechanism, which significantly improves the engineering practicality of video vibration measurement in complex field environments.

[0169] Example 2 Based on the same inventive concept as Embodiment 1, this embodiment provides a structural vibration measurement system based on video images. Since the principle of solving the problem in this system is similar to that of the structural vibration measurement method based on video images described in Embodiment 1, the implementation of this system can refer to the implementation of the structural vibration measurement method based on video images.

[0170] like Figure 10 As shown, this embodiment provides a structural vibration measurement system based on video images, including: The first determining module 10 is used to determine the measurable pixels in the vibration video.

[0171] The first construction module 20 is used to determine the target area where the target object is located and the reference area where the background is located from the vibration video, and then construct the target motion signal matrix and the background motion signal matrix respectively based on the motion signals of the measurable pixels in the target area and the reference area.

[0172] The second construction module 30 is used to decompose the background motion signal matrix and extract the time component common to multiple measurable pixels in the reference area to construct a camera shake reference base.

[0173] The third construction module 40 is used to determine the vibration frequency of the target object by using measurable pixels in the target area and camera shake reference base, so as to construct the vibration reference base of the target object.

[0174] The second determining module 50 is used to determine the proportion of energy of the motion signal of each measurable pixel in the target area interpreted by each reference base using the vibration reference base and the camera jitter reference base, so as to determine the first set of vibration pixels.

[0175] The pixel merging module 60 is used to select reliable weak response pixels from the measurable pixels that have not been selected into the first vibration pixel set, and then merge all reliable weak response pixels with the first vibration pixel set to form a second vibration pixel set.

[0176] The weighted fusion module 70 is used to perform weighted fusion on the remaining motion signals of each pixel after suppressing camera shake, based on the reliability score of each pixel in the second vibration pixel set, to obtain the vibration signal of the target object.

[0177] For more detailed information on the working process of each of the above modules, please refer to the relevant content disclosed in Example 1, which will not be repeated here.

[0178] Example 3 This embodiment provides a computer device, including a processor and a memory; wherein, when the processor executes the computer program stored in the memory, it implements the steps of the structural vibration measurement method based on video images described in Embodiment 1.

[0179] For a more detailed explanation of the above method, please refer to the relevant content disclosed in Example 1, which will not be repeated here.

[0180] Example 4 This embodiment provides a computer-readable storage medium for storing a computer program; when the computer program is executed by a processor, it implements the steps of the structural vibration measurement method based on video images described in Embodiment 1.

[0181] For a more detailed explanation of the above method, please refer to the relevant content disclosed in Example 1, which will not be repeated here.

[0182] Example 5 This embodiment provides a computer program product, including computer-executable instructions or a computer program. When the computer-executable instructions or the computer program are executed by a processor, they implement the steps of the structural vibration measurement method based on video images described in Embodiment 1.

[0183] For a more detailed explanation of the above method, please refer to the relevant content disclosed in Example 1, which will not be repeated here.

[0184] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the systems, devices, storage media, and computer program products disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the descriptions are relatively simple, and relevant parts can be referred to the method section.

[0185] Those skilled in the art will clearly understand that the techniques in the embodiments of this application can be implemented using software plus necessary general-purpose hardware platforms. Based on this understanding, the technical solutions in the embodiments of this application, or the parts that contribute to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments of this application or some parts of the embodiments.

[0186] In some embodiments, computer-executable instructions may take the form of programs, software, software modules, scripts, or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as stand-alone programs or as modules, components, subroutines, or other units suitable for use in a computing environment.

[0187] As an example, computer-executable instructions may, but do not necessarily, correspond to files in a file system. They may be stored as part of a file that holds other programs or data, for example, in one or more scripts in a Hyper Text Markup Language (HTML) document, in a single file dedicated to the program in question, or in multiple co-located files (e.g., files that store one or more modules, subroutines, or code sections).

[0188] As an example, computer-executable instructions can be deployed to execute on a single electronic device, or on multiple electronic devices located at one location, or on multiple electronic devices distributed across multiple locations and interconnected via a communication network.

[0189] The present application has been described in detail above with reference to specific embodiments and exemplary examples; however, these descriptions should not be construed as limiting the present application. Those skilled in the art will understand that various equivalent substitutions, modifications, or improvements can be made to the technical solutions and implementation methods of the present application without departing from the spirit and scope of the present application, and all such modifications and improvements fall within the scope of the present application. The scope of protection of the present application is determined by the appended claims.

Claims

1. A method for measuring structural vibration based on video images, characterized in that, include: Identify measurable pixels in the vibration video; After determining the target area where the target object is located and the reference area where the background is located from the vibration video, the target motion signal matrix and the background motion signal matrix are constructed respectively based on the motion signals of the measurable pixels in the target area and the reference area. The background motion signal matrix is ​​decomposed, and the common temporal components of multiple measurable pixels in the reference area are extracted to construct a camera shake reference base. The vibration frequency of the target object is determined by using measurable pixels within the target area and a camera shake reference base, so as to construct a vibration reference base for the target object; Using a vibration reference base and a camera jitter reference base, the proportion of energy of the motion signal of each measurable pixel in the target area is determined by each reference base to determine the first set of vibration pixels; After selecting reliable weak response pixels from the measurable pixels that were not selected into the first vibration pixel set, all reliable weak response pixels are merged with the first vibration pixel set to form the second vibration pixel set. Based on the reliability score of each pixel in the second set of vibration pixels, the remaining motion signals of each pixel after suppressing camera shake are weighted and fused to obtain the vibration signal of the target object.

2. The structural vibration measurement method according to claim 1, characterized in that, Identify measurable pixels in the vibration video, including: Extracting the motion signal of each pixel over time from a vibration video including the target object; An imaging noise model is constructed based on the imaging conditions of the vibration video, and measurable pixels in the vibration video are selected based on the imaging noise model.

3. The structural vibration measurement method according to claim 2, characterized in that, After determining the target region where the target object is located and the reference region where the background is located from the vibration video, the target motion signal matrix and the background motion signal matrix are constructed respectively based on the motion signals of measurable pixels in the target region and the reference region, including: Based on the motion signals of each measurable pixel in the target area at each time, the motion amplitude of each measurable pixel at each time is arranged in chronological order to obtain the motion signal column vector of each measurable pixel. Arrange the column vectors of motion signals of all measurable pixels within the target area row by row to obtain the target motion signal matrix; Based on the motion signals of each measurable pixel in the reference area at each time, the motion amplitudes of each measurable pixel at each time are arranged in chronological order to obtain the motion signal column vector of each measurable pixel. The motion signal column vectors of all measurable pixels within the reference area are arranged row by row to obtain the background motion signal matrix.

4. The structural vibration measurement method according to claim 1, characterized in that, The background motion signal matrix is ​​decomposed, and the common temporal components of multiple measurable pixels within the reference region are extracted to construct a camera shake reference base, including: Singular value decomposition is performed on the background motion signal matrix to obtain multiple common time curves and the singular values ​​corresponding to the multiple common time curves; The common time curves corresponding to the first preset number of singular values ​​are selected as the column vectors of the camera shake reference basis, in descending order of singular values.

5. The structural vibration measurement method according to claim 2, characterized in that, The vibration frequency of the target object is determined using measurable pixels within the target area and a camera shake reference base, in order to construct a vibration reference base for the target object, including: Based on the structural position of the target object, a preset target sub-region is determined within the target area, and measurable pixels with motion signal amplitude greater than a preset threshold are selected from the preset target sub-region as seed pixels of the target object. The motion signals of various sub-pixels are projected onto the camera shake reference base to obtain the fitting coefficients of the motion signals of various sub-pixels on the camera shake reference base. The camera shake component in the motion signal of each sub-pixel is reconstructed based on the fitting coefficients. The camera shake component is then removed from the original motion signal to obtain the remaining motion signal of each sub-pixel. Perform Fourier transform on the residual motion signals of various sub-pixels to obtain the spectral amplitude of each sub-pixel at each frequency. The median of the spectral amplitude of all seed pixels at each frequency is used to obtain the aggregated spectral amplitude of various sub-pixels. Within the preset frequency band of interest, the frequency corresponding to the maximum amplitude of the aggregated spectrum is determined as the vibration frequency of the target object. Using the vibration frequency as the center, the residual motion signals of various sub-pixels are subjected to narrowband filtering, and the median of the narrowband filtering results of various sub-pixels is taken at each time step to obtain the vibration reference signal of the target object. Perform a Hilbert transform on the vibration reference signal to obtain the orthogonal phase components; The vibration reference signal and its corresponding orthogonal phase components are used to form the initial vibration reference matrix of the target object. After removing components in the initial vibration reference matrix that can be interpreted by the camera jitter reference basis, orthogonal normalization is performed to obtain the vibration reference basis of the target object.

6. The structural vibration measurement method according to claim 2, characterized in that, Using a vibration reference base and a camera shake reference base, the proportion of energy interpreted by each reference base for the motion signal of each measurable pixel within the target area is determined to identify the first set of vibration pixels, including: Construct the first projection matrix corresponding to the vibration reference base and the second projection matrix corresponding to the camera jitter reference base, respectively; The motion signals of each measurable pixel in the target area are projected onto the first projection matrix and the second projection matrix respectively to obtain the energy of the vibration projection component of each measurable pixel and the energy of the camera shake projection component. The ratio of the energy of the vibration projection component of each measurable pixel to the total energy of the motion signal, and the ratio of the energy of the camera shake projection component to the total energy of the motion signal are obtained respectively, and used as the first energy ratio and the second energy ratio respectively. After combining the vibration reference base and the camera jitter reference base, orthogonal normalization is performed to obtain the joint reference base; Construct a joint projection matrix corresponding to the joint reference base to determine the proportion of residual energy in the motion signal of each measurable pixel that cannot be explained by the joint projection matrix; The competition score for vibration and jitter of each measurable pixel is calculated based on the difference between the first energy ratio and the second energy ratio and the residual energy ratio. The selection threshold is determined based on the competition score. Measurable pixels with competition scores greater than or equal to the selection threshold are selected to obtain the first set of vibration pixels.

7. The structural vibration measurement method according to claim 6, characterized in that, After selecting reliable weak response pixels from the measurable pixels that were not included in the first vibration pixel set, all reliable weak response pixels are merged with the first vibration pixel set to form a second vibration pixel set, including: Pixels within the target area that belong to the measurable pixel set but were not selected into the first vibration pixel set are considered as candidate pixels; Remove the components that can be interpreted by the camera shake reference base from the motion signal of each candidate pixel to obtain the remaining signal of each candidate pixel; A reliability score is calculated based on the remaining signal of each candidate pixel; Candidate pixels whose reliability scores meet the preset conditions are identified as reliable weak response pixels; All reliable weak response pixels are merged with the first set of vibration pixels to form the second set of vibration pixels.

8. The structural vibration measurement method according to claim 7, characterized in that, A reliability score is calculated based on the remaining signal of each candidate pixel, including: The proportion of energy in the remaining signal of the candidate pixel that can be explained by the vibration reference base is calculated as the first indicator; The remaining signal of the candidate pixel near the vibration frequency is calculated as the proportion of the remaining signal near the vibration frequency and the sum of the spectral energy near the camera shake frequency, which is used as a second indicator. Identify local neighbor pixels that are spatially adjacent to the candidate pixel and belong to the same structural part of the target object. Use the relative magnitude of the frequency band energy of the candidate pixel and the frequency band energy of other measurable pixels in the local neighbor pixels as a third indicator. The product of the first, second, and third indicators is used as the reliability score.

9. The structural vibration measurement method according to claim 8, characterized in that, Based on the reliability score of each pixel in the second vibration pixel set, the remaining motion signals of each pixel after suppressing camera shake are weighted and fused to obtain the vibration signal of the target object, including: Using the reliability score of each pixel in the second set of vibration pixels as the weight, the value of the remaining motion signal of each pixel after suppressing camera shake at the target time is weighted and averaged to obtain the vibration signal value at the target time. The vibration signal values ​​at each moment are arranged in chronological order to obtain the vibration signal of the target object.

10. A structural vibration measurement system based on video images, characterized in that, include: The first determining module is used to determine the measurable pixels in the vibration video; The first construction module is used to determine the target area where the target object is located and the reference area where the background is located from the vibration video, and then construct the target motion signal matrix and the background motion signal matrix respectively based on the motion signals of the measurable pixels in the target area and the reference area. The second construction module is used to decompose the background motion signal matrix and extract the time component common to multiple measurable pixels in the reference area to construct a camera shake reference base. The third construction module is used to determine the vibration frequency of the target object by using measurable pixels in the target area and camera shake reference base, so as to construct the vibration reference base of the target object; The second determining module is used to determine the proportion of energy of the motion signal of each measurable pixel in the target area interpreted by each reference base using the vibration reference base and the camera shake reference base, so as to determine the first set of vibration pixels. The pixel merging module is used to select reliable weak response pixels from the measurable pixels that have not been selected into the first vibration pixel set, and then merge all reliable weak response pixels with the first vibration pixel set to form a second vibration pixel set. The weighted fusion module is used to perform weighted fusion on the remaining motion signals of each pixel after suppressing camera shake, based on the reliability score of each pixel in the second set of vibration pixels, to obtain the vibration signal of the target object.

Images