An out-of-plane modal feature extraction method and system based on deep completion reconstruction

Abstract

The application relates to an out-of-plane modal feature extraction method and system based on deep completion reconstruction, and belongs to the technical field of out-of-plane modal feature extraction. The method comprises the following steps: capturing a group of continuously changing depth image sequences by using a depth camera; obtaining a group of complete image sequences with complete depth information by using depth completion processing; performing projection transformation to obtain a group of correction image sequences; obtaining a group of down-sampling image sequences which can retain more details by using sliding average filtering; completing plane domain signal reconstruction of image data by using fitting smoothing; completing time domain signal reconstruction of image data by using Euler amplification; and extracting overall modal vibration modes from the reconstructed data by using Hankel dynamic modal decomposition. The application realizes comprehensive and accurate out-of-plane response collection and imaging distortion correction of a measured structure. By introducing technologies such as sliding average, polynomial fitting and Euler amplification, high-quality signal reconstruction of depth data is realized, and complete and accurate structural modal vibration modes are constructed.

Description

Technical Field

[0001] This invention relates to a method and system for out-of-plane modal feature extraction based on depth completion reconstruction, belonging to the field of out-of-plane modal feature extraction technology. Background Technology

[0002] Structural vibration phenomena often contain rich physical mechanisms and dynamic information. Recording structural vibration response data using vibration measurement technology and extracting key modal features such as natural frequencies and mode shapes helps to gain a deeper understanding of vibration characteristics, providing important support for structural health monitoring and design. Traditional modal feature extraction relies on the deployment of contact sensors such as accelerometers, but the complex installation and acquisition process is time-consuming and costly. With the development of computer technology and the increasing prevalence of high-resolution cameras, computer vision-based modal feature extraction methods have emerged. These methods utilize cameras as acquisition devices, offering the advantage of non-contact measurement and effectively overcoming the shortcomings of traditional methods in terms of acquisition and cost.

[0003] Structural motion is essentially three-dimensional motion; therefore, determining the structural behavior both in-plane and out-of-plane is equally important for studying modal characteristics. Current research based on computer vision focuses on extracting precise information about the dynamic response of structures from video captured by cameras and applying it to modal characteristic analysis. One approach is to use high-speed cameras to acquire high-frame-rate grayscale videos and estimate the vibration response based on grayscale changes. For example, publication number CN117490823A, entitled "A Novel Vibration Extraction and Mode Shape Observation Method Based on Video Measurement," describes a method that extracts the time-series matrix from the edge regions of the structure under test in the video, uses a deviation extraction algorithm to obtain sub-pixel vibration signals, and then conducts structural vibration analysis and mode shape observation. Another approach is to use depth cameras to acquire depth videos containing structural displacement information and obtain modal frequency characteristics by processing the depth information. For example, publication number CN117522944A, entitled "An Adaptive Out-of-Plane Vibration Measurement System and Method Based on Depth Perception," describes a method that extracts the frequency of the out-of-plane vibration process of the measured structure by acquiring and processing depth image sequences. However, both technologies still have their limitations. The former is difficult to effectively capture out-of-plane vibration information of objects and is easily affected by changes in lighting and environmental noise. The latter, although it overcomes this deficiency, only uses some pixel information from the depth video for frequency measurement and does not conduct further analysis on the overall modal characteristics of the structural response.

[0004] Therefore, based on the second method, this invention introduces depth completion and signal reconstruction techniques to complete and correct all pixel information of the depth video, and uses modal decomposition techniques to achieve complete modal feature extraction. Summary of the Invention

[0005] To address the problem that current modal feature extraction methods based on depth video only utilize partial pixel information for frequency measurement and fail to conduct in-depth analysis of the complete modal features of structural response, this invention proposes an out-of-plane modal feature extraction method and system based on depth completion reconstruction.

[0006] An out-of-plane modal feature extraction method based on depth completion reconstruction includes the following steps:

[0007] Step 1: Use a depth camera to acquire a set of continuously changing depth image sequences by capturing the overall out-of-plane response process of the structure under test at a frame rate that satisfies the Nyquist sampling rate.

[0008] Step 2: Perform depth completion on the acquired depth image sequence, correct invalid and outlier pixels in the image sequence, and obtain a set of completed image sequences with complete depth information;

[0009] Step 3: Perform projection transformation on the completed image sequence according to the reference background to correct the imaging tilt and near-to-far-small problems caused by camera perspective distortion, and obtain a set of corrected image sequences.

[0010] Step 4: Select the measurement area, and use a slider with a custom dimension to perform a moving average filter within the measurement area of ​​the corrected image sequence at a specific step size to obtain a set of downsampled image sequences that retain more details.

[0011] Step 5: Perform polynomial fitting on the planar domain depth signal of the downsampled image sequence to solve the problem of signal step transition caused by low camera resolution, and complete the reconstruction of the planar domain signal of the image data.

[0012] Step 6: Extract the temporal depth signal from the image data after reconstruction of the planar domain signal and perform Euler amplification on it to remove the higher-order modes from the noise components. At the same time, add an exponential decay factor to weaken the influence of noise after the disappearance of higher-order components, thus completing the temporal signal reconstruction of the image data.

[0013] Step 7: Extract dynamic mode vectors from the reconstructed data using Hankel dynamic mode decomposition, and then extract mode shape vectors from the dynamic mode vectors with a fixed step size, and splice them together to obtain the overall mode shape.

[0014] Preferably, in step 2, the depth image sequence is processed to obtain a completed image sequence with complete depth information, including the following steps:

[0015] Step 2.1: Using the line connecting the main imager of the depth camera to the auxiliary imager as the index direction, traverse the depth image column by column along this direction, find consecutive zero-value pixels, and fill them with the closest non-empty valid depth value above them.

[0016] Step 2.2: Set a depth threshold M, and invert the depth image after directional padding to leave a buffer space between the effective depth value and the zero value.

[0017] D inverted =MD input M-Max(D input )>0

[0018] Where Max() represents finding the maximum value of the input data, D input and D inverted These are the original image and the inverted image, respectively.

[0019] Step 2.3: Construct a 5-dimensional diamond-shaped filling kernel and perform a dilation operation on the inverted image to fill out some outlier pixels with the most probable neighborhood depth value. Then, use a 5-dimensional rectangular filling kernel to perform a closing operation on the dilated image to complete the hole filling operation.

[0020]

[0021]

[0022] in, K represents the morphological dilation operation, · represents the morphological closing operation, and K represents the morphological closing operation. diamond and K rect These are diamond-shaped fill kernels and rectangular fill kernels, respectively. filling This is the depth image after the small hole is filled.

[0023] Step 2.4: Perform median filtering on the depth image after hole filling to remove noise interference caused by dilation. Then, perform inversion operation on the median-filtered image again to restore the original depth encoding.

[0024] D output =M-med(D filling ,K rect )

[0025] Where med() represents the median filtering operation, D output This is the output image after depth completion.

[0026] Step 2.5: Traverse all image elements in the depth image sequence and repeat the above steps to complete the depth completion operation of the depth image sequence and obtain the completed image sequence.

[0027] Preferably, in step 3, the completed image sequence is processed to obtain an image sequence after viewpoint distortion correction, including the following steps:

[0028] Step 3.1: Extract the coordinates of the four control points of the reference background in the completed image and set the corresponding projection coordinates. Construct the homography calculation formula to obtain the projection matrix H, and apply it to the projection transformation of all pixel coordinates, i.e.

[0029]

[0030] o j =Hw j i j ∈D output ,w j ∈D orrect

[0031] Among them, (u j ,v j ) and (x j ,y j ) represent the control point coordinates and their corresponding projected coordinates, h ij D represents the degree of freedom coefficients of the projection matrix. correct For the corrected image after projection transformation, o j and w j These are the pixel coordinates before and after projection.

[0032] Step 3.2: Traverse all image elements in the completed image sequence and repeat the above projection transformation process to complete the viewpoint distortion correction operation on the completed image sequence and obtain the corrected image sequence.

[0033] Preferably, in step 4, the measurement area is selected, and the corrected image sequence is processed to obtain a downsampled image sequence that retains more details, including the following steps:

[0034] Step 4.1: Perform element-by-element judgment on the first frame of the corrected image, extract the pixel template whose depth value is within the response amplitude range of the measured structure, and determine the maximum inscribed rectangle of the template as the measurement area;

[0035] Step 4.2: Create a two-dimensional sliding window, perform moving average filtering in the horizontal and vertical directions of the measurement area, adjust the row and column dimensions of the rectangular window and the sliding step size to complete the custom downsampling processing of the depth data.

[0036]

[0037] Where I(x,y) and O(x,y) represent the depth values ​​at coordinates (x,y) before and after downsampling, respectively, m and n are the row and column dimensions of the sliding window, respectively, and s x and s y These represent the sliding step lengths in the horizontal and vertical directions, respectively.

[0038] Step 4.3: Within the measurement area, traverse all image elements in the corrected image sequence and repeat the above moving average filtering process to obtain a downsampled image sequence with a custom data dimension.

[0039] Preferably, in step 5, the downsampled image sequence is processed to complete the planar domain signal reconstruction of the image data, including the following steps:

[0040] Step 5.1: Divide the two-dimensional downsampled image into a one-dimensional planar domain signal by columns, and perform polynomial fitting column by column. By minimizing the error between the fitted function and the signal, the polynomial coefficients with the best smoothing effect are obtained, i.e.

[0041]

[0042]

[0043] Where E(w) is the error function, y(x,w) is the polynomial fitting function, N is the length of the planar domain signal, and t n Let w be the depth value of the planar domain signal at point n, M be the polynomial order, and w be the depth value of the signal at point n. j Let be the coefficients of the j-th order polynomial.

[0044] Step 5.2: Traverse all image elements in the downsampled image sequence, repeat the above dimensionality reduction and polynomial fitting process, and complete the planar domain signal reconstruction of the image data.

[0045] Preferably, in step 6, the image data after planar domain signal reconstruction is processed to complete the temporal domain signal reconstruction of the image data, including the following steps:

[0046] Step 6.1: Using pixels as units, extract the temporal depth signal from the image data reconstructed from the planar domain signal, and then apply the Fir bandpass filter bank to separate the signal components corresponding to each order of the main frequency;

[0047] Step 6.2: Calculate the second-order Taylor expansion remainder of each signal component, multiply it by the amplification factor and the exponential decay factor, and then superimpose each remainder with the original signal component to complete the Euler amplification of the time-domain depth signal, i.e.

[0048]

[0049] in, This represents the time-domain depth signal after Euler amplification. Let δ(t) be the signal component corresponding to each order of the main frequency, I be the order of the main frequency, and α and β be the amplification factor and the exponential decay factor, respectively.

[0050] Step 6.3: Traverse the temporal depth signal corresponding to all pixels, repeat the Euler amplification process described above, and then recombine them according to the coordinates of each pixel to complete the temporal signal reconstruction of the image data.

[0051] Preferably, in step 7, the reconstructed image data is processed to extract the mode shape vectors and concatenate them to obtain the overall mode shape, including the following steps:

[0052] Step 7.1: Slice the image data column by column, and construct a Hankel matrix using the column vector pixels as measurement points. Shift Hankel matrix and the scaling factor α i This leads to the composite block Hankel matrices X and Y, i.e.

[0053]

[0054]

[0055] Where, d i (t) represents the depth value at measurement point i at time t, where m and n are the row and column dimensions of the Hankel matrix, respectively.

[0056] Step 7.2: Perform singular value decomposition on the composite block Hankel matrix X, and then construct the operator matrix. Then, eigenvalue analysis is performed on the operator matrix to extract eigenvalues ​​and eigenvectors, and then the dynamic mode vector is obtained, i.e.

[0057]

[0058] in,() * and() -1 W and S respectively represent finding the conjugate matrix and the inverse matrix. These are the left singular matrix, the singular value matrix, and the right singular matrix, respectively, (λ j ,ω j ) represents the eigenvalues ​​and eigenvectors.

[0059] Step 7.3: Traverse all slice data, repeat the Hankel dynamic mode decomposition process described above, and then extract the mode shape vectors from all dynamic mode vectors with a fixed step size, and splice them together to obtain the overall mode shape.

[0060] An out-of-plane modal feature extraction system based on depth completion reconstruction.

[0061] It includes: an image acquisition module, a depth completion module, a distortion correction module, a downsampling module, a fitting and smoothing module, an Euler amplification module, and a modality extraction module.

[0062] Image acquisition module: The depth camera is used to acquire the overall out-of-plane response process of the structure under test at a frame rate that meets the Nyquist sampling rate, and a set of continuously changing depth image sequences is obtained.

[0063] Depth Completion Module: Performs depth completion on the acquired depth image sequence, corrects invalid and outlier pixels in the image sequence, and obtains a set of completed image sequences with complete depth information;

[0064] Distortion correction module: Performs projection transformation on the completed image sequence according to the reference background to correct the imaging tilt and near-to-far-size problem caused by camera perspective distortion, and obtains a set of corrected image sequences;

[0065] Downsampling module: Select the measurement area, and use a slider with a custom dimension to perform a moving average filter within the measurement area of ​​the corrected image sequence at a specific step size to obtain a set of downsampled image sequences that retain more details;

[0066] Fitting and smoothing module: Performs polynomial fitting on the planar domain depth signal of the downsampled image sequence to solve the problem of signal step transition caused by low camera resolution and completes the reconstruction of the planar domain signal of the image data.

[0067] Euler Amplification Module: Extracts the temporal depth signal from the image data after reconstruction of the planar domain signal and amplifies it with Euler, so that the higher-order modes are separated from the noise components. At the same time, an exponential decay factor is added to weaken the influence of noise after the disappearance of higher-order components, thus completing the temporal signal reconstruction of the image data.

[0068] Modal extraction module: The module uses Hankel dynamic modal decomposition to extract dynamic modal vectors from the reconstructed data. Then, it extracts mode shape vectors from the dynamic modal vectors with a fixed step size and splices them together to obtain the overall mode shape.

[0069] Beneficial effects:

[0070] 1. Compared with the method of using a high-speed camera to acquire high frame rate grayscale video and estimating vibration response based on grayscale changes, this invention uses a depth camera as an acquisition device, combined with depth completion and projection transformation theory, to achieve comprehensive and accurate out-of-plane response acquisition and imaging distortion correction of the measured structure.

[0071] 2. Compared with the general method of obtaining frequency characteristics by processing depth information, this invention introduces steps such as moving average, polynomial fitting, and Euler amplification to achieve high-quality signal reconstruction of depth data and construct complete structural mode shapes with the help of Hankel dynamic mode decomposition. Attached Figure Description

[0072] Figure 1This is a schematic diagram of the out-of-plane modal feature extraction method based on depth completion reconstruction of the present invention;

[0073] Figure 2 This is a diagram showing the result of the depth image sequence of the present invention after processing by each module;

[0074] Figure 3 The time-domain result and spectrum of the depth signal of this invention after moving average, fitting smoothing and Euler amplification;

[0075] Figure 4 This is a set of third-order mode shape vectors obtained by Hankel dynamic mode decomposition according to the present invention;

[0076] Figure 5 The third complete mode shape of the structure under test in this invention. Detailed Implementation

[0077] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0078] In the description of this invention, it should be understood that the terms "upper", "middle", "outer", "inner", etc., which indicate orientation or positional relationship, are only for the convenience of describing this invention and simplifying the description, and do not indicate or imply that the components or elements referred to must have a specific orientation, or be constructed and operated in a specific orientation, and therefore should not be construed as limiting this invention.

[0079] Example 1: Combination Figure 1-5 This embodiment describes an out-of-plane modal feature extraction method based on depth completion reconstruction.

[0080] Includes the following steps:

[0081] Step 1: Use a depth camera to acquire the overall out-of-plane response process of the structure under test at a frame rate that meets the Nyquist sampling rate, and obtain a set of continuously changing depth image sequences. Compared with the method of using a high-speed camera to carry out modal feature analysis, the method of the present invention uses a depth camera as an acquisition device, which can effectively capture the out-of-plane depth information of the object without contact, thereby performing overall modal feature analysis.

[0082] Step 2: Perform depth completion on the acquired depth image sequence, correct invalid and outlier pixels in the image sequence, and obtain a set of completed image sequences with complete depth information;

[0083] Step 3: Perform projection transformation on the completed image sequence according to the reference background to correct the imaging tilt and near-to-far-small problems caused by camera perspective distortion, and obtain a set of corrected image sequences.

[0084] Step 4: Select the measurement area, and use a slider with a custom dimension to perform a moving average filter within the measurement area of ​​the corrected image sequence at a specific step size to obtain a set of downsampled image sequences that retain more details.

[0085] Step 5: Perform polynomial fitting on the planar domain depth signal of the downsampled image sequence to solve the problem of signal step transition caused by low camera resolution, and complete the reconstruction of the planar domain signal of the image data.

[0086] Step 6: Extract the temporal depth signal from the image data after reconstruction of the planar domain signal and perform Euler amplification on it to remove the higher-order modes from the noise components. At the same time, add an exponential decay factor to weaken the influence of noise after the disappearance of higher-order components, thus completing the temporal signal reconstruction of the image data.

[0087] Step 7: Extract dynamic mode vectors from the reconstructed data using Hankel dynamic mode decomposition, and then extract mode shape vectors from the dynamic mode vectors with a fixed step size, and splice them together to obtain the overall mode shape.

[0088] This invention utilizes depth completion and projection transformation theory to achieve comprehensive and accurate out-of-plane response acquisition and imaging distortion correction of the measured structure. By using moving average, polynomial fitting, Euler amplification and Hankel dynamic mode decomposition, it achieves high-quality signal reconstruction of depth data and constructs complete structural mode shapes.

[0089] Example 2: Combination Figure 1-5 This embodiment describes an out-of-plane modal feature extraction method based on depth completion reconstruction. Step 2 involves processing the depth image sequence to obtain a completed image sequence with complete depth information, including the following steps:

[0090] Step 2.1: Using the line connecting the main imager of the depth camera to the auxiliary imager as the index direction, traverse the depth image column by column along this direction, find consecutive zero-value pixels, and fill them with the closest non-empty valid depth value above them.

[0091] Step 2.2: Set a depth threshold M, and invert the depth image after directional padding to leave a buffer space between the effective depth value and the zero value.

[0092] D inverted =MD input M-Max(D input )>0

[0093] Where Max() represents finding the maximum value of the input data, D input and D inverted These are the original image and the inverted image, respectively.

[0094] Step 2.3: Construct a 5-dimensional diamond-shaped filling kernel and perform a dilation operation on the inverted image to fill out some outlier pixels with the most probable neighborhood depth value. Then, use a 5-dimensional rectangular filling kernel to perform a closing operation on the dilated image to complete the hole filling operation.

[0095]

[0096]

[0097] in, K represents the morphological dilation operation, · represents the morphological closing operation, and K represents the morphological closing operation. diamond and K rect These are diamond-shaped fill kernels and rectangular fill kernels, respectively. filling This is the depth image after the small hole is filled.

[0098] Step 2.4: Perform median filtering on the depth image after hole filling to remove noise interference caused by dilation. Then, perform inversion operation on the median-filtered image again to restore the original depth encoding.

[0099] D output =M-med(D filling ,K rect )

[0100] Where med() represents the median filtering operation, D output This is the output image after depth completion.

[0101] Step 2.5: Traverse all image elements in the depth image sequence and repeat the above steps (steps 2.1 to 2.4) for the total number of frames in the depth image sequence to complete the depth completion operation of the depth image sequence and obtain the completed image sequence.

[0102] Example 3: Combination Figure 1-5 This embodiment describes an out-of-plane modal feature extraction method based on depth completion reconstruction. Step 3 involves processing the completed image sequence to obtain a viewpoint distortion-corrected image sequence, including the following steps:

[0103] Step 3.1: Extract the coordinates of the four control points of the reference background in the completed image and set the corresponding projection coordinates. Construct the homography calculation formula to obtain the projection matrix H, and apply it to the projection transformation of all pixel coordinates, i.e.

[0104]

[0105] o j =Hw j i j ∈D output ,w j ∈D orrect

[0106] Among them, (u j ,v j ) and (x j ,y j ) represent the control point coordinates and their corresponding projected coordinates, h ij D represents the degree of freedom coefficients of the projection matrix. correct For the corrected image after projection transformation, o j and w j These are the pixel coordinates before and after projection.

[0107] Step 3.2: Traverse all image elements in the completed image sequence and repeat the above projection transformation process (Step 3.1). The number of repetitions is equal to the total number of frames in the completed image sequence. This completes the viewpoint distortion correction operation on the completed image sequence, resulting in the corrected image sequence.

[0108] Example 4: Combination Figure 1-5 This embodiment describes an out-of-plane modal feature extraction method based on depth completion reconstruction. In step 4, a measurement region is selected, and the corrected image sequence is processed to obtain a downsampled image sequence that retains more details. This includes the following steps:

[0109] Step 4.1: Perform element-by-element judgment on the first frame of the corrected image, extract the pixel template whose depth value is within the response amplitude range of the measured structure, and determine the maximum inscribed rectangle of the template as the measurement area;

[0110] Step 4.2: Create a two-dimensional sliding window, perform moving average filtering in the horizontal and vertical directions of the measurement area, adjust the row and column dimensions of the rectangular window and the sliding step size to complete the custom downsampling processing of the depth data.

[0111]

[0112] Where I(x,y) and O(x,y) represent the depth values ​​at coordinates (x,y) before and after downsampling, respectively, m and n are the row and column dimensions of the sliding window, respectively, and s x and s y These represent the sliding step lengths in the horizontal and vertical directions, respectively.

[0113] Step 4.3: Within the measurement area, traverse all image elements in the corrected image sequence and repeat the above moving average filtering process (step 4.2). The number of repetitions is equal to the total number of frames in the corrected image sequence, resulting in a downsampled image sequence with a custom data dimension.

[0114] Example 5: Combination Figure 1-5 This embodiment describes an out-of-plane modal feature extraction method based on depth completion reconstruction. Step 5 involves processing the downsampled image sequence to reconstruct the planar domain signal of the image data, including the following steps:

[0115] Step 5.1: Divide the two-dimensional downsampled image into a one-dimensional planar domain signal by columns, and perform polynomial fitting column by column. By minimizing the error between the fitted function and the signal, the polynomial coefficients with the best smoothing effect are obtained, i.e.

[0116]

[0117]

[0118] Where E(w) is the error function, y(x,w) is the polynomial fitting function, N is the length of the planar domain signal, and t n Let w be the depth value of the planar domain signal at point n, M be the polynomial order, and w be the depth value of the signal at point n. j Let be the coefficients of the j-th order polynomial.

[0119] Step 5.2: Traverse all image elements in the downsampled image sequence and repeat the dimensionality reduction and polynomial fitting process described above (Step 5.1). The number of repetitions is equal to the total number of frames in the downsampled image sequence, thus completing the planar domain signal reconstruction of the image data.

[0120] Example 6: Combination Figure 1-5 This embodiment describes an out-of-plane modal feature extraction method based on depth completion reconstruction. Step 6 involves processing the image data after planar domain signal reconstruction to complete the temporal domain signal reconstruction of the image data. This includes the following steps:

[0121] Step 6.1: Using pixels as units, extract the temporal depth signal from the image data reconstructed from the planar domain signal, and then apply the Fir bandpass filter bank to separate the signal components corresponding to each order of the main frequency;

[0122] Step 6.2: Calculate the second-order Taylor expansion remainder of each signal component, multiply it by the amplification factor and the exponential decay factor, and then superimpose each remainder with the original signal component to complete the Euler amplification of the time-domain depth signal, i.e.

[0123]

[0124] in, This represents the time-domain depth signal after Euler amplification. Let δ(t) be the signal component corresponding to each order of the main frequency, I be the order of the main frequency, and α and β be the amplification factor and the exponential decay factor, respectively.

[0125] Step 6.3: Traverse the temporal depth signal corresponding to all pixels and repeat the Euler amplification process (steps 6.1 to 6.2) as described above. The number of repetitions is the number of pixels contained in a single downsampled image. Then, recombine the pixels according to their coordinates to complete the temporal signal reconstruction of the image data.

[0126] Example 7: Combination Figure 1-5 This embodiment describes an out-of-plane modal feature extraction method based on depth completion reconstruction. Step 7 involves processing the reconstructed image data to extract modal shape vectors and concatenating them to obtain the overall modal shape. This includes the following steps:

[0127] Step 7.1: Slice the image data column by column, and construct a Hankel matrix using the column vector pixels as measurement points. Shift Hankel matrix and the scaling factor α i This leads to the composite block Hankel matrices X and Y, i.e.

[0128]

[0129]

[0130] Where, d i (t) represents the depth value at measurement point i at time t, where m and n are the row and column dimensions of the Hankel matrix, respectively.

[0131] Step 7.2: Perform singular value decomposition on the composite block Hankel matrix X, and then construct the operator matrix. Then, intrinsic analysis is performed to extract eigenvalues ​​and eigenvectors, and then the dynamic mode vector is obtained, i.e.

[0132]

[0133] in,() * and() -1 W and S respectively represent finding the conjugate matrix and the inverse matrix. These are the left singular matrix, the singular value matrix, and the right singular matrix, respectively, (λ j ,ω j ) represents the eigenvalues ​​and eigenvectors.

[0134] Step 7.3: Traverse all slice data and repeat the Hankel dynamic mode decomposition process (steps 7.1 to 7.2) as described above. The number of repetitions is equal to the total number of columns in the data cube. Then, extract the mode shape vectors from all dynamic mode vectors with a fixed step size and splice them together to obtain the overall mode shape.

[0135] To better describe the method described in this application, the following embodiments are used to illustrate the complete process of an out-of-plane modal feature extraction method based on depth completion reconstruction in practical applications, including the following steps:

[0136] Step 1: Prepare a three-layer structure, fix the bottom and apply external excitation to trigger the response;

[0137] Step 1.1: Use camera control software to pre-calibrate the left and right cameras of the depth camera. During the preparation stage, minimize the impact of imaging distortion caused by hardware problems. Then set the acquisition resolution to 848*480 pixels, the frame rate to 90Hz, and the exposure mode to the initial frame mode.

[0138] Step 1.2: Fix the depth camera at the field of view facing the structure under test, then adjust the camera angle so that the line connecting the main imager and the auxiliary imager is as vertical as possible, and acquire a depth image sequence of 2373 frames. The initial depth image is shown below. Figure 2 As shown in (a), the corresponding depth signal and spectrum are as follows: Figure 3 As shown in (a).

[0139] Step 2:

[0140] Step 2.1: Traverse the depth image column by column along the line connecting the main imager and the auxiliary imager, find consecutive zero-value pixels, and fill them with the closest non-empty effective depth value above them;

[0141] Step 2.2: Set the depth threshold M to 6000mm, slightly larger than the maximum effective depth value (5600mm) of all pixels in the depth image sequence. Then, perform an inversion operation on the depth image after directional padding.

[0142] D inverted =MD input M-Max(D input )>0

[0143] Where Max() represents finding the maximum value of the input data, D input and D inverted These are the original image and the inverted image, respectively.

[0144] Step 2.3: Construct a diamond-shaped filling kernel with dimension 5, perform a dilation operation on the inverted image, and then use a rectangular filling kernel with dimension 5 to perform a closing operation on the dilated image to complete the hole filling operation.

[0145]

[0146]

[0147] in, K represents the morphological dilation operation, · represents the morphological closing operation, and K represents the morphological closing operation. diamond and K rect These are diamond-shaped fill kernels and rectangular fill kernels, respectively. filling This is the depth image after the small hole is filled.

[0148] Step 2.4: Perform median filtering on the depth image after hole filling, using a 5-dimensional identity matrix as the filter kernel. Then, invert the median-filtered image again to restore the original depth encoding.

[0149] D output =M-med(D filling ,K rect )

[0150] Where med() represents the median filtering operation, D output This is the output image after depth completion.

[0151] Step 2.5: Traverse all image elements in the depth image sequence and repeat the above steps to complete the depth completion operation on the depth image sequence, obtaining a completed image sequence. The initial completed image is as follows: Figure 2 As shown in (b).

[0152] Step 3:

[0153] Step 3.1: Extract the coordinates of the four control points of the reference background in the initial completed image and set the corresponding projection coordinates. Construct the homography calculation formula to obtain the projection matrix H, and apply it to the projection transformation of all pixel coordinates, i.e.

[0154]

[0155] o j =Hw j i j ∈D output ,w j ∈D orrect

[0156] Among them, (u j ,v j ) and (x j,y j ) represent the control point coordinates and their corresponding projected coordinates, h ij D represents the degree of freedom coefficients of the projection matrix. correct For the corrected image after projection transformation, o j and w j These are the pixel coordinates before and after projection.

[0157] Step 3.2: Traverse all image elements in the completed image sequence, repeat the above projection transformation process, and complete the viewpoint distortion correction operation on the completed image sequence to obtain the corrected image sequence. The initial corrected image is as follows: Figure 2 As shown in (c).

[0158] Step 4:

[0159] Step 4.1: Perform element-by-element judgment on the initial corrected image, extract pixel templates whose depth values ​​are within the response amplitude range (400mm, 700mm) of the measured structure, and determine the maximum inscribed rectangle of this template as the measurement area. The size of this area is 77*17. Figure 2 As shown in (d);

[0160] Step 4.2: Create a two-dimensional sliding window, perform moving average filtering in the horizontal and vertical directions of the measurement area, adjust the row and column dimensions of the rectangular window to 20 and 2 respectively, and set the sliding step size in the horizontal and vertical directions to 1 and 10 respectively, completing the custom downsampling processing of the depth data.

[0161]

[0162] Where I(x,y) and O(x,y) represent the depth values ​​at coordinates (x,y) before and after downsampling, respectively, m and n are the row and column dimensions of the sliding window, respectively, and s x and s y These represent the sliding step lengths in the horizontal and vertical directions, respectively.

[0163] Step 4.3: Within the measurement area, traverse all image elements in the corrected image sequence and repeat the above moving average filtering process to obtain a downsampled image sequence with a custom data dimension. The initial downsampled image is shown below. Figure 2 As shown in (e), the corresponding depth signal and spectrum are as follows: Figure 3 As shown in (b).

[0164] Step 5:

[0165] As in step 5.1: Divide the two-dimensional downsampled image into a one-dimensional planar domain signal by columns, and perform polynomial fitting column by column. By minimizing the error between the fitted function and the signal, the polynomial coefficients with the best smoothing effect are obtained, i.e.

[0166]

[0167]

[0168] Where E(w) is the error function, y(x,w) is the polynomial fitting function, N is the length of the planar domain signal, and t n Let w be the depth value of the planar domain signal at point n, M be the polynomial order, and w be the depth value of the signal at point n. j Let be the coefficients of the j-th order polynomial.

[0169] Step 5.2: Traverse all image elements in the downsampled image sequence, repeating the above dimensionality reduction and polynomial fitting process to complete the planar domain signal reconstruction of the image data. The initial frame is as follows: Figure 2 As shown in (f), the corresponding depth signal and spectrum are as follows: Figure 3 As shown in (c).

[0170] Step 6:

[0171] Step 6.1: Using pixels as units, extract the temporal depth signal from the image data reconstructed from the planar domain signal, and then apply a Fir bandpass filter bank with a main frequency of [2.25Hz, 9.30Hz, 20.67Hz] to separate each signal component;

[0172] Step 6.2: Calculate the second-order Taylor expansion remainder of each signal component, multiply it by the amplification factor [1,3,30] and the exponential decay factor [0.001,0.0015,0.005], and then superimpose each remainder with the original signal component to complete the Euler amplification of the time-domain depth signal, i.e.

[0173]

[0174] in, This represents the time-domain depth signal after Euler amplification. Let δ(t) be the signal component corresponding to each order of the main frequency, I be the order of the main frequency, and α and β be the amplification factor and the exponential decay factor, respectively.

[0175] Step 6.3: Traverse the temporal depth signals corresponding to all pixels, repeat the Euler amplification process described above, and then recombine them according to the pixel coordinates to complete the temporal signal reconstruction of the image data. The initial frame is as follows: Figure 2 As shown in (g), the corresponding depth signal and spectrum are as follows: Figure 3 As shown in (d).

[0176] Step 7:

[0177] Step 7.1: Slice the image data column by column, and construct a Hankel matrix using the column vector pixels as measurement points. Shift Hankel matrix and the scaling factor α i This leads to the composite block Hankel matrices X and Y, i.e.

[0178]

[0179]

[0180] Where, d i (t) represents the depth value at measurement point i at time t, and m and n are the row and column dimensions of the Hankel matrix, which are 200 and 1800, respectively.

[0181] Step 7.2: Perform singular value decomposition on the composite block Hankel matrix X, and then construct the operator matrix. Then, eigenvalue analysis is performed on the operator matrix to extract eigenvalues ​​and eigenvectors, and then the dynamic mode vector is obtained, i.e.

[0182]

[0183] in,() * and() -1 W and S respectively represent finding the conjugate matrix and the inverse matrix. These are the left singular matrix, the singular value matrix, and the right singular matrix, respectively, (λ j ,ω j ) represents the eigenvalues ​​and eigenvectors.

[0184] Step 7.3: Traverse all slice data, repeat the Hankel dynamic mode decomposition process described above, and then extract the mode shape vectors from all dynamic mode vectors with a fixed step size, such as... Figure 4 As shown. Finally, concatenating all the mode shape vectors yields the overall mode shape, as shown. Figure 5 As shown.

[0185] This invention utilizes depth completion and projection transformation theory to achieve comprehensive and accurate out-of-plane response acquisition and imaging distortion correction of the measured structure. By using moving average, polynomial fitting, Euler amplification and Hankel dynamic mode decomposition, it achieves high-quality signal reconstruction of depth data and constructs complete structural mode shapes.

[0186] Example 2: Combined with Appendix Figure 1This embodiment describes an out-of-plane modal feature extraction system based on depth completion reconstruction, comprising an image acquisition module, a depth completion module, a distortion correction module, a downsampling module, a fitting and smoothing module, an Euler amplification module, and a modal extraction module. The image acquisition module, a hardware component of the system, consists of a depth camera and a computer connected via a USB 3.2 cable. The camera control software on the computer sends acquisition commands to the depth camera, records and saves the captured raw depth image sequence, and then imports it into MATLAB software for processing. The depth completion module, distortion correction module, downsampling module, fitting and smoothing module, Euler amplification module, and modal extraction module are sequentially called as different functional modules within the MATLAB software. This invention utilizes a depth camera as the acquisition device, effectively capturing out-of-plane vibration phenomena of the measured structure. It is less susceptible to the influence of complex backgrounds, lighting changes, noise, etc. Simultaneously, by introducing depth completion and projection transformation theories, complete and corrected pixel information, thereby performing overall modal feature extraction.

[0187] Image acquisition module: The depth camera is used to acquire the overall out-of-plane response process of the structure under test at a frame rate that meets the Nyquist sampling rate, and a set of continuously changing depth image sequences is obtained.

[0188] Depth Completion Module: Performs depth completion on the acquired depth image sequence, corrects invalid and outlier pixels in the image sequence, and obtains a set of completed image sequences with complete depth information;

[0189] Distortion correction module: Performs projection transformation on the completed image sequence according to the reference background to correct the imaging tilt and near-to-far-size problem caused by camera perspective distortion, and obtains a set of corrected image sequences;

[0190] Downsampling module: Select the measurement area, and use a slider with a custom dimension to perform a moving average filter within the measurement area of ​​the corrected image sequence at a specific step size to obtain a set of downsampled image sequences that retain more details;

[0191] Fitting and smoothing module: Performs polynomial fitting on the planar domain depth signal of the downsampled image sequence to solve the problem of signal step transition caused by low camera resolution and completes the reconstruction of the planar domain signal of the image data.

[0192] Euler Amplification Module: Extracts the temporal depth signal from the image data after reconstruction of the planar domain signal and amplifies it with Euler, so that the higher-order modes are separated from the noise components. At the same time, an exponential decay factor is added to weaken the influence of noise after the disappearance of higher-order components, thus completing the temporal signal reconstruction of the image data.

[0193] Modal extraction module: The module uses Hankel dynamic modal decomposition to extract dynamic modal vectors from the reconstructed data. Then, it extracts mode shape vectors from the dynamic modal vectors with a fixed step size and splices them together to obtain the overall mode shape.

[0194] In the description of this specification, references to terms such as "an embodiment," "example," and "specific example" refer to a specific feature, structure, material, or characteristic described in connection with that embodiment or example, which is included in at least one embodiment or example of the invention. In this specification, illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the described specific features, structures, materials, or characteristics may be combined in any suitable manner in one or more embodiments or examples.

[0195] The preferred embodiments of the present invention disclosed above are merely illustrative of the invention. These preferred embodiments do not exhaustively describe all details, nor do they limit the invention to the specific implementations described. Clearly, many modifications and variations can be made based on the content of this specification. This specification selects and specifically describes these embodiments to better explain the principles and practical applications of the invention, thereby enabling those skilled in the art to better understand and utilize the invention. The invention is limited only by the claims and their full scope and equivalents.

Claims

1. A method for out-of-plane modal feature extraction based on depth completion reconstruction, characterized in that, Includes the following steps: Step 1: Use a depth camera to acquire a set of continuously changing depth image sequences by capturing the overall out-of-plane response process of the structure under test at a frame rate that satisfies the Nyquist sampling rate. Step 2: Perform depth completion on the acquired depth image sequence, correct invalid and outlier pixels in the image sequence, and obtain a set of completed image sequences with complete depth information; Step 3: Perform projection transformation on the completed image sequence according to the reference background to correct the imaging tilt and near-to-far-small problems caused by camera perspective distortion, and obtain a set of corrected image sequences; Step 4: Select the measurement area, and use a slider with a custom dimension to perform a moving average filter within the measurement area of ​​the corrected image sequence at a specific step size to obtain a set of downsampled image sequences that retain more details; Step 5: Perform polynomial fitting on the planar domain depth signal of the downsampled image sequence to solve the problem of signal step transition caused by low camera resolution, and complete the reconstruction of the planar domain signal of the image data. Step 6: Extract the temporal depth signal from the image data after reconstruction of the planar domain signal and perform Euler amplification on it to remove the higher-order modes from the noise components. At the same time, add an exponential decay factor to weaken the influence of noise after the disappearance of higher-order components, thus completing the temporal signal reconstruction of the image data. Step 7: Extract dynamic mode vectors from the reconstructed data using Hankel dynamic mode decomposition, and then extract mode shape vectors from the dynamic mode vectors with a fixed step size, and splice them together to obtain the overall mode shape.

2. The out-of-plane modal feature extraction method based on depth completion reconstruction according to claim 1, characterized in that, Step 2 involves processing the depth image sequence to obtain a completed image sequence with full depth information, including the following steps: Step 2.1: Using the line connecting the main imager of the depth camera to the auxiliary imager as the index direction, traverse the depth image column by column along this direction, find consecutive zero-value pixels, and fill them with the closest non-empty valid depth value above them. Step 2.2: Set a depth threshold M, and invert the depth image after directional padding to leave a buffer space between the effective depth value and the zero value. D inverted =M-D input ,M-Max(D input )>0 Where Max() represents finding the maximum value of the input data, D input and D inverted These are the original image and the inverted image, respectively. Step 2.3: Construct a 5-dimensional diamond-shaped filling kernel and perform a dilation operation on the inverted image to fill out some outlier pixels with the most probable neighborhood depth value. Then, use a 5-dimensional rectangular filling kernel to perform a closing operation on the dilated image to complete the hole filling operation. in, K represents the morphological dilation operation, · represents the morphological closing operation, and K represents the morphological closing operation. diamond and K rect These are diamond-shaped fill kernels and rectangular fill kernels, respectively. filling This is the depth image after the small hole has been filled; Step 2.4: Perform median filtering on the depth image after hole filling to remove noise interference caused by dilation. Then, perform inversion operation on the median-filtered image again to restore the original depth encoding. D output =M-med(D filling ,K rect ) Where med() represents the median filtering operation, D output This is the output image after depth completion; Step 2.5: Traverse all image elements in the depth image sequence and repeat the above steps to complete the depth completion operation of the depth image sequence and obtain the completed image sequence.

3. The out-of-plane modal feature extraction method based on depth completion reconstruction according to claim 1, characterized in that, Step 3 involves processing the completed image sequence to obtain a viewpoint distortion-corrected image sequence, including the following steps: Step 3.1: Extract the coordinates of the four control points of the reference background in the completed image and set the corresponding projection coordinates. Construct the homography calculation formula to obtain the projection matrix H, and apply it to the projection transformation of all pixel coordinates, i.e. o j =Hw j ,i j ∈D output ,w j ∈D orrect Among them, (u j ,v j ) and (x j ,y j ) represent the control point coordinates and their corresponding projected coordinates, h ij D represents the degree of freedom coefficients of the projection matrix. correct For the corrected image after projection transformation, o j and w j These are the pixel coordinates before and after projection; Step 3.2: Traverse all image elements in the completed image sequence and repeat the above projection transformation process to complete the viewpoint distortion correction operation on the completed image sequence and obtain the corrected image sequence.

4. The out-of-plane modal feature extraction method based on depth completion and signal reconstruction according to claim 1, characterized in that, In step 4, the measurement area is selected, and the corrected image sequence is processed to obtain a downsampled image sequence that retains more details. This includes the following steps: Step 4.1: Perform element-by-element judgment on the first frame of the corrected image, extract the pixel template whose depth value is within the response amplitude range of the measured structure, and determine the maximum inscribed rectangle of the template as the measurement area; Step 4.2: Create a two-dimensional sliding window, perform moving average filtering in the horizontal and vertical directions of the measurement area, adjust the row and column dimensions of the rectangular window and the sliding step size to complete the custom downsampling processing of the depth data. Where I(x,y) and O(x,y) represent the depth values ​​at coordinates (x,y) before and after downsampling, respectively, m and n are the row and column dimensions of the sliding window, respectively, and s x and s y These represent the sliding step lengths in the horizontal and vertical directions, respectively. Step 4.3: Within the measurement area, traverse all image elements in the corrected image sequence and repeat the above moving average filtering process to obtain a downsampled image sequence with a custom data dimension.

5. The out-of-plane modal feature extraction method based on depth completion reconstruction according to claim 1, characterized in that, Step 5 involves processing the downsampled image sequence to reconstruct the planar domain signal of the image data, including the following steps: Step 5.1: Divide the two-dimensional downsampled image into a one-dimensional planar domain signal by columns, and perform polynomial fitting column by column. By minimizing the error between the fitted function and the signal, the polynomial coefficients with the best smoothing effect are obtained, i.e. Where E(w) is the error function, y(x,w) is the polynomial fitting function, N is the length of the planar domain signal, and t n Let w be the depth value of the planar domain signal at point n, M be the polynomial order, and w be the depth value of the signal at point n. j Let be the coefficients of the j-th order polynomial; Step 5.2: Traverse all image elements in the downsampled image sequence, repeat the above dimensionality reduction and polynomial fitting process, and complete the planar domain signal reconstruction of the image data.

6. The out-of-plane modal feature extraction method based on depth completion reconstruction according to claim 1, characterized in that, In step 6, the image data after planar domain signal reconstruction is processed to complete the temporal domain signal reconstruction of the image data, including the following steps: Step 6.1: Using pixels as units, extract the temporal depth signal from the image data reconstructed from the planar domain signal, and then apply the Fir bandpass filter bank to separate the signal components corresponding to each order of the main frequency; Step 6.2: Calculate the second-order Taylor expansion remainder of each signal component, multiply it by the amplification factor and the exponential decay factor, and then superimpose each remainder with the original signal component to complete the Euler amplification of the time-domain depth signal, i.e. in, This represents the time-domain depth signal after Euler amplification. Let δ(t) be the signal component corresponding to each order of the main frequency, I be the order of the main frequency, and α and β be the amplification factor and the exponential decay factor, respectively. Step 6.3: Traverse the temporal depth signal corresponding to all pixels, repeat the Euler amplification process described above, and then recombine them according to the coordinates of each pixel to complete the temporal signal reconstruction of the image data.

7. The out-of-plane modal feature extraction method based on depth completion reconstruction according to claim 1, characterized in that, In step 7, the reconstructed image data is processed to extract the mode shape vectors and concatenate them to obtain the overall mode shape, including the following steps: Step 7.1: Slice the image data column by column, and construct a Hankel matrix using the column vector pixels as measurement points. Shift Hankel matrix and the scaling factor α i This leads to the composite block Hankel matrices X and Y, i.e. Where, d i (t) represents the depth value at measurement point i at time t, where m and n are the row and column dimensions of the Hankel matrix, respectively; Step 7.2: Perform singular value decomposition on the composite block Hankel matrix X, and then construct the operator matrix. Then, eigenvalue analysis is performed on the operator matrix to extract eigenvalues ​​and eigenvectors, and then the dynamic mode vector is obtained, i.e. in,() * and() -1 W and S represent finding the conjugate matrix and the inverse matrix, respectively. These are the left singular matrix, the singular value matrix, and the right singular matrix, respectively, (λ j ,ω j ) represents the eigenvalues ​​and eigenvectors; Step 7.3: Traverse all slice data, repeat the Hankel dynamic mode decomposition process described above, and then extract the mode shape vectors from all dynamic mode vectors with a fixed step size, and splice them together to obtain the overall mode shape.

8. A system for extracting out-of-plane modal features based on depth completion reconstruction, characterized in that, include: Image acquisition module, depth completion module, distortion correction module, downsampling module, fitting and smoothing module, Euler amplification module, and modality extraction module. The image acquisition module, depth completion module, distortion correction module, downsampling module, fitting smoothing module, Euler amplification module, and modality extraction module are described. Image acquisition module: The depth camera is used to acquire the overall out-of-plane response process of the structure under test at a frame rate that meets the Nyquist sampling rate, and a set of continuously changing depth image sequences is obtained. Depth Completion Module: Performs depth completion on the acquired depth image sequence, corrects invalid and outlier pixels in the image sequence, and obtains a set of completed image sequences with complete depth information; Distortion correction module: Performs projection transformation on the completed image sequence according to the reference background to correct the imaging tilt and near-to-far-size problem caused by camera perspective distortion, and obtains a set of corrected image sequences; Downsampling module: Select the measurement area, and use a slider with a custom dimension to perform a moving average filter within the measurement area of ​​the corrected image sequence at a specific step size to obtain a set of downsampled image sequences that retain more details; Fitting and smoothing module: Performs polynomial fitting on the planar domain depth signal of the downsampled image sequence to solve the problem of signal step transition caused by low camera resolution and completes the reconstruction of the planar domain signal of the image data. Euler Amplification Module: Extracts the temporal depth signal from the image data after reconstruction of the planar domain signal and amplifies it with Euler, so that the higher-order modes are separated from the noise components. At the same time, an exponential decay factor is added to weaken the influence of noise after the disappearance of higher-order components, thus completing the temporal signal reconstruction of the image data. Modal extraction module: The module uses Hankel dynamic modal decomposition to extract dynamic modal vectors from the reconstructed data. Then, it extracts mode shape vectors from the dynamic modal vectors with a fixed step size and splices them together to obtain the overall mode shape.

Images