Structural vibration displacement identification method, device and equipment and storage medium

Abstract

The invention relates to the technical field of bridge structures and vision measurement, and discloses a structure vibration displacement recognition method, device and equipment and a storage medium, and the method comprises the steps: obtaining a vibration video of a to-be-recognized region, and carrying out the preprocessing of the vibration video, and obtaining an image sequence and a displacement proportion; deblurring the image sequence by adopting a space-time coupling method to obtain a clear image sequence and an optimized optical flow matrix; performing intermediate frame interpolation based on the clear image sequence and the optimized optical flow matrix to obtain a target image sequence; and performing structure vibration displacement identification based on the target image sequence and the displacement proportion to obtain vibration displacement. According to the method, deblurring is carried out through a space-time coupling method, motion blurring is effectively eliminated, noise is suppressed, a clear image sequence is used for frame insertion, the frame density of the image sequence is increased, the problem of large inter-frame displacement caused by insufficient video frame rate is relieved, vibration recognition is carried out on the target image sequence in combination with the displacement proportion, and physical displacement of structural vibration is obtained. And the displacement identification precision under complex conditions is improved.

Description

Technical Field

[0001] This invention relates to the fields of bridge structure and visual measurement technology, specifically to a method, device, equipment, and storage medium for identifying structural vibration displacement. Background Technology

[0002] With the development of computer vision and image acquisition technologies, non-contact structural vibration displacement recognition methods based on computer vision have been widely applied in engineering fields. Compared with traditional contact-based structural vibration recognition methods, visual measurement technology has advantages such as low cost, flexible deployment, and the ability to perform simultaneous multi-point measurements. However, in practical applications, its recognition accuracy is easily affected by motion blur, noise, and inter-frame undersampling in the image. Therefore, it is particularly important to conduct research on structural vibration displacement recognition under complex conditions of motion blur, noise interference, and large displacements.

[0003] Existing structural vibration recognition methods primarily employ deep learning-based approaches and traditional deconvolution methods for image deblurring and noise reduction. Deep learning-based methods typically suffer from complex model parameters, high training costs, and strong dependence on large-scale, clearly labeled, and blurred image data. More importantly, their generalization ability is weak when dealing with different scenarios. Traditional deconvolution methods estimate a blur kernel and then perform deconvolution or inverse filtering operations on the blurred image based on this kernel to achieve image deblurring and noise reduction. However, most traditional methods assume a uniform blur kernel, making them unable to handle spatially varying motion blur. When facing the difficulty of large displacement estimation, existing techniques use image pyramiding, but the pyramid downsampling process can introduce image distortion, leading to the loss of subtle features and affecting recognition accuracy.

[0004] In summary, although existing structural vibration identification methods can identify vibrations under complex conditions such as motion ambiguity, noise interference, and large displacements, their displacement identification accuracy is low. Summary of the Invention

[0005] In view of this, the present invention provides a structural vibration displacement identification method, apparatus, device and storage medium to solve the problem of low displacement identification accuracy in existing structural vibration identification methods.

[0006] In a first aspect, the present invention provides a method for identifying structural vibration displacement, the method comprising:

[0007] The vibration video of the area to be identified is acquired and preprocessed to obtain an image sequence and displacement ratio. The displacement ratio represents the physical displacement of the actual structure corresponding to the pixel displacement of the area to be identified.

[0008] The spatiotemporal coupling method is used to deblur the image sequence, resulting in a final clear image sequence and an optimized optical flow matrix;

[0009] Intermediate frame interpolation is performed based on the final clear image sequence and the optimized optical flow matrix to obtain the target image sequence;

[0010] Structural vibration displacement is identified based on the target image sequence and displacement ratio to obtain the vibration displacement of the area to be identified.

[0011] This invention establishes a conversion relationship between pixel displacement and physical displacement by acquiring vibration video and displacement ratio of the region to be identified. It employs a spatiotemporal coupling method to deblur the image sequence, obtaining a clear image sequence and an optimized optical flow matrix. This overcomes the limitations of traditional uniform blur kernels, effectively eliminating motion blur and suppressing noise, thus improving image clarity. The clear image sequence and optimized optical flow matrix are then used for frame interpolation, increasing the frame density of the image sequence and alleviating the problem of large inter-frame displacement caused by insufficient video frame rate. Compared to traditional pyramid downsampling methods, this method retains more details and more accurately restores the true vibration trajectory. Finally, the displacement ratio is combined to perform vibration identification on the target image sequence, obtaining the physical displacement of the structural vibration. This improves image quality and displacement identification accuracy under complex conditions of blur, noise interference, and large displacement, and solves the problem of difficulty in displacement estimation due to undersampling or large inter-frame displacement.

[0012] In one optional implementation, a spatiotemporal coupling method is used to deblur the image sequence to obtain a final clear image sequence and an optimized optical flow matrix, including:

[0013] The image sequence is used as the initial clear image sequence and blurry image sequence;

[0014] For each frame in the initial clear image sequence, based on the bidirectional optical flow of adjacent frames in the initial optical flow matrix, the blur kernel of each pixel in the image is simulated to obtain the blur kernel matrix. The initial optical flow matrix is ​​composed of the optical flow between all images in the initial clear image sequence.

[0015] A data item model is constructed based on the blurred image sequence, the initial clear image sequence, the blur kernel matrix, and the initial optical flow matrix.

[0016] Based on the assumption of consistent optical flow in the initial clear image sequence, a time-term model is constructed.

[0017] Based on the assumption of spatial consistency of the initial clear image sequence, a spatial term model is constructed;

[0018] The sum of the data item model, time item model, and spatial item model is used as the spatiotemporal coupling defuzzification model, and the objective function of the spatiotemporal coupling defuzzification model is determined.

[0019] For the objective function, the other is solved by fixing one of the initial sharp image sequence and the initial optical flow matrix to obtain the intermediate sharp image sequence or the intermediate optical flow matrix, and then it is determined whether the objective function has converged.

[0020] If the objective function does not converge, the intermediate sharp image sequence obtained in this solution is used as the new initial sharp image sequence or the intermediate optical flow matrix obtained in this solution is used as the new initial optical flow matrix. The process is repeated for each frame in the initial sharp image sequence. Based on the bidirectional optical flow of the adjacent frames in the initial optical flow matrix, the blur kernel of each pixel in the image is simulated to obtain the blur kernel matrix. This process continues until the objective function converges. Finally, the intermediate sharp image sequence and the intermediate optical flow matrix obtained in the solution are used as the final sharp image sequence and the optimized optical flow matrix.

[0021] This invention uses bidirectional optical flow between adjacent frames to simulate a pixel-by-pixel blur kernel, which more comprehensively represents pixel motion compared to unidirectional optical flow. Based on this, a data term model, a time term model, and a spatial term model are constructed respectively. The objective function of the spatiotemporal coupled deblurring model is determined based on the sum of the three. The objective function is solved by alternately optimizing. Compared with a single constraint model, the multi-dimensional joint constraint can more comprehensively deal with motion blur and noise interference, and output a higher quality clear image.

[0022] In one alternative implementation, the data item model is represented by the following formula:

[0023]

[0024] In the formula, E data (L,u,B) represents a data item model containing an initial sharp image sequence L, an initial optical flow matrix u, and a blurred image sequence B; λ represents the weight coefficient, which can be set by the user; i represents the index in the image sequence. Denotes the Topulitz matrix; K i Let L represent the blur kernel of all pixels in the i-th frame of the initial sharp image sequence within the blur kernel matrix; K represents the blur kernel matrix formed by the blur kernels of all pixels in all images of the initial sharp image sequence; i B represents the i-th frame in the initial clear image sequence; i This represents the i-th frame image in a blurred image sequence.

[0025] This invention constructs data items to quantify the pixel differences between the acquired image and the deblurred image, thereby guiding the model to approach the direction of minimizing pixel differences.

[0026] In one alternative implementation, the time term model is represented by the following formula:

[0027]

[0028] In the formula, E temporal (L,u) represents the time term model containing the initial sharp image sequence L and the initial optical flow matrix u; i represents the index in the image sequence; n represents the index of the adjacent frame of the i-th image in the initial sharp image sequence; N represents the number of adjacent frames of the i-th image in the initial sharp image sequence; μ n The weight parameter representing the nth frame image can be set manually; L i (x) represents the sharp image at pixel coordinate x in the i-th frame of the initial sharp image sequence; L i+n (x+u i→i+n ) represents the pixel coordinates x+u in the (i+n)th frame of the initial sharp image sequence. i→i+n The corresponding clear image; u i→i+n This represents the optical flow in the initial optical flow matrix from the i-th frame to the (i+n)-th frame in the initial clear image sequence.

[0029] This invention constructs a time term based on the optical flow consistency assumption, so that a certain pixel in the image has the same pixel value before and after motion, thus achieving smooth changes and avoiding abrupt changes.

[0030] In one alternative implementation, the spatial term model is represented by the following formula:

[0031]

[0032] In the formula, E spatial (L,u) represents the spatial term model containing the initial sharp image sequence L and the initial optical flow matrix u; i represents the index in the image sequence; L i represents the i-th frame in the initial sharp image sequence; n represents the index of the adjacent frame of the i-th frame in the initial sharp image sequence; N represents the number of adjacent frames of the i-th frame in the initial sharp image sequence; s represents the regularization intensity control parameter. Represents the gradient operator; σ represents the initial image of the i-th frame in the initial sharp image sequence during the alternating optimization process, which is also the i-th frame in the blurred image sequence; I Indicates the decay exponent; u i→i+n This represents the optical flow in the initial optical flow matrix from the i-th frame to the (i+n)-th frame in the initial clear image sequence.

[0033] This invention alleviates the ill-posedness problem of deblurred images and optical flow estimation by constructing spatial terms.

[0034] In one optional implementation, intermediate frame interpolation is performed based on the final sharp image sequence and the optimized optical flow matrix to obtain the target image sequence, including:

[0035] For each frame in the final clear image sequence, the forward and backward optical flows are determined based on the optimized optical flow in the optimized optical flow matrix of the image and the next frame of the image.

[0036] The image is projected forward to a first preset time using forward optical flow to obtain the forward transformed image coordinates. The first preset time is located between the image and the next frame image and is the sum of the image time and the first preset time point.

[0037] The next frame image is back-projected to the second preset time using reverse optical flow to obtain the reverse transformed image coordinates. The second preset time is located between the next frame image and the image, and is the difference between the time of the next frame image and the second preset time point. The sum of the first preset time point and the second preset time point is the time difference between the image and the next frame image.

[0038] Bilinear interpolation is performed on the forward-transformed image coordinates and the backward-transformed image coordinates to obtain the first intermediate frame and the second intermediate frame.

[0039] The target intermediate frame is obtained by weighted fusion based on the first intermediate frame, the first preset time point, the second intermediate frame, and the second preset time point.

[0040] The final clear image sequence and the target intermediate frames corresponding to each frame are integrated to obtain the target image sequence.

[0041] This invention simulates the motion of pixels between frames from two directions by determining forward and reverse optical flow, avoiding the loss of motion information that may be caused by unidirectional optical flow. The current frame is projected to a first preset time by forward optical flow, and the next frame is projected to a second preset time by reverse optical flow, forming a bidirectional constraint with forward projection. This can correct the unidirectional error that may exist in forward projection. Bilinear interpolation is performed on the image coordinates obtained by forward and reverse transformations to make the pixel values ​​of the interpolated frames transition smoothly. Finally, the forward and reverse intermediate frames are linearly weighted and fused according to the first and second preset times to obtain the final target intermediate frame. The target intermediate frame is inserted between adjacent frames of a clear image sequence to alleviate the problem of large inter-frame displacement caused by undersampling, making the dynamic process of structural vibration more continuous and facilitating the capture of subtle vibration features.

[0042] In one optional implementation, structural vibration displacement identification is performed based on the target image sequence and displacement ratio to obtain the vibration displacement of the region to be identified, including:

[0043] For each frame of the target image sequence, a feature detector is used to extract multiple image feature points from the image;

[0044] In the next frame of the image, each image feature point is tracked based on the fundamental assumption of optical flow to obtain the pixel displacement of the image feature point;

[0045] Based on the displacement ratio of the region to be identified, the pixel displacement of image feature points is converted into physical displacement;

[0046] Based on the physical displacement of all image feature points in the image, the vibration displacement between the image and the next frame is obtained.

[0047] The vibration displacement of the region to be identified is obtained based on the vibration displacement between every two frames in the target image sequence.

[0048] This invention extracts feature points in an image using a feature detector, tracks the movement of these feature points between adjacent frames, converts the tracked pixel displacements into physical displacements using previously calculated displacement ratios, and arranges the vibration displacements between every two frames in a time sequence to visually demonstrate the dynamic process of structural vibration in the area to be identified.

[0049] In a second aspect, the present invention provides a structural vibration displacement identification device, the device comprising:

[0050] The acquisition module is used to acquire vibration video of the area to be identified and preprocess it to obtain image sequence and displacement ratio. The displacement ratio represents the physical displacement of the actual structure corresponding to the pixel displacement of the area to be identified.

[0051] The deblurring module is used to deblur the image sequence using a spatiotemporal coupling method to obtain the final clear image sequence and optimized optical flow matrix;

[0052] The interpolation module is used to interpolate intermediate frames based on the final clear image sequence and the optimized optical flow matrix to obtain the target image sequence;

[0053] The identification module is used to identify structural vibration displacement based on the target image sequence and displacement ratio, and obtain the vibration displacement of the area to be identified.

[0054] Thirdly, the present invention provides a computer device, including: a memory and a processor, which are communicatively connected to each other. The memory stores computer instructions, and the processor executes the computer instructions to perform the structural vibration displacement identification method described in the first aspect or any corresponding embodiment.

[0055] Fourthly, the present invention provides a computer-readable storage medium storing computer instructions for causing a computer to execute the structural vibration displacement identification method described in the first aspect or any corresponding embodiment thereof. Attached Figure Description

[0056] To more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0057] Figure 1 This is a schematic flowchart of a structural vibration displacement identification method according to an embodiment of the present invention;

[0058] Figure 2 This is a comparative schematic diagram of structural vibration displacement identification results according to an embodiment of the present invention;

[0059] Figure 3 This is a structural block diagram of a structural vibration displacement identification device according to an embodiment of the present invention;

[0060] Figure 4 This is a schematic diagram of the hardware structure of a computer device according to an embodiment of the present invention. Detailed Implementation

[0061] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0062] While existing structural vibration identification methods can perform identification under complex conditions such as motion blur, noise interference, and large displacement, their displacement identification accuracy is low. This invention employs a spatiotemporal coupling method to deblur image sequences, obtaining clear image sequences and an optimized optical flow matrix. This overcomes the limitations of traditional uniform blur kernels, effectively eliminating motion blur and suppressing noise, thus improving image clarity. The clear image sequences and optimized optical flow matrix are then used for frame interpolation, increasing the frame density of the image sequences and alleviating the problem of large inter-frame displacement caused by insufficient video frame rates. Compared to traditional pyramid downsampling methods, this method retains more details and more accurately reconstructs the true vibration trajectory. Finally, the displacement ratio is combined to perform vibration identification on the target image sequence, obtaining the physical displacement of the structural vibration. This improves image quality and displacement identification accuracy under complex conditions of blur, noise interference, and large displacement, and solves the problem of difficulty in displacement estimation caused by undersampling or large inter-frame displacement.

[0063] According to an embodiment of the present invention, a method for identifying structural vibration displacement is provided. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Furthermore, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0064] This embodiment provides a method for identifying structural vibration displacement. Figure 1 This is a flowchart of a structural vibration displacement identification method according to an embodiment of the present invention, such as... Figure 1 As shown, the process includes the following steps:

[0065] Step S101: Acquire vibration video of the area to be identified and preprocess it to obtain an image sequence and displacement ratio. The displacement ratio represents the physical displacement of the actual structure corresponding to the pixel displacement of the area to be identified. Specifically, set the camera parameters (e.g., resolution 1920×1080, frame rate 30Hz) and acquire vibration video of the bridge structure (e.g., pedestrian overpass). The acquisition duration is set according to actual needs. Extract the vibration video of the area to be identified from the original video and decompose it into an image sequence frame by frame. Simultaneously, use the scaling factor method to determine the displacement ratio as the ratio of the actual size of the area to be identified to its size in the original video, thus establishing a linear transformation relationship from pixel displacement to physical displacement.

[0066] Step S102 involves deblurring the image sequence using a spatiotemporal coupling method to obtain a final clear image sequence and an optimized optical flow matrix. Specifically, since motion blur and noise interference can significantly impact displacement recognition, related technologies cannot handle spatially varying motion blur. Therefore, this embodiment of the invention uses a spatiotemporal coupling method to fuse information from the temporal and spatial dimensions, effectively eliminating motion blur in the image and suppressing noise, thereby improving image clarity.

[0067] Step S103 involves interpolating intermediate frames based on the final clear image sequence and the optimized optical flow matrix to obtain the target image sequence. Specifically, related technologies employ image pyramids for processing large displacement motion scenarios; however, pyramid downsampling may introduce image distortion, leading to the loss of subtle features and affecting the accuracy of optical flow estimation. Therefore, this embodiment of the invention inserts intermediate frames between adjacent frames to avoid large structural vibration displacements caused by large inter-frame time intervals. After frame interpolation, the inter-frame time interval is reduced, resulting in a more accurate reconstruction of the true vibration trajectory.

[0068] Step S104: Structural vibration displacement is identified based on the target image sequence and displacement ratio to obtain the vibration displacement of the area to be identified. Specifically, since the above steps are based on pixels in the image, the pixel displacement is converted into physical displacement through displacement ratio to obtain the actual and accurate vibration displacement of the area to be identified, thus improving the displacement identification accuracy.

[0069] This invention establishes a conversion relationship between pixel displacement and physical displacement by acquiring vibration video and displacement ratio of the region to be identified. It employs a spatiotemporal coupling method to deblur the image sequence, obtaining a clear image sequence and an optimized optical flow matrix. This overcomes the limitations of traditional uniform blur kernels, effectively eliminating motion blur and suppressing noise, thus improving image clarity. The clear image sequence and optimized optical flow matrix are then used for frame interpolation, increasing the frame density of the image sequence and alleviating the problem of large inter-frame displacement caused by insufficient video frame rate. Compared to traditional pyramid downsampling methods, this method retains more details and more accurately restores the true vibration trajectory. Finally, the displacement ratio is combined to perform vibration identification on the target image sequence, obtaining the physical displacement of the structural vibration. This improves image quality and displacement identification accuracy under complex conditions of blur, noise interference, and large displacement, and solves the problem of difficulty in displacement estimation due to undersampling or large inter-frame displacement.

[0070] This embodiment provides a method for identifying structural vibration displacement, which specifically includes the following steps:

[0071] Step S201: Acquire vibration video of the area to be identified and preprocess it to obtain an image sequence and displacement ratio. The displacement ratio represents the physical displacement of the actual structure corresponding to the pixel displacement of the area to be identified. For details, please refer to [link to relevant documentation]. Figure 1 Step S101 of the illustrated embodiment will not be described again here.

[0072] In step S202, the image sequence is deblurred using a spatiotemporal coupling method to obtain the final clear image sequence and optimized optical flow matrix.

[0073] Specifically, step S202 includes:

[0074] Step S2021: The image sequence is used as both the initial sharp image sequence and the blurred image sequence. Specifically, since the image sequence contains motion blur and noise, it is used as the blurred image sequence and simultaneously as the initial sharp image sequence in the deblurring process.

[0075] Step S2022: For each frame in the initial clear image sequence, based on the bidirectional optical flow of adjacent frames in the initial optical flow matrix, the blur kernel of each pixel in the image is simulated to obtain a blur kernel matrix. The initial optical flow matrix is ​​composed of the optical flow between all images in the initial clear image sequence. Specifically, for each frame in the initial clear image sequence, the optical flow of adjacent frames of that image is determined from the initial optical flow matrix as a bidirectional optical flow to simulate the blur kernel of each pixel in that image. Adjacent frames refer to the previous and next frames of the current image. More specifically, traditional deblurring methods often assume a uniform distribution of blur kernels, which cannot handle spatially varying motion blur. However, video capture devices have short exposure times, and the motion speed between adjacent frames is approximately constant. Therefore, bidirectional optical flow can be used to simulate the blur kernel of each pixel. Bidirectional optical flow can accurately characterize the motion trajectory of pixels within the exposure time, thereby segmentally simulating the blur kernel of each pixel and improving deblurring accuracy. For each frame of image, as shown in the first part of equation (1), the blur kernel is calculated using the forward optical flow (from the current frame to the next frame), and as shown in the second part of equation (1), the blur kernel is calculated using the backward optical flow (from the next frame to the current frame). The blur kernels beyond the forward and backward optical flow are 0, thus obtaining the blur kernel of each pixel in the current frame image, accurately modeling the motion blur characteristics of different pixels. The blur kernel of each pixel in each frame image can be obtained through equation (1). The blur kernels of all pixels in the frame image are combined as the blur kernel of the frame image, and the blur kernels of all images are used to form a blur kernel matrix.

[0076]

[0077] In the formula, k i,p (u,v) represents the blur kernel at position p of the i-th frame in the initial sharp image sequence; δ(·) represents the Kronecker function; τ i Indicates exposure time; u i→i+1 =(u i→i+1 ,v i→i+1 ) and u i→i-1 =(u i→i-1 ,v i→i-1 ) represents the forward and backward optical flow of the i-th frame in the initial clear image sequence, and u and v are the horizontal and vertical optical flows.

[0078] Step S2023: Based on the blurred image sequence, the initial clear image sequence, the blur kernel matrix, and the initial optical flow matrix, a data term model is constructed. The initial optical flow matrix consists of the optical flow between all images in the blurred image sequence. Specifically, the data term model shown in equation (2) is constructed to quantify the pixel difference between the acquired image and the deblurred image, guiding the model to approach the direction of minimizing pixel difference. The data term model is expressed by the following formula:

[0079]

[0080] In the formula, E data (L,u,B) represents a data item model containing an initial sharp image sequence L, an initial optical flow matrix u, and a blurred image sequence B; λ represents the weight coefficient, which can be set by the user; i represents the index in the image sequence. Denotes the Topulitz matrix; K i Let L represent the blur kernel of all pixels in the i-th frame of the initial sharp image sequence within the blur kernel matrix; K represents the blur kernel matrix formed by the blur kernels of all pixels in all images of the initial sharp image sequence; i B represents the i-th frame in the initial clear image sequence; i This represents the i-th frame image in a blurred image sequence.

[0081] Step S2024: Based on the optical flow consistency assumption of the initial clear image sequence, construct the temporal term model. Specifically, optical flow consistency means that a certain pixel in the image has the same pixel value before and after motion, ensuring smooth changes in optical flow between clear frames and avoiding abrupt motion. Based on this, construct the temporal term model shown in Equation (3).

[0082]

[0083] In the formula, E temporal (L,u) represents the time term model containing the initial sharp image sequence L and the initial optical flow matrix u; i represents the index in the image sequence; n represents the index of the adjacent frame of the i-th image in the initial sharp image sequence; N represents the number of adjacent frames of the i-th image in the initial sharp image sequence; μ n The weight parameter representing the nth frame image can be set manually; L i (x) represents the sharp image at pixel coordinate x in the i-th frame of the initial sharp image sequence; L i+n (x+u i→i+n ) represents the pixel coordinates x+u in the (i+n)th frame of the initial sharp image sequence. i→i+n The corresponding clear image; u i→i+n This represents the optical flow in the initial optical flow matrix from the i-th frame to the (i+n)-th frame in the initial clear image sequence.

[0084] Step S2025: Based on the spatial consistency assumption of the initial clear image sequence, construct a spatial term model. Specifically, the spatial consistency assumption means that the image and its corresponding optical flow should have coherent and smooth characteristics in space. This is beneficial for the clear frame and its corresponding optical flow to maintain smooth changes in space, and also encourages the discontinuity of the clear frame and its corresponding optical flow at the edges. Based on this, construct the energy model of the spatial term shown in Equation (4).

[0085]

[0086] In the formula, E spatial (L,u) represents the spatial term model containing the initial sharp image sequence L and the initial optical flow matrix u; i represents the index in the image sequence; L i represents the i-th frame in the initial sharp image sequence; n represents the index of the adjacent frame of the i-th frame in the initial sharp image sequence; N represents the number of adjacent frames of the i-th frame in the initial sharp image sequence; s represents the regularization intensity control parameter. Represents the gradient operator; σ represents the initial image of the i-th frame in the initial sharp image sequence during the alternating optimization process, which is also the i-th frame in the blurred image sequence; I Indicates the decay exponent; u i→i+n This represents the optical flow in the initial optical flow matrix from the i-th frame to the (i+n)-th frame in the initial clear image sequence.

[0087] Step S2026 involves using the sum of the data item model, time item model, and spatial item model as the spatiotemporal coupled deblurring model, and determining the objective function of this model. Specifically, a single-dimensional model cannot simultaneously handle issues such as temporal consistency and spatial smoothness. By fusing the data item, time item, and spatial item, a multi-dimensional constrained global optimization model is formed, resulting in the spatiotemporal coupled deblurring model. The minimum value of this model is then used as the objective function. By integrating information from both temporal and spatial dimensions, global optimal elimination of motion blur and noise interference is achieved, significantly improving deblurring accuracy and image quality compared to a single model, laying the foundation for subsequent vibration analysis.

[0088] Step S2027: For the objective function, solve for the other by fixing one of the initial sharp image sequence and the initial optical flow matrix, obtaining the intermediate sharp image sequence or intermediate optical flow matrix, and then determine whether the objective function has converged. Specifically, the optical flow matrix and the sharp images are interrelated; optical flow affects image transformation, and image sharpness affects optical flow estimation, making direct solution difficult. Therefore, by alternately optimizing these two aspects and iteratively updating step by step, the optimal solution is gradually approximated, reducing computational complexity.

[0089] Step S2028: If the objective function has not converged, the intermediate clear image sequence obtained in this solution is used as the new initial clear image sequence, or the intermediate optical flow matrix obtained in this solution is used as the new initial optical flow matrix. The process returns to the step of simulating the blur kernel of each pixel in the image based on the bidirectional optical flow of adjacent frames in the initial clear image sequence for each frame, until the objective function converges. The final intermediate clear image sequence and intermediate optical flow matrix are then used as the final clear image sequence and optimized optical flow matrix. Specifically, the initial optical flow matrix is ​​fixed, and the objective function is solved to obtain the current clear image sequence, which is used as the intermediate clear image. It is then determined whether the objective function has converged. If it has not converged, the intermediate clear image sequence obtained above is used as the new initial clear image sequence, and the process returns to step S2022. In this step, the initial clear image sequence is fixed, the optical flow matrix is ​​solved, and the process continues to determine whether the spatiotemporal coupling deblurring model has converged. If convergence fails, the alternating optimization process described above is repeated. If convergence occurs, the final intermediate optical flow matrix is ​​used as the optimized optical flow matrix, and the final intermediate clear image sequence is used as the final clear image sequence. By solving the objective function through alternating optimization, compared to a single-constraint model, multi-dimensional joint constraints can more comprehensively address motion blur and noise interference, outputting higher-quality clear images.

[0090] Step S203: Interpolate intermediate frames based on the final clear image sequence and the optimized optical flow matrix to obtain the target image sequence.

[0091] Specifically, step S203 includes:

[0092] Step S2031: For each frame in the final clear image sequence, determine the forward and backward optical flow based on the optimized optical flow in the optimized optical flow matrix of the image and the next frame. Specifically, frame interpolation requires predicting the image between two frames based on the motion information of adjacent frames, and the forward optical flow u. i→i+1 =(u i→i+1 ,v i→i+1 Optical flow from the current frame to the next frame and reverse optical flow u i+1→i =(u i+1→i ,v i+1→i The optical flow from the next frame to the current frame can represent the forward and reverse pixel motion trajectories, respectively. Combining the two can more accurately describe the bidirectional consistency of inter-frame motion and avoid motion estimation bias caused by unidirectional optical flow. More specifically, for each frame, the forward and reverse optical flow between it and the next frame are extracted from the optimized optical flow matrix.

[0093] Step S2032: The image is projected forward using forward optical flow to a first preset time point to obtain forward transformed image coordinates. The first preset time point is located between the image and the next frame image and is the sum of the image time and the first preset time point. Specifically, the value of the first preset time point t is (0,1). If the time of the current frame image is a, then the first preset time point is a+t. The pixels of the current frame are projected forward using forward optical flow to the first preset time point between two frames, and the forward transformed image coordinates are generated by the following formula (5) to provide sampling points for frame interpolation.

[0094] (x',y')=(x 1 +t·u i→i+1 (x 2 ,y 2 ),y 1 +t·v i→i+1 (x 2 ,y 2 ))(5)

[0095] In the formula, (x', y') represents the coordinates of the forward-transformed image; t represents the first preset time point; x 1 This represents the x-coordinate of each pixel in the current frame image; u i→i+1 Represents the horizontal component of the forward optical flow; (x 2 ,y 2 ) represents the coordinates of each pixel in the forward optical flow; y 1 This represents the ordinate of each pixel in the current frame image; v i→i+1 This represents the vertical component of the forward optical flow.

[0096] Step S2033: The image is back-projected to the second preset time using reverse optical flow to obtain the reverse-transformed image coordinates. The second preset time is located between the next frame image and the image, and is the difference between the time of the next frame image and the second preset time point. The sum of the first preset time point and the second preset time point is the time difference between the image and the next frame image. Specifically, since the current frame and the next frame are adjacent frames, the second preset time point can be obtained as 1-t based on the first preset time point as the value range, and thus the second preset time point is a+1-t. The pixels of the next frame are projected to the second preset time between the two frames according to the reverse optical flow, and the reverse-transformed image coordinates are generated by the following formula (6) to provide sampling points for frame interpolation. Furthermore, the reverse projection and the forward projection form a symmetrical constraint on the time axis, which cancels the optical flow estimation error and avoids the distortion of the intermediate frame caused by the unidirectional projection deviation.

[0097] (x”,y”)=(x 3 +(1-t)·u i+1→i (x 4 ,y 4 ),y 3 +(1-t)·vi+1→i (x 4 ,y 4 ))(6)

[0098] In the formula, (x”, y”) represents the coordinates of the inversely transformed image; 1-t represents the second preset time point; x 3 This represents the x-coordinate of each pixel in the next frame of the image; u i+1→i Represents the horizontal component of the reverse optical flow; (x 4 ,y 4 ) represents the coordinates of each pixel in the reverse optical flow; y 3 This represents the ordinate of each pixel in the next frame of the image; v i+1→i This represents the vertical component of the reverse optical flow.

[0099] Step S2034 involves performing bilinear interpolation on the forward-transformed image coordinates and the backward-transformed image coordinates to obtain the first intermediate frame and the second intermediate frame. Specifically, for the forward-transformed image coordinates, four neighboring pixels are identified, and their weights are calculated. A weighted average of these four pixel coordinates and weights is then performed to obtain the pixel value obtained from the forward transformation. This pixel value, along with other pixels in the current frame image, constitutes the first intermediate frame. The bilinear interpolation process for the backward-transformed image coordinates can be referenced from the forward transformation process and will not be elaborated upon here.

[0100] Step S2035: A weighted fusion is performed based on the first intermediate frame, the first preset time point, the second intermediate frame, and the second preset time point to obtain the target intermediate frame. Specifically, the sum of the first preset time point and the second preset time point is the time difference between the current frame and the next frame, which is 1. Based on the above relationship, the time weights of the two time points are determined respectively. For example, the first preset time point is t, and the second preset time point is 1-t, therefore the time weight of the first preset time point is 1-t, and the time weight of the second preset time point is t. As shown in equation (7), the first intermediate frame and the second intermediate frame are linearly weighted and fused according to the above time weights to obtain the final target intermediate frame.

[0101]

[0102] In the formula, I i+t (x,y) represents the target intermediate frame; 1-t represents the time weight represented by the first preset time point; t represents the first intermediate frame; t represents the time weight represented by the second preset time point; This indicates the second intermediate frame.

[0103] Step S2036: The final clear image sequence and the target intermediate frame corresponding to each frame are integrated to obtain the target image sequence. Specifically, the target intermediate frame of each frame is inserted into the original image sequence to improve the temporal resolution and provide denser time sampling points for structural vibration displacement identification.

[0104] Step S204: Based on the target image sequence and displacement ratio, structural vibration displacement is identified to obtain the vibration displacement of the area to be identified.

[0105] Specifically, step S204 includes:

[0106] Step S2041: For each frame of the target image sequence, a feature detector is used to extract multiple image feature points from the image. Specifically, an ORB (Oriented Fast and Rotated BRIEF) feature detector is used to process each frame of the target image sequence to extract stable image feature points, providing sufficient reference points for subsequent optical flow tracking.

[0107] In step S2042, in the next frame of the image, each image feature point is tracked based on the fundamental assumptions of optical flow to obtain the pixel displacement of the image feature points. Specifically, the fundamental assumptions of optical flow (constant brightness, small displacement, spatial consistency) hold true within a short time interval. The KLT (Kanade-Lucas-Tomasi) optical flow algorithm is based on these assumptions and can efficiently and accurately track the motion of feature points. By determining the pixel displacement between feature points in two adjacent frames, structural vibration can be characterized.

[0108] Step S2043: Based on the displacement ratio of the region to be identified, the pixel displacement of the image feature points is converted into physical displacement. Specifically, the visual measurement yields pixel-level displacement, which needs to be converted into actual physical displacement using the displacement ratio calculated in step S101 to reflect the true vibration amplitude of the structure.

[0109] Step S2044: Based on the physical displacements of all image feature points in the image, the vibration displacement between the image and the next frame is obtained. Specifically, the displacement of a single feature point is random, and the vibration displacement of the entire frame image needs to be characterized by the statistical results of multiple feature points to eliminate the influence of local noise and make the inter-frame displacement results more reflective of the true vibration state of the structure. Optionally, the average or weighted average of the physical displacements of all feature points can be used as the vibration displacement between adjacent frames, and other calculation methods can also be used. This embodiment of the invention does not limit this.

[0110] Step S2045: Based on the vibration displacement between every two frames in the target image sequence, the vibration displacement of the region to be identified is obtained. Specifically, for each pair of adjacent frames in the target image sequence, the vibration displacement between each pair of adjacent frames is determined through the above steps, thereby obtaining the vibration displacement of the entire target image sequence. The vibration state at different times can be plotted, providing key data support for engineering applications such as bridge health monitoring and modal analysis, and achieving high-precision identification of non-contact vibration displacement.

[0111] In some alternative implementations, Figure 2 This is a comparative schematic diagram of structural vibration displacement recognition results according to an embodiment of the present invention. Without image preprocessing, the traditional KLT optical flow method exhibits significant errors and a certain degree of displacement drift during large displacement recognition. While the pyramid KLT optical flow method effectively expands the recognition range by constructing an image pyramid, showing advantages in recognizing large displacement motions and significantly improving recognition accuracy, the deblurring and frame interpolation displacement recognition method adopted in this embodiment of the present invention improves recognition accuracy compared to the two related techniques mentioned above.

[0112] This invention establishes a conversion relationship between pixel displacement and physical displacement by acquiring vibration video and displacement ratio of the region to be identified. It employs a spatiotemporal coupling method to deblur the image sequence, obtaining a clear image sequence and an optimized optical flow matrix. This overcomes the limitations of traditional uniform blur kernels, effectively eliminating motion blur and suppressing noise, thus improving image clarity. The clear image sequence and optimized optical flow matrix are then used for frame interpolation, increasing the frame density of the image sequence and alleviating the problem of large inter-frame displacement caused by insufficient video frame rate. Compared to traditional pyramid downsampling methods, this method retains more details and more accurately restores the true vibration trajectory. Finally, the displacement ratio is combined to perform vibration identification on the target image sequence, obtaining the physical displacement of the structural vibration. This improves image quality and displacement identification accuracy under complex conditions of blur, noise interference, and large displacement, and solves the problem of difficulty in displacement estimation due to undersampling or large inter-frame displacement.

[0113] This embodiment also provides a structural vibration displacement identification device, which is used to implement the above embodiments and preferred embodiments; details already described will not be repeated. As used below, the term "module" can refer to a combination of software and / or hardware that performs a predetermined function. Although the device described in the following embodiments is preferably implemented in software, hardware implementation, or a combination of software and hardware, is also possible and contemplated.

[0114] This embodiment provides a structural vibration displacement identification device, such as... Figure 3 As shown, it includes:

[0115] The acquisition module 301 is used to acquire vibration video of the area to be identified and preprocess it to obtain image sequence and displacement ratio. The displacement ratio represents the physical displacement of the actual structure corresponding to the pixel displacement of the area to be identified.

[0116] The deblurring module 302 is used to deblur the image sequence using a spatiotemporal coupling method to obtain the final clear image sequence and optimized optical flow matrix.

[0117] Interpolation module 303 is used to perform intermediate frame interpolation based on the final clear image sequence and the optimized optical flow matrix to obtain the target image sequence.

[0118] The recognition module 304 is used to identify the structural vibration displacement based on the target image sequence and displacement ratio, so as to obtain the vibration displacement of the area to be identified.

[0119] In some alternative implementations, the deblurring module 302 includes:

[0120] The first determining unit is used to determine the image sequence as an initial clear image sequence and a blurred image sequence.

[0121] The simulation unit is used to simulate the blur kernel of each pixel in the image for each frame in the initial sharp image sequence, based on the bidirectional optical flow of the adjacent frames in the initial optical flow matrix, to obtain the blur kernel matrix. The initial optical flow matrix is ​​composed of the optical flow between all images in the initial sharp image sequence.

[0122] The first building unit is used to construct a data item model based on the blurred image sequence, the initial clear image sequence, the blurred kernel matrix, and the initial optical flow matrix.

[0123] The second building block is used to construct a time-term model based on the optical flow consistency assumption of the initial clear image sequence.

[0124] The third building block is used to construct a spatial term model based on the spatial consistency assumption of the initial clear image sequence.

[0125] The second determining unit is used to take the sum of the data item model, the time item model, and the spatial item model as the spatiotemporal coupling defuzzification model, and to determine the objective function of the spatiotemporal coupling defuzzification model.

[0126] The first solving unit is used to solve the objective function by fixing one of the initial sharp image sequence and the initial optical flow matrix, to obtain the intermediate sharp image sequence or the intermediate optical flow matrix, and to determine whether the objective function has converged.

[0127] The second solution unit is used to, when the objective function has not converged, take the intermediate sharp image sequence obtained in this solution as the new initial sharp image sequence or take the intermediate optical flow matrix obtained in this solution as the new initial optical flow matrix, and return to the step of simulating the blur kernel of each pixel in the image based on the bidirectional optical flow of the adjacent frames in the image in the initial optical flow matrix, and obtaining the blur kernel matrix, for each frame in the initial sharp image sequence until the objective function converges. The final intermediate sharp image sequence and intermediate optical flow matrix obtained in the solution are then used as the final sharp image sequence and optimized optical flow matrix.

[0128] In some alternative implementations, the data item model is represented by the following formula:

[0129]

[0130] In the formula, E data (L,u,B) represents a data item model containing an initial sharp image sequence L, an initial optical flow matrix u, and a blurred image sequence B; λ represents the weight coefficient, which can be set by the user; i represents the index in the image sequence. Denotes the Topulitz matrix; K i Let L represent the blur kernel of all pixels in the i-th frame of the initial sharp image sequence within the blur kernel matrix; K represents the blur kernel matrix formed by the blur kernels of all pixels in all images of the initial sharp image sequence; i B represents the i-th frame in the initial clear image sequence; i This represents the i-th frame image in a blurred image sequence.

[0131] In some alternative implementations, the time term model is represented by the following formula:

[0132]

[0133] In the formula, E temporal (L,u) represents the time term model containing the initial sharp image sequence L and the initial optical flow matrix u; i represents the index in the image sequence; n represents the index of the adjacent frame of the i-th image in the initial sharp image sequence; N represents the number of adjacent frames of the i-th image in the initial sharp image sequence; μ n The weight parameter representing the nth frame image can be set manually; L i (x) represents the sharp image at pixel coordinate x in the i-th frame of the initial sharp image sequence; L i+n (x+u i→i+n ) represents the pixel coordinates x+u in the (i+n)th frame of the initial sharp image sequence. i→i+n The corresponding clear image; u i→i+n This represents the optical flow in the initial optical flow matrix from the i-th frame to the (i+n)-th frame in the initial clear image sequence.

[0134] In some alternative implementations, the spatial term model is represented by the following formula:

[0135]

[0136] In the formula, E spatial (L,u) represents the spatial term model containing the initial sharp image sequence L and the initial optical flow matrix u; i represents the index in the image sequence; L i represents the i-th frame in the initial sharp image sequence; n represents the index of the adjacent frame of the i-th frame in the initial sharp image sequence; N represents the number of adjacent frames of the i-th frame in the initial sharp image sequence; s represents the regularization intensity control parameter. Represents the gradient operator; σ represents the initial image of the i-th frame in the initial sharp image sequence during the alternating optimization process, which is also the i-th frame in the blurred image sequence; I Indicates the decay exponent; u i→i+n This represents the optical flow in the initial optical flow matrix from the i-th frame to the (i+n)-th frame in the initial clear image sequence.

[0137] In some alternative implementations, the interpolation module 303 includes:

[0138] The third determining unit is used to determine the forward optical flow and backward optical flow for each frame in the final clear image sequence, based on the optimized optical flow of the image and the next frame in the optimized optical flow matrix.

[0139] The first projection unit is used to project the image forward to a first preset time using forward optical flow to obtain forward transformed image coordinates. The first preset time is located between the image and the next frame image and is the sum of the image time and the first preset time point.

[0140] The second projection unit is used to reverse project the next frame image to the second preset time using reverse optical flow to obtain the reverse transformed image coordinates. The second preset time is located between the next frame image and the image, and is the difference between the time of the next frame image and the second preset time point. The sum of the first preset time point and the second preset time point is the time difference between the image and the next frame image.

[0141] The interpolation unit is used to perform bilinear interpolation on the forward-transformed image coordinates and the backward-transformed image coordinates to obtain the first intermediate frame and the second intermediate frame.

[0142] The fusion unit is used to perform weighted fusion based on the first intermediate frame, the first preset time point, the second intermediate frame, and the second preset time point to obtain the target intermediate frame.

[0143] The integration unit is used to integrate the final clear image sequence with the target intermediate frame corresponding to each frame image to obtain the target image sequence.

[0144] In some alternative implementations, the identification module 304 includes:

[0145] The extraction unit is used to extract multiple image feature points from each frame of the target image sequence using a feature detector.

[0146] The tracking unit is used to track each image feature point in the next frame of the image based on the fundamental assumption of optical flow, and to obtain the pixel displacement of the image feature point.

[0147] The conversion unit is used to convert the pixel displacement of image feature points into physical displacement based on the displacement ratio of the region to be identified.

[0148] The fourth determining unit is used to obtain the vibration displacement between the image and the next frame image based on the physical displacement of all image feature points in the image.

[0149] The fifth determining unit is used to obtain the vibration displacement of the region to be identified based on the vibration displacement between every two frames in the target image sequence.

[0150] Further functional descriptions of the above modules and units are the same as those in the corresponding embodiments described above, and will not be repeated here.

[0151] In this embodiment, the structural vibration displacement identification device is presented in the form of a functional unit. Here, a unit refers to an ASIC (Application Specific Integrated Circuit) circuit, a processor and memory that execute one or more software or fixed programs, and / or other devices that can provide the above functions.

[0152] This invention also provides a computer device having the above-described features. Figure 3 The structural vibration displacement identification device shown.

[0153] Please see Figure 4 , Figure 4 This is a schematic diagram of the structure of a computer device provided in an optional embodiment of the present invention, such as... Figure 4As shown, the computer device includes one or more processors 10, memory 20, and interfaces for connecting the components, including high-speed interfaces and low-speed interfaces. The components communicate with each other via different buses and can be mounted on a common motherboard or otherwise installed as needed. The processors can process instructions executed within the computer device, including instructions stored in or on memory to display graphical information of a GUI on external input / output devices (such as display devices coupled to the interfaces). In some alternative implementations, multiple processors and / or multiple buses can be used with multiple memories and multiple memory modules, if desired. Similarly, multiple computer devices can be connected, each providing some of the necessary operations (e.g., as a server array, a group of blade servers, or a multiprocessor system). Figure 4 Take a processor 10 as an example.

[0154] Processor 10 may be a central processing unit, a network processor, or a combination thereof. Processor 10 may further include a hardware chip. The hardware chip may be an application-specific integrated circuit (ASIC), a programmable logic device (PLD), or a combination thereof. The programmable logic device may be a complex programmable logic device (CAMP), a field-programmable gate array (FPGA), a general-purpose array logic (GDA), or any combination thereof.

[0155] The memory 20 stores instructions executable by at least one processor 10 to cause at least one processor 10 to perform the method shown in the above embodiments.

[0156] The memory 20 may include a program storage area and a data storage area. The program storage area may store the operating system and applications required for at least one function; the data storage area may store data created based on the use of the computer device. Furthermore, the memory 20 may include high-speed random access memory and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some alternative embodiments, the memory 20 may optionally include memory remotely located relative to the processor 10, and these remote memories may be connected to the computer device via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0157] The memory 20 may include volatile memory, such as random access memory; the memory may also include non-volatile memory, such as flash memory, hard disk or solid-state drive; the memory 20 may also include a combination of the above types of memory.

[0158] The computer device also includes an input device 30 and an output device 40. The processor 10, memory 20, input device 30, and output device 40 can be connected via a bus or other means. Figure 4 Taking the example of a connection between China and Israel via a bus.

[0159] Input device 30 can receive input numerical or character information, and generate key signal inputs related to user settings and function control of the computer device, such as a touchscreen, keypad, mouse, trackpad, touchpad, joystick, one or more mouse buttons, trackball, joystick, etc. Output device 40 may include display devices, auxiliary lighting devices (e.g., LEDs), and haptic feedback devices (e.g., vibration motors). The aforementioned display devices include, but are not limited to, liquid crystal displays, light-emitting diodes, displays, and plasma displays. In some alternative embodiments, the display device may be a touchscreen.

[0160] This invention also provides a computer-readable storage medium. The methods described above according to embodiments of the invention can be implemented in hardware or firmware, or implemented as computer code that can be recorded on a storage medium, or implemented as computer code downloaded via a network and originally stored on a remote storage medium or a non-transitory machine-readable storage medium and then stored on a local storage medium. Thus, the methods described herein can be processed by software stored on a storage medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware. The storage medium can be a magnetic disk, optical disk, read-only memory, random access memory, flash memory, hard disk, or solid-state drive, etc.; further, the storage medium can also include combinations of the above types of memory. It is understood that computers, processors, microprocessor controllers, or programmable hardware include storage components capable of storing or receiving software or computer code, which, when accessed and executed by the computer, processor, or hardware, implements the methods shown in the above embodiments.

[0161] A portion of this invention can be applied as a computer program product, such as computer program instructions, which, when executed by a computer, can invoke or provide the methods and / or technical solutions according to the invention through the operation of the computer. Those skilled in the art will understand that the forms in which computer program instructions exist in a computer-readable medium include, but are not limited to, source files, executable files, installation package files, etc. Correspondingly, the ways in which computer program instructions are executed by a computer include, but are not limited to: the computer directly executing the instructions, or the computer compiling the instructions and then executing the corresponding compiled program, or the computer reading and executing the instructions, or the computer reading and installing the instructions and then executing the corresponding installed program. Here, the computer-readable medium can be any available computer-readable storage medium or communication medium accessible to a computer.

[0162] Although embodiments of the invention have been described in conjunction with the accompanying drawings, those skilled in the art can make various modifications and variations without departing from the spirit and scope of the invention, and such modifications and variations all fall within the scope defined by the appended claims.

Claims

1. A method for identifying structural vibration displacement, characterized in that, The method includes: The vibration video of the region to be identified is acquired and preprocessed to obtain an image sequence and displacement ratio. The displacement ratio represents the physical displacement of the actual structure corresponding to the pixel displacement of the region to be identified. The image sequence is deblurred using a spatiotemporal coupling method to obtain a final clear image sequence and an optimized optical flow matrix; Intermediate frame interpolation is performed based on the final clear image sequence and the optimized optical flow matrix to obtain the target image sequence; Based on the target image sequence and the displacement ratio, structural vibration displacement is identified to obtain the vibration displacement of the region to be identified.

2. The method according to claim 1, characterized in that, The process of deblurring the image sequence using a spatiotemporal coupling method to obtain a final clear image sequence and an optimized optical flow matrix includes: The image sequence is used as the initial clear image sequence and blurry image sequence; For each frame in the initial clear image sequence, based on the bidirectional optical flow of the adjacent frames of the image in the initial optical flow matrix, the blur kernel of each pixel in the image is simulated to obtain a blur kernel matrix. The initial optical flow matrix is ​​composed of the optical flow between all images in the initial clear image sequence. A data item model is constructed based on the blurred image sequence, the initial clear image sequence, the blur kernel matrix, and the initial optical flow matrix. Based on the optical flow consistency assumption of the initial clear image sequence, a time-term model is constructed; Based on the spatial consistency assumption of the initial clear image sequence, a spatial term model is constructed; The sum of the data item model, the time item model, and the spatial item model is used as the spatiotemporal coupling defuzzification model, and the objective function of the spatiotemporal coupling defuzzification model is determined. For the objective function, the other is solved by fixing one of the initial clear image sequence and the initial optical flow matrix to obtain the intermediate clear image sequence or the intermediate optical flow matrix, and then it is determined whether the objective function has converged. If the objective function does not converge, the intermediate sharp image sequence obtained in this solution is used as the new initial sharp image sequence, or the intermediate optical flow matrix obtained in this solution is used as the new initial optical flow matrix. The process is repeated for each frame in the initial sharp image sequence, based on the bidirectional optical flow of the adjacent frames of the image in the initial optical flow matrix, to simulate the blur kernel of each pixel in the image and obtain the blur kernel matrix. This process continues until the objective function converges. Finally, the intermediate sharp image sequence and the intermediate optical flow matrix obtained in the solution are used as the final sharp image sequence and the optimized optical flow matrix.

3. The method according to claim 2, characterized in that, The data item model is represented by the following formula: In the formula, E data (L,u,B) represents a data item model containing an initial sharp image sequence L, an initial optical flow matrix u, and a blurred image sequence B; λ represents the weight coefficient, which can be set by the user; i represents the index in the image sequence. Denotes the Topulitz matrix; K i Let L represent the blur kernel of all pixels in the i-th frame of the initial sharp image sequence within the blur kernel matrix; K represents the blur kernel matrix formed by the blur kernels of all pixels in all images of the initial sharp image sequence; i B represents the i-th frame in the initial clear image sequence; i This represents the i-th frame image in the blurred image sequence.

4. The method according to claim 2, characterized in that, The time term model is expressed by the following formula: In the formula, E temporal (L,u) represents the time term model containing the initial sharp image sequence L and the initial optical flow matrix u; i represents the index in the image sequence; n represents the index of the adjacent frame of the i-th image in the initial sharp image sequence; N represents the number of adjacent frames of the i-th image in the initial sharp image sequence; μ n The weight parameter representing the nth frame image can be set manually; L i (x) represents the sharp image at pixel coordinate x in the i-th frame of the initial sharp image sequence; L i+n (x+u i→i+n ) represents the pixel coordinates x+u in the (i+n)th frame of the initial sharp image sequence. i→i+n The corresponding clear image; u i→i+n This represents the optical flow in the initial optical flow matrix from the i-th frame to the (i+n)-th frame in the initial clear image sequence.

5. The method according to claim 2, characterized in that, The spatial term model is expressed by the following formula: In the formula, E spatial (L,u) represents the spatial term model containing the initial sharp image sequence L and the initial optical flow matrix u; i represents the index in the image sequence; L i represents the i-th frame in the initial clear image sequence; n represents the index of the adjacent frame of the i-th frame in the initial clear image sequence; N represents the number of adjacent frames of the i-th frame in the initial clear image sequence; s represents the regularization strength control parameter; Represents the gradient operator; σ represents the initial image of the i-th frame in the initial sharp image sequence during the alternating optimization process, which is also the i-th frame in the blurred image sequence; I Indicates the decay index; u i→i+n This represents the optical flow in the initial optical flow matrix from the i-th frame to the (i+n)-th frame in the initial clear image sequence.

6. The method according to claim 1, characterized in that, The step of interpolating intermediate frames based on the final clear image sequence and the optimized optical flow matrix to obtain the target image sequence includes: For each frame in the final clear image sequence, the forward optical flow and backward optical flow are determined based on the optimized optical flow in the optimized optical flow matrix of the image and the next frame of the image; The forward optical flow is used to project the image forward to a first preset time to obtain forward transformed image coordinates. The first preset time is located between the image and the next frame image and is the sum of the time of the image and the first preset time point. The next frame image is back-projected to the second preset time using the reverse optical flow to obtain the reverse transformed image coordinates. The second preset time is located between the next frame image and the image, and is the difference between the time of the next frame image and the second preset time point. The sum of the first preset time point and the second preset time point is the time difference between the image and the next frame image. The forward-transformed image coordinates and the backward-transformed image coordinates are respectively subjected to bilinear interpolation to obtain the first intermediate frame and the second intermediate frame; The target intermediate frame is obtained by weighted fusion based on the first intermediate frame, the first preset time point, the second intermediate frame, and the second preset time point. The final clear image sequence and the target intermediate frames corresponding to each frame are integrated to obtain the target image sequence.

7. The method according to claim 1, characterized in that, The step of identifying structural vibration displacement based on the target image sequence and the displacement ratio to obtain the vibration displacement of the region to be identified includes: For each frame of the target image sequence, a feature detector is used to extract multiple image feature points from the image; In the next frame of the image, each image feature point is tracked based on the fundamental assumption of optical flow to obtain the pixel displacement of the image feature point; Based on the displacement ratio of the region to be identified, the pixel displacement of the image feature points is converted into physical displacement; Based on the physical displacement of all image feature points in the image, the vibration displacement between the image and the next frame image is obtained; The vibration displacement of the region to be identified is obtained based on the vibration displacement between every two frames in the target image sequence.

8. A structural vibration displacement identification device, characterized in that, The device includes: The acquisition module is used to acquire vibration video of the area to be identified and preprocess it to obtain image sequence and displacement ratio, wherein the displacement ratio represents the physical displacement of the actual structure corresponding to the pixel displacement of the area to be identified. The deblurring module is used to deblur the image sequence using a spatiotemporal coupling method to obtain a final clear image sequence and an optimized optical flow matrix; An interpolation module is used to perform intermediate frame interpolation based on the final clear image sequence and the optimized optical flow matrix to obtain the target image sequence; The identification module is used to identify structural vibration displacement based on the target image sequence and the displacement ratio, so as to obtain the vibration displacement of the region to be identified.

9. A computer device, characterized in that, include: A memory and a processor are interconnected, the memory stores computer instructions, and the processor executes the computer instructions to perform the structural vibration displacement identification method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions for causing the computer to execute the structural vibration displacement identification method according to any one of claims 1 to 7.

Images