Structural vibration displacement identification method based on dense matching and prior knowledge enhancement
Through the method based on dense matching and prior knowledge enhancement, the image feature enhancement network and structural linear vibration theory are used to achieve rapid and dense recognition of structural vibration displacement, solving the problem of insufficient recognition speed and accuracy in the prior art, and suitable for structure Application of health monitoring system.
Patent Information
- Application Number
- CN202411091247.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-09
- Publication Date
- 2024-11-22
- Estimated Expiration
- 2044-08-09
AI Technical Summary
The prior art cannot simultaneously realize rapid and dense identification of structural vibration displacement, and the measurement accuracy is not high under harsh environmental conditions, difficult to automate, and it is impossible to effectively utilize the motion prior information in monitoring video data.
Using a method based on dense matching and prior knowledge enhancement, non-iteration optical flow estimation is performed through image feature enhancement networks, dense matching and optical flow propagation are used to combine physical knowledge of structural linear vibration theory, and using singular value decomposition to denoising noise , realize fast and dense structural vibration displacement recognition.
It realizes accurate and robust structural vibration displacement recognition under harsh environmental conditions, reduces errors caused by poor monitoring video quality, is suitable for the application of structural health monitoring systems, and improves the accuracy and automation of recognition.
Smart Images

Figure CN119006522A_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the technical field of signal processing and structural health monitoring, and in particular relates to a structural vibration displacement identification method based on dense matching and prior knowledge enhancement. Background Art
[0002] The measurement of structural vibration displacement time history is very important in civil engineering. It not only reveals the static and dynamic characteristics of civil structures, but also directly reflects the structural status. For example, excessive deflection in bridge structures and excessive inter-story displacement angles in building structures directly indicate potential damage to in-service structures. Current measurement technologies include: contact sensors connected to monitoring points, traditional non-contact sensors, and vision-based non-contact sensors.
[0003] Contact sensors measure the relative motion between two physically connected points and obtain displacement. Although these sensors have high displacement measurement accuracy, they require fixed reference points, which is not feasible for large structures. In addition, wired connections make such systems expensive to deploy and maintain. Wireless transmission technology can reduce the complexity of wiring, but contact systems usually only deploy sparse "point" sensors, which means that the displacement measurement results have only low spatial resolution. Therefore, structural information may be lost, resulting in the inability to accurately identify local damage later. Traditional non-contact sensors obtain displacement by measuring satellite positioning, laser, radar and other remote sensing signals. Among them, satellite positioning has low measurement accuracy and frequency, which is suitable for static displacement monitoring. Laser measurement has high accuracy and frequency, but the laser power under safety restrictions cannot achieve long-distance displacement measurement. In addition, both of the above non-contact systems require markers to be installed on the structure as measurement points, so they also have the sparse measurement defects similar to contact systems. Radar can achieve dense measurement, but its cost and power consumption are very high, which is not suitable for practical applications in civil engineering.
[0004] Thanks to the progress of visual sensors, low-cost, accurate and relatively high-speed long-distance video recording is now possible. For example, video recording can be performed using a combination of a consumer camera or smartphone, a zoom lens, and a tripod, or even a drone. The resulting image sequence records the motion of all pixels in the field of view, which contains dense structural displacement information. Vision-based portable measurement systems are not only easy to operate, but also can quickly resume work after major disasters, obtain structural information, and ensure timely emergency response. In the past few decades, a series of visual tracking and conversion algorithms have been applied to civil engineering scenarios to achieve non-contact displacement measurement based on monitoring video data. The advantages of low cost, dense measurement, and easy operation and maintenance make vision-based systems an effective supplement to traditional displacement measurement systems. In stable imaging and monitoring environments, vision measurement systems are even a highly competitive alternative.
[0005] Vision-based displacement measurement includes two main steps: pixel motion estimation and motion-to-displacement conversion. Existing motion estimation algorithms are divided into two categories: non-iterative and iterative. Among them, non-iterative algorithms are suitable for quickly tracking the motion of specific pixels, and require additional calculations to achieve quasi-dense measurement and sub-pixel accuracy. The iterative algorithm is limited to estimating motion fields with small displacement amplitudes, and requires multi-layer image pyramid stacking and multiple iterations to improve measurement accuracy. At present, a single motion estimation algorithm cannot achieve fast estimation and dense estimation at the same time, which hinders the real-time identification of structural displacement and the maximum utilization of monitoring video data. In addition, existing methods also require manual adjustment of the hyperparameters of the measurement algorithm to cope with harsh environmental conditions. The above problems are very unfavorable for the application and automation of vision-based displacement measurement technology.
[0006] Currently, there is no research and application of visual displacement identification methods that consider both the speed and density of motion estimation in the field of structural health monitoring. The potential of using motion priors and physical knowledge implicit in monitoring data to achieve accurate and robust structural vibration displacement identification under low monitoring video quality has not been explored. Summary of the invention
[0007] The purpose of the present invention is to solve the problems in the prior art and to meet the actual detection needs. A structural vibration displacement identification method based on dense matching and prior knowledge enhancement is proposed.
[0008] The present invention is implemented by the following technical solution. The present invention proposes a structural vibration displacement identification method based on dense matching and prior knowledge enhancement, and the method comprises the following steps:
[0009] Step 1: Use simulation rendering technology to create a large dataset containing videos of diverse monitoring scenes and inter-frame optical flow for training the image feature enhancement network
[0010] Step 2: Establish a non-iterative optical flow estimation model based on dense matching: Select the initial network architecture hyperparameters and establish an image feature enhancement network Two related images in the input dataset (I 1 ,I 2 ), the same resolution feature (F 1 ,F 2 ); Image I 1 The position of all pixels in is recorded as P 1 , using the dense matching strategy based on the attention mechanism for non-iterative feature matching, we can get the pixel points in image I 2 The corresponding position P in 2 , and then get the initial inter-frame optical flow estimation result O = P 2 -P 1 ; Using the optical flow propagation strategy based on the attention mechanism, the initial optical flow is propagated to the occluded and out-of-boundary areas in the image that cannot be matched, and the optical flow estimation value O is obtained. * ; Compare the optical flow estimate with the true value in the dataset, and train the network through error back propagation and gradient descent method After that, hyperparameters are tuned to obtain an image feature enhancement network that has learned matching description capabilities. After that, its parameter θ is fixed and will not change;
[0011] Step 3: Extract the motion prior information implicit in the actual structural vibration monitoring video to improve the optical flow estimation result;
[0012] Step 4: Set a reference frame with a static structure in the monitoring video, take the static frame and the current frame as input, and calculate the full-field pixel motion between the two frames based on the optical flow estimation model in step 2 and the two types of motion prior constraints in step 3;
[0013] Step 5. Improve the displacement measurement results based on the physical knowledge of structural linear vibration theory; combine the vibration displacement time-history vectors of all measurement nodes into a displacement matrix; use singular value decomposition to approximate the modal decomposition of the structural vibration response, and approximate the low-rank properties by retaining only the main singular values to denoise the displacement measurement results and reduce the excessive error level caused by poor recording quality in some structural node areas in the monitoring video; use the main singular value components to recombine and obtain more accurate structural vibration displacement identification results.
[0014] Furthermore, the inter-frame optical flow is the full-field pixel motion occurring between two frames of video images.
[0015] Furthermore, in step three, on the one hand, a non-iterative motion estimation algorithm is used to quickly obtain an estimate of the upper bound of the motion amplitude of the structural nodes to be monitored, which is used as a motion prior to limit the scope of dense matching; on the other hand, the forward optical flow and backward optical flow between video frames are compared to estimate the occlusion and out-of-boundary areas in the monitored video, and the degree of overlap between these areas and the structural node areas is calculated as a motion prior to determine whether optical flow propagation is needed and to limit the scope of optical flow propagation.
[0016] Furthermore, in step four, the structural node area in which vibration displacement needs to be identified is framed in the video, and the homography matrix for converting the captured video view to the structural front view is obtained based on feature point matching and perspective transformation, and the scale factor for unit conversion is calculated in the front view; the homography matrix and the scale factor are used to realize the conversion of pixel motion to actual displacement, and the displacement of all pixels in the node area is averaged to obtain the node vibration displacement; the above process is repeated to obtain the node vibration displacement time history.
[0017] Furthermore, the step 2 is specifically as follows:
[0018] Step 2.1: Establish the initial image feature enhancement network To process two frames of images (I 1 ,I 2 ):
[0019]
[0020] Among them, the enhanced image features have the same spatial resolution as the original image;
[0021] Step 2.2: Image I 1 The position of all pixels in the 1 , the paired pixels are in image I 2 The corresponding position in is denoted as P 2 , based on the pixel-level feature description, a dense matching strategy based on the attention mechanism is used to achieve optical flow estimation:
[0022]
[0023] O=P 2 -P 1
[0024] The result of the optical flow estimation based on dense matching is O; the two spatial coordinate axes of the image feature matrix are merged into one, and the softmax function is used to normalize the feature matching transfer weights; the original positions of the pixels are recombined and converted to new positions, achieving sub-pixel precision optical flow estimation;
[0025] Step 2.3: Using input image features F 1The self-similarity of is used to propagate the optical flow estimation results of the successfully matched area to the unmatched area; the optical flow propagation strategy is implemented based on the attention mechanism:
[0026]
[0027] Finally, the optical flow estimation result O after considering the propagation strategy is obtained. * ;
[0028] Step 2.4: Compare the optical flow estimation results with the true values in the data set, establish the objective function, train through error back propagation and gradient descent method, perform hyperparameter tuning, and obtain an image feature enhancement network that has learned matching description capabilities. Its parameter θ is then fixed.
[0029] Furthermore, in step three, a non-iterative motion estimation algorithm is used to quickly obtain an estimate of the upper bound of the motion amplitude of the structure node to be monitored, which is used as a motion prior to limit the range of dense matching for each pixel:
[0030]
[0031] The mask function changes with the pixel position, and resets the feature transfer weights outside the matching area of the motion prior constraint to zero, thereby shielding information that is not conducive to accurate matching at the pixel level.
[0032] Furthermore, in step 3, the input order of the two frames is exchanged to obtain the reverse optical flow, and the forward optical flow and backward optical flow between the video frames are compared to estimate the occlusion and out-of-bounds areas in the monitoring video, and the overlap between these areas and the structural node areas is calculated as a motion prior to limit the range of optical flow propagation for each pixel:
[0033]
[0034] Among them, the mask function changes with the pixel position, resets the feature transfer weight outside the propagation area of the motion prior constraint to zero, and realizes pixel-level shielding of information that is not conducive to accurate propagation.
[0035] Furthermore, the step five is specifically as follows:
[0036] Step 5.1: The vibration displacement time history vector of each node identified in the video is recorded as d i (i=1,…,M), and form the structural displacement matrix D=[d 1 …d i …d M ]; According to the modal superposition principle in the theory of structural linear vibration, the modal decomposition of the structural vibration response can be approximated using singular value decomposition:
[0037] D=U·diag(s)·V T
[0038] Among them, U and V are singular vector matrices, and s approximately represents the amplitude of each modal component in the structural displacement response;
[0039] Step 5.2: According to physical knowledge, the structural displacement matrix D has a low-rank property; calculate the ratio of adjacent order singular values in s, set a threshold, set the singular values that are too small to zero, and obtain a new singular value vector s * , recombined to get:
[0040] D * =U·diag(s * )·V T
[0041] Among them, D * This is the structural vibration displacement identification result after denoising.
[0042] The present invention also proposes an electronic device, comprising a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the steps of the structural vibration displacement identification method based on dense matching and prior knowledge enhancement are implemented.
[0043] The present invention also proposes a computer-readable storage medium for storing computer instructions, which, when executed by a processor, implement the steps of the structural vibration displacement identification method based on dense matching and prior knowledge enhancement.
[0044] Beneficial effects of the present invention:
[0045] 1. Compared with the existing methods, the non-iterative method based on dense matching described in the present invention obtains the estimation result of the full-field pixel motion through a single-step calculation based on the matching strategy of the attention mechanism, which solves the problem that the existing methods cannot simultaneously realize fast displacement recognition and dense displacement recognition, making the method more suitable for application in structural health monitoring systems.
[0046] 2. The optical flow propagation strategy of the method of the present invention can use the image peripheral information to restore the pixel motion of the occluded and out-of-boundary areas in the monitoring video, so that the method can still calculate relatively accurate displacement recognition results when the monitoring area is partially occluded; in addition, the method of the present invention uses a deep learning model to enhance image features, and uses the enhanced image features to perform pixel motion estimation, so that the method has better robustness under harsh environmental conditions and is more suitable for practical applications.
[0047] 3. The method described in the present invention utilizes the prior knowledge of the motion of the monitoring area implicit in the video data, and constrains the range of dense matching and optical flow propagation based on the prior knowledge, thereby further improving the accuracy of displacement recognition.
[0048] 4. The present invention utilizes the physical prior knowledge that structural vibration response is usually dominated by a small number of modes. Based on the approximate relationship between modal decomposition and singular value decomposition of structural vibration response in linear vibration theory, the displacement identification result is denoised by retaining only the main singular value components, thereby improving the accuracy of structural vibration displacement identification, so that the method can resist the high measurement noise in some structural node areas caused by poor monitoring video quality. BRIEF DESCRIPTION OF THE DRAWINGS
[0049] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.
[0050] Figure 1 The flowchart of the structural vibration displacement identification method based on dense matching and prior knowledge enhancement of the present invention.
[0051] Figure 2 Schematic diagram of the optical flow estimation method.
[0052] Figure 3 This is a schematic diagram of a monitoring video used as an example in the present invention, in which there are 16 structural nodes.
[0053] Figure 4 It is a schematic diagram of the full-field pixel motion (ie, optical flow) estimation results of the method of the present invention when the monitoring node 5 is partially blocked and under random lighting conditions.
[0054] Figure 5 This is a schematic diagram of the comparison of the error between the structural vibration displacement identification result and the true value before and after using motion prior knowledge to constrain the optical flow estimation process. The error shown is the average error of 16 monitoring nodes.
[0055] Figure 6 This is a schematic diagram of the comparison between the structural vibration displacement identification result and the true value before and after the displacement identification result is denoised using physical prior knowledge. The corresponding result of monitoring node 9 is shown. DETAILED DESCRIPTION
[0056] The technical solutions in the embodiments of the present invention will be described clearly and completely below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0057] In order to solve the problem of fast, dense and robust visual-based structural displacement recognition, the present invention implements a structural vibration displacement recognition method based on dense matching and prior knowledge enhancement. First, the method trains a deep learning model to enhance image features, and uses a dense matching method based on an attention mechanism to obtain the full-field pixel motion of the monitoring video. The method is non-iterative, but achieves fast and dense motion estimation, solving the problems of existing algorithms. In addition, the enhanced image features are more robust to harsh environmental conditions. Further, combined with a method for converting pixel motion in all structural node areas into structural displacements at one time, the present invention quickly and robustly measures dense structural vibration displacements. The present invention also proposes a strategy based on motion prior and physical knowledge enhancement to improve the measurement results. Finally, the method of the present invention achieves accurate structural vibration displacement recognition. In addition to selecting the structural node area for displacement conversion, the method of the present invention does not require additional manual intervention, which is conducive to promoting the realization of automated structural vibration displacement recognition.
[0058] The purpose of the present invention is to solve the problem that the existing vision-based structural vibration displacement recognition method cannot realize fast displacement recognition and dense displacement recognition at the same time, and propose a non-iterative method based on dense matching, which can estimate the full-field pixel motion only through single-step non-iterative matching, thereby realizing fast and dense displacement recognition. In addition, when the monitoring environment conditions are relatively harsh (for example, unstable lighting), or the target monitoring area in the structural vibration video data is partially obscured, the method of the present invention uses image features enhanced by deep learning models for recognition, and uses optical flow propagation strategies to estimate pixel motion in the obscured area, which has a wider range of applicability in actual engineering. The present invention also proposes to use prior knowledge to improve displacement recognition results. For example Figure 2 The process principle of optical flow estimation, structural vibration displacement identification and prior knowledge constraint is shown. For the video data in the actual structural health monitoring system, real-time and comprehensive analysis of structural state information is required, so a method for quickly identifying dense structural vibration displacement is needed. At the same time, due to the high variability of the outdoor environment, the displacement identification method needs to be highly robust, so it is suitable for the method of the present invention.
[0059] Combination Figure 1 The present invention proposes a structural vibration displacement identification method based on dense matching and prior knowledge enhancement, which specifically includes the following steps:
[0060] Step 1: Use simulation rendering technology to create a large dataset containing videos of diverse monitoring scenes and the true value of inter-frame optical flow (i.e., the full-field pixel motion between two frames of video images) for training the image feature enhancement network.
[0061] Step 2: Establish a non-iterative optical flow estimation model based on dense matching. Select the initial network architecture hyperparameters and establish an image feature enhancement network. Two related images in the input dataset (I 1 ,I 2 ), the same resolution feature (F 1 ,F 2 ). Image I 1 The position of all pixels in is recorded as P 1 , using the dense matching strategy based on the attention mechanism for non-iterative feature matching, we can get the pixel points in image I 2 The corresponding position P in 2 , and then get the initial inter-frame optical flow estimation result O = P 2 -P 1 Using the optical flow propagation strategy based on the attention mechanism, the initial optical flow is propagated to the occluded and out-of-boundary areas in the image that cannot be matched, and the optical flow estimation value O is obtained. * The estimated optical flow value is compared with the true value in the dataset, and the network is trained by error back propagation and gradient descent method. Perform hyperparameter tuning to obtain an image feature enhancement network that has learned matching description capabilities After that, its parameter θ is fixed and will not change;
[0062] Step three: extract the motion prior information implicit in the actual structural vibration monitoring video to improve the optical flow estimation results. First, use the existing non-iterative motion estimation algorithm to quickly obtain an estimate of the upper bound of the motion amplitude of the structural nodes to be monitored, as a motion prior, to limit the scope of dense matching; second, compare the forward optical flow and backward optical flow between adjacent video image frames, estimate the occlusion and out-of-boundary areas in the monitoring video, calculate the degree of overlap between these areas and the structural node areas, as a motion prior, determine whether optical flow propagation is needed, and limit the scope of optical flow propagation;
[0063] Step 4. Set a reference frame in which the structure in the monitoring video is still, and use the still frame and the current frame as input. Based on the optical flow estimation model in step 2 and the two types of motion prior constraints in step 3, calculate the full-field pixel motion between the two frames. Select the structural node area in the video where vibration displacement needs to be identified, and based on existing technologies such as feature point matching and perspective transformation, obtain the homography matrix for converting the captured video view to the structural front view, and calculate the scale factor for unit conversion in the front view. The homography matrix and scale factor are used to realize the conversion of pixel motion to actual displacement, and the displacement of all pixels in the node area is averaged to obtain the node vibration displacement. Repeat the above process to obtain the node vibration displacement time history;
[0064] Step 5. Improve the displacement measurement results based on the physical knowledge of structural linear vibration theory. The vibration displacement time-history vectors of all measurement nodes are combined into a displacement matrix. According to physical knowledge, this matrix generally has a low-rank property. Use singular value decomposition to approximate the modal decomposition of the structural vibration response. By retaining only the main singular values to approximate the low-rank property, the displacement measurement results are denoised to reduce the excessive error level caused by poor recording quality in some structural node areas in the monitoring video. Use the main singular value components to recombine to obtain more accurate structural vibration displacement identification results.
[0065] For the optical flow estimation process, the step 2 is specifically as follows:
[0066] Step 2.1: Highly discriminative pixel-level feature description is the key to the success of the optical flow estimation method based on dense matching in the present invention. Establishing the initial image feature enhancement network To process two frames of images (I 1 ,I 2 ):
[0067]
[0068] Among them, the enhanced image features have the same spatial resolution as the original image;
[0069] Step 2.2: The optical flow estimation problem is essentially to find the positions of all pairs of pixels between two frames of images. 1 The position of all pixels in the 1 , the paired pixels are in image I 2 The corresponding position in is denoted as P 2 , based on the reliable pixel-level feature description, a dense matching strategy based on the attention mechanism is used to achieve optical flow estimation:
[0070]
[0071] O=p 2 -p1
[0072] The result of the optical flow estimation based on dense matching is O. In order to facilitate matrix multiplication, the two spatial coordinate axes of the image feature matrix are merged into one. The softmax function is used to normalize the feature matching transfer weights. Considering the correlation of image features, the original positions of the pixels are recombined and converted to new positions, and optical flow estimation with sub-pixel accuracy is achieved. This dense matching process only needs to be performed once, so the method of the present invention is non-iterative and achieves fast optical flow estimation;
[0073] Step 2.3: Considering that there may be occluded or out-of-bounds pixel areas in the actual monitoring video, this will lead to matching failure. 1 The self-similarity of is used to propagate the optical flow estimation results of the successfully matched area to the unmatched area. Similarly, the above optical flow propagation strategy is implemented based on the attention mechanism:
[0074]
[0075] Finally, the estimated result O after considering the optical flow propagation strategy is obtained. * ;
[0076] Step 2.4: The above feature enhancement, optical flow estimation, and optical flow propagation processes are all differentiable. The optical flow estimation results are compared with the true values in the data set, and the objective function is established. The error back propagation and gradient descent method are used for training, and hyperparameters are tuned to obtain an image feature enhancement network that has learned matching description capabilities. Its parameter θ is then fixed.
[0077] The matching strategy in step 2.2 and the propagation strategy in step 2.3 both take into account the global pixel features. However, the global features may contain local information that is not conducive to the motion estimation of each pixel. The motion prior knowledge can be used to limit the scope of the attention mechanism and improve the optical flow estimation result. The specific step 3 is as follows:
[0078] Step 3.1: Use the existing non-iterative motion estimation algorithm to quickly obtain an estimate of the upper bound of the motion amplitude of the structural nodes to be monitored, as a motion prior, to limit the range of dense matching for each pixel:
[0079]
[0080] The mask function changes with the pixel position, resets the feature transfer weight outside the matching area to zero, and achieves pixel-level shielding of information that is not conducive to accurate matching.
[0081] Step 3.2: Exchange the input order of the two frames to obtain the reverse optical flow, compare the forward optical flow and the backward optical flow between the video frames, estimate the occlusion and out-of-boundary areas in the monitoring video, calculate the overlap between these areas and the structural node areas, and use them as motion priors to limit the range of optical flow propagation for each pixel:
[0082]
[0083] Among them, the mask function changes with the pixel position, resets the feature transfer weight outside the propagation area to zero, and realizes pixel-level shielding of information that is not conducive to accurate propagation.
[0084] Considering that the shooting quality of different areas in the same monitoring video may not be uniform, resulting in a relatively high error level, the displacement measurement result obtained in step 4 is improved by using physical prior knowledge. The specific step 5 is as follows:
[0085] Step 5.1: The vibration displacement time history vector of each node identified in the video is recorded as d i (i=1,…,M), and form the structural displacement matrix D=[d 1 …d i …d M ]. According to the modal superposition principle in the theory of structural linear vibration, the modal decomposition of the structural vibration response can be approximated using singular value decomposition:
[0086] D=U·diag(s)·V T
[0087] Among them, U and V are singular vector matrices, and s approximately represents the amplitude of each modal component in the structural displacement response;
[0088] Step 5.2: According to physical knowledge, the structural vibration response is usually dominated by a small number of modes, that is, the structural displacement matrix D has a low-rank property. This physical prior knowledge constraint can be imposed by retaining only the main singular value components to remove noise in the visual displacement recognition results. Calculate the ratio of adjacent order singular values in s, set a threshold, set the singular values that are too small to zero, and obtain a new singular value vector s * , recombined to get:
[0089] D * =U·diag(s * )·V T
[0090] Among them, D * This is the structural vibration displacement identification result after denoising.
[0091] The present invention proposes a structural vibration displacement identification method based on dense matching and prior knowledge enhancement, the method includes the architecture design and training of an image feature enhancement network, the establishment of a non-iterative optical flow estimation model based on dense matching, the improvement of optical flow estimation results based on motion prior information in monitoring videos, the conversion of pixel motion to structural displacement, the improvement of displacement identification results based on physical prior knowledge, etc. The method of the present invention uses a trained deep learning model to enhance image features, and uses dense matching and optical flow propagation strategies based on an attention mechanism to obtain full-field pixel motion of monitoring videos. The method realizes fast and dense motion estimation and solves the problems of existing methods. In addition, the enhanced image features are more robust to harsh environmental conditions and have a wider range of applications. Combined with a method for converting pixel motion in all structural node areas into structural displacement at one time, the present invention quickly and robustly identifies dense structural vibration displacements in monitoring videos. The present invention also proposes a strategy based on motion prior and physical knowledge enhancement to improve the identification results, and realizes accurate structural vibration displacement identification. The method has unique advantages in terms of accuracy, density and speed in identifying structural vibration displacements under low monitoring video quality, and is also highly robust to harsh environmental conditions.
[0092] Example
[0093] Figure 3 The following is a schematic diagram of the monitoring video used in this example. There are 16 structural nodes in the video, and the structure mainly vibrates vertically. The resolution of the video is 1920 pixels × 1080 pixels, the shooting time and frequency are 10 seconds and 120Hz respectively, and the video shooting angle is perpendicular to the plane of structural movement. Figure 4 , Figure 5 , Figure 6 , the structural vibration displacement is identified by using the method based on dense matching and prior knowledge enhancement of the present invention.
[0094] The first step is to use simulation rendering software such as Blender to create a large dataset of videos containing diverse scenes (different ambient lighting, resolution, etc.) and true values of inter-frame optical flow for training the image feature enhancement network.
[0095] The second step is to build an image feature enhancement network based on the Transformer architecture in order to capture as much information as possible in the input image data that is helpful for dense matching. It consists of three parts: a self-attention layer applied to the first input, a mutual attention layer applied to the first and second inputs, and a fully connected layer. 1 ,I 2 ), the same resolution feature (F 1 ,F2 ). Based on image features (F 1 ,F 2 ), using dense matching and optical flow propagation strategy to estimate the optical flow between image frames. Using the one-norm error between the estimated optical flow value and the true value as the objective function, using the Adam optimization algorithm, initialize various training parameters (batch size, number of training rounds, learning rate, etc.), and train the network And perform training hyperparameter tuning to obtain an image feature enhancement network that effectively learns matching description capabilities Fixed its network parameters and no longer changes.
[0096] After obtaining the optical flow estimation model, Figure 4 In the example, partial occlusion is artificially applied to the video area where the monitoring node 5 is located, the brightness of each frame of the monitoring video is randomly adjusted, and the full-field pixel motion estimation is performed using the method of the present invention. The estimated full-field pixel motion is visualized as an optical flow image, where the color represents the motion direction and the saturation represents the motion amplitude. Figure 4 The good optical flow estimation results shown indicate that the method of the present invention is extremely robust to adverse environmental conditions.
[0097] The step three is specifically as follows: on the one hand, the KLT feature tracking algorithm is used to quickly obtain the upper limit of the motion amplitude of the target structure node as 6 pixels, which is used as a motion prior to limit the matching range radius of each pixel point to 6 pixels; on the other hand, the forward optical flow and the backward optical flow between adjacent video image frames are compared, and it is observed that the monitoring video used does not have occlusion and boundary problems. It is used as a motion prior, the optical flow propagation step is removed, and the optical flow estimation result is improved.
[0098] The specific steps of step four are as follows: the structure in the monitoring video is in a stationary state in the first frame, the first frame is used together with the current frame as input, and the full-field pixel motion between the two frames is calculated based on steps two and three. Since the monitoring video used is shot in a front view, the scale factor for converting pixel length to displacement can be directly calculated based on the known structural dimensions. Select the structural node area in the video where vibration displacement needs to be identified, multiply the motion estimation results of all pixels in the node area by the scale factor and average them to obtain the node vibration displacement. Repeat the above process to identify the vibration displacement time history of 16 nodes;
[0099] exist Figure 5 In the present invention, after applying the two types of motion prior constraints in step three, the error of identifying the structural vibration displacement by the method of the present invention is significantly reduced, which verifies the effectiveness of the strategy proposed in the present invention.
[0100] The step 5 is specifically as follows: the video used is composed of 1200 frames of images, and the displacement time-history vectors identified by each of the 16 structural nodes are combined into a 16-dimensional × 1200-dimensional structural displacement matrix. Singular value decomposition is performed, and it is observed that the main singular value components of the matrix are the first three orders. The first three order singular values are retained for noise reduction to obtain the final structural vibration displacement identification result.
[0101] exist Figure 6 In the present invention, after applying the physical prior constraints in step five, the vibration displacement time history identification result of monitoring node 9 is obviously closer to the true value in both time domain and frequency domain, which verifies the effectiveness of the strategy proposed in the present invention.
[0102] Combined with the above results, it can be seen that the method of the present invention can quickly and accurately identify dense structural vibration displacements from monitoring videos, and the identification results have good robustness to problems such as harsh environmental conditions, occlusion, and video recording noise.
[0103] The present invention also proposes an electronic device, comprising a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the steps of the structural vibration displacement identification method based on dense matching and prior knowledge enhancement are implemented.
[0104] The present invention also proposes a computer-readable storage medium for storing computer instructions, which, when executed by a processor, implement the steps of the structural vibration displacement identification method based on dense matching and prior knowledge enhancement.
[0105] The memory in the embodiments of the present application may be a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of RAM are available, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), and direct RAM bus RAM (DRRAM). It should be noted that the memory of the method described in the present invention is intended to include, but is not limited to, these and any other suitable types of memory.
[0106] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium, for example, the computer instructions may be transmitted from a website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (digital subscriber line, DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, server or data center. The computer-readable storage medium may be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrated. The available medium may be a magnetic medium (eg, a floppy disk, a hard disk, a magnetic tape), an optical medium (eg, a high-density digital video disc (DVD)), or a semiconductor medium (eg, a solid state disc (SSD)).
[0107] In the implementation process, each step of the above method can be completed by an integrated logic circuit of hardware in a processor or an instruction in the form of software. The steps of the method disclosed in conjunction with the embodiment of the present application can be directly embodied as a hardware processor for execution, or a combination of hardware and software modules in a processor for execution. The software module can be located in a storage medium mature in the art such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory or an electrically erasable programmable memory, a register, etc. The storage medium is located in a memory, and the processor reads the information in the memory and completes the steps of the above method in conjunction with its hardware. To avoid repetition, it is not described in detail here.
[0108] It should be noted that the processor in the embodiment of the present application can be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method embodiment can be completed by an integrated logic circuit of hardware in the processor or an instruction in the form of software. The above processor can be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The methods, steps and logic block diagrams disclosed in the embodiments of the present application can be implemented or executed. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc. The steps of the method disclosed in the embodiment of the present application can be directly embodied as a hardware decoding processor to perform, or the hardware and software modules in the decoding processor can be combined and performed. The software module can be located in a mature storage medium in the field such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory or an electrically erasable programmable memory, a register, etc. The storage medium is located in a memory, and the processor reads the information in the memory and completes the steps of the above method in combination with its hardware.
[0109] The above is a detailed introduction to the structural vibration displacement identification method based on dense matching and prior knowledge enhancement proposed in the present invention. In this article, specific examples are used to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea. At the same time, for those skilled in the art, according to the idea of the present invention, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present invention.
Claims
1. A structural vibration displacement identification method based on dense matching and prior knowledge enhancement, It is characterized in that The method comprises the following steps: Step 1: Use simulation rendering technology to create a large dataset containing videos of diverse monitoring scenes and inter-frame optical flow for training the image feature enhancement network Step 2: Establish a non-iterative optical flow estimation model based on dense matching: Select the initial network architecture hyperparameters and establish an image feature enhancement network Two related images in the input dataset (I 1 ,I 2 ), the same resolution feature (F 1 ,F 2 );Image I 1 The position of all pixels in the 1 , using the dense matching strategy based on the attention mechanism for non-iterative feature matching, we can get the pixel points in image I 2 The corresponding position P in 2 , and then get the initial inter-frame optical flow estimation result O = P 2 -P 1 ; Using the optical flow propagation strategy based on the attention mechanism, the initial optical flow is propagated to the occluded and out-of-boundary areas in the image that cannot be matched, and the optical flow estimation value O is obtained. * ; Compare the optical flow estimate with the true value in the dataset, and train the network through error back propagation and gradient descent method After that, hyperparameters are tuned to obtain an image feature enhancement network that has learned matching description capabilities. After that, its parameter θ is fixed and will not change; Step 3: Extract the motion prior information implicit in the actual structural vibration monitoring video to improve the optical flow estimation result; Step 4: Set a reference frame with a static structure in the monitoring video, take the static frame and the current frame as input, and calculate the full-field pixel motion between the two frames based on the optical flow estimation model in step 2 and the two types of motion prior constraints in step 3; Step 5. Improve the displacement measurement results based on the physical knowledge of structural linear vibration theory; combine the vibration displacement time-history vectors of all measurement nodes into a displacement matrix; use singular value decomposition to approximate the modal decomposition of the structural vibration response, and approximate the low-rank properties by retaining only the main singular values to reduce the noise of the displacement measurement results and reduce the excessive error level caused by poor recording quality in some structural node areas in the monitoring video; use the main singular value components to recombine and obtain more accurate structural vibration displacement identification results.
2. The method according to claim 1, It is characterized in that The inter-frame optical flow is the full-field pixel motion occurring between two video frames.
3. The method according to claim 1, It is characterized in that In step three, on the one hand, a non-iterative motion estimation algorithm is used to quickly obtain an estimate of the upper bound of the motion amplitude of the structural nodes to be monitored, which is used as a motion prior to limit the scope of dense matching. On the other hand, the forward optical flow and backward optical flow between video frames are compared to estimate the occlusion and out-of-bounds areas in the monitoring video, and the degree of overlap between these areas and the structural node areas is calculated as a motion prior to determine whether optical flow propagation is needed and to limit the scope of optical flow propagation.
4. The method according to claim 1, It is characterized in that In step 4, the structural node area where vibration displacement needs to be identified is framed in the video, and the homography matrix for converting the captured video view to the structural front view is obtained based on feature point matching and perspective transformation, and the scale factor for unit conversion is calculated in the front view; The homography matrix and scale factor are used to realize the conversion of pixel motion to actual displacement, and the displacement of all pixels in the node area is averaged to obtain the node vibration displacement. Repeat the above process to obtain the node vibration displacement time history.
5. The method according to claim 1, It is characterized in that The step 2 is specifically as follows: Step 2.1: Establish the initial image feature enhancement network To process two frames of images (I 1 ,I 2 ): Among them, the enhanced image features have the same spatial resolution as the original image; Step 2.2: Image I 1 The position of all pixels in the 1 , the paired pixels are in image I 2 The corresponding position in is denoted as P 2 , based on the pixel-level feature description, a dense matching strategy based on the attention mechanism is used to achieve optical flow estimation: O=p 2 -p 1 The result of the optical flow estimation based on dense matching is O; the two spatial coordinate axes of the image feature matrix are merged into one, and the softmax function is used to normalize the feature matching transfer weights; the original positions of the pixels are recombined and converted to new positions, achieving sub-pixel precision optical flow estimation; Step 2.3: Using input image features F 1 The self-similarity of is used to propagate the optical flow estimation results of the successfully matched area to the unmatched area; the optical flow propagation strategy is implemented based on the attention mechanism: Finally, the optical flow estimation result O after considering the propagation strategy is obtained. * ; Step 2.4: Compare the optical flow estimation results with the true values in the data set, establish the objective function, train through error back propagation and gradient descent method, perform hyperparameter tuning, and obtain an image feature enhancement network that has learned matching description capabilities. Its parameter θ is then fixed.
6. The method according to claim 3, It is characterized in that In step 3, a non-iterative motion estimation algorithm is used to quickly obtain an estimate of the upper bound of the motion amplitude of the structure node to be monitored, which is used as a motion prior to limit the range of dense matching for each pixel: The mask function changes with the pixel position, and resets the feature transfer weights outside the matching area of the motion prior constraint to zero, thereby shielding information that is not conducive to accurate matching at the pixel level.
7. The method according to claim 3, It is characterized in that In step 3, the input order of the two frames is exchanged to obtain the reverse optical flow. The forward optical flow and the backward optical flow between the video frames are compared to estimate the occlusion and out-of-bounds areas in the monitoring video. The overlap between these areas and the structural node areas is calculated as a motion prior to limit the range of optical flow propagation for each pixel: Among them, the mask function changes with the pixel position, resets the feature transfer weight outside the propagation area of the motion prior constraint to zero, and realizes pixel-level shielding of information that is not conducive to accurate propagation.
8. The method according to claim 1, It is characterized in that The step five is specifically as follows: Step 5.1: The vibration displacement time history vector of each node identified in the video is recorded as d i (i=1,…,M), and form the structural displacement matrix D=[d 1 …d i …d M ]; According to the modal superposition principle in the theory of structural linear vibration, the modal decomposition of the structural vibration response can be approximated using singular value decomposition: D=U·diag(s)·V T Among them, U and V are singular vector matrices, and s approximately represents the amplitude of each modal component in the structural displacement response; Step 5.2: According to physical knowledge, the structural displacement matrix D has a low-rank property; calculate the ratio of adjacent order singular values in s, set a threshold, set the singular values that are too small to zero, and obtain a new singular value vector s * , recombined to get: D * =U·diag(s * )·V T Among them, D * This is the structural vibration displacement identification result after denoising.
9. An electronic device comprising a memory and a processor, wherein the memory stores a computer program. It is characterized in that When the processor executes the computer program, the steps of the method according to any one of claims 1 to 8 are implemented.
10. A computer-readable storage medium for storing computer instructions, It is characterized in that When the computer instructions are executed by a processor, the steps of the method according to any one of claims 1 to 8 are implemented.
Citation Information
Patent Citations
Structural modal parameter identification method and device, computer equipment and storage medium
CN113901920A
Structural vibration displacement identification method and system based on deep recurrent neural network optical flow estimation model
CN114485417A
Dynamic SLAM dense map construction method based on pyramid L-K optical flow and multi-view geometric constraint
CN116878524A
Monocular unsupervised depth estimation method based on contextual attention mechanism
US20210390723A1
Obstacle recognition method for autonomous robots
US20220066456A1
Cited By
Structural displacement non-contact sensing method, device and equipment and storage medium
CN119323752A
