Structural displacement visual identification method and system fused with three-dimensional deformable model

Abstract

The invention discloses a structure displacement visual identification method and system fused with a three-dimensional deformable model, and the method comprises the steps: collecting the vibration video data of an engineering structure through a consumer-level monocular camera, building a three-dimensional deformable grid model, and generating a pose rendering image and displacement value data; removing environmental noise interference; estimating dense optical flow data in the rendered image through an optical flow model, and constructing an optical flow feature data set containing pose information; inputting an integral structure image sequence and a local structure image sequence of the two groups of data sequences, and generating corresponding integral features and local features respectively after multi-layer convolution operation of a convolution layer; and generating a feature query vector, a key vector and a value vector through linear transformation, establishing a dynamic weight between two features according to the vectors, performing global average pooling after two fusion features are obtained, and outputting a pose estimation value through a multi-layer perceptron. The displacement of the three-dimensional structure can be accurately identified only through the monocular consumer camera.

Description

Technical Field

[0001] The present invention belongs to the field of structural health monitoring, and specifically relates to a method and system for visual recognition of structural displacements that integrates a three-dimensional deformable model. The method uses computer vision technology and deep learning algorithms, uses a consumer-grade monocular camera, and combines three-dimensional deformable grid modeling and dense optical flow estimation to achieve accurate recognition of tiny displacements of buildings, and is suitable for health monitoring and vibration monitoring of high-rise engineering structures. Background Art

[0002] Structural deformation measurement can detect abnormal deformation of structures in time, prevent structural failure and collapse, and thus ensure the safety and stability of engineering structures. For example, engineering structures such as buildings and bridges are subject to various external forces and environmental effects during their service, such as earthquakes, wind, vehicle loads, temperature and humidity changes, etc. These loads and environmental effects will cause structural deformation. If the deformation exceeds the allowable range of the structural design, it may cause the engineering structure to fail or even collapse and damage, causing serious casualties and property losses. Through the deformation monitoring system, structural deformation can be monitored in real time and early warning can be issued in time to ensure the safety and stability of engineering structures and prevent disasters.

[0003] At present, structural health monitoring technology mainly relies on physical sensors such as strain gauges, accelerometers, laser displacement sensors, GPS, etc. These sensors can provide high-precision displacement and vibration data, but they have high costs in installation and maintenance, especially in large-scale engineering structures or infrastructure such as bridges, the economic cost of deploying these sensors is relatively high. In addition, these sensor systems usually require complex wiring, calibration and maintenance, especially during long-term monitoring, they are easily affected by environmental factors such as temperature and humidity, thus affecting the measurement accuracy and long-term stability of data. Traditional visual sensing technology, such as displacement monitoring through high-end industrial cameras or lidar, although it can provide non-contact measurement methods, has high equipment costs, complex data processing, and is difficult to promote and apply in the health monitoring of large-scale structures. This has prompted researchers to explore the use of relatively inexpensive and easy-to-deploy consumer-grade visual sensing devices, such as monocular cameras or mobile phone cameras, to monitor and analyze engineering structure displacement. Displacement monitoring methods based on monocular cameras have attracted widespread attention in recent years. Monocular cameras have the advantages of low price and easy installation, and can perform non-contact visual monitoring without interfering with the normal use of buildings. However, monocular cameras are subject to imaging principles and hardware limitations, and their application in structural health monitoring still faces multiple technical challenges. First, the imaging quality of consumer-grade monocular cameras is lower than that of industrial cameras. Limited by resolution and focal length, it is difficult to accurately capture the tiny displacements of engineering structures. Second, the monocular vision system lacks depth information and needs to use additional algorithms to compensate for its ability to perceive structural movement in three-dimensional space, which increases the complexity of data processing.

[0004] At present, there are still some shortcomings in the research on displacement recognition based on monocular consumer cameras in structural health monitoring. First, the existing consumer-grade monocular cameras have low pixels and the quality of the captured videos is poor, which limits the accuracy and reliability of displacement recognition. Secondly, displacement recognition based on computer vision is easily interfered by factors such as background, which introduces a lot of noise into the image and affects the recognition accuracy. In addition, the contrast of consumer-grade cameras is limited. In environments with drastic changes in weather or lighting, especially in high-contrast scenes, the image quality will be significantly reduced, resulting in the loss of some displacement information and affecting the recognition effect. Therefore, in order to solve the above problems, this paper intends to make full use of the limited viewing angle of the monocular camera based on monocular cameras and deep learning technology, establish a three-dimensional deformable grid model, perform dense optical flow estimation, and generate masks in conjunction with semantic segmentation to control the impact of background, weather and lighting on image quality. Using deep learning neural networks for pose estimation, the cross-attention mechanism applied to medical images is innovatively transferred to the structural dense displacement estimation method to achieve accurate displacement recognition of specific engineering structures, and use consumer-grade monocular cameras to achieve high-precision monitoring effects.

[0005] In summary, although monocular cameras have broad application prospects in the field of structural health monitoring, they still face problems such as poor imaging quality, large environmental interference, and insufficient recognition accuracy in the recognition of micro-displacements. The present invention is proposed based on the solution of these problems, aiming to significantly improve the recognition accuracy of micro-displacements based on consumer-grade monocular cameras through advanced technical means such as deep learning, three-dimensional modeling, and dense optical flow estimation, and provide a low-cost, efficient, and easy-to-deploy solution for the health monitoring of engineering structures. Summary of the invention

[0006] The purpose of the present invention is to address the limitations of the existing structural health monitoring (SHM) system, which is due to the inability of monocular devices to recognize three-dimensional structures, the high cost of high-precision cameras, the low monitoring accuracy of consumer-grade cameras, and the interference of the environment (background, weather, light, etc.). In particular, in terms of micro-displacement recognition, the existing methods are difficult to meet the actual needs. In order to overcome these technical challenges, a method for identifying displacement of engineering structures based on a monocular consumer-grade camera is proposed. This method combines deep learning technology, three-dimensional deformable grid modeling, semantic segmentation, dense optical flow estimation and cross-attention mechanism, uses a monocular consumer-grade camera to identify three-dimensional structural displacement, overcomes problems such as image distortion and environmental interference, and significantly improves the accuracy of displacement recognition, especially for the recognition of micro-displacements of engineering structures. It has high sensitivity and accuracy.

[0007] In view of the above defects or improvement needs of the prior art, as a first aspect of the present invention, the present invention provides a method for visually recognizing structural displacement by integrating a three-dimensional deformable model, comprising: S1. Collect engineering structure vibration video data through a consumer-grade monocular camera, build a 3D deformable mesh model and generate pose rendering images and displacement value data; S2. Remove the interference of environmental noise on displacement recognition; S3. Estimating dense optical flow data in the rendered image through the optical flow model, and constructing an optical flow feature dataset containing pose information; S4. A structured dense displacement estimation method based on a cross-attention mechanism outputs the model displacement prediction value by fusing local and global features. The specific steps are as follows: Input two sets of data sequences: the overall structure image sequence and the local structure image sequence. After multi-layer convolution operations of the convolution layer, the corresponding overall features are generated respectively. With local features ; Overall characteristics and local features After linear transformation, the feature query vector, key vector and value vector are generated, and dynamic weights are established between the two features based on these vectors to obtain two fusion features. as well as ; Determine the loss function, perform global average pooling on the two fusion features obtained, and then output the pose estimation value through a multi-layer perceptron.

[0008] Furthermore, the displacement value in step S1 is randomly generated and obeys uniform distribution, and the extreme value of the displacement value can be adjusted according to actual engineering project needs and experimental requirements.

[0009] Furthermore, the specific method for removing the interference of environmental noise on displacement recognition in step S2 is: The semantic segmentation mask of the engineering structure is generated through the semantic segmentation model to remove the interference of environmental noise on displacement recognition. The calculation formula is:

[0010] Apply Mask For images, the calculation formula is:

[0011] In the formula, is the original image, is the structural area at time 0 and 0 displacement state, is the denoised image.

[0012] Furthermore, the specific method of step S3 includes: Estimate dense optical flow data in rendered images through optical flow models; From the overall displacement optical flow data The most significant local feature area is extracted, and then enlarged and preprocessed to obtain local displacement optical flow data ; Obtain the displacement optical flow data after fusion of local features and overall features .

[0013] Furthermore, the dense optical flow data in the rendered image is estimated by the optical flow model, and the specific method is as follows: For two frames of images at time 0 and time t, the calculation formula is:

[0014] In the formula, represents the image at time 0, express images of moments; Respectively represent the optical flow at time 0 and The weight in direction, Respectively The light flows at all times and Directional weight; According to Taylor expansion, exist Performing a first-order expansion at , we can obtain:

[0015] Taylor expansion of the equation and ignoring higher-order terms gives the optical flow constraint equation:

[0016] In the formula, , Denotes the optical flow in and Directional weight; Calculate the 0th and 1st frames The optical flow between frames, frame 0 and frame Average optical flow between frames , which can be expressed as:

[0017] Substituting this definition into the optical flow constraint equation, the optical flow expression at multi-frame intervals is obtained as follows:

[0018] In the formula, and Indicates the pixel brightness of frame 0 and Directional gradient; Represents the temporal gradient of an image.

[0019] Furthermore, the fused displacement optical flow data in step S3 The calculation method is: The semantic segmentation model is based on The optical flow model is improved from the Improved; the overall displacement The calculation process of the frame image displacement optical flow data can be expressed as:

[0020]

[0021] Local displacement The calculation process of the frame image displacement optical flow data can be expressed as:

[0022]

[0023] The expression of the dataset after fusion:

[0024] In the formula, Respectively The light flows at all times and The weight in direction, represents the overall displacement mask of the output, Represents the output local displacement mask, Indicates the operation of extracting masks based on the U-Net network model. represents the optical flow image, represents the fused optical flow data after mask denoising, Represents the overall displacement optical flow data after mask denoising, represents the overall local optical flow data after mask denoising, Represents the optical flow model function, whose input is the pose rendering image and output is the optical flow data of the corresponding structure.

[0025] Furthermore, in step S4, a dynamic weight is established between the two features according to these vectors to obtain a fusion feature, and the specific method is as follows: Overall characteristics and local features After input into the network, linear transformation is performed to generate query, key and value vectors; For example, it generates the corresponding query, key, and value vectors through a set of learnable weight matrices:

[0026] Similarly, local features It is also possible to generate query, key, and value vectors:

[0027] in, Represent overall features and local features respectively; The query vector, key vector and value vector represent the overall features respectively; The query vector, key vector and value vector represent the local features respectively; are learnable weight matrices used to generate query vectors, key vectors, and value vectors; these matrices can automatically learn the most suitable parameters through the model's training data to generate vector representations that contribute to displacement estimation on different features; In the cross-attention mechanism, the similarity or correlation between the generated query vector and the key vector is calculated to generate the attention weight; the similarity is calculated using the vector dot product and scaled to maintain a stable gradient. The specific formula is:

[0028] In the formula, Indicates the dimension of the key vector, used for scaling to prevent the vector dot product value from being too large; denote the query vector, key vector and value vector respectively, represents the matrix transpose, represents the activation function, Represents the calculation of the cross-attention mechanism; The core of this formula is to calculate the similarity between the query vector and the key vector, divided by the scaling factor and pass it through Normalized to the attention weight; then, through the weighted sum vector, To extract features; in the cross attention mechanism, this calculation process is carried out bidirectionally between the overall features and the local features to capture detailed information; using the overall features The query vector Can be used to focus on key location information of local features:

[0029] Local feature query Contextual information used to extract overall features:

[0030] In the formula, The query vector, key vector and value vector represent the overall features respectively; They represent the query vector, key vector and value vector of local features respectively.

[0031] Furthermore, the loss function expression in step S4 is:

[0032] In the formula, is the actual displacement value data, Predict displacement value data for the model, is the total number of samples, is the regularization coefficient; represents the regularization weight, Represents the actual displacement value data of the ith value, Represents the displacement value data predicted by the i-th model.

[0033] As a second aspect of the present invention, a structure displacement visual recognition system integrating a three-dimensional deformable model is also provided, comprising: A three-dimensional deformable mesh model building unit is used to collect engineering structure vibration video data through a consumer-grade monocular camera, build a three-dimensional deformable mesh model, and generate pose rendering images and displacement value data; Data preprocessing unit, used to remove the interference of environmental noise on displacement recognition; An optical flow feature data set construction unit, used to estimate dense optical flow data in a rendered image through an optical flow model, and to construct an optical flow feature data set containing pose information; The model learning unit is used for the structure-intensive displacement estimation method based on the cross-attention mechanism. By fusing local and global features, the model displacement prediction value is output. The specific steps are as follows: Input two sets of data sequences: the overall structure image sequence and the local structure image sequence. After multi-layer convolution operations at the volume base layer, the corresponding overall features are generated respectively. With local features ; Overall characteristics and local features After linear transformation, the feature query vector, key vector and value vector are generated, and dynamic weights are established between the two features based on these vectors to obtain two fusion features. as well as ; Determine the loss function, perform global average pooling on the two fusion features obtained, and then output the pose estimation value through a multi-layer perceptron.

[0034] As a third aspect of the present invention, it also relates to a computer-readable storage medium, on which a computer program is stored, and the computer program is executed by a processor to implement the above-mentioned structural displacement visual recognition method for fusing three-dimensional deformable models.

[0035] In general, the above technical solutions conceived by the present invention can achieve the following beneficial effects compared with the prior art: 1. The structural displacement visual recognition method of the present invention that integrates a three-dimensional deformable model introduces feature fusion methods such as a cross-attention mechanism to fuse features from different sources (such as overall feature maps, local feature maps, optical flow data, etc.). This mechanism can fully explore the associations between different features, allowing the model to comprehensively judge structural displacement from multiple dimensions such as global and local, spatial and motion, overcoming the limitations of traditional methods that only analyze from a single perspective. These data reflect the state of the engineering structure from different perspectives. For example, the overall image provides a macroscopic structural posture, the local image captures the details of key parts, the optical flow data reflects the pixel-level motion information, and the semantic segmentation mask is used to eliminate environmental interference. This fusion of multi-source data is a relatively innovative idea in the field of structural displacement recognition, and can more comprehensively describe the displacement of the engineering structure.

[0036] 2. The structural displacement visual recognition method integrating a three-dimensional deformable model of the present invention realizes the recognition of three-dimensional vibration displacement by rendering based on a three-dimensional deformable mesh model and a cross-attention mechanism and a double attention mechanism, and completes the transformation of two-dimensional plane monitoring to three-dimensional space monitoring only through a consumer-grade monocular camera. Compared with traditional high-precision sensors and industrial cameras, the equipment cost is greatly reduced. In addition, the method does not require contact monitoring of the engineering structure, reduces potential damage to the engineering structure, is easy to install and maintain, and is suitable for health monitoring of large-scale engineering structures. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1 A method flow chart of a preferred embodiment of the present invention; Figure 2 A method step diagram of a preferred embodiment of the present invention; Figure 3 A schematic diagram of the details and dimensions of a physical model of a preferred embodiment of the present invention; Figure 4 A schematic diagram of a three-dimensional deformable grid model according to a preferred embodiment of the present invention; Figure 5 A vibration simulation diagram of a preferred embodiment of the present invention; Figure 6 This is a rendering of the overall structural posture of a preferred embodiment of the present invention; Figure 7 It is a rendering of the local structure posture of a preferred embodiment of the present invention; Figure 8 A schematic diagram of the overall structure semantic segmentation mask of a preferred embodiment of the present invention; Fig. 9 A schematic diagram of a local structure semantic segmentation mask according to a preferred embodiment of the present invention; Fig.10 This is a visualization diagram of the optical flow data of the overall structural displacement image of a preferred embodiment of the present invention; Fig.11 A visualization diagram of the optical flow data of the local structure displacement image of a preferred embodiment of the present invention; Fig.12 A schematic diagram of a loss function curve during the training of an invalid data detection model in a preferred embodiment of the present invention; Fig.13 A schematic diagram showing the comparison between the actual vibration data and the predicted value output by the model in a preferred embodiment of the present invention; Fig.14 Schematic diagram of system units according to a preferred embodiment of the present invention. DETAILED DESCRIPTION

[0038] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention. In addition, the technical features involved in the various embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.

[0039] Example 1 Please refer to Figure 1 This embodiment 1 provides a structural displacement visual recognition method integrating a three-dimensional deformable model, comprising: S1. Collect engineering structure vibration video data through a consumer-grade monocular camera, build a 3D deformable mesh model and generate pose rendering images and displacement value data; S2. Remove the interference of environmental noise on displacement recognition; S3. Estimating dense optical flow data in the rendered image through the optical flow model, and constructing an optical flow feature dataset containing pose information; S4. A structured dense displacement estimation method based on a cross-attention mechanism outputs the model displacement prediction value by fusing local and global features. The specific steps are as follows: Input two sets of data sequences: the overall structure image sequence and the local structure image sequence. After multi-layer convolution operations of the convolution layer, the corresponding overall features are generated respectively. With local features ; Overall characteristics and local features After linear transformation, the feature query vector, key vector and value vector are generated, and dynamic weights are established between the two features based on these vectors to obtain two fusion features. as well as ; Determine the loss function, perform global average pooling on the two fusion features obtained, and then output the pose estimation value through a multi-layer perceptron.

[0040] For further details on the above method steps of this implementation example, please refer to Figure 2 ,The purpose of this implementation is to propose a method for ,engineering structure displacement recognition based on deep learning and ,three-dimensional deformable mesh model to be applied to the processing and ,analysis process of predicting and recognizing engineering structure displacement by ,consumer-grade monocular camera.

[0041] The core of the present invention is to realize the recognition of three-dimensional vibration displacement based on three-dimensional deformable grid model rendering and cross-attention mechanism and double attention mechanism. The method is mainly divided into two steps. The first step is model training. This process requires the use of a three-dimensional deformable grid model for modeling and vibration simulation, outputting a posture rendering image and undergoing semantic segmentation and optical flow conversion, and inputting it into the model so that the model learns the structural displacement corresponding to different postures; the second step is displacement recognition of observed videos, that is, using the trained model to process the captured video data, calculate and predict the displacement data of the captured engineering structure, thereby realizing displacement recognition of the engineering structure.

[0042] Specifically, step S1 is as follows: (1a) Create a 1:1 model of the exterior facade and characteristic structure of the engineering structure to be monitored, insert and bind the "skeleton" in the "column" of the model, adjust the material, length and related properties, and set the hierarchical relationship between the "parent class" and "child class". At the same time, in order to limit the horizontal and vertical displacement range of the floor, add follow-up effects and constraints, set the common section S and control section S c To achieve precise control of model deformation; (1b) Write a script in the driver that comes with Blender to enable it to read the data in the displacement file and assign the displacement value of the corresponding position to the corresponding key frame control section, thereby realizing the dynamic displacement simulation of the model. The displacement data file can be in Excel format, in which the displacement value is randomly generated and obeys a uniform distribution. The extreme value can be adjusted according to the actual engineering project requirements and experimental requirements.

[0043] (1c) All key frames are rendered and the final simulated pose rendering is output. The output rendering has a width of 384 and a height of 704.

[0044] Specifically, step S2 uses a semantic mask and dense optical flow estimation method based on deep learning to generate a semantic segmentation mask of the engineering structure through a pre-trained semantic segmentation model to remove the interference of environmental noise on displacement recognition. Specifically: The semantic segmentation model generates a semantic segmentation mask of the engineering structure to remove the interference of environmental noise on displacement recognition. The calculation formula is:

[0045] Apply Mask For images, the calculation formula is:

[0046] In the formula, is the original image, is the structural area at time 0 and 0 displacement state, is the denoised image.

[0047] Specifically, step S3 includes: Estimate dense optical flow data in rendered images through optical flow models; From the overall displacement optical flow data The most significant local feature area is extracted, and then enlarged and preprocessed to obtain local displacement optical flow data ; Obtain the displacement optical flow data after fusion of local features and overall features .

[0048] (3a) The dense optical flow data in the rendered image is estimated through the optical flow model, and an optical flow feature dataset containing pose information is constructed. For two frames of images at time 0 and time t, the calculation formula is:

[0049] In the formula, represents the image at time 0, express images of moments; Respectively represent the optical flow at time 0 and The weight in direction, Respectively The light flows at all times and Directional weight; According to Taylor expansion, exist Performing a first-order expansion at , we can obtain:

[0050] Taylor expansion of the equation and ignoring higher-order terms gives the optical flow constraint equation:

[0051] In the formula, and Indicates the pixel brightness of frame 0 and Directional gradient; Represents the temporal gradient of an image; , The optical flow is and Directional weight.

[0052] Calculate the 0th and 1st frames The optical flow between frames, frame 0 and frame Average optical flow between frames , which can be expressed as:

[0053] Substituting this definition into the optical flow constraint equation, the optical flow expression at multi-frame intervals is obtained as follows:

[0054] (3b) Construct an optical flow feature dataset containing pose information, from the overall displacement optical flow data The most significant local feature area is extracted, and then enlarged and preprocessed to obtain local displacement optical flow data , can be expressed by the following formula:

[0055] (3c) Specifically, the semantic segmentation model is improved based on the U-Net model, and the optical flow model is improved based on Flownet2. The calculation process of the frame image displacement optical flow data can be expressed as:

[0056]

[0057] Local displacement The calculation process of the frame image displacement optical flow data can be expressed as:

[0058]

[0059] The expression of the dataset after fusion:

[0060] In the formula, Respectively The light flows at all times and The weight in direction, represents the overall displacement mask of the output, Represents the output local displacement mask, Indicates the operation of extracting masks based on the U-Net network model. represents the optical flow image, represents the fused optical flow data after mask denoising, Represents the overall displacement optical flow data after mask denoising, represents the overall local optical flow data after mask denoising, Represents the optical flow model function, whose input is the pose rendering image and output is the optical flow data of the corresponding structure.

[0061] Step S4 is further expanded, and its specific steps are: (4a) In the overall architecture, the model first inputs two sets of data sequences, namely the overall structure image sequence and the local structure image sequence. After the multi-layer convolution operation of EfficientNet-B0, these two sets of images generate corresponding overall feature maps and local feature maps respectively. At this time, the overall feature map contains the macroscopic pose information of the engineering structure, while the local feature map contains subtle local pose changes. The optical flow sequence expression of the input model is:

[0062] (4b) The core of the cross-attention mechanism is to establish dynamic weights between different features through the interaction of query vector Q, key vector K and value vector V, so as to more accurately capture subtle local changes. First, the overall features and local features are converted into three main vectors, namely query vector Q, key vector K and value vector V. These three vectors work together to realize feature interaction and information capture. Overall features and local features After input into the network, it undergoes linear transformation to generate query, key and value vectors. For example, it generates the corresponding query, key, and value vectors through a set of learnable weight matrices:

[0063] Similarly, local features It is also possible to generate query, key, and value vectors:

[0064] in, Represent overall features and local features respectively; The query vector, key vector and value vector represent the overall features respectively; The query vector, key vector and value vector represent the local features respectively; are learnable weight matrices used to generate query vectors, key vectors, and value vectors; these matrices can automatically learn the most suitable parameters through the model's training data to generate vector representations that contribute to displacement estimation on different features; In the cross-attention mechanism, the generated query vector and key vector The similarity or correlation is calculated between them, and then the attention weight is generated. The similarity is calculated by vector dot product and scaled to maintain a stable gradient. The specific formula is:

[0065] In the formula, Indicates the dimension of the key vector, used for scaling to prevent the vector dot product value from being too large; denote the query vector, key vector and value vector respectively, represents the matrix transpose, represents the activation function, Represents the calculation of the cross-attention mechanism; The core of this formula is to calculate the similarity between the query vector and the key vector, divided by the scaling factor and pass it through Normalized to the attention weight. Then, through the weighted sum vector To extract features. In the cross-attention mechanism, this calculation process is performed bidirectionally between global features and local features to capture detailed information. For example, using the global features The query vector Can be used to focus on key location information of local features:

[0066] Similarly, local feature queries It can also be used to extract contextual information of overall features:

[0067] Through the bidirectional cross-attention calculation, the overall and local features not only retain their respective important information, but also supplement the details not captured by each other. This bidirectional information interaction design ensures that the model can accurately capture displacement changes at different scales when extracting features.

[0068] (4c) The fused features calculated by the cross-attention mechanism are then globally averaged and pooled, as shown below:

[0069]

[0070] The obtained sequence is passed to the multi-layer perceptron MLP layer, which contains several fully connected layers and nonlinear activation functions, which can be used to further process features and abstract complex relationships. Through nonlinear transformation, high-dimensional fusion features are mapped to low-dimensional space, thereby effectively compressing information redundancy and improving the accuracy of feature expression. After processing by the MLP layer, the model outputs a pose estimate, which represents the displacement state of the overall structure or local structure in a specific environment or perspective. The predicted displacement value The expression is as follows:

[0071]

[0072] In addition, in this embodiment, the loss function expression in step S4 is:

[0073] In the formula, is the actual displacement value data, Predict displacement value data for the model, is the total number of samples, is the regularization coefficient.

[0074] This embodiment also provides examples in practical application scenarios, as follows: This embodiment is completed based on a computer program written in Python language, and the data used is provided by the 3rd International Structural Health Monitoring Competition. The following specific examples illustrate the effects of the present invention.

[0075] First, a three-dimensional deformable mesh model was built using Blender software, and the physical model was set up on the shaking table of Tongji University for dynamic tests under unidirectional earthquake action. The main structure of the model is 6.5 meters high, each floor is 1.5 meters high, and the base height is 0.35 meters; the column spacing along the vibration direction is 3.0 meters, the column spacing in the vertical vibration direction is 1.5 meters, the shaking table size is 4.0 meters × 4.0 meters, and the load-bearing capacity is 25 tons. During the test, various monitoring was carried out on each floor of the model, and the absolute acceleration and relative displacement were routinely measured by accelerometers and linear displacement sensors. The test camera (SONY HDR - PJ220) was arranged 3.0 meters in front of the model frame, and a fixed camera (Black Magic DesignURSA mini 4K) was set up 8.0 meters away from the frame model. Referring to the size of the model of the 3rd International Structural Health Testing Competition and the camera layout angle of view, a 1:1 scale model was established in Blender. The model is as follows Figure 3 , Figure 4 shown.

[0076] Insert and bind "bone" in the "column" of the model, adjust the material, length and related properties, and set the hierarchical relationship between "parent class" and "child class". At the same time, follow-up effects and constraints are added to limit the horizontal and vertical displacement range of the floor. Write a script in the driver that comes with Blender to enable it to read the data in the displacement file and assign the displacement value of the corresponding position to the corresponding keyframe control section, so as to realize the dynamic displacement simulation of the model. The displacement data file can be in Excel format, in which the displacement value is randomly generated and obeys uniform distribution, and the extreme value can be adjusted according to the actual engineering project requirements and experimental requirements. All keyframes are rendered and the final simulation rendering is output. The output rendering is 384 in width and 704 in height. A total of 80,001 frames of pose rendering are generated, including a frame of initial pose rendering at 0 time and 0 displacement. Extract the most significant local feature area, then enlarge and preprocess it to obtain the local structure pose rendering.

[0077] The generated overall structure and local structure pose rendering images are input into the semantic mask and dense optical flow estimation model based on deep learning, and the semantic segmentation mask is generated and involved in the calculation, and the overall structure and local structure optical flow data sets for training are generated. A total of 80001×2 data are generated, which are used for model training respectively, and the actual vibration data provided by the 3rd International Structural Health Monitoring Competition are preprocessed in the same way and used as the test set. The specific data division method is shown in Table 1.

[0078] Table 1 Datasets used in the experiment

[0079] When using a sliding window to sample sequence data, it is necessary to set an appropriate sliding window size and sliding step size. At the same time, in order to make the model converge better, this model uses learning rate decay, where the learning rate is adjusted to 0.1 times the original value each iteration. Before training the model, the test data should be preprocessed, that is, the data of each test indicator should be standardized.

[0080] Some parameter settings during the experiment are shown in Table 2.

[0081] Table 2 Model related parameter settings

[0082] Use the divided training set and validation set to train the model. The loss function curve during the training process is as follows: Fig.12 As shown in the figure. During the training process, the loss functions of the training set and the validation set quickly dropped below 0.03 after 4 iterations. As the iterations progressed, the loss functions of the two sets stabilized after 40 iterations, that is, the model reached convergence. The final loss function stabilized at around 0.08. In addition, the overall trend of the loss function curves of the training set and the validation set is the same, which conforms to the learning curve law in the deep learning training process, and the model does not overfit.

[0083] After the model training is completed, it is necessary to conduct in-depth analysis of the model's predicted values ​​and actual true values ​​to ensure that the model has good generalization ability and prediction accuracy. A variety of evaluation indicators are used to measure the performance of the model, including mean square error (MSE), mean absolute error (MAE), coefficient of determination (R²), and root mean square error (RMSE). The evaluation results are shown in the following table: Table 3 Evaluation results of multiple indicators

[0084] Combining these indicators, it can be seen that the error between the predicted value and the actual value of the model is very small, the fitting effect is excellent, and it has good accuracy and robustness.

[0085] Please refer to Fig.13 In the comparison of the curve fitting results between the actual vibration data and the predicted values ​​output by the model, both large and small displacements can be well fitted. The prediction of 10-2 millimeter displacement at a distance of about 10m is achieved, which is equivalent to the displacement recognition of engineering structures with centimeter-level accuracy at a distance of 1000m, and has the possibility of practical application value.

[0086] From the above analysis, it can be seen that the engineering structure displacement recognition method based on deep learning and three-dimensional deformable mesh model used in this paper adopts a consumer-grade monocular camera, and realizes the recognition of small displacements of engineering structures through computer vision and deep learning technology. By establishing the corresponding three-dimensional deformable mesh model, 80,000 random vibration pose renderings are selected as training sets and validation sets for model testing. The model is trained by setting appropriate parameters. The loss function of the training process shows that the model converges after 4 iterations, and the training process is stable. A variety of error analysis methods including mean square error (MSE), mean absolute error (MAE), determination coefficient (R²) and root mean square error (RMSE) were used and the result curve fitting was performed. The results showed that the mean square error (MSE) of the model was 0.08; the mean absolute error (MAE) was 0.19; the determination coefficient (R²) was 0.99; and the root mean square error (RMSE) was 0.28. The detection effect of the model was good and stable, and the prediction of 10-2 millimeter-level displacement at a distance of about 10m was achieved, which is equivalent to achieving centimeter-level displacement identification for engineering structures at 1000m, and has the possibility of practical application value.

[0087] Example 2 Please refer to Fig.14 This embodiment 2 provides a structural displacement visual recognition system integrating a three-dimensional deformable model, including: A three-dimensional deformable mesh model building unit is used to collect engineering structure vibration video data through a consumer-grade monocular camera, build a three-dimensional deformable mesh model, and generate pose rendering images and displacement value data; A data preprocessing unit, used to remove the interference of environmental noise on displacement recognition; An optical flow feature data set construction unit, used to estimate dense optical flow data in a rendered image through an optical flow model, and to construct an optical flow feature data set containing pose information; The model learning unit is used for the structure-intensive displacement estimation method based on the cross-attention mechanism. By fusing local and global features, the model displacement prediction value is output. The specific steps are as follows: Input two sets of data sequences: the overall structure image sequence and the local structure image sequence. After multi-layer convolution operations at the volume base layer, the corresponding overall features are generated respectively. With local features ; Overall characteristics and local features After linear transformation, the feature query vector, key vector and value vector are generated, and dynamic weights are established between the two features based on these vectors to obtain two fusion features. as well as ; Determine the loss function, perform global average pooling on the two fusion features obtained, and then output the pose estimation value through a multi-layer perceptron.

[0088] Example 3 This embodiment 3 also provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of any of the above-mentioned structural displacement visual recognition methods for integrating a three-dimensional deformable model can be implemented.

[0089] The computer-readable storage medium may include: a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and other media that can store program codes.

[0090] For an introduction to the computer-readable storage medium provided in this application, please refer to the above method embodiment, and this application will not go into details here.

[0091] It will be easily understood by those skilled in the art that the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for visual recognition of structural displacements by integrating a three-dimensional deformable model, characterized in that: include: S1. Collect engineering structure vibration video data through a consumer-grade monocular camera, build a 3D deformable mesh model and generate pose rendering images and displacement value data; S2. Remove the interference of environmental noise on displacement recognition; S3. Estimating dense optical flow data in the rendered image through the optical flow model, and constructing an optical flow feature dataset containing pose information; S4. A structured dense displacement estimation method based on a cross-attention mechanism outputs the model displacement prediction value by fusing local and global features. The specific steps are as follows: Input two sets of data sequences: the overall structure image sequence and the local structure image sequence. After multi-layer convolution operations of the convolution layer, the corresponding overall features are generated respectively. With local features ; Overall characteristics and local features After linear transformation, the feature query vector, key vector and value vector are generated, and dynamic weights are established between the two features based on these vectors to obtain two fusion features. as well as ; Determine the loss function, perform global average pooling on the two fusion features obtained, and then output the pose estimation value through a multi-layer perceptron.

2. According to the method of claim 1, the structure displacement visual recognition method integrating a three-dimensional deformable model is characterized in that: The displacement value in step S1 is randomly generated and obeys uniform distribution, and the extreme value of the displacement value can be adjusted according to the actual engineering project requirements and experimental requirements.

3. According to the method of claim 1, the structure displacement visual recognition method integrating a three-dimensional deformable model is characterized in that: The specific method for removing the interference of environmental noise on displacement recognition in step S2 is: The semantic segmentation mask of the engineering structure is generated through the semantic segmentation model to remove the interference of environmental noise on displacement recognition. The calculation formula is: ， Apply Mask For images, the calculation formula is: ， In the formula, is the original image, is the structural area at time 0 and 0 displacement state, is the denoised image.

4. According to claim 3, a method for visually recognizing structural displacements by integrating a three-dimensional deformable model, characterized in that: The specific method of step S3 includes: Estimate dense optical flow data in rendered images through optical flow models; From the overall displacement optical flow data The most significant local feature area is extracted, and then enlarged and preprocessed to obtain local displacement optical flow data ; Obtain the displacement optical flow data after fusion of local features and overall features .

5. According to claim 4, a method for visually recognizing structural displacements by integrating a three-dimensional deformable model, characterized in that: The dense optical flow data in the rendered image is estimated by the optical flow model, and the specific method is as follows: For two frames of images at time 0 and time t, the calculation formula is: ， In the formula, represents the image at time 0, express images of moments; Respectively represent the optical flow at time 0 and The weight in direction, Respectively The light flows at all times and Directional weight; According to Taylor expansion, exist Performing a first-order expansion at , we can obtain: ， Taylor expansion of the equation and ignoring higher-order terms gives the optical flow constraint equation: ， In the formula, , Denotes the optical flow in and Directional weight; Calculate the 0th and 1st frames The optical flow between frames, frame 0 and frame Average optical flow between frames , which can be expressed as: ， Substituting this definition into the optical flow constraint equation, the optical flow expression at multi-frame intervals is obtained as follows: ， In the formula, and Indicates the pixel brightness of frame 0 and Directional gradient; Represents the temporal gradient of an image.

6. According to claim 4, a method for visually recognizing structural displacements by integrating a three-dimensional deformable model, characterized in that: The fused displacement optical flow data in step S3 The calculation method is: The semantic segmentation model is based on The optical flow model is improved from the Improved; the overall displacement The calculation process of the frame image displacement optical flow data can be expressed as: ， ， Local displacement The calculation process of the frame image displacement optical flow data can be expressed as: ， ， The expression of the dataset after fusion: ， In the formula, Respectively The light flows at all times and The weight in direction, represents the overall displacement mask of the output, Represents the output local displacement mask, Indicates the operation of extracting masks based on the U-Net network model. represents the optical flow image, represents the fused optical flow data after mask denoising, Represents the overall displacement optical flow data after mask denoising, represents the overall local optical flow data after mask denoising, Represents the optical flow model function, whose input is the pose rendering image and output is the optical flow data of the corresponding structure.

7. According to claim 1, a method for visually recognizing structural displacements by integrating a three-dimensional deformable model, characterized in that: In step S4, a dynamic weight is established between the two features according to these vectors to obtain a fusion feature. The specific method is: Overall characteristics and local features After input into the network, linear transformation is performed to generate query, key and value vectors; For example, it generates the corresponding query, key, and value vectors through a set of learnable weight matrices: ， Similarly, local features It is also possible to generate query, key, and value vectors: ， in, Represent overall features and local features respectively; The query vector, key vector and value vector represent the overall features respectively; The query vector, key vector and value vector represent the local features respectively; are learnable weight matrices used to generate query vectors, key vectors, and value vectors; these matrices can automatically learn the most suitable parameters through the model's training data to generate vector representations that contribute to displacement estimation on different features; In the cross-attention mechanism, the similarity or correlation between the generated query vector and the key vector is calculated to generate the attention weight; the similarity is calculated using the vector dot product and scaled to maintain a stable gradient. The specific formula is: ， In the formula, Indicates the dimension of the key vector, used for scaling to prevent the vector dot product value from being too large; denote the query vector, key vector and value vector respectively, represents the matrix transpose, represents the activation function, Represents the calculation of the cross-attention mechanism; The core of this formula is to calculate the similarity between the query vector and the key vector, divided by the scaling factor and pass it through Normalized to the attention weight; then, through the weighted sum vector, To extract features; in the cross attention mechanism, this calculation process is carried out bidirectionally between the overall features and the local features to capture detailed information; using the overall features The query vector Can be used to focus on key location information of local features: ， Local feature query Contextual information used to extract overall features: ， In the formula, The query vector, key vector and value vector represent the overall features respectively; They represent the query vector, key vector and value vector of local features respectively.

8. According to claim 1, a method for visually recognizing structural displacements by integrating a three-dimensional deformable model, characterized in that: The loss function expression in step S4 is: ， In the formula, is the actual displacement value data, Predict displacement value data for the model, is the total number of samples, is the regularization coefficient; represents the regularization weight, Represents the actual displacement value data of the ith value, Represents the displacement value data predicted by the i-th model.

9. A structural displacement visual recognition system integrating a three-dimensional deformable model, characterized in that: include: A three-dimensional deformable mesh model building unit is used to collect engineering structure vibration video data through a consumer-grade monocular camera, build a three-dimensional deformable mesh model, and generate pose rendering images and displacement value data; A data preprocessing unit, used to remove the interference of environmental noise on displacement recognition; An optical flow feature data set construction unit, used to estimate dense optical flow data in a rendered image through an optical flow model, and to construct an optical flow feature data set containing pose information; The model learning unit is used for the structure-intensive displacement estimation method based on the cross-attention mechanism. By fusing local and global features, the model displacement prediction value is output. The specific steps are as follows: Input two sets of data sequences: the overall structure image sequence and the local structure image sequence. After multi-layer convolution operations at the volume base layer, the corresponding overall features are generated respectively. With local features ; Overall characteristics and local features After linear transformation, the feature query vector, key vector and value vector are generated, and dynamic weights are established between the two features based on these vectors to obtain two fusion features. as well as ; Determine the loss function, perform global average pooling on the two fusion features obtained, and then output the pose estimation value through a multi-layer perceptron.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: The computer program is executed by a processor to implement the structural displacement visual recognition method of fused three-dimensional deformable model as described in any one of claims 1-8.

Images