Structural displacement non-contact sensing method, device and equipment and storage medium

Abstract

The invention discloses a structure displacement non-contact sensing method, device and equipment and a storage medium, and the method comprises the steps: collecting real structure vibration real-shot video data, and constructing the structure vibration real-shot video data into a real-shot image data set through the combination of a rendering technology and an optical flow estimation pre-training model; wherein the optical flow estimation pre-training model is obtained by training a preset optical flow initial estimation model through a public data set; performing iterative optimization training on the optical flow estimation pre-training model by adopting a preset optical flow iterative optimization framework based on the real shot image data set to obtain an optical flow estimation target model; performing full-field optical flow prediction on a specified real-shot image sequence by using the optical flow estimation target model to obtain optical flow displacement under a camera coordinate system; and carrying out space coordinate conversion on the optical flow displacement under the camera coordinate system to obtain the structure displacement under the world coordinate system. According to the invention, the precision and efficiency of structure displacement sensing can be improved.

Description

Technical Field

[0001] The present invention relates to the technical field of engineering structure detection, and in particular to a non-contact sensing method, device, equipment and storage medium for structural displacement. Background Art

[0002] Optical flow is a research direction in the field of computer vision. Optical flow refers to a two-dimensional vector field that describes the trajectory and speed of the movement of pixels over time in continuous frame images. It is used to capture the movement of objects in the image and the displacement information between adjacent frames. The optical flow method uses the changes in the pixels in the time domain in the image sequence and the correlation between adjacent frames to find the correspondence between the previous frame and the current frame, thereby calculating the motion information of objects between adjacent frames. According to the scope of optical flow calculation, it is mainly divided into dense optical flow and sparse optical flow. According to the development history of optical flow, the optical flow method is divided into two categories: traditional methods and deep learning methods.

[0003] Traditional methods include HS (Horn-Schunck) dense optical flow, LK (Lucas-Kanade) sparse optical flow, and improved algorithms based on these two methods. Deep learning methods have emerged in recent years, including supervised learning, unsupervised learning, and self-supervised learning. Currently, supervised learning is the most widely studied and applied in many engineering fields.

[0004] The most commonly used supervised deep learning optical flow models are: FlowNet series, Spy-Net, PWC-Net, LiteFlowNet series, IRR, MaskFlowNet, RAFT and GMA. These deep learning-based optical flow methods are more advantageous than traditional methods in terms of performance and speed. Here, all deep learning models are used to calculate dense optical flow, among which FlowNet2.0 and RAFT are more popular.

[0005] Supervised deep learning requires a dataset with real optical flow labels for training. Currently, the datasets that are publicly available and can be used to train the above-mentioned deep learning optical flow models include: Flyingchairs, FlyingThings3D, Sintel, and KITTI. Generally speaking, optical flow datasets are divided into computer-rendered synthetic datasets and real-world datasets. Among them, optical flow datasets with real optical flow labels are usually difficult to produce. The main reason is that manual annotation of optical flow labels is very time-consuming and labor-intensive, and it is extremely difficult to obtain labels. At the same time, the rendering dataset has very high requirements for equipment and data, and the labels of the real-world dataset also require very high-precision measurement equipment, but only sparse optical flow labels can be obtained.

[0006] Therefore, for supervised deep learning models, pre-trained models that have been trained with public data sets are generally selected for structural displacement measurement tasks. In the process of transfer learning, the pre-trained model is generally required to be fine-tuned and trained on a self-made optical flow data set to achieve better prediction results. However, since optical flow data sets are difficult to produce, the existing technology currently lacks the production of data sets, and no retraining of the model is taken. The lack of data set production and model retraining often leads to poor model prediction performance, resulting in large errors in structural displacement measurement results. Summary of the invention

[0007] The embodiments of the present invention provide a method, device, equipment and storage medium for non-contact sensing of structural displacement, aiming to improve the accuracy and efficiency of structural displacement sensing.

[0008] In a first aspect, an embodiment of the present invention provides a non-contact sensing method for structural displacement, comprising: Collect real structural vibration video data, and construct the real structural vibration video data into a real image dataset by combining rendering technology and optical flow estimation pre-training model; wherein the optical flow estimation pre-training model is obtained by training a preset optical flow initial estimation model with a public dataset; Based on the real-shot image dataset, a preset optical flow iterative optimization framework is used to iteratively optimize the optical flow estimation pre-trained model to obtain an optical flow estimation target model; Using the optical flow estimation target model to perform full-field optical flow prediction on a specified real-shot image sequence, and obtain the optical flow displacement in the camera coordinate system; The optical flow displacement in the camera coordinate system is transformed into a spatial coordinate to obtain a structural displacement in the world coordinate system.

[0009] In a second aspect, an embodiment of the present invention provides a non-contact sensing device for structural displacement, comprising: An acquisition unit is used to acquire real structural vibration real-shot video data, and construct the structural vibration real-shot video data into a real-shot image data set by combining rendering technology and an optical flow estimation pre-training model; wherein the optical flow estimation pre-training model is obtained by training a preset optical flow initial estimation model with a public data set; A training unit, configured to perform iterative optimization training on the optical flow estimation pre-training model based on the real-shot image data set using a preset optical flow iterative optimization framework to obtain an optical flow estimation target model; A prediction unit, used to perform full-field optical flow prediction on a specified real-shot image sequence using the optical flow estimation target model to obtain an optical flow displacement in a camera coordinate system; The conversion unit is used to perform spatial coordinate conversion on the optical flow displacement in the camera coordinate system to obtain the structural displacement in the world coordinate system.

[0010] In a third aspect, an embodiment of the present invention provides a computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the non-contact sensing method for structural displacement as described in the first aspect is implemented.

[0011] In a fourth aspect, an embodiment of the present invention provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the non-contact sensing method for structural displacement as described in the first aspect is implemented.

[0012] The embodiments of the present invention provide a method, device, equipment and storage medium for non-contact perception of structural displacement. The embodiments of the present invention use rendering technology to produce real structural vibration video data as a data set, which can improve the structural displacement recognition effect under different working conditions. At the same time, the optical flow iterative optimization framework is combined to perform model iterative optimization training, so as to select the optical flow estimation model with the best performance, thereby improving the accuracy and efficiency of structural displacement perception. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying any creative work.

[0014] Figure 1 A schematic diagram of a flow chart of a non-contact sensing method for structural displacement provided by an embodiment of the present invention; Figure 2 A schematic block diagram of a non-contact sensing device for structural displacement provided by an embodiment of the present invention; Figure 3 A schematic diagram of a rendering process in a non-contact sensing method for structural displacement provided by an embodiment of the present invention; Figure 4 A schematic diagram of data collection in a non-contact sensing method for structural displacement provided by an embodiment of the present invention; Figure 5 A schematic diagram of data set preparation in a non-contact sensing method for structural displacement provided by an embodiment of the present invention; Figure 6 A schematic diagram of data rendering in a non-contact sensing method for structural displacement provided by an embodiment of the present invention; Figure 7Another data rendering schematic diagram of a non-contact structural displacement sensing method provided by an embodiment of the present invention; Figure 8 A schematic diagram of optical flow map recognition in a non-contact sensing method for structural displacement provided by an embodiment of the present invention; Fig. 9 Another schematic diagram of optical flow map recognition in a non-contact sensing method for structural displacement provided by an embodiment of the present invention; FIG. 10 ( a ) is an example diagram of a target area for first displacement extraction in a non-contact sensing method for structural displacement provided by an embodiment of the present invention; FIG10( b ) is a first displacement measurement curve diagram in a non-contact sensing method for structural displacement provided by an embodiment of the present invention; FIG. 11( a ) is an exemplary diagram of a target area for extracting displacement in a non-contact structural displacement sensing method provided by an embodiment of the present invention; FIG11( b ) is a second displacement measurement curve diagram in a non-contact structural displacement sensing method provided by an embodiment of the present invention; Fig.12 A schematic diagram of displacement measurement error in a non-contact sensing method for structural displacement provided by an embodiment of the present invention; Fig.13 A schematic diagram of another displacement measurement error in a non-contact sensing method for structural displacement provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0015] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0016] It should be understood that when used in this specification and the appended claims, the terms "include" and "comprises" indicate the presence of described features, integers, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or combinations thereof.

[0017] It should also be understood that the terms used in this specification of the present invention are only for the purpose of describing specific embodiments and are not intended to limit the present invention. As used in the specification of the present invention and the appended claims, unless the context clearly indicates otherwise, the singular forms "a", "an" and "the" are intended to include plural forms.

[0018] It should be further understood that the term "and / or" used in the present description and the appended claims refers to and includes any and all possible combinations of one or more of the associated listed items.

[0019] See below Figure 1 , an embodiment of the present invention provides a non-contact sensing method for structural displacement, specifically comprising: steps S101~S104.

[0020] S101, collecting real structural vibration video data, and combining rendering technology and optical flow estimation pre-training model to construct the structural vibration video data into a real-shot image dataset; wherein the optical flow estimation pre-training model is obtained by training a preset optical flow initial estimation model with a public dataset; S102, based on the real-shot image data set, using a preset optical flow iterative optimization framework to iteratively optimize the optical flow estimation pre-trained model to obtain an optical flow estimation target model; S103, using the optical flow estimation target model to perform full-field optical flow prediction on a specified real-shot image sequence to obtain an optical flow displacement in a camera coordinate system; S104 , performing spatial coordinate transformation on the optical flow displacement in the camera coordinate system to obtain a structural displacement in the world coordinate system.

[0021] In this embodiment, real structural vibration video data is first collected for real structures (such as bridges, buildings and other structures), and then the collected structural vibration video data is processed using rendering technology and a pre-trained optical flow estimation pre-training model to obtain a real-shot image data set with real labels. Then, iterative optimization processing is performed through a set optical flow iterative optimization framework to obtain an optical flow estimation target model with better performance. Subsequently, the optical flow estimation target model can be used to perform optical flow prediction on a specified real-shot image sequence, and then the optical flow prediction result is converted into structural displacement in the world coordinate system to complete structural displacement perception.

[0022] This embodiment uses rendering technology to produce real structural vibration video data as a data set, which can improve the structural displacement recognition effect under different working conditions. At the same time, it combines the optical flow iterative optimization framework to perform model iterative optimization training to select the optical flow estimation model with the best performance, thereby improving the accuracy and efficiency of structural displacement perception.

[0023] In some optional embodiments, when using a public data set for pre-training, the public data set Flying Chairs can be used, and the RAFT model can be used as the preset initial optical flow estimation model, and the initial optical flow estimation model can be fine-tuned based on real structural vibration video data to improve the model prediction performance and reduce the error of structural displacement measurement. In the fine-tuning stage, the weights of the low-level convolutional layers in the model can be frozen to focus on optimizing the high-level feature extraction layer. Furthermore, the Flying Things3D data set is used to continue training the model, and the learning rate is reduced to ensure training stability, and finally an optical flow estimation pre-training model is obtained.

[0024] In other optional embodiments, when performing model training, a GRU (Gated Recurrent Unit) lightweight neural network can be used to implement it. In this way, a simple structure (update gate and reset gate) can be used to reduce the model parameters and improve the efficiency and speed of the overall training process without affecting the accuracy.

[0025] Deep learning is an advanced technology based on artificial neural networks in machine learning. It automatically learns and extracts high-level abstract features of data by building complex neural network models, thereby achieving various complex tasks. The basic idea is to learn image features through a deep learning network, model data at different scales, and effectively fuse information to extract corresponding features. When training the neural network model in this embodiment, the main deep learning algorithms used include convolutional neural networks (CNN) and recurrent neural networks (RNN), among which the recurrent neural network (RNN) mainly uses the gated recurrent unit GRU (Gated Recurrent Unit) model, which can be used to process optical flow estimation tasks in images and videos.

[0026] Among them, convolutional neural network (CNN) is a neural network specially designed to process data with similar grid structures, such as time series data and image data. Its essence is a mapping from input to output. In the process of digital image processing, the image is processed by input layer, convolution layer, pooling layer, fully connected layer, output layer, etc., and the feature vector is output to extract image features. In engineering structure displacement recognition, CNN can be used to extract optical flow features and convert them into more complete structural displacement features, thereby improving the accuracy of recognition.

[0027] The most important basic formula of convolution operation in CNN can be expressed as: ; The basic formula of the pooling operation can be expressed as: ; Among them, x represents the input data, y represents the result after convolution or pooling, w represents the convolution kernel (also known as the filter), k represents the pooling window size, b is the bias term, C represents the number of channels of the input data, S represents the pooling step size, and i, j, and k in the formula are the indexes of the tensor y in different dimensions of rows, columns, and channels, respectively.

[0028] Recurrent Neural Network (RNN) is a neural network with recurrent connections, which is suitable for processing sequence data. In traditional neural networks, each input and output are independent and unconnected, while RNN introduces a recurrent connection in time into the network by introducing a recurrent unit, so that the network can model sequence data. Its basic structure includes an input layer, a hidden layer (loop body), and an output layer. The loop body in the hidden layer allows information to be transmitted within the network, retaining the information of the previous time step, so that the sequence data can be better processed. RNN receives the input and the hidden state of the previous time step at each time step, and outputs the hidden state and prediction result of the current time step.

[0029] In traditional RNN, multiple nested activation functions are used, which makes the gradient easily vanish or explode during back propagation, thus affecting the training effect of the model. In order to solve this problem, the gated recurrent unit GRU neural network is introduced. GRU is an improvement on the structure of recurrent neural network (RNN) and contains two main gates: Update Gate and Reset Gate. These gates are used to control the flow of information between time steps, so that the information of long time steps can be captured and retained more effectively. By effectively controlling the flow of gradients, the gradients can be prevented from vanishing or exploding. The formulas of the three gates are introduced as follows: (1) Update Gate: The update gate determines how much state information from the previous time step needs to be passed to the current time step. The output value of the update gate is between 0 and 1 and is calculated using a Sigmoid activation function. 0 means that the hidden state of the previous time step is completely retained, and 1 means that the candidate hidden state of the current time step is completely dependent. The formula is as follows: ; Among them, Z t represents the output of the update gate, W z is the weight matrix, h t-1 is the hidden state of the previous time step, x t is the input of the current time step, is the Sigmoid activation function, the formula is as follows: ; in, Represents the output value of the Sigmoid activation function.

[0030] (2) Reset Gate: The reset gate controls how much of the state information of the previous time step needs to be reset. When calculating a new candidate hidden state, the output value of the reset gate is between 0 and 1. 0 means completely ignoring the hidden state of the previous time step and only relying on the input of the current time step to calculate the candidate hidden state, and 1 means fully utilizing the hidden state of the previous time step to calculate the candidate hidden state of the current time step. The formula is as follows: ; in, represents the output of the reset gate, is the weight matrix.

[0031] (3) Candidate Hidden State: The candidate hidden state is the potential state of the current time step, which combines the current input and the hidden state of the previous time step after reset.

[0032] ; in, is a candidate hidden state, is the weight matrix.

[0033] (4) Hidden state update: The final hidden state is the weighted sum of the hidden state of the previous time step and the candidate hidden state of the current time step, and the weight is controlled by the update gate.

[0034] ; in, is the hidden state at the current time step t, is the hidden state at the previous time step t-1. is the candidate hidden state at the current time step t, is the update gate for the current time step t.

[0035] This embodiment combines the convolutional neural network and the GRU neural network for iterative optimization training, which can give full play to their advantages in optical flow feature extraction and optical flow estimation of video images, so that the model has better performance in extracting optical flow features, which is then transformed into more accurate and reliable displacement recognition.

[0036] In one embodiment, the step S101 includes: Taking the first frame image as a reference, the remaining frame images are respectively combined with the first frame image to obtain a sequence of real-shot image pairs; Inputting the real-shot image pair sequence into an optical flow estimation pre-training model to obtain an initial optical flow label; For each of the real-shot image pairs, based on the first frame image and the initial optical flow label, another frame image in the real-shot image pair is rendered to obtain a new frame image, and a new real-shot image pair is constructed based on the new frame image; A disturbance factor is added to the initial optical flow label to obtain an optical flow label, and the optical flow label is combined with a new real-shot image pair to construct the real-shot image dataset.

[0037] Specifically, for each of the real-shot image pairs, based on the first frame image and the initial optical flow label, rendering another frame image in the real-shot image pair to obtain a new frame image, and constructing a new real-shot image pair based on the new frame image, including: Render each of the real-shot image pairs according to the following formula: ; in, represents the rendered real-shot image pair, R represents the rendering process, I1 represents the first frame image, and I i-1 Indicates another frame image in the real-shot image pair, label i-1 represents the initial label of optical flow, represents the optical flow estimation model used, Represents the optical flow estimation model used in the previous round.

[0038] In this embodiment, when creating a data set based on the real-shot video data of structural vibration, in order to solve the problem of error accumulation in image pairs composed of adjacent frames, the real-shot video data of structural vibration is first constructed as a real-shot image sequence {I1, I2, ... I i ……, I N}, then take the first frame image I1 in the sequence as the reference, and combine the subsequent images with the first frame image to form a real-shot image pair sequence {(I1, I2), (I1, I3), ... (I1, I i )……, (I1, I N )}; Then, the real image pair {I1, I i} Input to the optical flow estimation pre-training model, and the optical flow estimation pre-training model calculates the two-dimensional optical flow field describing the horizontal and vertical displacement of each pixel from the first frame to the i-th frame, and finally saves it as the optical flow initial label label i-1 Among them, the real image pair {I1, I i}I1 remains unchanged, I i(new) Through the rendering module with I1 and label i-1 Generate the benchmark rendering and the initial optical flow label label i-1 Add perturbation factor to generate optical flow label label' i-1 Finally, the new real-shot image pair {I1, Ii(new)} and optical flow label label' i-1 Together they form a complete optical flow training dataset.

[0039] In one embodiment, the step S102 includes: Inputting the real-shot image data set into the optical flow estimation pre-training model for the first optimization training to obtain the optical flow estimation optimization model for the first round of iteration; Rendering the real-shot image data set in the first optimization training, and inputting the rendered real-shot image data set into the optical flow estimation optimization model of the first round of iteration for a second optimization training, thereby obtaining the optical flow estimation optimization model of the second round of iteration; Rendering the real-shot image data set in the second optimization training, and inputting the rendered real-shot image data set into the second-round iterative optical flow estimation optimization model for a third optimization training, thereby obtaining the third-round iterative optical flow estimation optimization model; And so on, until the preset number of iterations is completed, the optical flow estimation optimization model with the best performance is selected as the optical flow estimation target model.

[0040] In this embodiment, the optical flow estimation pre-trained model is optimized based on the optical flow iterative optimization framework driven by the real-shot image data set. In the iterative optimization process, except for the first round of iteration, each other round of iteration will render the data set used in the previous round of iteration to form a new data set, so as to use the new data set for a new round of iterative optimization. In other words, the optical flow estimation optimization model trained on the real-shot image data set in the previous round is used, and the real-shot image data set of the previous round is further rendered by the rendering module to generate a new optical flow data set. The specific rendering method is as follows: ; Among them, R represents rendering of the image pair, I1, I i-1 represents any pair of input images, Indicates the optical flow estimation optimization model trained on the real-shot image dataset in the previous round, label i-1 represents the initial optical flow label calculated and saved by the deep learning optical flow estimation model, Represents any pair of generated real-shot images.

[0041] Then, the real-shot image dataset generated by this round of rendering is used to train the optical flow estimation optimization model of the previous round of lifting: ; ; Among them, L represents the objective function of model training, and Respectively represent the horizontal component and vertical vector of the optical flow vector of the tth iteration, F x and F y Represent the horizontal and vertical components of the real optical flow vector respectively, Represents the target model of this round of training, by minimizing the objective function Get the best training model.

[0042] In practical applications, the real-shot image dataset can be divided into training set and test set in a ratio of 9:1, and the training parameters can be set for model training. For example, the data batch size for each round of training is 2, and the Adam optimizer is used, and the initial value of the optimizer is set to 1×10 -8 The number of iterations of the optical flow model is 12, and it is trained for 100 steps in the case of a total of 200 pairs of optical flow datasets and 300 steps in the case of 600 pairs of optical flow datasets.

[0043] In a specific embodiment, the selecting the optical flow estimation optimization model with the best performance as the optical flow estimation target model includes: The performance of the optical flow estimation optimization model for each round of iteration is evaluated according to the following formula: ; ; ; ; Among them, the 1PX, 3PX, and 5PX indicators represent the proportion of pixels whose error between the optical flow predicted by the optical flow estimation optimization model and the true optical flow is less than 1 pixel, 3 pixels, and 5 pixels respectively. i represents the Euclidean distance between the predicted displacement and the actual displacement of each pixel in the i-th two-dimensional optical flow field, An indicator function representing a threshold value for determining whether the user's error is greater than 1 pixel, 3 pixels, or 5 pixels, where k represents any constant of 1, 3, or 5.

[0044] In one embodiment, the step S104 includes: A target frame is set for the area where the optical flow displacement is located, and an average optical flow displacement of all pixels in the target frame is calculated, and then the average optical flow displacement is used as a displacement recognition result of the target frame; The scaling factor between the camera coordinate system and the world coordinate system is obtained, and the displacement recognition result in the camera coordinate system is converted into the structural displacement in the world coordinate system based on the scaling factor according to the following formula: ; Among them, D 3Drepresents the structural displacement in the world coordinate system, f is the scaling factor, L 3D represents the length selected in the world coordinate system based on the scaling factor, L img Indicates that L 3D The corresponding pixel distance in the image, Represents the displacement recognition result in the camera coordinate system.

[0045] After selecting the optical flow estimation target model through iterative optimization, this embodiment can perform full-field optical flow prediction on the specified real-shot image sequence, extract the optical flow displacement of the target structure area, and obtain the displacement of the target area in the real-world coordinate system through scaling factor conversion. Specifically, the optical flow displacement of the target structure area in the image can be manually selected through human-computer interaction, and the average optical flow displacement of all pixels in the area is used as the displacement recognition result of the area in the camera plane, and then the scaling factor between the three-dimensional world and the camera plane is calculated, and the displacement in the camera plane is converted to the real displacement of the structure in the three-dimensional world.

[0046] Furthermore, after the step of performing spatial coordinate conversion on the optical flow displacement in the camera coordinate system to obtain the structural displacement in the world coordinate system, the method further comprises: A differential evolution algorithm is used to perform displacement correction on the structural displacement, and the result after displacement correction is output as the final structural displacement perception result.

[0047] In order to further ensure the accuracy of the prediction results, this embodiment performs displacement correction processing on the converted structural displacement. Specifically, the objective function is constructed through the root mean square error, and is continuously optimized and updated based on the differential evolution method until the root mean square error is reduced to a stable state, thereby achieving the optimal correction effect.

[0048] In practical applications, when rendering, this embodiment estimates the optical flow and depth map, applies the splatting technology to generate a new perspective image, and uses the bidirectional hole filling (BHF) technology to fill the holes caused by motion and depth differences, and finally generates high-quality image pairs and their optical flow labels. These image pairs and labels are used to train the optical flow estimation model to ensure that the generated data set has high realism and consistency, thereby improving the performance of the optical flow estimation model.

[0049] Combination Figure 3 , the specific process is as follows: Step 1: Select two consecutive video frames I1 and I2 and estimate the optical flow F between them 1→2 and F 2→1 , the formula is as follows: ; ; Among them, F 1→2 and F 2→1 are forward optical flow and reverse optical flow respectively, Optimizing models for optical flow estimation.

[0050] Step 2: Use the Dense Prediction Transformer (DPT) monocular depth estimation model to estimate the depth information of the two frames of images. The formula is as follows: ; ; Among them, D1 and D2 are the inverse depth information of video frames I1 and I2, that is, the distance of the object from the camera. The larger the inverse depth value, the closer the object is to the camera. It is the DPT monocular depth estimation model.

[0051] Step 3: In order to increase the diversity of the real structural vibration optical flow dataset, add a perturbation factor to correct the optical flow, and perform forward and reverse mapping on the video frames I1 and I2 according to the depth information calculated in step 2 to generate a new perspective image. and , the formula is as follows: ; ; ; ; in, and By adding a disturbance factor Corrected optical flow, S stands for Splatting technology.

[0052] The splatting technique is used to generate a new image by weighted averaging each pixel in two consecutive frames (I1, I2) in the video frame, where the weighted average is calculated based on the depth map calculated by DPT to solve the occlusion problem. The specific steps are as follows: First, the depth maps D1 and D2 calculated by the DPT monocular depth estimation model and the optical flow field information corrected by adding the perturbation factor are mapped to the position information of each pixel in the new perspective image. The formula is as follows: ; Among them, q represents the position of the pixel in the original image (I1, I2), and p represents the position of the actual pixel mapped to the new perspective image based on the optical flow field information and depth information. represents the position of the predicted pixel in the new perspective image, and u represents the deviation between the actual optical flow mapping and the predicted target position.

[0053] In the process of image processing and optical flow estimation, the importance of local areas is highlighted by weighting, ensuring that the calculation focuses on more relevant neighboring pixels, while weakening or ignoring the influence of pixels far from the center on the calculation results. The formula is as follows: ; in, and They represent the deviation in the x direction and the deviation in the y direction respectively. That is, the larger the deviation, the less accurate the predicted pixel position and the smaller the weight.

[0054] New images The generation formula is as follows: ; in, represents the pixel value of the generated perspective image at position p, q represents the position of the pixel in the original image, is the depth value of pixel q in the original image, and the depth value is exponentially calculated to enhance the effect of depth on weight and reduce the effect of occlusion. is the pixel value of pixel q in the original image.

[0055] Step 4: Due to the motion and depth differences of dynamic objects, forward mapping may introduce occlusions and holes in the target image. This embodiment uses the bidirectional hole filling (BHF) technology to fill these holes using reverse optical flow and image content to ensure the quality of the generated image. First, check the hole mask M, the formula is as follows: ; In the hole mask M, the hole pixels are marked as 0 and other pixels are marked as 1. This means that if If the value is large, it means that multiple original pixels contribute to the target position p in the new perspective image, which means that this position is not a hole; if A smaller value means that almost no original pixel is mapped to this position, which means that this position is a hole.

[0056] Secondly, the images of the forward and backward mappings are fused, and the formula is as follows: ; in, To generate high-quality images, and It is the new perspective image generated in step 3.

[0057] Through the above steps, the splatting technique and bidirectional hole filling (BHF) technique are used to generate high-quality perspective images. These images are used to construct the final optical flow dataset.

[0058] Step 5: Process the image With the original image The optical flow labels are obtained by fusion in the first frame of the original image. And the second original frame It is generated by adding a perturbation factor to the optical flow between.

[0059] In another practical application, this embodiment uses a laboratory steel ruler structure and a five-story frame structure as the engineering scene. Figure 4 As shown. A camera was used to shoot a real-life vibration video of a real engineering structure under external excitation and construct a real-life image sequence. There were 201 images of the steel ruler structure, which formed 200 pairs of images; there were 601 images of the five-layer frame structure, which formed 600 pairs of images. The first image of each image pair is the first image of the entire image sequence, and the second image is the 2nd, 3rd, 4th, ..., 201st image of the entire image sequence (601 images for the five-layer frame structure). The image resolution of the steel ruler structure is 1280×352, and the image resolution of the five-layer frame structure is 1920×600.

[0060] In another practical application, this embodiment uses a laboratory steel ruler structure and a five-story frame structure as the engineering scene. Figure 4 As shown. A camera was used to shoot a real-life vibration video of a real engineering structure under external excitation and construct a real-life image sequence. There were 201 images of the steel ruler structure, which formed 200 pairs of images; there were 601 images of the five-layer frame structure, which formed 600 pairs of images. The first image of each image pair is the first image of the entire image sequence, and the second image is the 2nd, 3rd, 4th, ..., 201st image of the entire image sequence (601 images for the five-layer frame structure). The image resolution of the steel ruler structure is 1280×352, and the image resolution of the five-layer frame structure is 1920×600.

[0061] The image is rendered to generate a new dataset, and the model is trained on the newly generated dataset for 5 rounds. The rendering parameters are shown in Table 1, and the training parameters are shown in Table 2.

[0062] Table 1

[0063] Table 2

[0064] The data sets generated in each round are as follows. The steel ruler has 200 image pairs and the five-layer framework has 600 image pairs, which are divided into training and test sets at a ratio of 9:1. Figure 1 Kahara Figure 2 , each round of iteration framework is Figure 1 Keep unchanged, create new Figure 2 , and optical flow labels. A complete dataset includes two images and an optical flow label. The process of generating a dataset is as follows Figure 5 shown.

[0065] The generated results are as follows Figure 6 , Figure 7 As shown in the figure. A pair of data images of a steel ruler and a five-layer frame are selected for display. The original image pair is predicted by the RAFT-pre-training model to obtain the initial optical flow label, and then the rendering module is used to Figure 1 , the initial optical flow label generates a new Figure 2 , so the optical flow label can actually satisfy the original Figure 1 ,new Figure 2 The optical flow prediction results between Figure 2 Generated by the rendering module, there are certain "artifact" effects.

[0066] Furthermore, the RAFT-pretrained model optimized by the aforementioned iterative framework and the original RAFT-pretrained model that has not been optimized by the iterative framework are used to perform optical flow prediction on the original image sequence taken by the camera, and the optimized models are compared to see whether there is any improvement in the optical flow prediction performance and displacement measurement error.

[0067] The following three test results are presented: optical flow map recognition results, displacement measurement curve, and displacement error.

[0068] (1) Optical flow map The result of optical flow map recognition is the full-field optical flow map calculated by inputting an image pair consisting of two images into the model. The steel ruler and the five-layer framework each select a pair of image pairs for display, such as Figure 8 , Fig. 9 The optical flow map is a qualitative result, reflecting the model's estimation of the optical flow in the entire scene. The quality of the optical flow map is evaluated by whether it can capture clear structures, such as Figure 8 (a)-(b) and Fig. 9 (a)-(b) in the figure are the first and second images of a pair of images, respectively. Figure 8 (c) and Fig. 9 (c) The optical flow results of the pre-trained model without iterative loop framework in the steel ruler structure and the five-layer framework structure, Figure 8 (d) and Fig. 9(d) The optical flow map results of the pre-trained model after 5 rounds of iterative cycle framework optimization on the steel ruler structure and the five-layer framework structure. In terms of the clarity of the structure, the optical flow map recognition results after the optical flow iterative optimization framework optimization are clearer. Figure 8 In (d), the three calibration objects and the steel ruler can be identified; Fig. 9 In (d), the five-layer framework recognition is also clearer, and there is no optical flow in large unstructured areas.

[0069] (2) Displacement measurement curve The structural displacement of the target area is extracted from the optical flow estimation results and compared with the measurement results of the laser displacement meter. There are 200 pairs of optical flow estimation results for the steel ruler structure, corresponding to the time of 1s-3s, and 600 pairs of optical flow estimation results for the five-layer frame structure, corresponding to the time of 0s-6s.

[0070] As shown in Figures 10(a), 10(b), 11(a) and 11(b), they are schematic diagrams of the displacement curve results of the steel ruler and the five-story frame, respectively, where Figures 10(a) and 11(a) are the target areas for extracting displacement of the steel ruler and the five-story frame, respectively, and Figures 10(b) and 11(b) are the displacement curve results, respectively.

[0071] Figure 10 (b) is the displacement measurement curve of the steel ruler, which is divided into three sub-graphs: left, middle, and right. Each sub-graph corresponds to the displacement extraction area selected in the calibration plate shown in Figure 10 (a). The black color is the measurement result of the laser displacement meter (LVDT), which is used as the true value for comparison. The red color is the displacement perception curve without optimization, and the green color is the displacement perception curve after 5 rounds of iterative framework. From the trend and peak point of the curve, the displacement perception curve after 5 rounds of optical flow iterative optimization framework is closer to the LVDT measurement result than the unoptimized displacement curve. The unoptimized displacement curve is far away from the LVDT measurement result at multiple displacement peaks, which is reduced after optimization.

[0072] Figure 11 (b) is the displacement measurement curve of the five-layer frame, which is divided into five sub-graphs from top to bottom, and each sub-graph corresponds to the displacement extraction area selected from the top to bottom of the structure shown in Figure 11 (a) from five layers to one layer. Similarly, black, red, and green represent the displacement measurement curves of LVDT, unoptimized, and optimized, respectively. The results show that in the displacement measurement results of the first and fifth layers, the unoptimized measurement results and the LVDT measurement results are quite different in the curve trend and peak point measurement results, while the measurement results of the optimized model are significantly improved; the displacement measurement results of the second, third, and fourth layers are not significantly improved.

[0073] (3) Displacement measurement error The measurement result of the laser displacement meter is taken as the true value, and the optical flow estimation result is taken as the predicted value to calculate the root mean square error (RMSE) and the normalized root mean square error (NRMSE). The calculation formulas of RMSE and NRMSE are as follows. is the sample size, It is The true value of the observation, It is The predicted value of the observed value, represents a data sequence, and and Represent the maximum and minimum values ​​of the data series respectively.

[0074] ; ; like Fig.12 , Fig.13 The figures show the displacement measurement errors of the steel ruler structure and the five-story frame structure. The left figure in each figure is a bar graph of NRMSE, and the right figure is a bar graph of RMSE. The red columns in each figure are errors without optimization, and the green columns are errors after optimization. Each sub-figure in the figure is divided into errors in the left, middle, and right regions, and each sub-figure in the figure is divided into errors from five floors to one floor from top to bottom. The smaller the NRMSE and RMSE error results, the better, indicating that the error in the measured displacement is smaller.

[0075] like Fig.12 As shown in Figure 10, the NRMSE and RMSE of the three optimized positions are smaller than those without optimization, and the errors are consistent with the displacement measurement curves in Figure 10 (b). Fig.13 As shown in Figure 11, the measurement errors of the optimized fifth and first layers are greatly reduced compared with those without optimization, and the measurement errors of the second, third, and fourth layers are at the same level. The error results are consistent with the displacement measurement curve in Figure 11 (b).

[0076] It can be seen that the non-contact structural displacement perception method provided in this embodiment can obtain a structural vibration optical flow dataset with real optical flow labels in a specific engineering scenario, as well as a pre-trained model that has been fine-tuned and trained, thereby improving the measurement accuracy of structural displacement.

[0077] Specifically, the advantages of this embodiment over the prior art include: (1) Compared with the existing supervised deep learning models, which generally select pre-trained models that have been trained with public data sets to perform structural displacement measurement tasks, this embodiment uses a rendering module to produce an optical flow data set for real structural vibration live video, thereby enhancing the generalization ability of the model in real scenes. Therefore, the structural displacement perception method provided by this embodiment can be applied to complex actual engineering projects and significantly improve the accuracy of displacement recognition; (2) Compared with the existing optical flow dataset production, this embodiment improves the accuracy of optical flow estimation and the authenticity of the dataset by alternately optimizing the optical flow estimation network and the generated dataset. It also adopts the splatting technology and the bidirectional hole filling (BHF) technology to provide a higher quality and more realistic optical flow data generation method without real labels, which effectively makes up for the shortcomings of the existing technology in terms of data quality and diversity; (3) During the training process, this embodiment uses a lightweight recurrent neural network GRU (Gated Recurrent Unit) to extract optical flow features and convert them into accurate structural displacement features. GRU reduces the parameters of the model without affecting the accuracy through its simple structure (update gate and reset gate), thereby improving the efficiency and speed of the entire training process.

[0078] Figure 2 A schematic block diagram of a non-contact structural displacement sensing device 200 provided in an embodiment of the present invention, the device 200 includes: The acquisition unit 201 is used to acquire real structural vibration real-shot video data, and construct the structural vibration real-shot video data into a real-shot image data set by combining rendering technology and an optical flow estimation pre-training model; wherein the optical flow estimation pre-training model is obtained by training a preset optical flow initial estimation model with a public data set; A training unit 202 is used to perform iterative optimization training on the optical flow estimation pre-training model based on the real-shot image data set using a preset optical flow iterative optimization framework to obtain an optical flow estimation target model; A prediction unit 203 is used to perform full-field optical flow prediction on a specified real-shot image sequence using the optical flow estimation target model to obtain an optical flow displacement in a camera coordinate system; The conversion unit 204 is used to perform spatial coordinate conversion on the optical flow displacement in the camera coordinate system to obtain the structural displacement in the world coordinate system.

[0079] In one embodiment, the acquisition unit 201 includes: A combining unit, used to combine the remaining frames of images with the first frame of image respectively, taking the first frame of image as a reference, to obtain a sequence of real-shot image pairs; An input unit, used to input the real-shot image pair sequence into an optical flow estimation pre-training model to obtain an initial optical flow label; A first rendering unit is used to render, for each of the real-shot image pairs, another frame image in the real-shot image pair based on the first frame image and the initial optical flow label to obtain a new frame image, and construct a new real-shot image pair based on the new frame image; The construction unit is used to add a perturbation factor to the initial optical flow label to obtain the optical flow label, and combine the optical flow label with the new real-shot image pair to construct the real-shot image dataset.

[0080] In one embodiment, the first rendering unit includes: The second rendering unit is used to render each of the real-shot image pairs according to the following formula: ; in, represents the rendered real-shot image pair, R represents the rendering process, I1 represents the first frame image, and I i-1 Indicates another frame image in the real-shot image pair, label i-1 represents the initial label of optical flow, represents the optical flow estimation model used, Represents the optical flow estimation model used in the previous round.

[0081] In one embodiment, the training unit 202 includes: A first optimization unit is used to input the real-shot image data set into the optical flow estimation pre-training model for a first optimization training to obtain a first round of iterative optical flow estimation optimization model; The second optimization unit is used to render the real-shot image data set in the first optimization training, and input the rendered real-shot image data set into the optical flow estimation optimization model of the first round of iteration for a second optimization training, so as to obtain the optical flow estimation optimization model of the second round of iteration; A third optimization unit is used to render the real-shot image data set in the second optimization training, and input the rendered real-shot image data set into the optical flow estimation optimization model of the second round of iteration for a third optimization training, so as to obtain the optical flow estimation optimization model of the third round of iteration; The selection unit is used to perform the same process until a preset number of iterations are completed, and select the optical flow estimation optimization model with the best performance as the optical flow estimation target model.

[0082] In one embodiment, the selection unit includes: The evaluation unit is used to evaluate the performance of the optical flow estimation optimization model in each round of iteration according to the following formula: ; ; ; ; Among them, the 1PX, 3PX, and 5PX indicators represent the proportion of pixels whose error between the optical flow predicted by the optical flow estimation optimization model and the true optical flow is less than 1 pixel, 3 pixels, and 5 pixels respectively. i represents the Euclidean distance between the predicted displacement and the actual displacement of each pixel in the i-th two-dimensional optical flow field, An indicator function representing a threshold value for determining whether the user's error is greater than 1 pixel, 3 pixels, or 5 pixels, where k represents any constant of 1, 3, or 5.

[0083] In one embodiment, the conversion unit 204 includes: A setting unit, used for setting a target frame for the area where the optical flow displacement is located, calculating an average optical flow displacement of all pixels in the target frame, and then using the average optical flow displacement as a displacement recognition result of the target frame; A calculation unit is used to obtain a scaling factor between the camera coordinate system and the world coordinate system, and convert the displacement recognition result in the camera coordinate system into a structural displacement in the world coordinate system based on the scaling factor according to the following formula: ; Among them, D 3D represents the structural displacement in the world coordinate system, f is the scaling factor, L 3D represents the length selected in the world coordinate system based on the scaling factor, L img Indicates that L 3D The corresponding pixel distance in the image, Represents the displacement recognition result in the camera coordinate system.

[0084] In one embodiment, the non-contact structural displacement sensing device 200 further includes: The correction unit is used to perform displacement correction on the structural displacement by using a differential evolution algorithm, and output the displacement correction result as the final structural displacement perception result.

[0085] Since the embodiments of the apparatus part correspond to the embodiments of the method part, please refer to the description of the embodiments of the method part for the embodiments of the apparatus part, which will not be repeated here.

[0086] The embodiment of the present invention further provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed, the steps provided in the above embodiment can be implemented. The storage medium may include: a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and other media that can store program codes.

[0087] The embodiment of the present invention also provides a computer device, which may include a memory and a processor, wherein a computer program is stored in the memory, and when the processor calls the computer program in the memory, the steps provided in the above embodiment may be implemented. Of course, the computer device may also include various network interfaces, power supplies and other components.

[0088] The various embodiments in the specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same and similar parts between the various embodiments can be referred to each other. For the system disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part description. It should be pointed out that for ordinary technicians in this technical field, without departing from the principles of this application, several improvements and modifications can be made to the present application, and these improvements and modifications also fall within the scope of protection of the claims of this application.

[0089] It should also be noted that, in this specification, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "comprises", "comprising" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the statement "comprising a ..." does not exclude the presence of other identical elements in the process, method, article or device including the element.

Claims

1. A non-contact sensing method for structural displacement, characterized in that: include: Collect real structural vibration video data, and construct the real structural vibration video data into a real image dataset by combining rendering technology and optical flow estimation pre-training model; wherein the optical flow estimation pre-training model is obtained by training a preset optical flow initial estimation model with a public dataset; Based on the real-shot image dataset, a preset optical flow iterative optimization framework is used to iteratively optimize the optical flow estimation pre-trained model to obtain an optical flow estimation target model; Using the optical flow estimation target model to perform full-field optical flow prediction on a specified real-shot image sequence, and obtain the optical flow displacement in the camera coordinate system; The optical flow displacement in the camera coordinate system is transformed into a spatial coordinate to obtain a structural displacement in the world coordinate system.

2. The non-contact sensing method for structural displacement according to claim 1, characterized in that: The method collects real structural vibration video data, and constructs the real structural vibration video data into a real image dataset by combining rendering technology and an optical flow estimation pre-training model, including: Taking the first frame image as a reference, the remaining frame images are respectively combined with the first frame image to obtain a sequence of real-shot image pairs; Inputting the real-shot image pair sequence into an optical flow estimation pre-training model to obtain an initial optical flow label; For each of the real-shot image pairs, based on the first frame image and the initial optical flow label, another frame image in the real-shot image pair is rendered to obtain a new frame image, and a new real-shot image pair is constructed based on the new frame image; A disturbance factor is added to the initial optical flow label to obtain an optical flow label, and the optical flow label is combined with a new real-shot image pair to construct the real-shot image dataset.

3. The non-contact sensing method for structural displacement according to claim 2 is characterized in that: For each of the real-shot image pairs, based on the first frame image and the initial optical flow label, another frame image in the real-shot image pair is rendered to obtain a new frame image, and a new real-shot image pair is constructed based on the new frame image, including: Render each of the real-shot image pairs according to the following formula: ； in, represents the rendered real-shot image pair, R represents the rendering process, I1 represents the first frame image, and I i-1 Indicates another frame image in the real-shot image pair, label i-1 represents the initial label of optical flow, represents the optical flow estimation model used, Represents the optical flow estimation model used in the previous round.

4. The non-contact sensing method for structural displacement according to claim 1, characterized in that: The method of iteratively optimizing and training the optical flow estimation pre-training model based on the real-shot image dataset using a preset optical flow iterative optimization framework to obtain an optical flow estimation target model includes: Inputting the real-shot image data set into the optical flow estimation pre-training model for the first optimization training to obtain the optical flow estimation optimization model for the first round of iteration; Rendering the real-shot image data set in the first optimization training, and inputting the rendered real-shot image data set into the optical flow estimation optimization model of the first round of iteration for a second optimization training, thereby obtaining the optical flow estimation optimization model of the second round of iteration; Rendering the real-shot image data set in the second optimization training, and inputting the rendered real-shot image data set into the second-round iterative optical flow estimation optimization model for a third optimization training, thereby obtaining the third-round iterative optical flow estimation optimization model; And so on, until the preset number of iterations is completed, the optical flow estimation optimization model with the best performance is selected as the optical flow estimation target model.

5. The non-contact sensing method for structural displacement according to claim 4, characterized in that: The step of selecting the optical flow estimation optimization model with the best performance as the optical flow estimation target model includes: The performance of the optical flow estimation optimization model for each round of iteration is evaluated according to the following formula: ； ； ； ； Among them, the 1PX, 3PX, and 5PX indicators represent the proportion of pixels whose error between the optical flow predicted by the optical flow estimation optimization model and the true optical flow is less than 1 pixel, 3 pixels, and 5 pixels respectively. i represents the Euclidean distance between the predicted displacement and the actual displacement of each pixel in the i-th two-dimensional optical flow field, An indicator function representing a threshold value for determining whether the user's error is greater than 1 pixel, 3 pixels, or 5 pixels, where k represents any constant of 1, 3, or 5.

6. The non-contact sensing method for structural displacement according to claim 1, characterized in that: The performing spatial coordinate transformation on the optical flow displacement in the camera coordinate system to obtain the structural displacement in the world coordinate system includes: A target frame is set for the area where the optical flow displacement is located, and an average optical flow displacement of all pixels in the target frame is calculated, and then the average optical flow displacement is used as a displacement recognition result of the target frame; The scaling factor between the camera coordinate system and the world coordinate system is obtained, and the displacement recognition result in the camera coordinate system is converted into the structural displacement in the world coordinate system based on the scaling factor according to the following formula: ； Among them, D 3D represents the structural displacement in the world coordinate system, f is the scaling factor, L 3D represents the length selected in the world coordinate system based on the scaling factor, L img Indicates that L 3D The corresponding pixel distance in the image, Represents the displacement recognition result in the camera coordinate system.

7. The non-contact sensing method for structural displacement according to claim 1, characterized in that: After the step of performing spatial coordinate transformation on the optical flow displacement in the camera coordinate system to obtain the structural displacement in the world coordinate system, the method further comprises: A differential evolution algorithm is used to perform displacement correction on the structural displacement, and the result after displacement correction is output as the final structural displacement perception result.

8. A non-contact sensing device for structural displacement, characterized in that: include: An acquisition unit is used to acquire real structural vibration real-shot video data, and construct the structural vibration real-shot video data into a real-shot image data set by combining rendering technology and an optical flow estimation pre-training model; wherein the optical flow estimation pre-training model is obtained by training a preset optical flow initial estimation model with a public data set; A training unit, configured to perform iterative optimization training on the optical flow estimation pre-training model based on the real-shot image data set using a preset optical flow iterative optimization framework to obtain an optical flow estimation target model; A prediction unit, used to perform full-field optical flow prediction on a specified real-shot image sequence using the optical flow estimation target model to obtain an optical flow displacement in a camera coordinate system; The conversion unit is used to perform spatial coordinate conversion on the optical flow displacement in the camera coordinate system to obtain the structural displacement in the world coordinate system.

9. A computer device, characterized in that: The method comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the method for non-contact sensing of structural displacement according to any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the non-contact sensing method for structural displacement according to any one of claims 1 to 7 is implemented.

Images